import "github.com/kinorai/omnifeed/internal/engine/reddit"
Package reddit implements the Reddit-specific engine: fetches threads via the public JSON API, expands collapsed reply branches with /api/morechildren, strips fields, drops deleted comments, and emits TOON or JSON.
MaxExpansionRounds is the hard cap on /api/morechildren iterations, applied even when expand=full is requested. Each round fetches up to 100 child IDs, so 40 rounds ≈ 4,000 comments — covers any realistic thread while bounding total latency and rate-limit exposure.
const MaxExpansionRounds = 40
func IsRedditURL(rawURL string) bool
IsRedditURL is the package-level matcher used by Engine.Matches; exported so callers (and tests) can detect Reddit URLs without instantiating an Engine.
func IsShareURL(rawURL string) bool
IsShareURL reports whether rawURL is a Reddit share link needing resolution.
func MergeExpanded(thread *Thread, newC []Comment, newG []Gap, requestedIDs []string, usedGapIdx []int)
MergeExpanded folds newly-fetched comments/gaps into the existing thread, dedupes by ID, removes fulfilled child IDs from the gaps that triggered the fetch, and drops gaps that no longer carry any children.
func NormalizePermalink(rawURL string) (string, error)
NormalizePermalink converts any reddit.com URL into the canonical /r/{sub}/comments/{id}[/{slug}] permalink fragment (no trailing slash, no .json suffix).
func ParseMoreChildren(raw []byte, opts Options) ([]Comment, []Gap, error)
ParseMoreChildren parses the /api/morechildren response: { json: { data: { things: […] } } }
Comment is a Reddit comment stripped to LLM-relevant fields.
type Comment struct {
ID string `json:"id" toon:"id"`
ParentID string `json:"parent_id" toon:"parent_id"`
Author string `json:"author" toon:"author"`
Score int `json:"score" toon:"score"`
Body string `json:"body" toon:"body"`
Depth *int `json:"depth,omitempty" toon:"depth,omitempty"`
Created *int64 `json:"created,omitempty" toon:"created,omitempty"`
}
Config configures a Reddit Engine.
type Config struct {
Fetcher *Fetcher
Limiter httpx.Limiter
Timeout time.Duration
DefaultOpts Options
Logger *slog.Logger
Metrics *observability.Metrics
}
Engine implements domain.Engine for Reddit URLs.
type Engine struct {
// contains filtered or unexported fields
}
func New(cfg Config) *Engine
New returns a Reddit Engine configured per cfg.
func (e *Engine) Crawl(ctx context.Context, rawURL string, eo domain.EngineOptions) (domain.Document, error)
Crawl fetches a Reddit thread, expands gaps up to the configured budget, and returns it encoded as TOON or JSON.
func (*Engine) Matches(rawURL string) bool
Matches claims only the reddit.com URLs this engine can actually render: a comments permalink, or a share link (/r/{sub}/s/{code}) that resolves to one. Other reddit.com URLs — profiles, wikis, search pages, /dev/api — fall through to the generic fallback engine instead of hard-failing in NormalizePermalink.
func (*Engine) Name() string
Name returns the engine identifier.
Fetcher retrieves Reddit data through a real headless browser. Reddit’s edge hard-blocks non-browser HTTP clients (Go’s net/http gets a 403 “network security” wall keyed on the TLS/JA3 fingerprint), so we never hit Reddit directly. Instead a crawl opens a browser Session, navigates to a reddit.com page (which clears the bot challenge), and runs a same-origin fetch() of the target JSON endpoint from inside that page — the browser context passes the wall and the in-page fetch inherits it, so the JSON comes back exactly as a logged-out browser would see it (no auth, no cookies).
The Session is backed by a browser.Browser (crawl4ai’s /execute_js).
type Fetcher struct {
// contains filtered or unexported fields
}
func NewFetcher(cfg FetcherConfig) *Fetcher
NewFetcher constructs a Fetcher from cfg.
func (f *Fetcher) Open(ctx context.Context) (*Session, error)
Open starts a crawl session. The caller owns it and must Close it.
FetcherConfig configures a Fetcher.
type FetcherConfig struct {
Browser browser.Browser
}
Gap represents a collapsed branch in the comment tree (a “more” placeholder).
type Gap struct {
Type string `json:"type" toon:"type"`
ParentID string `json:"parent_id" toon:"parent_id"`
Depth int `json:"depth" toon:"depth"`
Count int `json:"count,omitempty" toon:"count,omitempty"`
Children []string `json:"children,omitempty" toon:"children,omitempty"`
}
ListingRequest is a parsed subreddit listing URL: which subreddit and sort the path names, plus the two query params Reddit honors on listings.
type ListingRequest struct {
Sub string
Sort string
// T is Reddit's time window (hour|day|week|month|year|all), set only for the
// sorts Reddit applies it to (top, controversial). Empty when the URL omits
// it, names a bogus value, or pairs it with a sort that ignores it.
T string
// Limit is the number of posts to fetch, always within [1,maxListingLimit]:
// listingLimit when the URL asks for none, otherwise its clamped value.
Limit int
}
func ParseListingURL(rawURL string) (ListingRequest, bool)
ParseListingURL extracts the subreddit, sort, and listing query params from a subreddit listing URL. Sort defaults to “hot” (Reddit’s default view) when the path omits it. ok is false for any reddit.com URL that isn’t a bare subreddit listing.
Options carries Reddit-specific per-request knobs derived from query strings or env-var defaults.
type Options struct {
KeepDepth bool // include depth field on each comment
KeepCreated bool // include created field on each comment
MaxRounds int // hard cap on /api/morechildren expansion rounds
Format string // "toon" or "json"
// Size controls. FetchLimit/Depth/Sort map 1:1 onto Reddit's
// comments-endpoint query params; MaxComments/MaxTopLevel are enforced by
// omnifeed after fetching + expansion. Param semantics:
// https://www.reddit.com/dev/api/#GET_comments_{article}
FetchLimit int // Reddit `limit`: max comments in the initial tree
Depth int // Reddit `depth`: max nesting depth of the initial tree
Sort string // Reddit `sort`: comment sort order
MaxComments int // hard cap on total comments emitted (0 = unlimited)
MaxTopLevel int // hard cap on top-level comment threads (0 = unlimited)
}
Post is a Reddit post stripped to LLM-relevant fields.
type Post struct {
ID string `json:"id" toon:"id"`
Title string `json:"title" toon:"title"`
Author string `json:"author" toon:"author"`
Subreddit string `json:"subreddit" toon:"subreddit"`
Score int `json:"score" toon:"score"`
UpvoteRatio float64 `json:"upvote_ratio" toon:"upvote_ratio"`
NumComments int `json:"num_comments" toon:"num_comments"`
Created int64 `json:"created" toon:"created"`
URL string `json:"url" toon:"url"`
Selftext string `json:"selftext,omitempty" toon:"selftext,omitempty"`
Permalink string `json:"permalink" toon:"permalink"`
}
func ParseSubredditListing(raw []byte) ([]Post, error)
ParseSubredditListing decodes a subreddit listing (.json) — a single Listing whose children are t3 posts — into a slice of Posts (no comment trees).
Session is one crawl’s browser session. All fetches in a crawl share one Session so state like the recorded thread page carries across them. Not safe for concurrent use.
type Session struct {
// contains filtered or unexported fields
}
func (s *Session) Close(ctx context.Context) error
Close releases the browser session.
func (s *Session) FetchListing(ctx context.Context, sub, sort string, limit int, t string) ([]byte, error)
| FetchListing retrieves a subreddit listing (hot/new/top/…) via its .json endpoint, fetched same-origin from inside a real browser on reddit.com — the same bot-wall evasion FetchThread uses. limit caps the number of posts, and t is Reddit’s time window (hour | day | week | month | year | all), appended only when set. Whether a window is meaningful for the sort is ParseListingURL’s call — it only sets t for top/controversial — so this appends whatever it is handed. |
func (s *Session) FetchMoreChildren(ctx context.Context, linkID string, childIDs []string, sort string) ([]byte, error)
FetchMoreChildren expands collapsed reply branches via /api/morechildren. linkID must include the t3_ prefix; childIDs are bare IDs (no prefix). It re-navigates the thread page FetchThread recorded and runs the same-origin POST from it.
func (s *Session) FetchThread(ctx context.Context, permalink string, limit, depth int, sort string) ([]byte, error)
FetchThread retrieves a thread via the .json endpoint, fetched from inside a real browser on the reddit.com origin. limit/depth/sort map directly onto Reddit’s comments-endpoint query params (limit = max comments, depth = max subtree nesting): https://www.reddit.com/dev/api/#GET_comments_{article}
func (s *Session) ResolveShareURL(ctx context.Context, shareURL string) (string, error)
ResolveShareURL resolves a Reddit share link (/r/{sub}/s/{code}) to its canonical /comments/ permalink: the browser follows the 301 redirect and we read the resulting location. Returns the full canonical URL (tracking query params and all — NormalizePermalink only looks at the path).
SubredditListing is a subreddit page (hot/new/top/…) reduced to its posts, without comment trees — the output shape for a bare /r/{sub}/ listing URL.
type SubredditListing struct {
Subreddit string `json:"subreddit" toon:"subreddit"`
Sort string `json:"sort" toon:"sort"`
// T is the time window the listing was fetched with (Reddit's `t` param);
// omitted for sorts that ignore it or when the URL didn't ask for one.
T string `json:"t,omitempty" toon:"t,omitempty"`
Posts []Post `json:"posts" toon:"posts"`
}
Thread groups a Reddit post with its comment tree and remaining gaps.
type Thread struct {
Post Post `json:"post" toon:"post"`
Comments []Comment `json:"comments" toon:"comments"`
Gaps []Gap `json:"gaps,omitempty" toon:"gaps,omitempty"`
}
func ParseThread(raw []byte, opts Options) (Thread, error)
ParseThread decodes the 2-element listing array from .json and produces a Thread with all initial comments and gap markers.
Generated by gomarkdoc