import "github.com/kinorai/omnifeed/internal/engine/reddit"
Package reddit implements the Reddit-specific engine: fetches threads via the public JSON API, expands collapsed reply branches with /api/morechildren, strips fields, drops deleted comments, and emits TOON or JSON.
The ways the fetcher can read Reddit, in the order the chain tries them by default. Reddit closed logged-out .json in 2026, so each needs something:
const (
MethodClientID = "client_id"
MethodSession = "session"
MethodBrowser = "browser"
)
DefaultMethodCooldown is how long a method that failed in a way another method can get around (blocked, rate-limited, refused credential) is skipped before the chain tries it again.
const DefaultMethodCooldown = 10 * time.Minute
MaxExpansionRounds is the hard cap on /api/morechildren iterations, applied even when expand=full is requested. Each round fetches up to 100 child IDs, so 40 rounds ≈ 4,000 comments — covers any realistic thread while bounding total latency and rate-limit exposure.
const MaxExpansionRounds = 40
MethodChain is the method label OnOutcome gets for the chain as a whole: outcome “all_held” when every method was on hold and nothing was sent.
const MethodChain = "chain"
RedditMethodKey is the metadata key (fetch_url _meta) naming the chain method that served a Reddit document: client_id, session or browser.
const RedditMethodKey = "reddit_method"
DefaultMethods is the chain order when none is configured.
var DefaultMethods = []string{MethodClientID, MethodSession, MethodBrowser}
func IsRedditURL(rawURL string) bool
IsRedditURL is the package-level matcher used by Engine.Matches; exported so callers (and tests) can detect Reddit URLs without instantiating an Engine.
func IsShareURL(rawURL string) bool
IsShareURL reports whether rawURL is a Reddit share link needing resolution.
func MergeExpanded(thread *Thread, newC []Comment, newG []Gap, requestedIDs []string, usedGapIdx []int)
MergeExpanded folds newly-fetched comments/gaps into the existing thread, dedupes by ID, removes fulfilled child IDs from the gaps that triggered the fetch, and drops gaps that no longer carry any children.
func NormalizePermalink(rawURL string) (string, error)
NormalizePermalink converts any reddit.com URL into the canonical /r/{sub}/comments/{id}[/{slug}] permalink fragment (no trailing slash, no .json suffix).
func ParseMoreChildren(raw []byte, opts Options) ([]Comment, []Gap, error)
ParseMoreChildren parses the /api/morechildren response: { json: { data: { things: […] } } }
func SessionExpiry(cookie string) (time.Time, bool)
SessionExpiry reads the expiry from a reddit_session cookie. The cookie is a signed token whose payload carries `exp`, and Reddit does not renew it on use, so this is the date a human must log in again by. The signature is not checked: the value only feeds a warning and a metric.
Chain is one crawl’s view of every configured method. Each call goes to the first available method; on a failover error the method is skipped for its cooldown and the call moves to the next one. Not safe for concurrent use.
type Chain struct {
// contains filtered or unexported fields
}
func (c *Chain) Close(ctx context.Context) error
Close releases every session the chain opened.
func (c *Chain) FetchListing(ctx context.Context, sub, sort string, limit int, t string) ([]byte, error)
FetchListing is Session.FetchListing over the chain.
func (c *Chain) FetchMoreChildren(ctx context.Context, linkID string, childIDs []string, sort string) ([]byte, error)
FetchMoreChildren is Session.FetchMoreChildren over the direct methods only. The browser would POST /api/morechildren logged out, which Reddit refuses with 403, and that 403 would hold the browser method, the one path left that still reads threads. Only a direct thread has child IDs to expand (the HTML path reports its gaps without them), so a failure here costs the expansion only: the engine keeps the partial thread.
func (c *Chain) FetchThread(ctx context.Context, permalink string, limit, depth int, sort string, expand int) ([]byte, error)
FetchThread is Session.FetchThread over the chain.
func (c *Chain) Partial() string
Partial returns why the crawl’s answers were incomplete, or “”.
func (c *Chain) ResolveShareURL(ctx context.Context, shareURL string) (string, error)
ResolveShareURL is Session.ResolveShareURL over the chain.
func (c *Chain) Served() string
Served returns the method that answered the last successful call.
Comment is a Reddit comment stripped to LLM-relevant fields.
type Comment struct {
ID string `json:"id" toon:"id"`
ParentID string `json:"parent_id" toon:"parent_id"`
Author string `json:"author" toon:"author"`
Score int `json:"score" toon:"score"`
Body string `json:"body" toon:"body"`
Depth *int `json:"depth,omitempty" toon:"depth,omitempty"`
Created *int64 `json:"created,omitempty" toon:"created,omitempty"`
}
Config configures a Reddit Engine.
type Config struct {
Fetcher *Fetcher
Limiter httpx.Limiter
Timeout time.Duration
DefaultOpts Options
Logger *slog.Logger
Metrics *observability.Metrics
}
Engine implements domain.Engine for Reddit URLs.
type Engine struct {
// contains filtered or unexported fields
}
func New(cfg Config) *Engine
New returns a Reddit Engine configured per cfg.
func (e *Engine) Crawl(ctx context.Context, rawURL string, eo domain.EngineOptions) (domain.Document, error)
Crawl fetches a Reddit thread, expands gaps up to the configured budget, and returns it encoded as TOON or JSON.
func (*Engine) Matches(rawURL string) bool
Matches claims only the reddit.com URLs this engine can actually render: a comments permalink, or a share link (/r/{sub}/s/{code}) that resolves to one. Other reddit.com URLs — profiles, wikis, search pages, /dev/api — fall through to the generic fallback engine instead of hard-failing in NormalizePermalink.
func (*Engine) Name() string
Name returns the engine identifier.
func (*Engine) SameHostAsPage() bool
SameHostAsPage reports true: the engine’s JSON comes from www.reddit.com, the page’s own host, so a Reddit block must not be followed by a browser render of the same host (domain.SameHostEngine).
Fetcher retrieves Reddit data. Reddit closed logged-out .json in 2026, so every method needs something the logged-out web lacks:
OpenChain tries them in the configured order (see Chain). Open uses one backend only: the first credential, else the browser.
type Fetcher struct {
// contains filtered or unexported fields
}
func NewFetcher(cfg FetcherConfig) *Fetcher
NewFetcher constructs a Fetcher from cfg.
func (f *Fetcher) LoadSharedHolds(ctx context.Context)
LoadSharedHolds reads the holds other replicas recorded and reports the held methods down, so a pod that starts during a hold does not report every method up until its first call. A no-op without shared state.
func (f *Fetcher) Methods() []string
Methods returns the configured chain methods, in order.
func (f *Fetcher) OpenChain(context.Context) (*Chain, error)
OpenChain starts a crawl over the configured methods. Each method’s session opens on first use. The caller owns the chain and must Close it.
FetcherConfig configures a Fetcher.
type FetcherConfig struct {
Browser browser.Browser
// Quota, when non-nil, admits every request that reaches Reddit — the
// thread fetch, each /api/morechildren round, a listing, a share-link
// resolve — not the crawl as a whole. Reddit counts requests, and one
// expand=full crawl can be 40 of them. The requests a browser page sends
// itself (scroll batches, "more replies") are charged once the page
// answers. Nil disables it (OMNIFEED_REDDIT_QUOTA=0, the default).
Quota httpx.Limiter
// ClientID and SessionCookie, when set, enable the client_id and session
// methods: Reddit still serves its .json endpoints to authenticated
// clients, plain HTTP ones included (see directClient). ClientID is an
// OAuth app's client ID, used with the installed-client grant. HTTP is
// the client for both; it must not follow redirects (share links resolve
// from the Location header).
ClientID string
SessionCookie string
UserAgent string
HTTP *http.Client
// Methods is the chain order (MethodClientID, MethodSession,
// MethodBrowser); a method without its credential is left out. Empty
// means DefaultMethods. Cooldown is how long a method that failed in a way
// the next one can get around is skipped; 0 means DefaultMethodCooldown.
Methods []string
Cooldown time.Duration
// OnOutcome is told each call's result per method: "ok", "incomplete"
// (a browser answer holding less than the page claims), "not_found",
// "forbidden" (a subreddit refused to everyone), "layout" (an unreadable
// page), "error", or the failure kind that moved the call on. Method
// MethodChain with "all_held" is a call that found every method on hold.
// OnState is told when a method starts or stops being skipped: why is
// "failed", "backoff" (Reddit answered but asked for one) or "held"
// (found on hold, maybe by another replica), and hold how long for; both
// are empty when up. Both hooks are for metrics and logs and can be nil.
OnOutcome func(method, outcome string)
OnState func(method string, up bool, why string, hold time.Duration)
// OnBudget is told the X-Ratelimit-Remaining of each direct answer that
// carries it: the requests left in Reddit's window. It can be nil.
OnBudget func(method string, remaining float64)
// OnInPage is told how many requests a browser page sent itself, by kind
// ("listing" or "thread"): the scroll batches and "more replies" that come
// on top of the page load. It can be nil.
OnInPage func(kind string, n int)
// Shared, when non-nil, shares the methods' holds between replicas
// (main.go sets it when OMNIFEED_REDIS_URL is set). It fails open: on an
// error the chain uses only this pod's memory for 30s. OnSharedError is
// told each failed operation, and OnSharedDown each change between shared
// and per-pod state. Both can be nil.
Shared SharedState
OnSharedError func(op string, err error)
OnSharedDown func(down bool)
}
Gap represents a collapsed branch in the comment tree (a “more” placeholder).
type Gap struct {
Type string `json:"type" toon:"type"`
ParentID string `json:"parent_id" toon:"parent_id"`
Depth int `json:"depth" toon:"depth"`
Count int `json:"count,omitempty" toon:"count,omitempty"`
Children []string `json:"children,omitempty" toon:"children,omitempty"`
}
ListingRequest is a parsed subreddit listing URL: which subreddit and sort the path names, plus the two query params Reddit honors on listings.
type ListingRequest struct {
Sub string
Sort string
// T is Reddit's time window (hour|day|week|month|year|all), set only for the
// sorts Reddit applies it to (top, controversial). Empty when the URL omits
// it, names a bogus value, or pairs it with a sort that ignores it.
T string
// Limit is the number of posts to fetch, always within [1,maxListingLimit]:
// listingLimit when the URL asks for none, otherwise its clamped value.
Limit int
}
func ParseListingURL(rawURL string) (ListingRequest, bool)
ParseListingURL extracts the subreddit, sort, and listing query params from a subreddit listing URL. Sort defaults to “hot” (Reddit’s default view) when the path omits it. ok is false for any reddit.com URL that isn’t a bare subreddit listing.
Options carries Reddit-specific per-request knobs derived from query strings or env-var defaults.
type Options struct {
KeepDepth bool // include depth field on each comment
KeepCreated bool // include created field on each comment
MaxRounds int // hard cap on /api/morechildren expansion rounds
Format string // "toon" or "json"
// Size controls. FetchLimit/Depth/Sort map 1:1 onto Reddit's
// comments-endpoint query params; MaxComments/MaxTopLevel are enforced by
// omnifeed after fetching + expansion. Param semantics:
// https://www.reddit.com/dev/api/#GET_comments_{article}
FetchLimit int // Reddit `limit`: max comments in the initial tree
Depth int // Reddit `depth`: max nesting depth of the initial tree
Sort string // Reddit `sort`: comment sort order
MaxComments int // hard cap on total comments emitted (0 = unlimited)
MaxTopLevel int // hard cap on top-level comment threads (0 = unlimited)
}
Post is a Reddit post stripped to LLM-relevant fields.
type Post struct {
ID string `json:"id" toon:"id"`
Title string `json:"title" toon:"title"`
Author string `json:"author" toon:"author"`
Subreddit string `json:"subreddit" toon:"subreddit"`
Score int `json:"score" toon:"score"`
UpvoteRatio float64 `json:"upvote_ratio" toon:"upvote_ratio"`
NumComments int `json:"num_comments" toon:"num_comments"`
// Thread totals, set only on a thread's post header (nil, so omitted, on
// subreddit listings). They sit in the body because AI agents see the TOON
// text, not the response _meta. TotalComments is Reddit's num_comments
// (deleted/removed included); ReturnedComments is the comments emitted;
// HiddenMore is the sum of the remaining "more" gaps' counts; Truncated is
// true when omnifeed's caps dropped comments or any gap is left unexpanded.
TotalComments *int `json:"total_comments,omitempty" toon:"total_comments,omitempty"`
ReturnedComments *int `json:"returned_comments,omitempty" toon:"returned_comments,omitempty"`
HiddenMore *int `json:"hidden_more,omitempty" toon:"hidden_more,omitempty"`
Truncated *bool `json:"truncated,omitempty" toon:"truncated,omitempty"`
Created int64 `json:"created" toon:"created"`
URL string `json:"url" toon:"url"`
Selftext string `json:"selftext,omitempty" toon:"selftext,omitempty"`
Permalink string `json:"permalink" toon:"permalink"`
}
func ParseSubredditListing(raw []byte) ([]Post, error)
ParseSubredditListing decodes a subreddit listing (.json) — a single Listing whose children are t3 posts — into a slice of Posts (no comment trees).
Session is one chain method’s connection for one crawl: a browser session, or a direct client. Not safe for concurrent use.
type Session struct {
// contains filtered or unexported fields
}
func (s *Session) Close(ctx context.Context) error
Close releases the browser session.
func (s *Session) FetchListing(ctx context.Context, sub, sort string, limit int, t string) ([]byte, error)
| FetchListing retrieves a subreddit listing (hot/new/top/…) from its rendered HTML page, read from inside a real browser on reddit.com — the same path FetchThread uses. limit caps the number of posts, and t is Reddit’s time window (hour | day | week | month | year | all), appended only when set. Whether a window is meaningful for the sort is ParseListingURL’s call — it only sets t for top/controversial — so this appends whatever it is handed. |
func (s *Session) FetchMoreChildren(ctx context.Context, linkID string, childIDs []string, sort string) ([]byte, error)
FetchMoreChildren expands collapsed reply branches via /api/morechildren. linkID must include the t3_ prefix; childIDs are bare IDs (no prefix). Only a direct method can: Reddit refuses the logged-out call with 403, so the browser session has no path to it (the chain never asks it).
func (s *Session) FetchThread(ctx context.Context, permalink string, limit, depth int, sort string, expand int) ([]byte, error)
FetchThread retrieves a thread from its rendered HTML page, read from inside a real browser on the reddit.com origin (see html.go for why not .json). It returns the [post, comments] listing pair the .json endpoint used to, so ParseThread reads it unchanged. limit is the comment count at which the in-page expansion stops, and expand the most “more replies” requests it may send (capped at maxExpandRequests): the browser’s counterpart of the /api/morechildren rounds a direct method makes. depth is not applied: the page renders its own nesting. A direct method ignores expand; the engine expands its gaps afterwards.
func (s *Session) ResolveShareURL(ctx context.Context, shareURL string) (string, error)
ResolveShareURL resolves a Reddit share link (/r/{sub}/s/{code}) to its canonical /comments/ permalink: the browser follows the 301 redirect and we read the resulting location. Returns the full canonical URL (tracking query params and all — NormalizePermalink only looks at the path).
SharedState shares the methods’ holds between replicas, so a back-off or a refused credential seen by one pod stops every pod. Reddit’s budget belongs to the credential, not to the pod. Implementations: redisstate.Store.
type SharedState interface {
// Holds returns the time left on each held method. A method that is not
// held is absent from the map.
Holds(ctx context.Context, methods []string) (map[string]time.Duration, error)
// Hold skips method for at least d. It never shortens a longer hold.
Hold(ctx context.Context, method string, d time.Duration) error
// Clear ends the hold on method.
Clear(ctx context.Context, method string) error
}
SubredditListing is a subreddit page (hot/new/top/…) reduced to its posts, without comment trees — the output shape for a bare /r/{sub}/ listing URL.
type SubredditListing struct {
Subreddit string `json:"subreddit" toon:"subreddit"`
Sort string `json:"sort" toon:"sort"`
// T is the time window the listing was fetched with (Reddit's `t` param);
// omitted for sorts that ignore it or when the URL didn't ask for one.
T string `json:"t,omitempty" toon:"t,omitempty"`
Posts []Post `json:"posts" toon:"posts"`
}
Thread groups a Reddit post with its comment tree and remaining gaps.
type Thread struct {
// Note is set only when the crawl came back incomplete (a morechildren
// round failed mid-expansion). It is the FIRST field so it is the first
// line an agent reads in TOON and the first key in JSON.
Note string `json:"note,omitempty" toon:"note,omitempty"`
Post Post `json:"post" toon:"post"`
Comments []Comment `json:"comments" toon:"comments"`
Gaps []Gap `json:"gaps,omitempty" toon:"gaps,omitempty"`
}
func ParseThread(raw []byte, opts Options) (Thread, error)
ParseThread decodes the 2-element listing array from .json and produces a Thread with all initial comments and gap markers.
Generated by gomarkdoc