import "github.com/kinorai/omnifeed/internal/domain"
Package domain holds the core types exchanged between transports and engines. It has no external dependencies and no I/O.
Reddit comment-fetch defaults are defined here so config (env fallback) and the reddit engine (zero-Options floor) share one source instead of each duplicating the literals. https://www.reddit.com/dev/api/#GET_comments_{article}
const (
DefaultRedditFetchLimit = 500 // Reddit `limit`
DefaultRedditDepth = 20 // Reddit `depth`
DefaultRedditSort = "top" // Reddit `sort`
)
Content types engines report under ContentTypeKey. The Reddit engine also reports its format knob verbatim (“json”), which is simply not markdown.
const (
ContentTypeMarkdown = "markdown"
ContentTypeTOON = "toon"
)
ContentTypeKey is the Document.Metadata key every engine sets to describe the shape of PageContent: “markdown” for prose-ish text, “toon”/”json” for structured output. Size control keys off it — see TruncatableContentType.
const ContentTypeKey = "content_type"
ValidRedditSorts are the comment sort orders Reddit’s comments endpoint accepts; “confidence” is what the Reddit UI labels “best”. Kept here so config and the reddit engine validate against one list. https://www.reddit.com/dev/api/#GET_comments_{article}
var ValidRedditSorts = []string{"confidence", "top", "new", "controversial", "old", "random", "qa", "live"}
ValidTimeRanges are the recency-window filters the web_search front-ends accept (forwarded to SearXNG); “” means no filter. Kept here so the MCP and REST transports validate against one list.
var ValidTimeRanges = []string{"day", "week", "month", "year"}
func TruncatableContentType(contentType string) bool
TruncatableContentType reports whether generic char truncation is safe for a content type. Only markdown is: TOON and JSON carry length markers / closing delimiters, so cutting them mid-document produces a payload that lies about its own contents. Structured engines bound their output with their own element caps instead (see the Reddit knobs).
func ValidRedditSort(s string) bool
ValidRedditSort reports whether s is an accepted Reddit comment sort order.
func ValidSiteFilter(s string) bool
ValidSiteFilter reports whether s is a bare hostname usable as a `site:` search filter (e.g. “reddit.com”, “forums.plex.tv”).
func ValidTimeRange(s string) bool
ValidTimeRange reports whether s is an accepted search recency window.
Document is the canonical shape returned by every engine and re-serialized by every transport. The field names match Open WebUI’s external-loader contract so the OpenWebUI transport can serialize directly.
type Document struct {
PageContent string `json:"page_content"`
Metadata map[string]string `json:"metadata"`
}
Engine renders a single URL into a Document. Implementations should respect the caller-provided ctx for cancellation and deadlines.
type Engine interface {
Name() string
Matches(rawURL string) bool
Crawl(ctx context.Context, rawURL string, opts EngineOptions) (Document, error)
}
EngineOptions carries per-request knobs an engine may honor. Unknown fields are ignored by engines that don’t care.
type EngineOptions struct {
// Reddit-specific.
RedditKeepDepth bool // include depth field on comments
RedditKeepCreated bool // include created field on comments
RedditMaxRounds int // /api/morechildren expansion budget
RedditFormat string // "toon" | "json"
RedditFetchLimit int // Reddit `limit`: max comments in initial fetch (0 = engine default)
RedditDepth int // Reddit `depth`: max nesting depth (0 = engine default)
RedditSort string // Reddit `sort`: comment sort order ("" = engine default)
RedditMaxComments int // hard cap on total comments emitted (0 = unlimited)
RedditMaxTopLevel int // hard cap on top-level threads (0 = unlimited)
// Hacker News-specific. The HN engine has no upstream size params to
// forward (Algolia serves the whole tree in one response), so these are all
// post-fetch caps on what gets emitted.
HNMaxComments int // hard cap on total comments emitted (0 = engine default)
HNMaxTopLevel int // hard cap on top-level threads (0 = unlimited)
HNMaxPerSubtree int // hard cap on comments kept within each top-level thread (0 = unlimited)
// Generic-crawl (crawl4ai fallback) knobs.
//
// ScanFullPage scrolls the whole page before extraction so append-style
// infinite feeds load. Tri-state: nil = the deployment default
// (OMNIFEED_CRAWL4AI_SCAN_FULL_PAGE); callers opt in per URL — it costs
// multiple seconds and corrupts virtualized pages, so it's for feed/gallery
// URLs specifically.
ScanFullPage *bool
}
FailureKind is a bounded classification of why a crawl/fetch failed. It is the single source of truth for the taxonomy that observability renders as the `reason` metric label — carried as data on FetchError so callers never parse error strings to recover the cause.
type FailureKind string
The complete set of failure reasons. Keep this small: every value becomes a distinct metric series and a distinct thing to alert on.
const (
KindCaptcha FailureKind = "captcha" // bot wall / human-verification challenge page
KindHTTP403 FailureKind = "http_403" // explicit HTTP 403
KindHTTP429 FailureKind = "http_429" // rate limited
KindBotBlock FailureKind = "bot_block" // blocked with no clean status (nav blocked, non-JSON body)
KindThinContent FailureKind = "thin_content" // crawl4ai content-gate: too little usable content rendered (JS-only SPA shell, PDF/binary, near-empty) — not a wall, not an upstream fault
KindTimeout FailureKind = "timeout" // context deadline exceeded — omnifeed's own timeout budget (crawl4ai/reddit)
KindCanceled FailureKind = "canceled" // caller hung up before the fetch finished (client abort — not an omnifeed fault)
KindUpstreamError FailureKind = "upstream_error" // upstream 5xx or unreachable
// KindUpstreamRejected is crawl4ai's application-level 500 with the verdict
// scrubbed out of the response body (crawl4ai 0.9.2+ logs the real reason —
// bot wall, content-gate, or crash — server-side under a correlation id and
// returns a generic body). Indistinguishable client-side and dominated by
// per-page non-faults, so it gets one bounded retry (for the transient
// minority sharing the channel) and is not treated as an upstream outage.
KindUpstreamRejected FailureKind = "upstream_rejected"
// KindQuotaExhausted is omnifeed's OWN pacing verdict, not an upstream
// answer: the politeness quota for the host is spent and the wait until the
// next slot is longer than the caller's budget, so nothing was sent. The
// retry-after is in the error message. Not a fault — the deployment is
// working as configured, and the caller should retry later.
KindQuotaExhausted FailureKind = "quota_exhausted"
KindBadResponse FailureKind = "bad_response" // unparseable or empty upstream response
KindError FailureKind = "error" // anything else
)
func KindForStatus(code int) FailureKind
KindForStatus maps an HTTP status code to the matching FailureKind.
FetchError carries the classified cause of a failed crawl/fetch. Engines return it (optionally wrapping the underlying error) so observability.Reason can read Kind via errors.As instead of matching error text. StatusCode and Marker are optional context (0 / “” when not applicable).
type FetchError struct {
Kind FailureKind
StatusCode int
Marker string // matched anti-bot marker, set when Kind == KindCaptcha
Err error // underlying error, if any
}
func (e *FetchError) Error() string
func (e *FetchError) Unwrap() error
Unwrap exposes the underlying error to errors.Is / errors.As.
SearchOptions carries per-query knobs a Searcher may honor.
type SearchOptions struct {
Limit int // max results to return; <= 0 means no clamp
TimeRange string // "", "day", "week", "month", "year"
Language string // e.g. "en", "fr"; empty = upstream default
// Site restricts results to one hostname. Naming the site inside the query
// text instead ("reddit kubernetes plex") does not work: the engines read
// the site name as a topic word and return the site's own homepage and its
// Wikipedia article. Must satisfy ValidSiteFilter; empty = no restriction.
Site string
}
SearchResult is a single hit returned by a Searcher.
type SearchResult struct {
Title string `json:"title"`
URL string `json:"url"`
Snippet string `json:"snippet,omitempty"`
Engine string `json:"engine,omitempty"`
PublishedDate string `json:"published_date,omitempty"`
}
Searcher turns a query into ranked result URLs. It is the discovery counterpart of Engine: a Searcher finds URLs (query → results), Engine implementations then render them (URL → content).
type Searcher interface {
Name() string
Search(ctx context.Context, query string, opts SearchOptions) ([]SearchResult, error)
}
Truncation is the outcome of TruncateContent: the text to hand back plus the numbers a caller needs to continue where it stopped. Content was cut exactly when NextStartChar < TotalChars.
type Truncation struct {
Text string
TotalChars int // total characters (runes) in the source content
NextStartChar int // offset to pass as start_char to continue
}
func TruncateContent(content string, maxChars, startChar int) Truncation
TruncateContent returns the [startChar, startChar+maxChars) character window of content, appending a continuation marker when content remains beyond it. The marker counts against maxChars — Text never exceeds maxChars runes, so a caller-declared ceiling (anthropic/maxResultSizeChars) holds. Offsets and lengths are counted in characters (runes), never bytes, so a window never splits a multibyte character and the offsets a caller echoes back stay meaningful.
maxChars <= 0 means unlimited (startChar is still honored). A startChar at or past the end of a non-empty document yields a short explanatory message instead of a silently empty result.
func (t Truncation) Truncated() bool
Truncated reports whether content remains beyond the returned window.
Generated by gomarkdoc