import "github.com/kinorai/omnifeed/internal/domain"
Package domain holds the core types exchanged between transports and engines. It has no external dependencies and no I/O.
Reddit comment-fetch defaults are defined here so config (env fallback) and the reddit engine (zero-Options floor) share one source instead of each duplicating the literals. https://www.reddit.com/dev/api/#GET_comments_{article}
const (
DefaultRedditFetchLimit = 500 // Reddit `limit`
DefaultRedditDepth = 20 // Reddit `depth`
DefaultRedditSort = "top" // Reddit `sort`
)
Document metadata keys that more than one layer reads. Engines and the registry write them; the response cache and the transports read them.
const (
// PartialKey is "true" on a document an engine returned successfully but
// incompletely (e.g. a Reddit morechildren round was blocked mid-crawl).
// PartialReasonKey carries the classified reason. Partial documents are
// never cached: the next call should get a chance at the whole thing.
PartialKey = "partial"
PartialReasonKey = "partial_reason"
// FallbackFromKey names the dedicated engine that failed when the generic
// fallback rendered the document instead. A fallback render is a stand-in,
// so it is never cached either.
FallbackFromKey = "fallback_from"
// CacheKey is the response cache's verdict on a document: "hit", "miss"
// or "bypass". Absent when no cache sits in front of the engines.
CacheKey = "cache"
)
Content types engines report under ContentTypeKey. The Reddit engine also reports its format knob verbatim (“json”), which is simply not markdown.
const (
ContentTypeMarkdown = "markdown"
ContentTypeTOON = "toon"
)
ContentTypeKey is the Document.Metadata key every engine sets to describe the shape of PageContent: “markdown” for prose-ish text, “toon”/”json” for structured output. Size control keys off it — see TruncatableContentType.
const ContentTypeKey = "content_type"
ErrInvalidQuery marks a search refused for its input (an empty query, …) before any upstream was asked. No other searcher would answer it either, so it is never failed over.
var ErrInvalidQuery = errors.New("invalid search query")
ValidRedditSorts are the comment sort orders Reddit’s comments endpoint accepts; “confidence” is what the Reddit UI labels “best”. Kept here so config and the reddit engine validate against one list. https://www.reddit.com/dev/api/#GET_comments_{article}
var ValidRedditSorts = []string{"confidence", "top", "new", "controversial", "old", "random", "qa", "live"}
ValidTimeRanges are the recency-window filters the web_search front-ends accept (forwarded to SearXNG); “” means no filter. Kept here so the MCP and REST transports validate against one list.
var ValidTimeRanges = []string{"day", "week", "month", "year"}
func IsBlockKind(kind FailureKind) bool
IsBlockKind reports whether kind is a block or rate verdict: the upstream refused us (http_429, http_403, captcha, bot_block), or omnifeed’s own pacing did (quota_exhausted). Such a verdict is about the HOST that was asked, which is what decides whether a browser render of the page may follow it — see SameHostEngine.
func IsNoFallback(err error) bool
IsNoFallback reports whether err was marked with NoFallback.
func NoFallback(err error) error
NoFallback marks err as final: the engine already ran the generic fallback itself (on a URL it rewrote) or knows the content does not exist, so the registry must not spend another browser render on the original URL. The message and the wrapped chain — FetchError included — are unchanged.
func TruncatableContentType(contentType string) bool
TruncatableContentType reports whether generic char truncation is safe for a content type. Only markdown is: TOON and JSON carry length markers / closing delimiters, so cutting them mid-document produces a payload that lies about its own contents. Structured engines bound their output with their own element caps instead (see the Reddit knobs).
func ValidRedditSort(s string) bool
ValidRedditSort reports whether s is an accepted Reddit comment sort order.
func ValidSiteFilter(s string) bool
ValidSiteFilter reports whether s is a bare hostname usable as a `site:` search filter (e.g. “reddit.com”, “forums.plex.tv”).
func ValidTimeRange(s string) bool
ValidTimeRange reports whether s is an accepted search recency window.
Document is the canonical shape returned by every engine and re-serialized by every transport. The field names match Open WebUI’s external-loader contract so the OpenWebUI transport can serialize directly.
type Document struct {
PageContent string `json:"page_content"`
Metadata map[string]string `json:"metadata"`
}
Engine renders a single URL into a Document. Implementations should respect the caller-provided ctx for cancellation and deadlines.
type Engine interface {
Name() string
Matches(rawURL string) bool
Crawl(ctx context.Context, rawURL string, opts EngineOptions) (Document, error)
}
EngineOptions carries per-request knobs an engine may honor. Unknown fields are ignored by engines that don’t care.
type EngineOptions struct {
// FormatExplicit is true when the caller explicitly asked for a structured
// format (format=json|toon), as opposed to inheriting the deployment
// default. Such a caller parses the reply, so a dedicated engine failing
// must surface its error instead of handing back the generic engine's
// markdown under the same success shape.
FormatExplicit bool
// Reddit-specific.
RedditKeepDepth bool // include depth field on comments
RedditKeepCreated bool // include created field on comments
RedditMaxRounds *int // /api/morechildren expansion rounds (nil = engine default, 0 = none)
RedditFormat string // "toon" | "json"
RedditFetchLimit int // Reddit `limit`: max comments in initial fetch (0 = engine default)
RedditDepth int // Reddit `depth`: max nesting depth (0 = engine default)
RedditSort string // Reddit `sort`: comment sort order ("" = engine default)
RedditMaxComments int // hard cap on total comments emitted (0 = unlimited)
RedditMaxTopLevel int // hard cap on top-level threads (0 = unlimited)
// Hacker News-specific. The HN engine has no upstream size params to
// forward (Algolia serves the whole tree in one response), so these are all
// post-fetch caps on what gets emitted.
HNMaxComments int // hard cap on total comments emitted (0 = engine default)
HNMaxTopLevel int // hard cap on top-level threads (0 = unlimited)
HNMaxPerSubtree int // hard cap on comments kept within each top-level thread (0 = unlimited)
// Generic-crawl (crawl4ai fallback) knobs.
//
// ScanFullPage scrolls the whole page before extraction so append-style
// infinite feeds load. Tri-state: nil = the deployment default
// (OMNIFEED_CRAWL4AI_SCAN_FULL_PAGE); callers opt in per URL — it costs
// multiple seconds and corrupts virtualized pages, so it's for feed/gallery
// URLs specifically.
ScanFullPage *bool
// NoCache skips the fetch_url response cache for this request: the URL is
// fetched fresh, and a cacheable result still refreshes the cache entry.
// It never changes what an engine renders, so it is not part of the cache
// key.
NoCache bool
}
FailureKind is a bounded classification of why a crawl/fetch failed. It is the single source of truth for the taxonomy that observability renders as the `reason` metric label — carried as data on FetchError so callers never parse error strings to recover the cause.
type FailureKind string
The complete set of failure reasons. Keep this small: every value becomes a distinct metric series and a distinct thing to alert on.
const (
KindCaptcha FailureKind = "captcha" // bot wall / human-verification challenge page
KindHTTP403 FailureKind = "http_403" // explicit HTTP 403
KindHTTP429 FailureKind = "http_429" // rate limited
KindNotFound FailureKind = "not_found" // the requested page answered 404/410 or is a soft 404: the content does not exist
KindSiteError FailureKind = "site_error" // the requested site itself answered 5xx — not omnifeed's or crawl4ai's fault
KindBotBlock FailureKind = "bot_block" // blocked with no clean status (nav blocked, non-JSON body)
KindThinContent FailureKind = "thin_content" // crawl4ai content-gate: too little usable content rendered (JS-only SPA shell, PDF/binary, near-empty) — not a wall, not an upstream fault
KindTimeout FailureKind = "timeout" // context deadline exceeded — omnifeed's own timeout budget (crawl4ai/reddit)
KindCanceled FailureKind = "canceled" // caller hung up before the fetch finished (client abort — not an omnifeed fault)
KindUpstreamError FailureKind = "upstream_error" // upstream 5xx or unreachable
// KindUpstreamRejected is crawl4ai's application-level 500 with the verdict
// scrubbed out of the response body (crawl4ai 0.9.2+ logs the real reason —
// bot wall, content-gate, or crash — server-side under a correlation id and
// returns a generic body). Indistinguishable client-side and dominated by
// per-page non-faults, so it gets one bounded retry (for the transient
// minority sharing the channel) and is not treated as an upstream outage.
KindUpstreamRejected FailureKind = "upstream_rejected"
// KindQuotaExhausted is omnifeed's OWN pacing verdict, not an upstream
// answer: the politeness quota for the host is spent and the wait until the
// next slot is longer than the caller's budget, so nothing was sent. The
// retry-after is in the error message. Not a fault — the deployment is
// working as configured, and the caller should retry later.
KindQuotaExhausted FailureKind = "quota_exhausted"
KindBadResponse FailureKind = "bad_response" // unparseable or empty upstream response
// KindLayout is a page that rendered but that the engine could not read:
// the markup it extracts from is gone or changed (a site redesign), or an
// interstitial it does not know. Not a block and not the content's fault,
// so it is not retried elsewhere: the engine's extraction needs updating.
KindLayout FailureKind = "layout"
// KindDNSBlocked is a URL refused before any fetch because the resolver
// answered the unspecified address (0.0.0.0 or ::) for its host: a DNS
// filter blocks the name. Not the caller's mistake and not an upstream
// fault, and retrying cannot help while the filter lists it.
KindDNSBlocked FailureKind = "dns_blocked"
KindError FailureKind = "error" // anything else
)
func KindForStatus(code int) FailureKind
KindForStatus maps an HTTP status code to the matching FailureKind.
FetchError carries the classified cause of a failed crawl/fetch. Engines return it (optionally wrapping the underlying error) so observability.Reason can read Kind via errors.As instead of matching error text. StatusCode and Marker are optional context (0 / “” when not applicable).
type FetchError struct {
Kind FailureKind
StatusCode int
Marker string // matched anti-bot marker, set when Kind == KindCaptcha
Err error // underlying error, if any
// RetryAfter is how long the upstream (or omnifeed's own pacing) asked the
// caller to wait before trying again; 0 when nobody said. Transports render
// it as retry_after_s so a caller can back off instead of hammering.
RetryAfter time.Duration
}
func (e *FetchError) Error() string
func (e *FetchError) Unwrap() error
Unwrap exposes the underlying error to errors.Is / errors.As.
InvalidRequestError marks a failure as the caller’s mistake (a URL that is malformed, uses a forbidden scheme, or targets a private address) rather than anything upstream. Transports map it to the invalid_request error code via errors.As; Error() is the wrapped cause verbatim, so wrapping changes no message and no metric label.
type InvalidRequestError struct{ Err error }
func (e *InvalidRequestError) Error() string
func (e *InvalidRequestError) Unwrap() error
Unwrap exposes the cause to errors.Is / errors.As.
SameHostEngine is an optional Engine capability: SameHostAsPage reports whether the engine fetches from the same host that serves the page URL (Reddit’s in-browser .json, a Discourse forum’s own topic JSON), as opposed to a separate API host (api.github.com, hn.algolia.com, the Bluesky AppView).
It decides whether a block or rate verdict may be followed by the generic browser render. For a same-host engine the render would hit the host that just refused us — or that our own pacing is holding back — and prolong the block, so it is refused. For a separate-host engine the API’s quota says nothing about the page host, so the render is still the right answer (e.g. an anonymous GitHub deployment over its 60/h API quota renders github.com). Engines that do not implement it are treated as separate-host.
type SameHostEngine interface {
SameHostAsPage() bool
}
SearchOptions carries per-query knobs a Searcher may honor.
type SearchOptions struct {
Limit int // max results to return; <= 0 means no clamp
TimeRange string // "", "day", "week", "month", "year"
Language string // e.g. "en", "fr"; empty = upstream default
// Site restricts results to one hostname. Naming the site inside the query
// text instead ("reddit kubernetes plex") does not work: the engines read
// the site name as a topic word and return the site's own homepage and its
// Wikipedia article. Must satisfy ValidSiteFilter; empty = no restriction.
Site string
}
SearchReport is a Searcher’s answer to one query: the ranked results, and whether the upstream answered at full strength.
type SearchReport struct {
Results []SearchResult
// DegradedReason is set when the search succeeded but a source the
// operator relies on did not contribute — e.g. an engine that was rate
// limited or silently blocked — so the results are weaker than usual. It
// names the source and why, for logs and metrics. Empty means the search
// ran at full strength.
DegradedReason string
}
func (r SearchReport) Degraded() bool
Degraded reports whether the search ran without a source it relies on.
SearchResult is a single hit returned by a Searcher.
type SearchResult struct {
Title string `json:"title"`
URL string `json:"url"`
Snippet string `json:"snippet,omitempty"`
Engine string `json:"engine,omitempty"`
PublishedDate string `json:"published_date,omitempty"`
}
Searcher turns a query into ranked result URLs. It is the discovery counterpart of Engine: a Searcher finds URLs (query → results), Engine implementations then render them (URL → content).
type Searcher interface {
Name() string
Search(ctx context.Context, query string, opts SearchOptions) (SearchReport, error)
}
Truncation is the outcome of TruncateContent: the text to hand back plus the numbers a caller needs to continue where it stopped. Content was cut exactly when NextStartChar < TotalChars.
type Truncation struct {
Text string
TotalChars int // total characters (runes) in the source content
NextStartChar int // offset to pass as start_char to continue
}
func TruncateContent(content string, maxChars, startChar int) Truncation
TruncateContent returns the [startChar, startChar+maxChars) character window of content, appending a continuation marker when content remains beyond it. The marker counts against maxChars — Text never exceeds maxChars runes, so a caller-declared ceiling (anthropic/maxResultSizeChars) holds. Offsets and lengths are counted in characters (runes), never bytes, so a window never splits a multibyte character and the offsets a caller echoes back stay meaningful.
maxChars <= 0 means unlimited (startChar is still honored). A startChar at or past the end of a non-empty document yields a short explanatory message instead of a silently empty result.
func (t Truncation) Truncated() bool
Truncated reports whether content remains beyond the returned window.
Generated by gomarkdoc