import "github.com/kinorai/omnifeed/internal/config"
Package config loads all runtime configuration from OMNIFEED_-prefixed environment variables. Every knob the binary respects is declared here in one place so operators have a single source of truth.
DefaultCrawl4AIUserAgent matches the Chromium bundled in crawl4ai 0.9.5.
const DefaultCrawl4AIUserAgent = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36"
ValidSearchAudit is the complete set of OMNIFEED_SEARCH_AUDIT modes.
var ValidSearchAudit = []string{"off", "summary", "full"}
ValidSearchers is every searcher OMNIFEED_SEARCHERS may name.
var ValidSearchers = []string{"searxng", "tinyfish"}
Config is the fully-resolved runtime configuration.
type Config struct {
// HTTP loader (Open WebUI contract).
ListenAddr string
// MCP transports.
MCPListenAddr string
MCPStdio bool
// Observability.
MetricsAddr string
LogLevel string
LogFormat string
EnablePprof bool
// OTelEndpoint (OTEL_EXPORTER_OTLP_ENDPOINT) turns on OpenTelemetry
// tracing over OTLP/HTTP when set; empty = no tracing at all. The standard
// OTel names, not OMNIFEED_*, because the exporter reads the rest of the
// OTEL_EXPORTER_OTLP_* family itself. OTelServiceName (OTEL_SERVICE_NAME)
// defaults to "omnifeed".
OTelEndpoint string
OTelServiceName string
// Auth.
APIKey string
AllowNoAuth bool
// AllowedOrigins lists browser Origin values (scheme://host[:port])
// allowed to call the HTTP APIs cross-origin, on top of the always-allowed
// loopback origins. Requests without an Origin header (native clients)
// are never affected. Empty = loopback-only.
AllowedOrigins []string
// Upstream crawl4ai.
Crawl4AIURL string
Crawl4AITimeout time.Duration
Crawl4AIKeepLinks bool // render hyperlink anchor text + keep external links in markdown
Crawl4AIPruneThreshold float64 // PruningContentFilter cutoff for the generic engine (0–1; higher strips more boilerplate)
Crawl4AIWaitUntil string // page-ready signal for the generic engine: domcontentloaded (default) | load | networkidle | commit
Crawl4AIToken string // bearer token sent to crawl4ai (its CRAWL4AI_API_TOKEN); empty = no Authorization header
// Crawl4AIExcludedSelector overrides the generic engine's chrome selector
// list. Empty = the engine's conservative default; to effectively exclude
// nothing, set a selector that matches nothing.
Crawl4AIExcludedSelector string
// Crawl4AITargetElements is a comma-separated CSS selector list. Empty (the
// default) keeps the feature off; non-empty restricts extraction to matching
// containers, which can yield no content at all on pages without them.
Crawl4AITargetElements string
// Crawl4AIScanFullPage makes the generic engine scroll the full page (in
// Crawl4AIScrollDelay steps) before extraction, so lazy-loaded content
// renders — at a multi-second cost on long pages. Off by default (crawl4ai's
// own default): most agent fetches want the main content, not the infinite
// scroll tail.
Crawl4AIScanFullPage bool
Crawl4AIScrollDelay float64 // seconds between scroll steps when scanning the full page
// Crawl4AIDelayBeforeHTML is the unconditional settle (seconds) crawl4ai
// sleeps after the page-ready signal before extracting HTML — paid on every
// crawl whether or not the page needs it. crawl4ai's own default (0.1).
Crawl4AIDelayBeforeHTML float64
// Crawl4AIRemoveOverlays sends crawl4ai's remove_overlay_elements. Its
// geometry heuristic (delete any large absolute/fixed element) silently
// empties pages whose content sits in such containers — Wikipedia and
// several news fronts return only their <title>. Off by default;
// remove_consent_popups stays on regardless and covers cookie modals.
Crawl4AIRemoveOverlays bool
// Crawl4AIUserAgent is the browser identity crawl4ai presents. crawl4ai's
// own default is a malformed Chrome/116 string that Cloudflare challenges
// and some sites reject as an outdated browser. Keep the version in step
// with the Chromium the crawl4ai image bundles (156 in 0.9.5): a UA that
// disagrees with the engine is itself a bot signal. Set it empty to keep
// crawl4ai's own (and disable the alternate-identity retry).
Crawl4AIUserAgent string
// Crawl4AIStealth turns on crawl4ai's playwright-stealth patches.
Crawl4AIStealth bool
// Crawl4AIChallengeWait waits up to this long for a self-clearing bot
// challenge ("Just a moment...") to pass before extraction; 0 disables.
Crawl4AIChallengeWait time.Duration
// Crawl4AIMinProseChars rejects pages whose prose (link targets, URLs and
// markdown syntax removed) is shorter than this; 0 disables the floor.
Crawl4AIMinProseChars int
// Upstream SearXNG (optional). Empty disables the `search` MCP tool.
SearXNGURL string
SearXNGTimeout time.Duration
// SearXNGSiteEngines names the engines that run when the caller passes a
// `site` filter. SearXNG hands `site:` to the engines instead of applying it
// itself, and an engine that does not implement the operator either ignores
// it (answering with unrelated pages) or returns nothing at all — so on a
// mixed pool a site-scoped search comes back thin and noisy. Empty (the
// default) queries the whole pool; set it to the subset that honours
// `site:`, e.g. OMNIFEED_SEARXNG_SITE_ENGINES=privacywall,google cse
SearXNGSiteEngines []string
// SearXNGDelay and SearXNGQuota/SearXNGQuotaWindow pace queries to SearXNG.
// They protect the ENGINES behind it, not SearXNG itself: one query fans out
// to every enabled engine, so each engine sees exactly omnifeed's query
// rate, and an engine that judges that rate bot-like suspends itself or
// serves a CAPTCHA.
//
// Both controls are needed because engines enforce two different limits.
// Measured on this deployment's pool (2026-08-17): the strictest engine kept
// answering at a 3s spacing from a quiet start, but blocked after ~20
// requests in ~85s — it counts requests in a window. A delay alone run
// continuously would send 30 per 90s and trip it, so the delay shapes the
// gap and the quota bounds the burst.
//
// Both default to off, which keeps the previous unpaced behaviour. Set them
// to the values measured against your own pool.
SearXNGDelay time.Duration
SearXNGQuota int
SearXNGQuotaWindow time.Duration
// SearXNGConcurrency caps searches in flight across every replica. It exists
// because SearXNGDelay alone cannot: a nonzero delay serializes admissions
// cluster-wide, so without this the deployment runs one search at a time
// however many replicas it has. See internal/httpx/redislimit's package doc.
//
// Measured on this deployment's pool (2026-08-24): 16 concurrent searches
// ran clean from a cold start, and 24 blocked the strictest engine for the
// 1200s its suspension lasts. Leave margin — the cost of overshooting is a
// 20-minute engine outage, not a failed query.
//
// 0 keeps the pre-cap behaviour, where SearXNGDelay serializes.
SearXNGConcurrency int
// SearXNGMaxWait caps how long one query may sit in the pacing limiter
// before it gives up with a timeout. It is the knob behind the
// "context deadline exceeded" a fanned-out caller sees: the queue was
// longer than this, not the upstream slower. Raise it to trade latency for
// completions, lower it to push back on callers sooner. 0 uses the
// searcher's own default.
SearXNGMaxWait time.Duration
// SearXNGCriticalEngines names the engines whose absence makes a search
// degraded: the search still succeeds, but it ran without an engine the
// results lean on, so a configured backup searcher tops it up. An engine
// is absent when SearXNG reports it unresponsive (rate limited, CAPTCHA,
// ...) or when it contributed no rows — the silent block. On this pool
// that is `google cse`. Empty (the default) never reports a search
// degraded, so only errors, empty and thin answers reach the backup.
SearXNGCriticalEngines []string
// Searchers is the failover order: the first configured searcher answers
// every query, and each next one is the backup of the chain before it —
// asked only when that answer is an error, empty, thin (see
// SearchFallbackMinResults) or degraded (see SearXNGCriticalEngines).
// A listed searcher that is not configured (no URL or key) is skipped, so
// the default needs nothing new to keep today's SearXNG-only behaviour,
// and setting OMNIFEED_TINYFISH_API_KEY alone enables the backup.
Searchers []string
// SearchFallbackMinResults is the result count below which a successful
// search counts as thin and is topped up by the backup. 0 disables the
// thin trigger.
SearchFallbackMinResults int
// SearchPrimaryBudget bounds how long a searcher with a backup may take
// before the backup is asked instead. Without it a hung SearXNG spends its
// whole pacing wait plus timeout-and-retry budget (tens of seconds) before
// the backup even starts. Applies only between two chained searchers; 0
// disables it.
SearchPrimaryBudget time.Duration
// TinyFish Search API (optional searcher). TinyFishAPIKey empty disables
// it. The free tier allows 30 requests/minute and 12,000/day per key;
// TinyFishQuota/TinyFishQuotaWindow pace under the per-minute limit, and
// TinyFishMaxWait bounds how long a query queues for a slot.
TinyFishAPIKey string
TinyFishURL string
TinyFishTimeout time.Duration
TinyFishQuota int
TinyFishQuotaWindow time.Duration
TinyFishMaxWait time.Duration
// SearchAudit controls the per-search audit log: "off", "summary" or
// "full". It is deliberately NOT a log level. A level answers "how bad is
// this?", while this stream answers "what did the pool return?" — putting it
// behind DEBUG would mean enabling every other component's debug output to
// get it, and would invite sampling, which biases the per-engine statistics
// it exists to produce. Both modes emit at INFO.
//
// off nothing beyond the existing warnings (the default).
// summary one line per search: query, filters, how many rows each engine
// contributed, and which engines were unresponsive.
// full adds one line per (engine, result) with that engine's own rank —
// the position table. ~10 lines per search.
//
// Both modes log the query text; only "full" adds the result URLs. That is
// the most revealing data the service holds, so this is opt-in and its
// retention is the operator's decision.
//
// Emitted at INFO through the shared logger, so anything other than "off"
// requires OMNIFEED_LOG_LEVEL=debug or info — Load rejects the combination
// rather than let a warn-level deployment discard the feed in silence.
SearchAudit string
// Search tool limits.
SearchMaxResults int
// FetchMaxChars is the default character cap on markdown content returned by
// the fetch_url MCP tool (0 = unlimited). A caller-supplied max_chars wins.
FetchMaxChars int
// GitHubToken authenticates the GitHub engine's REST calls. Empty = anonymous
// (60 requests/hour/IP); a token raises it to 5000/hour.
GitHubToken string
// StackExchangeKey is the optional Stack Exchange API app key. Empty =
// keyless (300 requests/day/IP); a key raises the quota to 10,000/day.
StackExchangeKey string
// DiscourseHosts is the exact-hostname allowlist the Discourse engine claims
// topic URLs on. Discourse is self-hosted on arbitrary domains and Matches is
// a pure predicate (it can't probe), so the list has to be explicit. Empty
// (the variable set to "") means the engine claims nothing and every forum
// goes to the generic browser fallback.
DiscourseHosts []string
// Twitter/X engine. TwitterEnabled off sends every x.com/twitter.com URL
// to the generic fallback. TwitterFxTwitterURL is the FxTwitter API base —
// the public api.fxtwitter.com, or a self-hosted FxEmbed. TwitterMaxReplies
// caps the replies rendered under a post.
TwitterEnabled bool
TwitterFxTwitterURL string
TwitterMaxReplies int
// Reddit engine defaults.
RedditTimeout time.Duration
RedditMaxRounds int
RedditFormat string
RedditFetchLimit int // Reddit `limit` query param: max comments in initial fetch
RedditDepth int // Reddit `depth` query param: max comment-tree nesting
RedditSort string // Reddit `sort` query param: comment sort order
RedditMaxComments int // hard cap on total comments emitted (0 = unlimited)
RedditMaxTopLevel int // hard cap on top-level comment threads (0 = unlimited)
RedditKeepCreated bool // include the per-comment `created` timestamp
RedditKeepDepth bool // include the per-comment `depth` field
// RedditQuota / RedditQuotaWindow cap requests to Reddit in a rolling
// window: every request, not every crawl — the thread fetch, each
// /api/morechildren round, listings and share-link resolves all count.
// Reddit's unauthenticated budget is per IP and counted in a window, which
// the per-domain delay alone does not bound. Shared across replicas through
// Redis when OMNIFEED_REDIS_URL is set. 0 disables it (the default).
RedditQuota int
RedditQuotaWindow time.Duration
// RedditSession is a logged-in reddit_session cookie value. When set, the
// Reddit engine calls Reddit's JSON endpoints directly with it instead of
// reading pages through the browser. Secret: never log it.
RedditSession string
// RedditClientID is a Reddit OAuth app's client ID, used with the
// installed-client grant (no secret, no account). When set, it wins over
// RedditSession: it has ten times the request budget.
RedditClientID string
// RedditMethods is the order the Reddit chain tries its methods in
// (client_id, session, browser); methods without a credential are left
// out. RedditMethodCooldown is how long a method that failed in a way the
// next one can get around is skipped.
RedditMethods []string
RedditMethodCooldown time.Duration
// Distributed rate limiting (optional). RedisURL is the single opt-in
// switch: unset keeps pacing entirely in process, exactly as before. Set,
// the limiters share their state through Redis so every replica counts
// against one limit instead of one limit each.
RedisURL string
// RedisKeyPrefix namespaces the limiter keys, so several deployments can
// share one Redis instance without colliding.
RedisKeyPrefix string
// RedisTimeout is the budget for ONE Redis operation, not for one Acquire:
// an Acquire legitimately sleeps minutes between attempts while it waits out
// a quota window. On a breach the limiter fails open to in-process pacing
// for a cooldown, so a dead Redis costs one timeout per cooldown.
RedisTimeout time.Duration
// Limits and rate control.
MaxURLsPerRequest int
CrawlMaxResponseMB int
PerDomainConcurrency int
PerDomainDelay time.Duration
BlockPrivateIPs bool
// fetch_url response cache (internal/fetchcache). Stored in Redis when
// RedisURL is set (shared by every replica), else in an in-process LRU of
// CacheMaxBytes. CacheTTLThreads applies to the dedicated engines (Reddit,
// Hacker News, Discourse, Bluesky, GitHub, Stack Exchange), CacheTTLPages to the generic
// page fallback; a TTL of 0 disables caching for that class.
CacheEnabled bool
CacheTTLThreads time.Duration
CacheTTLPages time.Duration
CacheMaxBytes int
CacheMaxItemBytes int
CacheKeyPrefix string
}
func Load() (Config, error)
Load reads OMNIFEED_* env vars and returns a populated Config, or an error if a required variable is malformed. Defaults are documented inline.
Generated by gomarkdoc