import "github.com/kinorai/omnifeed/internal/config"
Package config loads all runtime configuration from OMNIFEED_-prefixed environment variables. Every knob the binary respects is declared here in one place so operators have a single source of truth.
ValidSearchAudit is the complete set of OMNIFEED_SEARCH_AUDIT modes.
var ValidSearchAudit = []string{"off", "summary", "full"}
Config is the fully-resolved runtime configuration.
type Config struct {
// HTTP loader (Open WebUI contract).
ListenAddr string
// MCP transports.
MCPListenAddr string
MCPStdio bool
// Observability.
MetricsAddr string
LogLevel string
LogFormat string
EnablePprof bool
// Auth.
APIKey string
AllowNoAuth bool
// AllowedOrigins lists browser Origin values (scheme://host[:port])
// allowed to call the HTTP APIs cross-origin, on top of the always-allowed
// loopback origins. Requests without an Origin header (native clients)
// are never affected. Empty = loopback-only.
AllowedOrigins []string
// Upstream crawl4ai.
Crawl4AIURL string
Crawl4AITimeout time.Duration
Crawl4AIKeepLinks bool // render hyperlink anchor text + keep external links in markdown
Crawl4AIPruneThreshold float64 // PruningContentFilter cutoff for the generic engine (0–1; higher strips more boilerplate)
Crawl4AIWaitUntil string // page-ready signal for the generic engine: domcontentloaded (default) | load | networkidle | commit
Crawl4AIToken string // bearer token sent to crawl4ai (its CRAWL4AI_API_TOKEN); empty = no Authorization header
// Crawl4AIExcludedSelector overrides the generic engine's chrome selector
// list. Empty = the engine's conservative default; to effectively exclude
// nothing, set a selector that matches nothing.
Crawl4AIExcludedSelector string
// Crawl4AITargetElements is a comma-separated CSS selector list. Empty (the
// default) keeps the feature off; non-empty restricts extraction to matching
// containers, which can yield no content at all on pages without them.
Crawl4AITargetElements string
// Crawl4AIScanFullPage makes the generic engine scroll the full page (in
// Crawl4AIScrollDelay steps) before extraction, so lazy-loaded content
// renders — at a multi-second cost on long pages. Off by default (crawl4ai's
// own default): most agent fetches want the main content, not the infinite
// scroll tail.
Crawl4AIScanFullPage bool
Crawl4AIScrollDelay float64 // seconds between scroll steps when scanning the full page
// Crawl4AIDelayBeforeHTML is the unconditional settle (seconds) crawl4ai
// sleeps after the page-ready signal before extracting HTML — paid on every
// crawl whether or not the page needs it. crawl4ai's own default (0.1).
Crawl4AIDelayBeforeHTML float64
// Crawl4AIRemoveOverlays sends crawl4ai's remove_overlay_elements. Its
// geometry heuristic (delete any large absolute/fixed element) silently
// empties pages whose content sits in such containers — Wikipedia and
// several news fronts return only their <title>. Off by default;
// remove_consent_popups stays on regardless and covers cookie modals.
Crawl4AIRemoveOverlays bool
// Upstream SearXNG (optional). Empty disables the `search` MCP tool.
SearXNGURL string
SearXNGTimeout time.Duration
// SearXNGSiteEngines names the engines that run when the caller passes a
// `site` filter. SearXNG hands `site:` to the engines instead of applying it
// itself, and an engine that does not implement the operator either ignores
// it (answering with unrelated pages) or returns nothing at all — so on a
// mixed pool a site-scoped search comes back thin and noisy. Empty (the
// default) queries the whole pool; set it to the subset that honours
// `site:`, e.g. OMNIFEED_SEARXNG_SITE_ENGINES=privacywall,google cse
SearXNGSiteEngines []string
// SearXNGDelay and SearXNGQuota/SearXNGQuotaWindow pace queries to SearXNG.
// They protect the ENGINES behind it, not SearXNG itself: one query fans out
// to every enabled engine, so each engine sees exactly omnifeed's query
// rate, and an engine that judges that rate bot-like suspends itself or
// serves a CAPTCHA.
//
// Both controls are needed because engines enforce two different limits.
// Measured on this deployment's pool (2026-08-17): the strictest engine kept
// answering at a 3s spacing from a quiet start, but blocked after ~20
// requests in ~85s — it counts requests in a window. A delay alone run
// continuously would send 30 per 90s and trip it, so the delay shapes the
// gap and the quota bounds the burst.
//
// Both default to off, which keeps the previous unpaced behaviour. Set them
// to the values measured against your own pool.
SearXNGDelay time.Duration
SearXNGQuota int
SearXNGQuotaWindow time.Duration
// SearXNGConcurrency caps searches in flight across every replica. It exists
// because SearXNGDelay alone cannot: a nonzero delay serializes admissions
// cluster-wide, so without this the deployment runs one search at a time
// however many replicas it has. See internal/httpx/redislimit's package doc.
//
// Measured on this deployment's pool (2026-08-24): 16 concurrent searches
// ran clean from a cold start, and 24 blocked the strictest engine for the
// 1200s its suspension lasts. Leave margin — the cost of overshooting is a
// 20-minute engine outage, not a failed query.
//
// 0 keeps the pre-cap behaviour, where SearXNGDelay serializes.
SearXNGConcurrency int
// SearXNGMaxWait caps how long one query may sit in the pacing limiter
// before it gives up with a timeout. It is the knob behind the
// "context deadline exceeded" a fanned-out caller sees: the queue was
// longer than this, not the upstream slower. Raise it to trade latency for
// completions, lower it to push back on callers sooner. 0 uses the
// searcher's own default.
SearXNGMaxWait time.Duration
// SearchAudit controls the per-search audit log: "off", "summary" or
// "full". It is deliberately NOT a log level. A level answers "how bad is
// this?", while this stream answers "what did the pool return?" — putting it
// behind DEBUG would mean enabling every other component's debug output to
// get it, and would invite sampling, which biases the per-engine statistics
// it exists to produce. Both modes emit at INFO.
//
// off nothing beyond the existing warnings (the default).
// summary one line per search: query, filters, how many rows each engine
// contributed, and which engines were unresponsive.
// full adds one line per (engine, result) with that engine's own rank —
// the position table. ~10 lines per search.
//
// Both modes log the query text; only "full" adds the result URLs. That is
// the most revealing data the service holds, so this is opt-in and its
// retention is the operator's decision.
//
// Emitted at INFO through the shared logger, so anything other than "off"
// requires OMNIFEED_LOG_LEVEL=debug or info — Load rejects the combination
// rather than let a warn-level deployment discard the feed in silence.
SearchAudit string
// Search tool limits.
SearchMaxResults int
// FetchMaxChars is the default character cap on markdown content returned by
// the fetch_url MCP tool (0 = unlimited). A caller-supplied max_chars wins.
FetchMaxChars int
// GitHubToken authenticates the GitHub engine's REST calls. Empty = anonymous
// (60 requests/hour/IP); a token raises it to 5000/hour.
GitHubToken string
// DiscourseHosts is the exact-hostname allowlist the Discourse engine claims
// topic URLs on. Discourse is self-hosted on arbitrary domains and Matches is
// a pure predicate (it can't probe), so the list has to be explicit. Empty
// (the variable set to "") means the engine claims nothing and every forum
// goes to the generic browser fallback.
DiscourseHosts []string
// Reddit engine defaults.
RedditTimeout time.Duration
RedditMaxRounds int
RedditFormat string
RedditFetchLimit int // Reddit `limit` query param: max comments in initial fetch
RedditDepth int // Reddit `depth` query param: max comment-tree nesting
RedditSort string // Reddit `sort` query param: comment sort order
RedditMaxComments int // hard cap on total comments emitted (0 = unlimited)
RedditMaxTopLevel int // hard cap on top-level comment threads (0 = unlimited)
RedditKeepCreated bool // include the per-comment `created` timestamp
RedditKeepDepth bool // include the per-comment `depth` field
// Distributed rate limiting (optional). RedisURL is the single opt-in
// switch: unset keeps pacing entirely in process, exactly as before. Set,
// the limiters share their state through Redis so every replica counts
// against one limit instead of one limit each.
RedisURL string
// RedisKeyPrefix namespaces the limiter keys, so several deployments can
// share one Redis instance without colliding.
RedisKeyPrefix string
// RedisTimeout is the budget for ONE Redis operation, not for one Acquire:
// an Acquire legitimately sleeps minutes between attempts while it waits out
// a quota window. On a breach the limiter fails open to in-process pacing
// for a cooldown, so a dead Redis costs one timeout per cooldown.
RedisTimeout time.Duration
// Limits and rate control.
MaxURLsPerRequest int
PerDomainConcurrency int
PerDomainDelay time.Duration
BlockPrivateIPs bool
}
func Load() (Config, error)
Load reads OMNIFEED_* env vars and returns a populated Config, or an error if a required variable is malformed. Defaults are documented inline.
Generated by gomarkdoc