omnifeed

Configuration

omnifeed reads OMNIFEED_-prefixed environment variables. You usually set only three: OMNIFEED_API_KEY, OMNIFEED_CRAWL4AI_URL and, for search, OMNIFEED_SEARXNG_URL.

The binary reads every variable below. The Apple container launcher has a few OMNIFEED_ variables of its own, for container names, host ports and images, which the binary never sees. They are in apple-container.md.

Egress. Reddit and the generic fallback reach the web through crawl4ai. Six engines call their upstream directly: Hacker News reads hn.algolia.com, GitHub reads api.github.com, Bluesky reads public.api.bsky.app, Discourse reads each host in OMNIFEED_DISCOURSE_HOSTS, Stack Exchange reads api.stackexchange.com, and Twitter/X reads api.fxtwitter.com (or OMNIFEED_TWITTER_FXTWITTER_URL), cdn.syndication.twimg.com, api.vxtwitter.com and t.co. If outbound traffic may reach only crawl4ai, those six break; Twitter/X then falls back to crawl4ai on the x.com URL. Restrict crawl4ai’s own egress to public addresses with a network policy or firewall: its browser follows redirects that omnifeed cannot check before they are fetched (SECURITY.md).

Variable Default Purpose
OMNIFEED_API_KEY (unset) Bearer token for /crawl, /search and /mcp. If unset, omnifeed refuses to start unless OMNIFEED_DEV_NO_AUTH=true. Stdio MCP ignores it.
OMNIFEED_CRAWL4AI_URL (required) crawl4ai endpoint. Reddit and the generic fallback fetch through it. If empty, omnifeed exits at startup.
OMNIFEED_CRAWL4AI_TOKEN (unset) Bearer token sent to crawl4ai, matching its CRAWL4AI_API_TOKEN. crawl4ai binds beyond loopback only when a token is set. Unset sends no Authorization header.
OMNIFEED_SEARXNG_URL (unset) SearXNG base URL, e.g. http://searxng:8080. Unset hides web_search and /search. The instance must enable the json format.
OMNIFEED_DEV_NO_AUTH false Run the HTTP transports with no auth when no key is set. Local use only. Ignored if a key is set.
OMNIFEED_LISTEN_ADDR :8080 HTTP listen address for /crawl and /search
OMNIFEED_MCP_LISTEN_ADDR :8081 MCP HTTP/SSE listen address
OMNIFEED_MCP_STDIO false Run MCP over stdio, same as --mcp-stdio
OMNIFEED_METRICS_ADDR :9090 Prometheus and health listen address
OMNIFEED_CRAWL4AI_TIMEOUT 90s Per-call timeout to crawl4ai
OMNIFEED_CRAWL4AI_KEEP_LINKS true Keep link anchor text and external links in fetched markdown. false strips them, which loses link-dense content like HN titles.
OMNIFEED_CRAWL4AI_PRUNE_THRESHOLD 0.48 PruningContentFilter cutoff, 0 to 1, for the generic engine. Raise it to strip more boilerplate from noisy pages. Lower it to keep more.
OMNIFEED_CRAWL4AI_WAIT_UNTIL domcontentloaded crawl4ai page-ready signal: domcontentloaded, load, networkidle or commit. domcontentloaded fires before client-side frameworks hydrate, so JS-only SPAs can come back empty. networkidle waits for them and slows every page.
OMNIFEED_CRAWL4AI_SCAN_FULL_PAGE false Scroll the full page before extraction by default (crawl4ai scan_full_page). In benchmarks it gained content only on append-style infinite-scroll feeds, tripled latency on every page, and corrupted virtualized pages through an open crawl4ai bug. Prefer the per-request scan_full_page argument on fetch_url or POST /crawl?scan_full_page=true. Either overrides this setting.
OMNIFEED_CRAWL4AI_SCROLL_DELAY 0.5 Seconds between scroll steps during a full-page scan. Sent only when the scan is on.
OMNIFEED_CRAWL4AI_REMOVE_OVERLAYS false Send crawl4ai’s remove_overlay_elements. Leave it off. It deletes every large fixed- or absolute-position element, which empties pages whose content lives in one: Wikipedia and several news fronts return only their <title>. remove_consent_popups handles cookie modals and stays on regardless.
OMNIFEED_CRAWL4AI_DELAY_BEFORE_HTML 0.1 Seconds to wait after the page-ready signal before extracting HTML (crawl4ai delay_before_return_html), on every crawl. Raise it, e.g. to 1.0, if pages that render late come back thin.
OMNIFEED_CRAWL4AI_EXCLUDED_SELECTOR .sidebar,.toc,#toc,.related,.newsletter,.cookie-banner,[aria-label*='cookie'],svg title,[class*='icon'] title,.highcharts-axis-labels,#didomi-host,#onetrust-consent-sdk,#usercentrics-root,.truste_box_overlay CSS selectors the generic engine drops before extraction (crawl4ai excluded_selector). The default matches only page chrome and text that is never content: SVG and icon-font labels, chart axis labels, and consent-manager roots. It can’t remove article text. Your list replaces it. Empty keeps the default. If exclusion empties a page, because its content was a #toc or .sidebar, the crawl retries once without it.
OMNIFEED_CRAWL4AI_USER_AGENT Chrome 156 on Linux Browser identity sent as crawl4ai user_agent. crawl4ai’s own default is a malformed Chrome/116 string that Cloudflare challenges. Keep the version in step with the Chromium bundled in your crawl4ai image (156 in 0.9.5). A page blocked under this identity is retried once under crawl4ai’s default, because Akamai-fronted sites accept that one and reject the modern string. A Cloudflare challenge is not: Cloudflare rejects that identity. Set it empty to use crawl4ai’s own and skip that retry.
OMNIFEED_CRAWL4AI_STEALTH true Send crawl4ai enable_stealth (playwright-stealth patches).
OMNIFEED_CRAWL4AI_CHALLENGE_WAIT 8s Wait up to this long for a self-clearing Cloudflare challenge (“Just a moment…”) to pass before extraction. Pages without one pay nothing. The wait is a CSS selector (wait_for), because crawl4ai 0.9.5 refuses a JavaScript one from a request. A page with no visible element (an empty page, an XML feed) runs the wait out too, so a wait that runs out is followed by one crawl without it, and the page that comes back decides: still a Cloudflare challenge fails as captcha; anything else is handled as a normal page. Longer waits rarely clear more and make every blocked page that much slower to fail. 0 disables. Max 60s.
OMNIFEED_CRAWL4AI_MIN_PROSE_CHARS 100 Reject a rendered page as thin_content when its readable text, without link targets, URLs and markdown syntax, is shorter than this. Catches navigation-only shells and empty app frames that would otherwise return as success. 0 disables.
OMNIFEED_CRAWL4AI_TARGET_ELEMENTS (unset, off) Comma-separated CSS selectors. When set, the generic engine extracts markdown only from matching containers (crawl4ai target_elements). Good on article and repo pages, but a page with no match returns empty content, and the thin-content guard reports an error. Test it on your own URLs first. A starting list: article, main, [role=main], .markdown-body, .post-content, #content.
OMNIFEED_SEARXNG_TIMEOUT 15s Per-query timeout to SearXNG
OMNIFEED_SEARXNG_SITE_ENGINES (unset, whole pool) Comma-separated SearXNG engines to query when the caller passes a valid site filter, e.g. privacywall,google cse. SearXNG forwards site: to its engines. Some honor it, some ignore it and return unrelated pages, and some return zero results. List the ones that honor it. Unset queries every engine.
OMNIFEED_SEARXNG_DELAY 0 (off) Minimum gap between queries to SearXNG. This paces the engines behind SearXNG: each query fans out to every enabled engine, so each engine sees omnifeed’s query rate, and one that finds it bot-like suspends itself or serves a CAPTCHA. A gap alone doesn’t bound a burst, so pair it with OMNIFEED_SEARXNG_QUOTA.
OMNIFEED_SEARXNG_QUOTA 0 (off) Maximum queries in any rolling OMNIFEED_SEARXNG_QUOTA_WINDOW. Engines enforce two limits. One measured pool answered at a 3s gap from a quiet start, then blocked after about 20 queries in 85s because it counts requests per window. The delay sets the gap and the quota bounds the burst. Measure both on your own pool; the engines’ daily and hourly budgets are in rate limits.
OMNIFEED_SEARXNG_QUOTA_WINDOW 90s Rolling window for OMNIFEED_SEARXNG_QUOTA. Ignored when the quota is 0. Must be above 0 when the quota is set.
OMNIFEED_SEARXNG_CONCURRENCY 0 (off) Maximum searches in flight to SearXNG across all replicas. A nonzero OMNIFEED_SEARXNG_DELAY alone serializes admissions, so the whole deployment runs one search at a time. Above 0, the delay spaces sends and this sets the bound. With Redis, slots are TTL leases, so a pod that dies mid-search frees its slot. On one pool (2026-08-24), 16 concurrent ran clean and 24 got the strictest engine suspended for 1200s. Leave margin: overshooting costs a 20-minute engine outage, not one failed query. A single instance needs no Redis. With several, set OMNIFEED_REDIS_URL, or N instances send N times this.
OMNIFEED_SEARXNG_MAX_WAIT 15s How long a query may wait in the pacing limiter before it gives up. A fanned-out caller that sees context deadline exceeded hit this queue limit, not a slow upstream. Raise it to trade latency for completions.
OMNIFEED_REDIS_URL (unset, off) Share rate-limiter state through Redis, e.g. redis://user:pass@redis:6379/2, or rediss:// for TLS. Unset, each replica paces alone, so N replicas send up to N times the configured rate, and OMNIFEED_SEARXNG_QUOTA and OMNIFEED_PER_DOMAIN_DELAY apply per pod. Set, the delay and quota count once for the deployment. The Reddit method chain also shares its holds through it (see OMNIFEED_REDDIT_METHODS). If Redis is unreachable, the limiters pace in process and crawls keep working. OMNIFEED_PER_DOMAIN_CONCURRENCY stays per pod by design, so the replica count multiplies it.
OMNIFEED_REDIS_KEY_PREFIX omnifeed:ratelimit Key namespace, so several deployments can share one Redis without counting each other’s requests. Every key has a TTL, which matters on a shared noeviction instance.
OMNIFEED_REDIS_TIMEOUT 250ms Budget for one Redis operation. An acquisition can wait minutes for a quota window, and only each Redis call inside it gets this budget. On a breach the limiter paces in process for a 30s cooldown before trying Redis again, so a dead Redis costs one timeout per cooldown, not one per request.
OMNIFEED_SEARCH_AUDIT off Per-search audit log: off, summary or full. It is a data feed, not a log level. Putting it behind debug would force every component’s debug output on, and invite sampling, which biases the per-engine statistics. summary logs one line per search with query_id, query, site, time_range, total, duration_ms and per-engine row counts, plus one line per unresponsive engine with engine and reason_class. full adds one line per engine and result with that engine’s own rank, joined on query_id. Lines stay flat because log stores turn JSON arrays into opaque strings that can’t be grouped or averaged. Both modes log the query, and full logs every result URL, so retention is your call. Lines log at INFO, so startup rejects any mode but off unless OMNIFEED_LOG_LEVEL is debug or info. The omnifeed_search_engine_position_rank and omnifeed_search_engine_unique_results_total metrics carry no queries or URLs and are always recorded.
OMNIFEED_SEARCH_MAX_RESULTS 25 Cap on the search limit argument, 1 to 100
OMNIFEED_SEARCHERS searxng,tinyfish Search failover order. The first configured searcher answers every query; each next one is the backup of the chain before it. A listed searcher with no URL or key is skipped, so the default stays SearXNG-only until OMNIFEED_TINYFISH_API_KEY is set. Valid names: searxng, tinyfish. An unknown or repeated name fails startup.
OMNIFEED_SEARXNG_CRITICAL_ENGINES (unset) Comma-separated SearXNG engines whose absence makes a search degraded: it succeeded, but one of these was rate limited, CAPTCHA’d or returned no rows, so the backup tops it up. On a pool that leans on Google, google cse. A critical engine outside OMNIFEED_SEARXNG_SITE_ENGINES doesn’t count on site-scoped searches, which never ask it. Unset never reports degraded.
OMNIFEED_SEARCH_FALLBACK_MIN_RESULTS 3 A successful search with fewer results than this is thin and is topped up by the backup. A caller limit below it lowers the bar to the limit, so 2 results to limit=2 are complete. 0 disables the thin trigger.
OMNIFEED_SEARCH_PRIMARY_BUDGET 12s How long a searcher with a backup may take before the backup is asked instead. Without it a hung SearXNG spends its pacing wait plus up to 3 timed-out attempts before the backup starts. Applies only between chained searchers. 0 disables it.
OMNIFEED_TINYFISH_API_KEY (unset, off) TinyFish Search API key. Set, it adds tinyfish to the searcher chain. Free tier: 30 requests/minute and 12,000/day per key, then HTTP 402 until 00:00 UTC, which omnifeed reports as quota_exhausted with that wait. A 429 is not retried: the limit is per minute, so a retry seconds later can’t succeed. Every query it answers leaves your network, and TinyFish’s terms let it train on usage.
OMNIFEED_TINYFISH_URL https://api.search.tinyfish.ai TinyFish Search endpoint. Must be https, since it carries the key; plain http is accepted to a loopback host only. Redirects are never followed, because net/http would copy X-API-Key to the redirect target.
OMNIFEED_TINYFISH_TIMEOUT 10s Per-attempt timeout to TinyFish. Its measured median was about 2s, worst about 6.5s.
OMNIFEED_TINYFISH_QUOTA 25 Maximum TinyFish queries per OMNIFEED_TINYFISH_QUOTA_WINDOW, shared across replicas with Redis. The default stays under the free tier’s 30/minute. 0 sends unpaced.
OMNIFEED_TINYFISH_QUOTA_WINDOW 1m Rolling window for OMNIFEED_TINYFISH_QUOTA
OMNIFEED_TINYFISH_MAX_WAIT 5s How long a TinyFish query may wait for a pacing slot before it fails with quota_exhausted. 0 means the 5s default, not unbounded.
OMNIFEED_FETCH_MAX_CHARS 120000 Default cap on markdown returned by fetch_url. 0 is unlimited. Over the cap, the reply ends with a resumable truncation marker. TOON and JSON output from the dedicated engines is never cut; use their own limits. Doesn’t apply to /crawl. See Controlling fetched content size.
OMNIFEED_GITHUB_TOKEN (unset) Personal access token for the GitHub engine. Anonymous access allows 60 requests per hour per IP. A token raises it to 5000. A repository root costs 3 requests, most other pages 1. Discussions need it: they are read through GraphQL, which refuses anonymous callers, so without a token discussion URLs go to the generic fallback. A fine-grained token with no extra permissions (public repositories, read-only) is enough.
OMNIFEED_STACKEXCHANGE_KEY (unset) App key for the Stack Exchange engine, registered on Stack Apps. Keyless access allows 300 requests a day per IP, shared with every other keyless client on that IP. A key raises it to 10,000. A question costs 2 requests (question, then answers by votes), an /a/{id} short link 3. Stack Exchange does not treat the key as a secret: it travels in the query string, so it shows in traces of the outgoing call.
OMNIFEED_DISCOURSE_HOSTS meta.discourse.org,discuss.python.org,users.rust-lang.org,internals.rust-lang.org,discuss.pytorch.org Hostnames where the Discourse engine claims /t/… topic URLs. Discourse runs on arbitrary domains and the engine can’t detect it, so list the forums you use. Matching is exact and case-insensitive, with no subdomain wildcards. Unlisted forums go through the generic browser fallback, which returns less of the thread. An empty string disables the engine.
OMNIFEED_TWITTER_ENABLED true Render X/Twitter post URLs (x.com, twitter.com, fxtwitter.com, fixupx.com, vxtwitter.com, fixvx.com, and t.co links that land on a post) with the Twitter engine. false sends them to the generic fallback, which loads x.com in crawl4ai. See Twitter/X posts.
OMNIFEED_TWITTER_FXTWITTER_URL https://api.fxtwitter.com Base URL of the FxTwitter API the engine reads first. Point it at a self-hosted FxEmbed to stop depending on the public instance.
OMNIFEED_TWITTER_MAX_REPLIES 20 Replies rendered under a post, from the first page FxTwitter returns (about 35). Must be at least 1.
OMNIFEED_REDDIT_TIMEOUT 4m Wall-clock cap for a Reddit thread expansion
OMNIFEED_REDDIT_MAX_ROUNDS 3 Default /api/morechildren rounds, capped at 40. 0 here means the default 3. Only a request’s expand: 0 turns expansion off. ?expand=full allows up to 40. The browser method reads it as the most “more replies” requests a thread page may send, capped at 10.
OMNIFEED_REDDIT_FORMAT toon Default Reddit output: toon or json
OMNIFEED_REDDIT_FETCH_LIMIT 500 Reddit limit: max comments in the initial tree
OMNIFEED_REDDIT_DEPTH 20 Reddit depth: max nesting depth of the initial tree
OMNIFEED_REDDIT_SORT top Reddit sort: confidence (best), top, new, controversial, old, random, qa or live
OMNIFEED_REDDIT_MAX_COMMENTS 0 Cap on total comments after expansion. 0 is unlimited.
OMNIFEED_REDDIT_MAX_TOP_LEVEL 0 Cap on top-level threads, replies included. 0 is unlimited.
OMNIFEED_REDDIT_KEEP_CREATED true Include each comment’s created timestamp
OMNIFEED_REDDIT_KEEP_DEPTH false Include each comment’s depth
OMNIFEED_REDDIT_QUOTA 0 (off) Maximum requests to Reddit in any rolling OMNIFEED_REDDIT_QUOTA_WINDOW. It counts every request, not every crawl: the thread fetch, each /api/morechildren round, a subreddit listing and a share-link resolve each take one, so an expand=full crawl can take 41. With the browser method, the requests the page sends itself also count: each scroll batch of a listing (100 posts cost 1+4) and each “more replies” of a thread. They are charged once the page answers, so the next request waits behind them. Shared across replicas when OMNIFEED_REDIS_URL is set. Reddit’s unauthenticated budget is about 10 requests per minute per IP in practice. On 2026-10-08 Reddit blocked after about 50 requests in 5 minutes, and once at about 3 per minute while other clients shared the egress IP. Recommended: 6 with the default 1m window, or 3 if the IP is shared. A request refused for lack of quota fails with quota_exhausted and its retry_after_s, and nothing is sent.
OMNIFEED_REDDIT_QUOTA_WINDOW 1m Rolling window for OMNIFEED_REDDIT_QUOTA. Ignored when the quota is 0. Must be above 0 when the quota is set.
OMNIFEED_REDDIT_SESSION (unset) Value of a logged-in reddit_session cookie. Reddit closed its .json endpoints to logged-out clients in 2026 but still serves them to logged-in sessions. It enables the session method, which calls them directly over HTTP: one request per thread, with live scores and the full comment tree, and no browser. Unset, it reads the rendered page through crawl4ai instead. Use a secondary account, because every fetch counts against it (100 requests per window, from X-Ratelimit-*). The cookie expires about 6 months after login, and Reddit does not renew it on use: log in again before omnifeed_reddit_session_expiry_timestamp_seconds. omnifeed logs a warning at startup when fewer than 14 days are left. Secret: keep it out of logs and version control.
OMNIFEED_REDDIT_METHODS client_id,session,browser The order the Reddit engine tries its methods in. A method without its credential is left out (browser needs only crawl4ai). On a failure the next method can get around (403, 429, challenge page, upstream 5xx), the call moves to the next method at once, and the failed one is skipped for OMNIFEED_REDDIT_METHOD_COOLDOWN. A missing thread or omnifeed’s own quota refusal does not move on, and neither does a subreddit Reddit refuses to everyone (private, quarantined, banned) or a page the browser rendered but could not read (layout, a markup change): those fail the same on every method, so nothing is held. An upstream fault (upstream_error: crawl4ai down, Reddit 5xx, a dropped connection) moves on, but never holds the last method left, so a crawl4ai restart costs the failed calls, not a cooldown. When every method is skipped, the call fails at once with quota_exhausted and the shortest wait in retry_after_s, and sends nothing. Reddit’s back-off (Retry-After, X-Ratelimit-Remaining: 0) holds only the method that got it, never the others. Without Redis, the memory is per process, and a failure costs each pod one request per cooldown. With OMNIFEED_REDIS_URL set, a hold recorded by one replica skips the method on every replica, because Reddit’s budget belongs to the credential, not to the pod. A shorter hold never cuts a longer one. If Redis fails, each pod uses its own memory for 30s, then tries Redis again. The keys are <OMNIFEED_REDIS_KEY_PREFIX>:{reddit-method}:<method>, and every operation is a script, so the limiter’s ACL is enough. The shared browser hold assumes that the replicas share one egress IP.
OMNIFEED_REDDIT_METHOD_COOLDOWN 10m How long a failed Reddit method is skipped before the chain tries it again. A 429 with a longer Retry-After holds it for that long instead. Watch omnifeed_reddit_method_up and omnifeed_reddit_method_calls_total{outcome!="ok"}.
OMNIFEED_REDDIT_CLIENT_ID (unset) Client ID of a Reddit OAuth app, used with the installed-client grant: no secret and no Reddit account. It enables the client_id method: an app-only token (it lasts 24 h and is renewed on its own), then oauth.reddit.com directly. That gives one request per thread, live scores, the full comment tree, and 1000 requests per window. Short-lived: Reddit stopped taking new API requests on 2026-10-31, starts cutting off unregistered apps on 2027-01-12, and closes the public API in March 2027.
OMNIFEED_MAX_URLS_PER_REQUEST 30 Cap on urls[] length
OMNIFEED_CRAWL_MAX_RESPONSE_MB 256 Cap on the page content one POST /crawl response carries, summed over its documents. Documents are never cut: one that would cross the cap comes back as an error document asking to request that URL on its own. 0 disables the cap.
OMNIFEED_PER_DOMAIN_CONCURRENCY 2 Max concurrent requests to one domain
OMNIFEED_PER_DOMAIN_DELAY 1500ms Minimum gap between requests to one domain
OMNIFEED_BLOCK_PRIVATE_IPS true SSRF protection: refuses hosts that are, or resolve to, addresses that are not globally reachable, non-canonical IPv4 forms, localhost names and hosts that do not resolve. A host whose every address is 0.0.0.0 or ::, the answer a DNS filter gives for a blocked name, is refused the same way but reported as dns_blocked, not invalid_request. Keep it on in production, and restrict crawl4ai’s egress to public addresses too (see SECURITY.md).
OMNIFEED_ALLOWED_ORIGINS (unset) Comma-separated browser Origin values, e.g. https://app.example.com, allowed to call the HTTP APIs cross-origin. Requests with no Origin, which covers every native MCP client, always pass, as do loopback origins such as the MCP inspector. Anything else gets 403. This is the DNS-rebinding guard the MCP transport spec requires, applied to every HTTP endpoint.
OMNIFEED_LOG_LEVEL info debug, info, warn or error
OMNIFEED_LOG_FORMAT json json or text
OMNIFEED_ENABLE_PPROF false Expose /debug/pprof/*
OTEL_EXPORTER_OTLP_ENDPOINT (unset) Turns on tracing: base URL of an OTLP/HTTP collector, e.g. http://vtsingle-traces.monitoring.svc:10428/insert/opentelemetry. The exporter appends /v1/traces. Unset means no tracing and no overhead. The standard OpenTelemetry name, so the exporter’s other OTEL_EXPORTER_OTLP_* variables (headers, timeout) apply too.
OTEL_SERVICE_NAME omnifeed service.name on every span
OMNIFEED_CACHE_ENABLED true Cache successful fetch_url and POST /crawl results. See Response cache.
OMNIFEED_CACHE_TTL_THREADS 10m How long a result from a dedicated engine (Reddit, Hacker News, Discourse, Bluesky, GitHub, Stack Exchange) stays cached. 0 turns caching off for them.
OMNIFEED_CACHE_TTL_PAGES 30m How long a generic page (the crawl4ai engine) stays cached. 0 turns caching off for them.
OMNIFEED_CACHE_MAX_BYTES 67108864 (64 MiB) Size of the in-process LRU, used when OMNIFEED_REDIS_URL is unset, or when Redis refuses the cache’s keys (see Response cache). Approximate: content plus metadata plus a small per-entry overhead. Per pod.
OMNIFEED_CACHE_MAX_ITEM_BYTES 4194304 (4 MiB) Largest single entry. In Redis it is the gzip-compressed size, in memory the raw size. A bigger result is still returned, just not cached.
OMNIFEED_CACHE_KEY_PREFIX omnifeed:cache Redis key namespace for cache entries, separate from the rate limiter’s OMNIFEED_REDIS_KEY_PREFIX. A Redis ACL user must be allowed ~<prefix>:* with GET and SET (see Response cache).

Retry-After propagation

When an upstream answers 429 or 503 with Retry-After, omnifeed holds that host for the given duration, capped at 5 minutes, across later requests. With OMNIFEED_REDIS_URL set, the hold spans replicas. It applies only to upstreams an engine calls directly, because crawl4ai hides the crawled site’s own Retry-After. Reddit is the exception: the engine reads Retry-After, X-Ratelimit-Reset and X-Ratelimit-Remaining itself. On a 429, or when X-Ratelimit-Remaining reaches 0, it holds only the chain method that got the answer, never www.reddit.com, because a host hold would stall a working credential. The hold is Retry-After, else X-Ratelimit-Reset, else 60s on a bare 429 (see OMNIFEED_REDDIT_METHODS). The 429 error carries the wait as retry_after_s. Callers waiting on a held host time out normally at their own budget: the engine timeout, about 30s, or MaxWait, 15s, for a search.

Pacing fail-fast

If a request has a deadline and the pacing wait is longer than the time left, the limiter refuses at once. Searches always have one, MaxWait, 15s. Nothing is sent, no quota is spent, and the caller gets quota_exhausted with the wait: pacing quota exhausted; retry in 47s. Before this, omnifeed held the query for 15s and then timed out anyway. On 2026-08-21, 21 searches each waited about 20s and all failed.

quota_exhausted means pacing works as configured, and the caller should retry later. If it fires more often than the engines require, raise OMNIFEED_SEARXNG_QUOTA or lower OMNIFEED_SEARXNG_DELAY. A request with no deadline still queues. Refusals show up as outcome="budget_exceeded" on omnifeed_domain_limiter_wait_seconds, at about 0 seconds each, unlike outcome="canceled".

Search failover

web_search and POST /search answer through one searcher, which may be a chain (OMNIFEED_SEARCHERS). The first searcher answers every query. Its backup is asked only when that answer is not good enough:

Trigger When What the caller gets
error The primary failed: quota_exhausted, a timeout, an upstream error status The backup’s results
empty The primary succeeded with no results, how a silently blocked pool answers The backup’s results
thin Fewer than OMNIFEED_SEARCH_FALLBACK_MIN_RESULTS results Both, interleaved
degraded A critical engine (OMNIFEED_SEARXNG_CRITICAL_ENGINES) was missing Both, interleaved

Interleaving takes the primary’s first result, then the backup’s, and so on, and drops a URL already taken (scheme, www., a trailing slash and the fragment are ignored). Each result’s engine says who found it, e.g. tinyfish.

If the backup fails too, or answers a failed primary with no results, the primary’s answer stands: its results if it had any, its original error if not, so quota_exhausted still reads quota_exhausted instead of an empty success. Two searches are never failed over: one whose caller hung up or ran out of deadline, and one refused for its input (an empty query), which the backup would refuse too. A primary that runs past OMNIFEED_SEARCH_PRIMARY_BUDGET counts as failed.

A critical engine that SearXNG skips for a filter it doesn’t support (some engines ignore time_range or language) reads as missing, so those searches go to the backup too. Watch omnifeed_search_fallbacks_total{trigger="degraded"} against the backup’s daily quota. Every handoff is counted in omnifeed_search_fallbacks_total and logged as search served by backup, search backup found nothing, primary answer stands or search backup failed, primary answer stands, with trigger, cause and query_id.

With a backup, a long pacing queue costs more than it saves: lower OMNIFEED_SEARXNG_MAX_WAIT (e.g. 3s) so a burst spills over to the backup instead of waiting 15s to fail. The chain’s name, e.g. searxng+tinyfish, is the searcher label on omnifeed_search_requests_total and the searcher key in web_search _meta.

Controlling fetched content size

A long article can overflow an MCP client’s per-response budget. The client then writes the text to a file and the model never reads it. So fetch_url caps markdown at OMNIFEED_FETCH_MAX_CHARS, 120000 by default and 0 for unlimited, and ends a cut reply with a resumable marker:

[omnifeed: content truncated at 120000 of 174233 characters. Call fetch_url again with start_char=120000 to continue.]

Both per-request parameters apply to markdown only:

Param Default Purpose
max_chars OMNIFEED_FETCH_MAX_CHARS Max characters to return, up to 500000. 0 or omitted uses the server default.
start_char 0 Character offset to start from. Pass the value from a truncation marker to read the next chunk.

Offsets and lengths count characters, not bytes, so a window never splits a multibyte character. The marker counts against max_chars, so a reply never exceeds it. The response _meta carries the same numbers as truncated, total_chars and next_start_char. fetch_url declares anthropic/maxResultSizeChars: 500000 in tools/list, which raises Claude Code’s per-tool text limit to match. Other clients ignore it.

omnifeed never truncates TOON or JSON output, which covers every dedicated engine. Its length markers would describe rows that were cut, so those engines use their own element caps, described below.

POST /crawl has no size parameters and never truncates. RAG pipelines chunk and embed whatever they receive, so a cap would drop retrievable text, and one shared offset would break every other URL in a batch.

Fetching infinite-scroll feeds

Generic crawls don’t scroll by default. Scrolling costs seconds on every page, gains content only on append-style infinite feeds, and corrupts virtualized pages. When a feed, listing or gallery comes back missing items, opt in per request with scan_full_page: true on fetch_url or POST /crawl?scan_full_page=true. Both accept false too, which turns the scroll off where OMNIFEED_CRAWL4AI_SCAN_FULL_PAGE turns it on. OMNIFEED_CRAWL4AI_SCROLL_DELAY sets the pause between scroll steps.

Thread totals in the body

AI agents read the TOON (or JSON) body, not the response _meta, so thread headers state how complete the comment list is:

_meta keeps its own fields (comments, truncated_from, total_comments, gaps) unchanged.

Controlling Reddit response size

Reddit comment trees can be huge. There are two kinds of limit:

Use the Reddit parameters to fetch less. Use the caps for a guaranteed ceiling: OMNIFEED_REDDIT_MAX_ROUNDS expansion adds comments beyond limit, so only the caps bound the total. fetch_url accepts all five per request as limit, depth, sort, max_comments and max_top_level, and a positive value overrides the env default. They apply to threads only. A subreddit listing has no comment tree and takes its post count and time window from the URL: ?limit=, 1 to 100 with a default of 25, and ?t=hour|day|week|month|year|all, which Reddit applies only to top and controversial.

Controlling Hacker News response size

The Algolia item API returns a Hacker News thread’s whole tree in one response, and top-level subtree sizes are skewed. One measured thread had subtrees of 105, 55, 16, 13, 12, 11, 10, 6 and so on across 62 top-level threads. There are no upstream parameters to forward, only omnifeed caps applied after the fetch, all three per request on fetch_url:

Param Default Purpose
max_per_subtree unlimited Max comments kept in each top-level thread, root included. Selection is breadth-first, so a thread keeps its root and shallow replies before its deep tail.
max_top_level unlimited Keep the first N top-level threads, in HN’s order, with all their replies.
max_comments 500 Ceiling on the flat comment list, applied last. 0 or omitted means 500; a caller can ask for up to 5000 (Algolia already returned the whole tree, so this costs no extra request). The cut is breadth-first across the whole thread.

Start with max_per_subtree, which fits that skew. On two real threads, max_per_subtree=12 returned about 56% and 47% of the full-tree bytes and kept 9 of 12 and 11 of 14 of the comments a human rated substantive. max_comments is one breadth-first cut over the whole thread: every top-level comment first, then their direct replies, then the next depth. It used to keep the first N comments in pre-order, which spent most of the budget in the biggest branch: at 100 comments that covered 4 of 62 top-level threads, with 72% of its comments in one subtree.

Hacker News has no depth cap on purpose, because depth says nothing about quality there. The most valuable comment in both measured threads sat at depth 7, and adding depth<=5 to a subtree cap lost real content while saving under 1k of 84k characters. depth and sort are Reddit-only and ignored on HN URLs.

Caps never reorder output. Comments stay in HN’s order, and breadth-first selection keeps each comment’s ancestors, so the parent_id chain never breaks.

Dedicated-engine fallback

When a dedicated engine (Reddit, Hacker News, GitHub, Discourse, Bluesky, Stack Exchange) fails, the generic browser engine renders the same URL instead, with three exceptions:

GitHub, Hacker News, Bluesky and Stack Exchange read a separate API host (api.github.com, hn.algolia.com, the Bluesky AppView, api.stackexchange.com). Their blocks and quota refusals are about that API host, not the page, so they still fall back: an anonymous GitHub deployment over its 60 requests per hour renders github.com instead. (For Stack Exchange the render usually fails too: Cloudflare answers crawl4ai on the question pages with a 403.) An engine declares the page’s host by implementing domain.SameHostEngine.

A fallback result is marked. _meta (or the loader’s metadata) carries fallback_from (the engine that failed) and fallback_reason (its failure kind), and the body starts with one line: > Note: the dedicated reddit engine failed (timeout); this is the generic page render instead.

Errors reach MCP callers as a tool result with isError: true whose text is fetch_url failed: <reason> (HTTP <status>): <cause> [<code>] Retryable after Ns. (the back-off is stated once; retry_after_s is also in structuredContent). See errors.md.

Twitter/X posts

x.com walls headless browsers, and crawl4ai fails every twitter.com link (the anti-bot verdict is a 39-byte page), so the Twitter engine reads public JSON mirrors directly, in order, and stops at the first that answers:

  1. FxTwitter API v2: /2/conversation/{id} for the post, its parents and the first page of replies, plus /2/thread/{id} when the post belongs to the author’s self-thread. Covers long posts, X Articles, quotes, media alt text, community notes, polls and link cards.
  2. Embed syndication (cdn.syndication.twimg.com/tweet-result). It cuts long posts at about 275 characters, so for those the engine asks vxTwitter for the full text. If that fails too, the cut text is served with text_truncated: true in _meta and a note in the body. vxTwitter alone is tried if syndication fails.
  3. crawl4ai on the canonical https://x.com/i/status/{id}, which X renders for logged-out readers. X’s own error pages are rejected, not returned.

If every source fails, the error says tweet unavailable and why: not found (FxTwitter and a second source both answered 404: deleted, protected or suspended), blocked or rate limited, or the last source’s failure. The error is final: the registry does not render the URL a second time. t.co links are resolved with one HEAD whose redirect is not followed; the destination passes the SSRF check, then renders as a post or, for any other site, through the generic engine. A twitter.com URL the engine does not claim, such as a profile, is rewritten to x.com before crawl4ai sees it.

Output is markdown by default: a header line (@handle (Name) · date · x.com URL · likes · reposts · replies · views), the text with t.co links expanded, the quoted post as a blockquote, media as [image: alt] or [video, 1:23], poll, community note, link card, the X Article body, the self-thread numbered, and the top replies as @handle (N likes): text, with the author’s own replies marked [author]. Pass format: toon or json for the same data structured. _meta carries upstream (fxtwitter, syndication, syndication+vxtwitter, vxtwitter or crawl4ai), replies, reply_total, thread_posts and text_truncated. Fetched posts are cached in process for 15 minutes.

FxTwitter, vxTwitter and the syndication endpoint are third-party or unofficial services X can break or pressure at any time. A self-hosted FxEmbed behind OMNIFEED_TWITTER_FXTWITTER_URL removes the first dependency.

Reddit access

Reddit closed its .json endpoints to logged-out clients in 2026. The Reddit engine reads Reddit through a chain of methods, in the order OMNIFEED_REDDIT_METHODS sets. ADR 0001 records the measurements and the reasons.

_meta.reddit_method names the method that served the thread or listing.

Sustained scraping can raise your IP’s risk score. If fetches return the block page, slow down, keep expand modest, or route the browser through a residential proxy.

Response cache

The same thread is often fetched again within minutes: a retry job re-reads the threads it already has, and agents re-open URLs they fetched a turn ago. Every repeat of a Reddit thread spends Reddit’s per-IP budget, which is small (on 2026-10-08 it ran out and the egress IP was blocked). So omnifeed caches successful results and serves repeats from the cache.

Every response from the cache layer carries _meta.cache (or metadata.cache on POST /crawl): hit (served from the cache, or shared with an identical in-flight request), miss (fetched upstream) or bypass. _meta.cached_at is the time (RFC 3339, UTC) the content was fetched and stored. It is absent when the response is not in the cache.

Bypass. Pass no_cache: true to fetch_url, or POST /crawl?no_cache=true (or =1), to skip the cache and fetch fresh. The fresh result still replaces the cached copy. Use it only when cached_at is too old for the job: every uncached Reddit fetch spends the per-IP budget.

Partial Reddit threads

A thread whose /api/morechildren expansion is cut short, because a round was blocked, rate limited, timed out or returned something unparseable, is still returned with the comments loaded so far. omnifeed flags it:

A thread that simply used up its expand budget is not partial. Partial threads are never cached.

The browser method has its own partial reasons. Its in-page expansion can be refused (http_429, http_403) or drop a request (expansion_failed). A page can also count comments it did not render, or list posts without an id (browser_incomplete, and such posts are dropped from a listing). Browser gaps carry no reply counts, so the note then reads only 44 of 72 comments could be loaded (http_429). A subreddit listing is flagged the same way.

Every Reddit document also carries _meta.reddit_method: the method that served it (client_id, session or browser).

Timeouts

Each engine has its own time budget. A client calling omnifeed should wait longer than the budget of the engine it is calling, or it gives up on a request that would have succeeded and, for Reddit, spends the rate budget for nothing.

Engine Budget Set by Notes
Hacker News 30 s per crawl fixed (internal/engine/hackernews) Wall clock for the whole crawl, including the pacing wait and retries against hn.algolia.com.
GitHub, Discourse, Bluesky, Stack Exchange 30 s per crawl fixed, per engine Same shape as Hacker News.
Reddit 4 min per crawl OMNIFEED_REDDIT_TIMEOUT Wall clock for the whole crawl: pacing wait, share-link resolve, thread fetch and every morechildren round. A round cut off by the deadline returns what was loaded, flagged partial. Each browser call inside it is also bounded by OMNIFEED_CRAWL4AI_TIMEOUT.
Generic page (crawl4ai) 90 s per crawl4ai call OMNIFEED_CRAWL4AI_TIMEOUT Inside each call, crawl4ai’s own page_timeout is 60 s (the most crawl4ai accepts over REST), so crawl4ai normally answers before omnifeed’s 90 s run out. There is no overall cap on a generic crawl: a transient failure is retried once, a Playwright navigation race once with networkidle, a blocked page once under the alternate browser identity, a thin page once without the excluded selector, and a page that turns out to be a PDF once through crawl4ai’s PDF strategy, so the worst case is several 90 s calls.
SearXNG search 15 s per attempt OMNIFEED_SEARXNG_TIMEOUT Up to 3 attempts with backoff, after a pacing wait of up to OMNIFEED_SEARXNG_MAX_WAIT (15 s by default).

Suggested client timeouts:

The loader (/crawl, /search) and MCP listeners stop writing a response after 300 s, so waiting longer than that gains nothing, and an OMNIFEED_REDDIT_TIMEOUT above about 4m30s gets cut by the server first. A cached result returns in milliseconds whatever the engine.

Raw-text bypass

Raw code, JSON, markdown and plain text have nothing for a browser to render, and Chromium’s page-idle wait makes them slow: a raw githubusercontent.com file takes 30 to 39 s in the browser and about 200 ms direct. When a URL’s extension looks raw (.md, .txt, .json, source files), the generic engine sends a HEAD request. If the server confirms a non-HTML text type, a plain GET fetches the body and returns it unchanged. Anything uncertain, such as a failed probe, text/html, binary bytes or blocked egress, falls back to the browser. With OMNIFEED_BLOCK_PRIVATE_IPS on, direct fetches refuse private and reserved addresses when dialing, so DNS rebinding can’t bypass URL validation. This needs outbound access to the target sites, not just crawl4ai. Without it the probe fails and everything goes through crawl4ai.

Health

On OMNIFEED_METRICS_ADDR (default :9090):

Request ids and log correlation

Every response from the loader (:8080) and MCP (:8081) servers carries X-Request-Id. A caller’s own X-Request-Id is reused when it is 1 to 128 characters from A-Z a-z 0-9 . _ : -; otherwise omnifeed generates one. MCP tool results return it as _meta.request_id (on stdio, one is generated per message). Log lines written for that request carry request_id, and trace_id and span_id when tracing is on, so one id joins the caller’s logs, omnifeed’s logs and the trace.

MCP log lines worth knowing:

Tracing

Off unless OTEL_EXPORTER_OTLP_ENDPOINT is set. Then omnifeed exports OpenTelemetry traces over OTLP/HTTP, sampling parent-based and always-on, and propagates W3C traceparent:

Spans never carry a URL, a query or a path. Hosts appear as server.address. Before export, url.full, url.query, url.path and caller addresses are dropped, and URLs inside error texts are cut to scheme://host. On shutdown, omnifeed flushes buffered spans after the servers drain.

Prometheus metrics

Served at /metrics on OMNIFEED_METRICS_ADDR, default :9090, alongside the Go and process collectors. The Go collector adds the runtime GC, scheduler and GC-CPU series (go_gc_*, go_sched_*, go_cpu_classes_gc_*). Don’t alert on go_sched_latencies_seconds: Go 1.26 and later misreport it (golang/go#79391).

Metric Type Labels What it measures
omnifeed_build_info gauge version Always 1; the running version
omnifeed_http_requests_total counter server, route, code HTTP requests by server (loader, mcp, obs), route (/crawl, /search, /mcp, /mcp/sse, /livez, /readyz, /metrics, anything else other) and status code
omnifeed_http_request_seconds histogram server, route Time until the handler returns. The SSE streams (GET /mcp, /mcp/sse) are counted but not timed: their duration is the connection’s lifetime
omnifeed_http_in_flight gauge server Requests being served, open SSE streams included
omnifeed_mcp_tool_calls_total counter tool, outcome, client MCP tools/call by tool, outcome (ok, error, invalid_params) and client: claude-code, opencode, openwebui, thedigestif, cursor or other, read from the MCP clientInfo (initialize on stdio, per-request _meta in the 2026-07-28 era) or else the User-Agent
omnifeed_mcp_tool_seconds histogram tool, outcome MCP tool latency, cache hits included
omnifeed_panics_total counter where Recovered panics: http, mcp_tool, loader_crawl, github_fanout, background. Any increase is a bug; the request failed instead of the process
omnifeed_dependency_up gauge dependency 1 when the last 30-second probe of crawl4ai, searxng or redis succeeded, 0 when it failed. Only configured dependencies have a series. Not a readiness gate, see Health
omnifeed_requests_total counter engine, tenant, status, reason, cache Crawl requests, with a bounded failure reason, ok on success. cache is hit, miss, bypass (no_cache) or off (cache disabled). Errors are never served from the cache, so they carry miss, bypass or off
omnifeed_request_seconds histogram engine, status, reason End-to-end crawl latency. Cache hits are excluded: they never reach an engine, and their microseconds would drag the quantiles down as the hit rate rose
omnifeed_request_attempts_total counter upstream, attempt HTTP attempts by the retrying client, first or retry
omnifeed_upstream_seconds histogram upstream, op, status Upstream round-trip per attempt, from start until the body is read: crawl4ai/crawl, crawl4ai/execute_js, searxng/search, searxng/config, github/api, hackernews/api, discourse/api, stackexchange/api
omnifeed_domain_limiter_wait_seconds histogram engine, outcome Time blocked acquiring the per-domain limiter, semaphore plus delay. outcome="canceled": the wait died in the queue. outcome="budget_exceeded": the wait exceeded the caller’s remaining deadline, so nothing queued; about 0 seconds, with reason="quota_exhausted" on the request metric
omnifeed_ratelimit_backend_errors_total counter op Failed Redis operations in the distributed limiter. On acquire the limiter paces in process. release and penalize failures cost only pacing accuracy. reddit_method_read, reddit_method_hold and reddit_method_clear are the Reddit method chain’s shared holds: on a failure each pod keeps its holds in memory for 30s
omnifeed_ratelimit_penalties_total counter upstream Upstream Retry-After headers, on 429 or 503, turned into a hold on that host. upstream="reddit" counts Reddit’s own back-off headers read inside the browser. upstream="stackexchange" counts the Stack Exchange API’s backoff field and the wait a throttle_violation names, which it sends in the JSON body instead of a header. This often precedes a CAPTCHA or block
omnifeed_ratelimit_degraded gauge scope 1 while pacing falls back to per-pod limits because Redis is unreachable, 0 while shared. Exists only when OMNIFEED_REDIS_URL is set, published at 0 on startup. One series per limiter scope, domain for crawling, searxng for queries and reddit for the Reddit request quota when set, reddit_methods for the Reddit method chain’s shared holds, each degrading and recovering on its own
omnifeed_response_chars histogram engine Engine output length before any max_chars truncation, successful crawls only
omnifeed_engine_fallbacks_total counter from_engine, reason Dedicated-engine failures re-crawled by the generic fallback. Block and rate reasons appear only for separate-host engines (GitHub, Hacker News, Bluesky, Stack Exchange), see Dedicated-engine fallback
omnifeed_searxng_unresponsive_engines_total counter engine, error Engines SearXNG reported unresponsive, per search. error is one of timeout, captcha, suspended, too_many_requests, access_denied, error, unknown
omnifeed_searxng_engine_results_total counter engine Result rows per SearXNG engine. A blocked engine keeps answering 200 with zero results, so its series goes flat while the rest of the pool moves. Alert on that divergence. Starts at 0 at startup for known engines (see below). Pair it with absent_over_time() all the same: an engine nothing names at startup has no series until a response names it
omnifeed_searxng_queries_total counter scoped Queries sent to SearXNG after the limiter, the rate the engines see. Compare with omnifeed_search_requests_total to see what pacing refused
omnifeed_searxng_engine_zero_results_total counter engine Searches where one engine returned no rows while the search had results: a silent block, per engine. Excludes engines SearXNG reported unresponsive. Counts only engines seen answering before, so pair with absent_over_time()
omnifeed_searxng_empty_searches_total counter scoped Searches with zero results and no unresponsive-engine report. scoped="true" is a site: query, where silent blocks concentrate
omnifeed_reddit_expansion_rounds histogram none /api/morechildren rounds per Reddit crawl
omnifeed_reddit_method_calls_total counter method, outcome Reddit calls per chain method. outcome is ok; incomplete (a browser answer that holds less than the page counts, served with partial_reason=browser_incomplete); a verdict that returns to the caller without failover: not_found, forbidden (a subreddit refused to everyone), layout (a rendered page the browser could not read: Reddit changed its markup), error; or the failure kind that moved the call to the next method (http_403, http_429, captcha, bot_block, upstream_error). method="chain" with outcome="all_held" is a call that found every method on hold and sent nothing: Reddit is out of reach. It is apart from omnifeed_requests_total{reason="quota_exhausted"}, which our own pacing refusals share
omnifeed_reddit_ratelimit_remaining gauge method X-Ratelimit-Remaining of the last answer Reddit sent to a direct method: the requests left in the current window. Watch it on session, whose budget the account owner’s own browsing also spends
omnifeed_reddit_method_last_success_timestamp_seconds gauge method When a method last answered (ok or incomplete), per pod; 0 until it has. A fallback method behind a working one is never called, so this ages without meaning a fault
omnifeed_reddit_method_up gauge method 1 while a configured method is in use, 0 while it is skipped after a failure or a back-off. With Redis, a hold from another replica sets it to 0 too, read at startup as well as on each call. It returns to 1 only after a call through the method succeeds, not when the cooldown ends
omnifeed_reddit_inpage_requests_total counter kind Requests a browser page sent to Reddit itself, on top of its page load: kind="listing" for scroll batches, kind="thread" for “more replies”. They are charged to OMNIFEED_REDDIT_QUOTA. Page loads plus these are what Reddit counts against the IP
omnifeed_reddit_session_expiry_timestamp_seconds gauge none When the reddit_session cookie expires. 0 when none is set
omnifeed_search_requests_total counter searcher, status, reason, scoped Search queries as callers sent them, refused ones included, unlike omnifeed_searxng_queries_total. scoped="true" is a site: query. sum() queries ignore the added label. Queries matching an exact label set don’t
omnifeed_search_fallbacks_total counter from, to, trigger, reason, outcome Searches handed to a backup searcher. trigger is error, empty, thin or degraded. reason is the primary’s classified failure on trigger="error" (quota_exhausted is our own pacing, not a SearXNG fault) and ok otherwise. outcome is served, empty (the backup found nothing), failed, or canceled (the caller hung up while the backup ran). See Search failover
omnifeed_search_request_seconds histogram searcher, status Search latency
omnifeed_search_engine_position_rank histogram engine The rank each engine gave each row it returned. 1 to 3 is a result a caller reads, 20 and above is filler
omnifeed_search_engine_unique_results_total counter engine Results no other engine returned, which shows whether an engine earns its slot
omnifeed_cache_requests_total counter result Response cache lookups: hit (served from the cache or shared with an identical in-flight fetch), miss (fetched upstream), bypass (no_cache), error (the cache backend failed, served as a miss). hit / (hit + miss) is the hit rate
omnifeed_cache_bytes gauge none Approximate bytes in the in-process cache. Only with the in-memory backend: the Redis backend is shared and not measured per pod
omnifeed_cache_backend gauge backend 1 on the response cache’s active backend (redis or memory), 0 on the other. memory while OMNIFEED_REDIS_URL is set means Redis refused the cache’s keys and this pod fell back to its in-process LRU
omnifeed_cache_backend_errors_total counter op, kind Response-cache Redis errors by op (get, set, probe) and kind: noperm (ACL or auth refusal: the pod switches to memory), transient (timeout, refused connection, LOADING…: 30-second cooldown, then retry), other (startup probe only: an answer the cache cannot use, e.g. READONLY)

Per-engine SearXNG series start at 0. increase() and rate() in Prometheus and VictoriaMetrics do not reliably count a counter’s first sample. A series created on a pod’s first search, at say 20 rows, reads as no increase at all, so after every restart the first results per engine went missing. That made an “engine silent” alert such as sum(increase(omnifeed_searxng_engine_results_total{engine="google cse"}[6h])) == 0 fire after deploys. At startup omnifeed therefore creates every per-engine series at 0: omnifeed_searxng_engine_results_total, omnifeed_searxng_engine_zero_results_total, omnifeed_search_engine_unique_results_total, omnifeed_search_engine_position_rank (_count 0) and omnifeed_searxng_unresponsive_engines_total for each error value. This happens at once for the engines in OMNIFEED_SEARXNG_SITE_ENGINES. For every engine SearXNG reports enabled in GET /config in the general category, it happens in the background. It is retried with backoff for up to 5 minutes, because SearXNG may start later, and never delays readiness. Only general counts because omnifeed sends no categories, so SearXNG runs only general engines. Engines in other categories (images, music, maps…) would only add series that never move. If /config does not answer, omnifeed logs one WARN. Engines then get their series the first time a response names them, as before.

This site is open source. Improve this page.