Ideas considered for omnifeed that are not in the code, and why. One entry per idea, newest first.
A rejected idea should stay rejected for a reason, and a good idea that came at the wrong time should be findable. Write an entry when the reasoning cost more than the code.
Format. Each entry states the idea, the evidence for and against it, what would justify revisiting it, and, if code was written, the git ref that holds it. Without a ref, nobody can recover the code.
Status. To do, noted 2026-10-25 (review of the crawl4ai 0.9.5 upgrade).
Code. None yet. Today’s check is challengeTimeoutPattern in
internal/engine/crawl4ai/verdict.go.
When crawl4ai’s wait_for runs out, the generic engine reads it as the challenge wait
(challengeWaitFor), marks the error errChallengeWaitTimeout, crawls once more without
the wait, and lets the returned page decide whether it is a Cloudflare challenge
(captcha). It recognizes the timeout by crawl4ai’s “Wait condition failed: Timeout”
alone. That holds only while challengeWaitFor is the only wait_for the engine sends.
crawl4ai 0.9.5 quotes the selector in the message (“Timeout after 8000ms waiting for
selector ‘:root:not(:has(script[src*="/cdn-cgi/challenge-platform/"]…”), so the check
can match the selector’s text instead.
wait_for (a caller’s, or a per-site one) would otherwise read as the
challenge wait, crawl the page again and skip the retry under crawl4ai’s default identity.smart_wait in
async_crawler_strategy.py), and nothing sends a second wait_for today.Any change that sends another wait_for, or a crawl4ai release that rewords the message.
Do it in the same change.
.json from inside a logged-out browserStatus. Shipped as the only Reddit path. Removed 2026-10-15.
Code. git show v0.36.0:internal/engine/reddit/fetcher.go (getJS, postJS,
Fetcher.Open, and the browser branch of Session.FetchMoreChildren).
The engine opened a reddit.com page in crawl4ai’s browser, then ran a same-origin
fetch() of the thread’s .json, and a POST to /api/morechildren for each
expansion round. The page cleared Reddit’s bot wall, and the in-page fetch inherited
that state. Reddit’s 429 back-off held www.reddit.com on the crawl limiter.
Reddit closed logged-out .json. From 2026-10-15, every logged-out .json request
got 403, from a browser too. Expansion now runs on the direct methods only. A
logged-out expansion would get 403 and hold the browser method, which is the one
logged-out path that still reads threads. The host hold is removed too: it would stall
a working credential. ADR 0001 records the measurements.
Reddit serving .json and /api/morechildren to logged-out browsers again. Before
you bring it back, run one logged-out .json fetch and one morechildren POST from
a browser and get 200 for both.
/readyz on crawl4ai and SearXNGStatus. Shipped with the health endpoints; removed 2026-10-13.
Code. git show v0.35.1:cmd/omnifeed/main.go (upstreamReady and the readyChecks slice).
/readyz GET-probed crawl4ai and, when search was configured, SearXNG, and failed when
either was unreachable, so Kubernetes took the pod out of the Service until they came back.
Every replica shares the same crawl4ai and SearXNG, so an outage of either failed readiness
on all pods at once and the Service had no endpoints left. The cache, the Reddit quota and
the API engines (GitHub, HN, Bluesky, Discourse, Twitter’s API sources) could still have
served; a partial outage became a total one. A dependency outage now fails only the calls
that need it, classified and retryable, and its state is exported as
omnifeed_dependency_up{dependency} for alerting.
Replicas that do not share their dependencies (a crawl4ai sidecar per pod, say), where pulling one pod out of rotation routes traffic to a healthy one instead of to nothing.
cleaned_html when crawl4ai returns no markdownStatus. Shipped since the first crawl4ai engine; removed 2026-10-09.
Code. git show v0.34.2:internal/engine/crawl4ai/engine.go (the content = result.CleanedHTML fallback).
When crawl4ai’s fit_markdown and raw_markdown were both empty, the engine returned
cleaned_html instead, on the theory that some HTML beats no content.
A 272-site benchmark against Claude Code’s built-in fetch (2026-10-08) found it only ever
fired on pages crawl4ai had failed to render. caniuse.com came back as 251 bytes of
<html><head><title>…, served as a successful fetch. The caller could not tell it from
content, so it never fell back to another source. With the fallback gone, an empty render
is thin_content and the caller moves on.
A page class where crawl4ai’s markdown is empty but its cleaned_html holds the content
the caller wanted. None turned up in 272 sites. If one does, convert that HTML to text and
run it through the same verdict checks (prose floor, block markers) rather than serving it
raw.
Status. Considered 2026-10-09, not built.
Alert when many fetch_url successes are tiny (omnifeed_response_chars ≤ 400), as a
proxy for block pages and empty shells served as success.
Normal traffic swings too much. Over one production week, the daily share of ≤ 400-char crawl4ai successes ranged from 2.4% to 14.6% at about 100 crawls a day, driven by what was fetched, not by failures. No threshold separates a bad day from a day of short pages. The silent-failure checks in the crawl4ai engine now give those pages their own failure reasons, which the reason-labelled metrics already count.
A deployment with enough volume, and a stable enough mix of URLs, that the share becomes a steady baseline worth watching.
Status. Parked 2026-08-22, never built. No code, no git ref.
When fetch_url fails on an article, the caller has no second option. A digest of
“what a community thinks of X” needs X, so the caller writes the retry chain into its
own prompt: try the URL, and on failure try a reader proxy such as
https://r.jina.ai/{url}.
The idea moves that chain into the engine registry: crawl4ai direct, then a
reader-proxy fallback, then give up. The response adds a provenance field
(source: direct|jina, degraded: bool) so the caller can report the article as
unreachable instead of implying it read one.
Measured 2026-08-21 on three Hacker News front-page URLs:
| reader | BBC (soft paywall) | WSJ (hard paywall) | Substack (JS) |
|---|---|---|---|
| crawl4ai direct | fail | fail (500) | pass, clean |
r.jina.ai/{url} |
pass | soft-fail: HTTP 200, CAPTCHA warning, empty body | pass, adds Published Time |
web.archive.org/web/2/{url} |
no snapshot | fail 503 | fail 503 |
md.dhr.wtf/?url= |
fail | fail (401) | fail |
archive.is/newest/{url} |
CAPTCHA | CAPTCHA | CAPTCHA |
The WSJ soft-fail decides it. r.jina.ai returned HTTP 200 with an empty body, so a
caller-side “on failure, try the proxy” rule never fires, and the caller summarizes an
empty article. omnifeed already classifies this: internal/antibot detects block and
CAPTCHA pages, and it correctly turned the direct WSJ attempt into a 500.
A chain in a caller’s prompt can’t be observed, so nobody can say how often the
fallback helped. In omnifeed it would be a reason label on an existing counter,
and internal/observability already classifies crawl failures, so existing alerts
would cover it. Comparable tools work this way. Firecrawl’s proxy: auto escalates
from basic to enhanced on the server and returns one warning field, and Jina reports
its escalation in X-Engine.
It makes a third-party reader service a hard dependency of the fetch path. r.jina.ai
allows 20 requests per minute without a key, 500 with a free one, and averages 7.9s,
so a serial chain nears 20s at worst. That suits a daily batch, not interactive use.
Rejected outright, so nobody retries them:
web.archive.org/web/2020/https://example.com. Unthrottled, it would help only for
pages someone already archived, which for a fresh front-page link is a coin flip.md.dhr.wtf, urltotext. Failed on a page that works direct, or paid only.urlreader.dev, reader.tsuki.dev, textance. The domains don’t resolve.Any one of:
If built, bound the third-party dependency with a hard timeout and a circuit breaker, and use it only as a fallback, never the primary path.
Status. Parked 2026-08-22, never built. No code, no git ref.
The Hacker News engine builds its tree from Algolia /items/{id}, which returns nested
children. Algolia leaves dead (flagged) comments out of that tree, and because the
tree is nested, dropping a node drops its whole subtree, live replies included.
The fix builds from the flat endpoint,
/search?tags=comment,story_<id>&hitsPerPage=1000, and rebuilds the hierarchy from
parent_id. It has every field the parser needs.
Measured 2026-08-21 on story 49371857:
| source | comments |
|---|---|
Algolia /items/{id} (current) |
503 |
| Algolia comment search | 573 |
Firebase descendants |
573 |
| HN HTML rows | 574 |
71 live comments, about 12%, were missing, including a 70-reply argument under one
[flagged] parent. Firebase confirms it: the dead node 49372309 has dead: True,
its parent is in the tree, and its child 49372670 is not. Latency is the same,
0.79 to 0.83s against 0.76 to 0.81s, and one page at hitsPerPage=1000 covered the
whole thread.
The caller who asked wanted the flagged comment itself, not its replies. Neither
Algolia endpoint returns it, because comment search also excludes the dead node.
Firebase returns dead: True with the text [flagged]. That leaves HN’s own HTML,
where that thread has 73 noshow rows that may carry the real text. That is
unverified, and it needs egress to news.ycombinator.com, which the network policy
denies.
Without the flagged text, the caller judged the live replies alone not worth the change. That is a product call. The 71 live comments are real content and cheap to recover.
Any one of:
total_comments field. Today a
consumer can’t tell a gutted thread from a complete one.noshow rows carry the flagged text. The HTML fetch would
then also give HN’s own comment ranking, since Algolia children are strictly
id-ascending and every cap selects chronologically, and per-comment downvotes from
the CSS colour class.Status. Built, shipped in 0.22.0, removed 2026-08-19.
Code. git show v0.22.0 -- internal/search/router internal/search/hackernews
internal/search/reddit internal/search/bluesky. The sitewide Reddit variant is on
branch feat/reddit-sitewide-search (PR #34, closed).
web_search(site=X) goes to a web engine, which ranks X’s pages by whatever it
scraped. X’s own search often ranks its content better and exposes signals no scraper
has: Hacker News points, Reddit score and comment count, Bluesky likes. The design was
a domain.Searcher per site, a router dispatching on SearchOptions.Site, and SearXNG
as the fallback whenever a vertical declined, returned nothing or failed.
Three were built: Hacker News (Algolia), Reddit (in-site search through the browser
port) and Bluesky (app.bsky.feed.searchPosts).
The code was tested and worked. The problem it solved turned out to be one deployment’s engine pool configuration, and fixing that made the verticals redundant there.
Measured 2026-08-18, four queries, site=reddit.com:
| query | Reddit in-site (sitewide) | Reddit in-site (r/<sub>) |
web site:reddit.com |
|---|---|---|---|
| longhorn disk pressure | 0/7: Windows Longhorn, disc golf | 5/5 on topic | 6/8 |
| best self hosted rss reader | 2/5 | 4/7 | 7/8 |
| wireguard mtu slow mobile | 3/4 | 3/6 | 6/6 |
| opus 5 coding (past week) | 5/5 fresh | 6/6 fresh | 0 results |
Web search beat Reddit’s own search on three of four. Sitewide in-site search collapses on an ambiguous term that has a large non-technical following on the site. It wins on recency, where the web engine returned nothing.
Two caveats:
omnifeed_search_routes_total had no series
for them), not because they lost a measurement. Nobody has tested whether Algolia
beats a scraper for HN, or whether web engines barely index bsky.app.Any one of:
site: and indexes recent content.
The verticals were built for that case, which is common for a keyless self-hosted
pool.searchPosts beats a good web engine.
Nobody knows yet.Don’t prefer a vertical unconditionally. Verticals win on recency and lose elsewhere,
so the router should use a vertical when SearchOptions.TimeRange is set and skip it
otherwise, per vertical. That was designed and never built, and it is the change that
would have made the feature correct.