import "github.com/kinorai/omnifeed/internal/engine/crawl4ai"
Package crawl4ai implements the fallback engine: dispatches generic URLs to an upstream crawl4ai instance and reshapes the response into the canonical Document.
DefaultExcludedSelector is the conservative chrome selector list sent as crawl4ai’s excluded_selector when the operator hasn’t set one. It names only chrome-shaped classes/ids (sidebars, tables of contents, related-post and newsletter boxes, cookie banners). On the rare page whose main content IS one of these (a docs index living in `#toc`, say), the crawl comes back empty — Crawl retries once without the selector rather than erroring, so the default can stay aggressive.
const DefaultExcludedSelector = ".sidebar,.toc,#toc,.related,.newsletter,.cookie-banner,[aria-label*='cookie']"
Config configures the crawl4ai Engine.
type Config struct {
Endpoint string
// Token, when set, is sent as `Authorization: Bearer <token>` on every crawl4ai
// request — required when the upstream runs with CRAWL4AI_API_TOKEN (crawl4ai
// 0.9.x binds non-loopback only when a token is set). The default is owned by
// config (OMNIFEED_CRAWL4AI_TOKEN); empty sends no Authorization header.
Token string
Client *httpx.Client
Limiter httpx.Limiter
// KeepLinks renders hyperlink anchor text and retains external links in the
// extracted markdown. When false, both are stripped for leaner output. The
// default is owned by config (OMNIFEED_CRAWL4AI_KEEP_LINKS).
KeepLinks bool
// PruneThreshold is the PruningContentFilter score cutoff (0–1): nodes scoring
// below it are dropped, so a higher value strips more boilerplate/duplicated
// chrome from noisy pages. The default is owned by config
// (OMNIFEED_CRAWL4AI_PRUNE_THRESHOLD).
PruneThreshold float64
// WaitUntil is crawl4ai's page-ready signal (Playwright wait_until):
// domcontentloaded (the default) fires before client-side frameworks hydrate,
// so JS-only SPAs render empty; networkidle waits for them at the cost of
// latency on every page. The default is owned by config
// (OMNIFEED_CRAWL4AI_WAIT_UNTIL); empty falls back to domcontentloaded.
WaitUntil string
// ExcludedSelector is the CSS selector list crawl4ai drops before extraction
// (OMNIFEED_CRAWL4AI_EXCLUDED_SELECTOR). Empty = DefaultExcludedSelector; to
// effectively exclude nothing, set a selector that matches nothing.
ExcludedSelector string
// TargetElements is a comma-separated CSS selector list; when non-empty,
// crawl4ai extracts markdown ONLY from matching containers. Off by default
// (OMNIFEED_CRAWL4AI_TARGET_ELEMENTS): on pages without a match the crawl
// yields no content, which the thin-content guard turns into an error.
TargetElements string
// ScanFullPage scrolls the page to the bottom (in ScrollDelay steps) before
// extraction so lazy-loaded content renders — multi-second on long pages.
// The default is owned by config (OMNIFEED_CRAWL4AI_SCAN_FULL_PAGE).
ScanFullPage bool
// ScrollDelay is the pause (seconds) between scroll steps; only sent when
// ScanFullPage is on (OMNIFEED_CRAWL4AI_SCROLL_DELAY).
ScrollDelay float64
// DelayBeforeHTML is the unconditional settle (seconds) after the WaitUntil
// signal before HTML extraction — paid on every crawl. The default is owned
// by config (OMNIFEED_CRAWL4AI_DELAY_BEFORE_HTML).
DelayBeforeHTML float64
// RemoveOverlays sends crawl4ai's remove_overlay_elements, whose geometry
// heuristic deletes any large absolute/fixed-position element before
// extraction. On sites whose main content lives in such containers
// (Wikipedia Vector-2022, several news fronts) it silently empties the
// whole page — the default is off (OMNIFEED_CRAWL4AI_REMOVE_OVERLAYS);
// remove_consent_popups stays on regardless and covers cookie modals.
RemoveOverlays bool
// BlockPrivateIPs hardens the raw-text bypass's direct fetches: resolved
// private/reserved addresses are refused at dial time (mirrors
// OMNIFEED_BLOCK_PRIVATE_IPS, which the registry enforces pre-dispatch via
// DNS lookup — the dial-time guard is what a rebinding race can't beat).
BlockPrivateIPs bool
}
Engine sends URLs to crawl4ai’s /crawl endpoint and extracts the best-fit markdown body. It is registered as the Registry fallback.
type Engine struct {
// contains filtered or unexported fields
}
func New(cfg Config) *Engine
New returns a crawl4ai fallback Engine wired with the given config.
func (e *Engine) Crawl(ctx context.Context, rawURL string, opts domain.EngineOptions) (domain.Document, error)
Crawl proxies rawURL to crawl4ai. The configured per-domain limiter applies to avoid hammering sites that crawl4ai itself doesn’t pace.
A thin-content result with the excluded selector active is retried once without it: the selector list names chrome shapes (.sidebar, #toc, …), and on the rare page whose main content matches one, the exclusion is what emptied the page — not the page itself.
func (*Engine) Matches(string) bool
Matches returns false: this engine is the fallback only.
func (*Engine) Name() string
Name returns the engine identifier (“crawl4ai”).
Generated by gomarkdoc