import "github.com/kinorai/omnifeed/internal/antibot"
Package antibot recognizes bot-wall / CAPTCHA / challenge pages that upstreams (Cloudflare, Reddit’s “network security” wall, PerimeterX, …) return in place of real content — frequently with an HTTP 200, so a crawl that “succeeded” can still be a block. Detect lets engines reclassify those as failures (a domain.FetchError of kind captcha) instead of handing a challenge page back to the caller, and surfaces the matched marker for metrics/logs.
func Detect(body string) (marker string, blocked bool)
Detect reports whether body looks like a bot wall / challenge page and, if so, the marker that matched (for logging/metrics). Matching is case-insensitive and bounded to the first scanLimit bytes.
func IsBlockResponse(body string) bool
IsBlockResponse reports whether an upstream error body is crawl4ai’s anti-bot block verdict — a non-transient content block (served as a 5xx), not a fault.
func IsScrubbedServerError(body string) bool
IsScrubbedServerError reports whether body is the generic 500 crawl4ai 0.9.2+ returns for its application-level failures ({“error”:”Internal server error”,”correlation_id”:”…”}) after scrubbing the real verdict — block wall, content-gate, or crash — into its own server log.
func IsStructuralBlock(body string) bool
IsStructuralBlock reports whether a crawl4ai block verdict is its own content-gate (a thin / empty / unparseable render) rather than a genuine anti-bot wall. Only meaningful when IsBlockResponse(body) is already true.
func RetryableStatus(status int, body string) bool
RetryableStatus is an httpx.RetryConfig predicate shared by the Reddit and generic crawl paths: retry genuine transient 429/5xx, but never a 5xx whose body carries crawl4ai’s explicit block verdict (it recurs, and re-driving the crawl just pays the cost again). crawl4ai 0.9.2+ scrubs its 500 bodies, which makes a deterministic verdict (block/content-gate) byte-identical to a transient fault (worker OOM, pool churn) — both are retried, and the callers bound the cost with a low MaxAttempts instead of a veto that would strand the transients with no retry anywhere.
Generated by gomarkdoc