import "github.com/kinorai/omnifeed/internal/engine"
Package engine defines the dispatch mechanism that picks the right per-URL handler. New engines (Hacker News, Stack Overflow, …) plug in by implementing domain.Engine and being Registered before the fallback.
FallbackNotice is the first line of a document the generic fallback rendered for a URL a dedicated engine claimed, so a reader of the text alone (an AI agent over MCP) knows it is not looking at the engine’s output.
const FallbackNotice = "> Note: the dedicated %s engine failed (%s); this is the generic page render instead.\n\n"
func CacheLabel(d Dispatcher, doc domain.Document, err error, noCache bool) string
| CacheLabel is the omnifeed_requests_total `cache` label for one crawl through d: the cache’s own verdict on a document (hit | miss | bypass), and for an error — which is never served from the cache — bypass when the caller asked for no_cache, else miss. off when d has no cache in front of it. |
CacheReporter is implemented by a Dispatcher that serves through a response cache (internal/fetchcache). Transports read it only to label metrics.
type CacheReporter interface {
CacheEnabled() bool
}
Dispatcher is what the transports need from the engine layer: crawl a URL through the right engine, and name the engine that would handle it (for metrics labels). *Registry implements it; the fetch_url response cache (internal/fetchcache) decorates it, so every transport shares one cache.
type Dispatcher interface {
Crawl(ctx context.Context, rawURL string, opts domain.EngineOptions) (domain.Document, error)
Resolve(rawURL string) domain.Engine
}
Registry holds an ordered list of engines and a fallback. Lookup is first-match-wins; the fallback handles anything no engine claimed.
type Registry struct {
// contains filtered or unexported fields
}
func New() *Registry
New returns an empty Registry. Use Register and Fallback to populate it.
func (r *Registry) BlockPrivateIPs(block bool) *Registry
BlockPrivateIPs configures the SSRF choke point. Crawl validates every URL before dispatch, so no transport (HTTP loader, MCP HTTP, MCP stdio) can forget the check. The http(s)-scheme and non-empty-host checks always run; the private/reserved-IP rejection is gated on block.
func (r *Registry) Crawl(ctx context.Context, rawURL string, opts domain.EngineOptions) (domain.Document, error)
Crawl dispatches rawURL to the resolved engine, after validating it at the SSRF choke point (see BlockPrivateIPs). Validating here — rather than in each transport — guarantees every inbound path is covered.
func (r *Registry) Fallback(e domain.Engine) *Registry
Fallback sets the engine used when no Registered engine claims a URL.
func (r *Registry) Logger(l *slog.Logger) *Registry
Logger sets the logger used to report engine→fallback handoffs.
func (r *Registry) Metrics(m *observability.Metrics) *Registry
Metrics sets the collectors used to count engine→fallback handoffs (omnifeed_engine_fallbacks_total). Nil disables the counter.
func (r *Registry) Register(e domain.Engine) *Registry
Register appends an engine to the dispatch chain. Order matters — earlier engines get first crack at each URL.
func (r *Registry) Resolve(rawURL string) domain.Engine
Resolve returns the engine that should handle rawURL.
Generated by gomarkdoc