import "github.com/kinorai/omnifeed/internal/engine/discourse"
Package discourse implements the Discourse engine. It reads a forum’s public topic JSON API and renders a topic URL as topic header + the full post list (TOON) — mirroring the Hacker News and GitHub engines’ shape.
Like those two (and unlike Reddit and the generic engine), this engine fetches its upstream DIRECTLY over HTTP rather than through crawl4ai: the topic JSON is public and not bot-walled, and a browser render is both slower and lossy (posts are lazily paginated in the DOM — the benchmark’s browser fallback returned the last 2 of 6 posts, buried in ~40% navigation chrome).
Discourse is self-hosted software running on arbitrary domains, so there is no host pattern to match on. domain.Engine.Matches is a pure predicate — it cannot probe a host to find out whether it runs Discourse — so the set of hosts this engine claims is an explicit operator-supplied allowlist (OMNIFEED_DISCOURSE_HOSTS). Unlisted forums fall through to the generic browser fallback, which still renders them, just less completely.
Config configures a Discourse Engine.
type Config struct {
Client *httpx.Client
Limiter httpx.Limiter
// Hosts is the exact hostname allowlist. Empty means the engine claims
// nothing — it can still be registered, Matches just always returns false.
Hosts []string
Timeout time.Duration // wall-clock budget per crawl; defaults to defaultTimeout
Logger *slog.Logger
}
Engine implements domain.Engine for Discourse topic URLs via the public topic JSON API.
type Engine struct {
// contains filtered or unexported fields
}
func New(cfg Config) *Engine
New returns a Discourse Engine configured per cfg.
func (e *Engine) Crawl(ctx context.Context, rawURL string, _ domain.EngineOptions) (domain.Document, error)
Crawl fetches the topic behind rawURL from the forum’s topic JSON API and returns it encoded as TOON.
func (e *Engine) Matches(rawURL string) bool
Matches claims topic URLs on the configured hosts only. The host must equal a configured hostname exactly (case-insensitively) — no subdomain wildcarding, because a Discourse install at forum.example.com says nothing about example.com. Non-topic pages on a listed host, and every page on an unlisted host, fall through to the generic crawl4ai fallback.
func (*Engine) Name() string
Name returns the engine identifier (“discourse”).
Post is one post in the topic. ReplyTo keeps the reply structure reconstructable without nesting (the same flat+parent shape the Reddit, Hacker News, and GitHub engines use).
type Post struct {
Number int `json:"number" toon:"number"`
Login string `json:"login" toon:"login"`
Created string `json:"created" toon:"created"`
ReplyTo int `json:"reply_to,omitempty" toon:"reply_to,omitempty"`
Body string `json:"body" toon:"body"`
}
Thread groups a topic with its posts, in post-stream order. Note, when set, tells the reader (the LLM) that the post list was truncated — metadata keys land in MCP _meta, which models never see, so the signal has to live in the content itself.
type Thread struct {
Topic Topic `json:"topic" toon:"topic"`
Note string `json:"note,omitempty" toon:"note,omitempty"`
Posts []Post `json:"posts" toon:"posts"`
}
Topic is the header of a Discourse topic, stripped to LLM-relevant fields.
type Topic struct {
Title string `json:"title" toon:"title"`
PostsCount int `json:"posts_count" toon:"posts_count"`
Created string `json:"created" toon:"created"`
Host string `json:"host" toon:"host"`
}
Generated by gomarkdoc