import "github.com/kinorai/omnifeed/internal/engine/hackernews"
Package hackernews implements the Hacker News engine. It reads the public Algolia HN API (hn.algolia.com) and renders a front-page URL as a ranked story list and an /item?id= URL as a flattened comment tree (TOON) — mirroring the Reddit engine’s output shape so nesting is reconstructable from parent_id.
Unlike the Reddit and generic engines, this engine fetches its upstream DIRECTLY over HTTP rather than through crawl4ai: the Algolia HN API is a public, CORS-open JSON API with no bot wall, so a headless browser would only add latency. It does require omnifeed to have outbound access to hn.algolia.com.
MaxThreadComments is the ceiling on comments emitted for one thread, so a megathread can’t blow the consumer’s context. A caller’s max_comments can only lower it. Exported because the fetch_url tool schema states the number.
const MaxThreadComments = 500
Comment is a Hacker News comment, flattened with parent_id (like the Reddit engine) so the full nesting is reconstructable without deep indentation.
type Comment struct {
ID int `json:"id" toon:"id"`
ParentID int `json:"parent_id" toon:"parent_id"`
Author string `json:"author" toon:"author"`
Body string `json:"body" toon:"body"`
Created int64 `json:"created" toon:"created"`
}
Config configures a Hacker News Engine.
type Config struct {
Client *httpx.Client
Limiter httpx.Limiter
APIBase string // defaults to the public Algolia API; overridden in tests
Timeout time.Duration // wall-clock budget per crawl; defaults to defaultTimeout
Logger *slog.Logger
}
Engine implements domain.Engine for Hacker News URLs via the Algolia API.
type Engine struct {
// contains filtered or unexported fields
}
func New(cfg Config) *Engine
New returns a Hacker News Engine configured per cfg.
func (e *Engine) Crawl(ctx context.Context, rawURL string, eo domain.EngineOptions) (domain.Document, error)
Crawl fetches the HN item or feed behind rawURL from the Algolia API and returns it encoded as TOON.
func (*Engine) Matches(rawURL string) bool
Matches claims the news.ycombinator.com URLs this engine renders: /item?id=N threads and the front-page-style feeds (/, /news, /newest, /ask, /show). Anything else (user profiles, /jobs, …) falls through to the generic fallback.
func (*Engine) Name() string
Name returns the engine identifier (“hackernews”).
FrontPage is a Hacker News listing (front page / newest / ask / show).
type FrontPage struct {
Feed string `json:"feed" toon:"feed"`
Stories []Story `json:"stories" toon:"stories"`
}
Item is the story/post header of a thread.
type Item struct {
ID int `json:"id" toon:"id"`
Title string `json:"title" toon:"title"`
Author string `json:"author" toon:"author"`
URL string `json:"url,omitempty" toon:"url,omitempty"`
Points int `json:"points" toon:"points"`
Created int64 `json:"created" toon:"created"`
Text string `json:"text,omitempty" toon:"text,omitempty"`
}
Story is one Hacker News front-page entry, stripped to LLM-relevant fields. No rank field: the Algolia feed is ordered by points, not HN’s live gravity rank, so a positional rank would misrepresent the front page.
type Story struct {
ID int `json:"id" toon:"id"`
Title string `json:"title" toon:"title"`
URL string `json:"url,omitempty" toon:"url,omitempty"`
Author string `json:"author" toon:"author"`
Points int `json:"points" toon:"points"`
NumComments int `json:"num_comments" toon:"num_comments"`
Created int64 `json:"created" toon:"created"`
}
Thread groups an HN story with its flattened comment tree.
type Thread struct {
Story Item `json:"story" toon:"story"`
Comments []Comment `json:"comments" toon:"comments"`
}
Generated by gomarkdoc