import "github.com/kinorai/omnifeed/internal/engine/hackernews"
Package hackernews implements the Hacker News engine. It reads the public Algolia HN API (hn.algolia.com) and renders a front-page URL as a ranked story list and an /item?id= URL as a flattened comment tree (TOON) — mirroring the Reddit engine’s output shape so nesting is reconstructable from parent_id.
Unlike the Reddit and generic engines, this engine fetches its upstream DIRECTLY over HTTP rather than through crawl4ai: the Algolia HN API is a public, CORS-open JSON API with no bot wall, so a headless browser would only add latency. It does require omnifeed to have outbound access to hn.algolia.com.
DefaultThreadComments is the cap on comments emitted for one thread when the caller sets no max_comments, so a megathread can’t blow the consumer’s context by default. AbsoluteMaxThreadComments is the most a caller can ask for: Algolia returns the whole tree in one request, so a bigger cap costs no extra fetches. Exported because the fetch_url tool schema states both numbers.
const (
DefaultThreadComments = 500
AbsoluteMaxThreadComments = 5000
)
Comment is a Hacker News comment, flattened with parent_id (like the Reddit engine) so the full nesting is reconstructable without deep indentation.
type Comment struct {
ID int `json:"id" toon:"id"`
ParentID int `json:"parent_id" toon:"parent_id"`
Author string `json:"author" toon:"author"`
Body string `json:"body" toon:"body"`
Created int64 `json:"created" toon:"created"`
}
Config configures a Hacker News Engine.
type Config struct {
Client *httpx.Client
Limiter httpx.Limiter
APIBase string // defaults to the public Algolia API; overridden in tests
RankBase string // HN Firebase API, source of the real front-page order; "" = default, "-" = disabled
Timeout time.Duration // wall-clock budget per crawl; defaults to defaultTimeout
Logger *slog.Logger
}
Engine implements domain.Engine for Hacker News URLs via the Algolia API.
type Engine struct {
// contains filtered or unexported fields
}
func New(cfg Config) *Engine
New returns a Hacker News Engine configured per cfg.
func (e *Engine) Crawl(ctx context.Context, rawURL string, eo domain.EngineOptions) (domain.Document, error)
Crawl fetches the HN item or feed behind rawURL from the Algolia API and returns it encoded as TOON.
func (*Engine) Matches(rawURL string) bool
Matches claims the news.ycombinator.com URLs this engine renders: /item?id=N threads and the front-page-style feeds (/, /news, /newest, /ask, /show). Anything else (user profiles, /jobs, …) falls through to the generic fallback.
func (*Engine) Name() string
Name returns the engine identifier (“hackernews”).
FrontPage is a Hacker News listing (front page / newest / ask / show).
type FrontPage struct {
Feed string `json:"feed" toon:"feed"`
Stories []Story `json:"stories" toon:"stories"`
}
Item is the story/post header of a thread.
type Item struct {
ID int `json:"id" toon:"id"`
Title string `json:"title" toon:"title"`
Author string `json:"author" toon:"author"`
URL string `json:"url,omitempty" toon:"url,omitempty"`
Points int `json:"points" toon:"points"`
Created int64 `json:"created" toon:"created"`
// TotalComments is how many live comments the Algolia tree held before any
// cap; Truncated reports whether the caps dropped some of them. They sit in
// the body because AI agents see the TOON text, not the response _meta.
TotalComments int `json:"total_comments" toon:"total_comments"`
Truncated bool `json:"truncated" toon:"truncated"`
Text string `json:"text,omitempty" toon:"text,omitempty"`
}
Story is one Hacker News front-page entry, stripped to LLM-relevant fields. No rank field: the Algolia feed is ordered by points, not HN’s live gravity rank, so a positional rank would misrepresent the front page.
type Story struct {
ID int `json:"id" toon:"id"`
Title string `json:"title" toon:"title"`
URL string `json:"url,omitempty" toon:"url,omitempty"`
Author string `json:"author" toon:"author"`
Points int `json:"points" toon:"points"`
NumComments int `json:"num_comments" toon:"num_comments"`
Created int64 `json:"created" toon:"created"`
}
Thread groups an HN story with its flattened comment tree.
type Thread struct {
Story Item `json:"story" toon:"story"`
Comments []Comment `json:"comments" toon:"comments"`
}
Generated by gomarkdoc