import "github.com/kinorai/omnifeed/internal/engine/github"
Package github implements the GitHub engine. It reads the GitHub REST API (api.github.com) and renders:
Like the Hacker News engine (and unlike Reddit and the generic engine), this engine fetches its upstream DIRECTLY over HTTP rather than through crawl4ai: the API is public JSON with no bot wall, and a browser render of a GitHub page is slower and full of UI chrome (comments are lazily paginated in the DOM). It does require omnifeed to have outbound access to api.github.com.
Comment is one conversation-tab comment (issue or PR), pruned to the three fields an agent actually reads.
type Comment struct {
Login string `json:"login" toon:"login"`
Created string `json:"created" toon:"created"`
Body string `json:"body" toon:"body"`
}
Config configures a GitHub Engine.
type Config struct {
Client *httpx.Client
Limiter httpx.Limiter
APIBase string // defaults to the public REST API; overridden in tests
Token string // optional PAT; empty = anonymous (60 req/h/IP)
Timeout time.Duration // wall-clock budget per crawl; defaults to defaultTimeout
Logger *slog.Logger
}
Engine implements domain.Engine for github.com and gist.github.com pages via the GitHub REST API (and GraphQL for discussions).
type Engine struct {
// contains filtered or unexported fields
}
func New(cfg Config) *Engine
New returns a GitHub Engine configured per cfg.
func (e *Engine) Crawl(ctx context.Context, rawURL string, _ domain.EngineOptions) (domain.Document, error)
Crawl fetches the page behind rawURL from the GitHub API. Issues and pull requests are returned as TOON (headed by a markdown title line); every other kind as markdown, which the transports may truncate to max_chars.
func (e *Engine) Matches(rawURL string) bool
Matches claims only the GitHub URLs this engine renders: repository roots, files (blob), directories (tree), issues, pull requests, commits, releases, discussions, and gists. Everything else on the host (actions, wiki, compare, issue lists, settings, search, …) falls through to the generic fallback, which renders the page through crawl4ai. Discussions need GraphQL, which requires a token, so without one they fall through too.
func (*Engine) Name() string
Name returns the engine identifier (“github”).
File is one changed file. Patch is empty once the diff budget is spent — the filename and stats are still listed so the change set stays complete.
type File struct {
Name string `json:"name" toon:"name"`
Status string `json:"status" toon:"status"`
Additions int `json:"additions" toon:"additions"`
Deletions int `json:"deletions" toon:"deletions"`
Patch string `json:"patch,omitempty" toon:"patch,omitempty"`
}
InlineComment is a review comment anchored to a line of the diff. ReplyTo keeps review threads reconstructable without nesting (same flat+parent shape the Reddit and Hacker News engines use).
type InlineComment struct {
Path string `json:"path" toon:"path"`
Line int `json:"line" toon:"line"`
ReplyTo int64 `json:"reply_to,omitempty" toon:"reply_to,omitempty"`
Login string `json:"login" toon:"login"`
Created string `json:"created" toon:"created"`
Body string `json:"body" toon:"body"`
}
Issue is the header of a GitHub issue, stripped to LLM-relevant fields.
type Issue struct {
Title string `json:"title" toon:"title"`
Author string `json:"author" toon:"author"`
State string `json:"state" toon:"state"`
Created string `json:"created" toon:"created"`
Labels []string `json:"labels,omitempty" toon:"labels,omitempty"`
Comments int `json:"comments" toon:"comments"`
Body string `json:"body,omitempty" toon:"body,omitempty"`
}
IssueThread groups an issue with its comments. Note, when set, tells the reader (the LLM) that a list was truncated — metadata keys land in MCP _meta, which models never see, so the signal has to live in the content itself.
type IssueThread struct {
Issue Issue `json:"issue" toon:"issue"`
Note string `json:"note,omitempty" toon:"note,omitempty"`
Comments []Comment `json:"comments" toon:"comments"`
}
PullRequest is the header of a pull request. It carries the issue fields (a PR *is* an issue) plus the diff stats that only the pulls endpoint reports.
type PullRequest struct {
Title string `json:"title" toon:"title"`
Author string `json:"author" toon:"author"`
State string `json:"state" toon:"state"`
Draft bool `json:"draft" toon:"draft"`
Merged bool `json:"merged" toon:"merged"`
Created string `json:"created" toon:"created"`
Labels []string `json:"labels,omitempty" toon:"labels,omitempty"`
Comments int `json:"comments" toon:"comments"`
Additions int `json:"additions" toon:"additions"`
Deletions int `json:"deletions" toon:"deletions"`
ChangedFiles int `json:"changed_files" toon:"changed_files"`
Body string `json:"body,omitempty" toon:"body,omitempty"`
}
PullThread is everything the four PR endpoints contribute, in one document. Note carries truncation warnings, as on IssueThread.
type PullThread struct {
PR PullRequest `json:"pr" toon:"pr"`
Note string `json:"note,omitempty" toon:"note,omitempty"`
Comments []Comment `json:"comments" toon:"comments"`
Reviews []Review `json:"reviews" toon:"reviews"`
InlineComments []InlineComment `json:"inline_comments" toon:"inline_comments"`
Files []File `json:"files" toon:"files"`
}
Review is one submitted PR review (APPROVED / CHANGES_REQUESTED / COMMENTED).
type Review struct {
Login string `json:"login" toon:"login"`
State string `json:"state" toon:"state"`
Submitted string `json:"submitted" toon:"submitted"`
Body string `json:"body,omitempty" toon:"body,omitempty"`
}
Generated by gomarkdoc