omnifeed

twitter

import "github.com/kinorai/omnifeed/internal/engine/twitter"

Package twitter implements the Twitter/X engine. It renders an x.com (or twitter.com, fxtwitter.com, fixupx.com, vxtwitter.com, fixvx.com) post URL as the post itself — full text, quote, media with alt text, community note, poll, link card, X Article body — plus its parents, the author’s self-thread and the top replies. Markdown by default; TOON or JSON when the caller passes `format`.

x.com is bot-walled for headless browsers and twitter.com links fail in crawl4ai outright, so the engine reads public JSON mirrors DIRECTLY over HTTP, in order:

  1. FxTwitter API v2 (api.fxtwitter.com, or a self-hosted FxEmbed via OMNIFEED_TWITTER_FXTWITTER_URL): /2/conversation for the post, its parents and the first page of replies; /2/thread when the post belongs to the author’s self-thread.
  2. The embed syndication endpoint (cdn.syndication.twimg.com), which cuts long posts at ~275 characters — so for a note_tweet the vxTwitter API (api.vxtwitter.com) is asked for the full text, and the document is flagged text_truncated when neither could supply it.
  3. The generic crawl4ai engine on the canonical x.com URL, which X serves server-rendered to logged-out readers.

When every source fails the engine returns a classified “tweet unavailable” error — never a near-empty document — marked domain.NoFallback so the registry does not render the URL a second time.

Index

Constants

const (
    // DefaultFxTwitterURL is the public FxTwitter API. A self-hosted FxEmbed
    // serves the same API (OMNIFEED_TWITTER_FXTWITTER_URL).
    DefaultFxTwitterURL = "https://api.fxtwitter.com"
    // DefaultMaxReplies is how many replies a post renders by default.
    DefaultMaxReplies = 20
)

type Config

Config configures a Twitter Engine.

type Config struct {
    Client  *httpx.Client
    Limiter httpx.Limiter
    // Generic is the engine tried last, on the canonical x.com URL (crawl4ai
    // in production). Nil skips that step and lets the registry's fallback
    // handle a failure on the original URL instead.
    Generic        domain.Engine
    FxTwitterURL   string // defaults to DefaultFxTwitterURL
    SyndicationURL string // defaults to the public endpoint; overridden in tests
    VxTwitterURL   string // defaults to the public endpoint; overridden in tests
    MaxReplies     int    // defaults to DefaultMaxReplies
    Timeout        time.Duration
    // BlockPrivateIPs applies the SSRF guard to the destination of a t.co
    // link before it is crawled.
    BlockPrivateIPs bool
    Logger          *slog.Logger
}

type Engine

Engine implements domain.Engine for X / Twitter post URLs.

type Engine struct {
    // contains filtered or unexported fields
}

func New

func New(cfg Config) *Engine

New returns a Twitter Engine configured per cfg.

func (*Engine) Crawl

func (e *Engine) Crawl(ctx context.Context, rawURL string, opts domain.EngineOptions) (domain.Document, error)

Crawl renders the post behind rawURL.

func (*Engine) Matches

func (*Engine) Matches(rawURL string) bool

Matches claims post URLs on x.com, twitter.com and the fixer mirrors, and t.co short links. Profiles, search, lists and the rest fall through to the generic engine (which rewrites twitter.com to x.com).

func (*Engine) Name

func (*Engine) Name() string

Name returns the engine identifier (“twitter”).

type OutPost

OutPost is one post in the structured output. Media, Poll and Card are pre-rendered one-liners — the same strings the markdown shows — so an LLM reads one vocabulary whichever format it asked for.

type OutPost struct {
    ID        string   `json:"id" toon:"id"`
    URL       string   `json:"url" toon:"url"`
    Author    string   `json:"author" toon:"author"`
    Name      string   `json:"name,omitempty" toon:"name,omitempty"`
    CreatedAt string   `json:"created_at,omitempty" toon:"created_at,omitempty"`
    Text      string   `json:"text" toon:"text"`
    Likes     int      `json:"likes" toon:"likes"`
    Reposts   int      `json:"reposts,omitempty" toon:"reposts,omitempty"`
    Replies   int      `json:"replies,omitempty" toon:"replies,omitempty"`
    Quotes    int      `json:"quotes,omitempty" toon:"quotes,omitempty"`
    Views     int      `json:"views,omitempty" toon:"views,omitempty"`
    ReplyTo   string   `json:"reply_to,omitempty" toon:"reply_to,omitempty"`
    ByAuthor  bool     `json:"by_author,omitempty" toon:"by_author,omitempty"`
    Media     []string `json:"media,omitempty" toon:"media,omitempty"`
    Quote     *OutPost `json:"quote,omitempty" toon:"quote,omitempty"`
    Note      string   `json:"community_note,omitempty" toon:"community_note,omitempty"`
    Poll      []string `json:"poll,omitempty" toon:"poll,omitempty"`
    Card      string   `json:"card,omitempty" toon:"card,omitempty"`
    Article   string   `json:"article,omitempty" toon:"article,omitempty"`
}

type Output

Output is the structured rendering of a bundle.

type Output struct {
    Post      OutPost   `json:"post" toon:"post"`
    Parents   []OutPost `json:"parents,omitempty" toon:"parents,omitempty"`
    Thread    []OutPost `json:"thread,omitempty" toon:"thread,omitempty"`
    Replies   []OutPost `json:"replies,omitempty" toon:"replies,omitempty"`
    Source    string    `json:"source" toon:"source"`
    Truncated bool      `json:"truncated" toon:"truncated"`
}

Generated by gomarkdoc

This site is open source. Improve this page.