Build a Real-Time X Sentiment Pipeline via API

Quick Answer

Use TwexAPI to build a real-time X sentiment pipeline by polling POST /twitter/advanced_search/page with searchTerms and sortBy: "Latest", paging with next_cursor, deduplicating by tweet_id, and scoring only new posts. Store raw tweet JSON, query, fetch time, language, model version, sentiment score, label, and review status so every dashboard point can be audited. Treat scores as model output, then send high-impact or borderline samples to a heavier model or human review when needed.

FAQ

Why use Advanced Search by Page instead of one bulk search call?

A real-time dashboard needs repeatable polling and pagination. /twitter/advanced_search/page returns a page of recent tweets plus has_next_page and next_cursor, so workers can fetch a bounded number of pages each run, resume cleanly, and avoid rescoring the same tweet_id over and over.

What should I store before running sentiment models?

Store the raw tweet JSON, normalized tweet_id, query, sortBy, fetch time, created_at_datetime, language, author handle, and engagement counts. If the model or thresholds change later, those fields let you reprocess the same source data without guessing how it entered the dataset.

When should I use an LLM instead of a lightweight model?

Use the lightweight model for broad coverage and speed. Send only selected rows to an LLM: posts with high engagement, borderline scores near neutral, mixed sarcasm, important accounts, or alerts that may trigger a business response. Store both model versions so score changes stay auditable.

Sentiment analysis fails when the data is stale or the score cannot be audited. If a dashboard only says "market sentiment: -0.42" but cannot show which posts, which query, which model version, and when the data was collected, analysts will not trust it.

A better approach is to treat sentiment as a data pipeline: poll recent posts with TwexAPI Advanced Search by Page, deduplicate by tweet_id, store raw JSON, then run a lightweight model first. You can later replace VADER with a transformer or LLM, but the collection, dedupe, scoring, aggregation, and review layers should stay stable.

Treat sentiment_score as model output, not as the final truth about public opinion. The score is useful only when it can be traced back to the original post, query, model version, sample size, and review status.

Pipeline Shape

Answer: Pipeline Shape means using TwexAPI Bearer APIs on api.twexapi.io for this user case. A typical read uses 10 Credits (about $0.10 per 1,000), and qualifying paid plans publish 20+ QPS. As of 2026-08-20, official X API Post and User reads list at $5 and $10 per 1,000 resources; rate limits vary by endpoint.

The pipeline turns X posts from query results into auditable sentiment metrics. It usually has five parts:

• Query design: Store brand terms, events, language filters, exclusions, and refresh windows in query configuration.
• Collection: Call POST /twitter/advanced_search/page with sortBy: "Latest" to collect recent posts.
• Storage: Save raw JSON and deduplicate by tweet_id before scoring.
• Scoring: Use a lightweight model for most posts and send borderline or high-impact samples to heavier models.
• Aggregation and review: Output averages, sample counts, outliers, and model versions by time window.

Step 1: Polling Collection

Answer: Step 1: Polling Collection is implemented by calling the TwexAPI endpoint documented in this guide with a Bearer Token; batch or paginated requests reduce overhead to ~10 credits per call at 20+ QPS.

Polling collection requests recent search results on a schedule instead of pretending you have a streaming connection. TwexAPI's POST /twitter/advanced_search/page returns up to 20 tweets per page. The first request omits next_cursor; later requests pass the cursor from the previous response.

The example below uses sortBy: "Latest" and seen_ids to avoid rescoring the same tweet. In production, store seen_ids in a database or cache rather than in memory.

1import os
2from datetime import datetime, timezone
3
4import requests
5
6API_URL = "https://api.twexapi.io/twitter/advanced_search/page"
7TOKEN = os.environ["TWEXAPI_BEARER_TOKEN"]
8
9def search_page(query, next_cursor=None):
10    payload = {
11        "searchTerms": [query],
12        "sortBy": "Latest"
13    }
14
15    if next_cursor:
16        payload["next_cursor"] = next_cursor
17
18    response = requests.post(
19        API_URL,
20        headers={
21            "Authorization": f"Bearer {TOKEN}",
22            "Content-Type": "application/json",
23            "Accept": "application/json",
24        },
25        json=payload,
26        timeout=30,
27    )
28    response.raise_for_status()
29    data = response.json()
30
31    if data.get("code", 200) >= 400:
32        raise RuntimeError(data.get("msg", "TwexAPI returned an error"))
33
34    return data
35
36def collect_recent_tweets(query, seen_ids, max_pages=3):
37    next_cursor = None
38    collected = []
39
40    for page_number in range(max_pages):
41        page = search_page(query, next_cursor)
42
43        for tweet in page.get("data", []):
44            if not tweet:
45                continue
46
47            tweet_id = tweet.get("tweet_id") or tweet.get("id")
48            if not tweet_id or tweet_id in seen_ids:
49                continue
50
51            seen_ids.add(tweet_id)
52            collected.append({
53                "query": query,
54                "fetched_at": datetime.now(timezone.utc).isoformat(),
55                "raw": tweet,
56            })
57
58        if not page.get("has_next_page"):
59            break
60
61        next_cursor = page.get("next_cursor")
62        if not next_cursor:
63            break
64
65    return collected
66
67seen_ids = set()
68tweets = collect_recent_tweets(
69    query='"crypto regulation" lang:en -filter:retweets',
70    seen_ids=seen_ids,
71    max_pages=2,
72)
73
74print(f"New tweets ready for scoring: {len(tweets)}")

Step 2: Fast Sentiment Scoring

Answer: Step 2: Fast Sentiment Scoring is implemented by calling the TwexAPI endpoint documented in this guide with a Bearer Token; batch or paginated requests reduce overhead to ~10 credits per call at 20+ QPS.

Fast sentiment scoring uses an explainable, low-cost model to score all new posts first, then sends only selected samples to a heavier model. The dashboard stays fresh without sending every text snippet to an expensive LLM.

VADER is useful as a first pass for English social text. For multilingual pipelines, route by lang: English can go to VADER, while other languages can use language-specific models or a translated common model.

1from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer
2
3analyzer = SentimentIntensityAnalyzer()
4MODEL_VERSION = "vader-3.3.2-en-baseline"
5
6def sentiment_label(score):
7    if score >= 0.2:
8        return "positive"
9    if score <= -0.2:
10        return "negative"
11    return "neutral"
12
13def text_for_model(tweet):
14    return tweet.get("full_text") or tweet.get("text") or ""
15
16def score_tweets(records):
17    scored = []
18
19    for record in records:
20        tweet = record["raw"]
21        text = text_for_model(tweet).strip()
22        if not text:
23            continue
24
25        compound = analyzer.polarity_scores(text)["compound"]
26        scored.append({
27            "tweet_id": tweet.get("tweet_id") or tweet.get("id"),
28            "query": record["query"],
29            "fetched_at": record["fetched_at"],
30            "created_at_datetime": tweet.get("created_at_datetime"),
31            "lang": tweet.get("lang"),
32            "text": text,
33            "sentiment_score": compound,
34            "sentiment_label": sentiment_label(compound),
35            "model_version": MODEL_VERSION,
36            "favorite_count": tweet.get("favorite_count"),
37            "retweet_count": tweet.get("retweet_count"),
38            "reply_count": tweet.get("reply_count"),
39            "raw": tweet,
40        })
41
42    return scored
43
44scored_tweets = score_tweets(tweets)
45print(f"Scored tweets: {len(scored_tweets)}")

Step 3: Window Aggregation

Answer: Step 3: Window Aggregation is implemented by calling the TwexAPI endpoint documented in this guide with a Bearer Token; batch or paginated requests reduce overhead to ~10 credits per call at 20+ QPS.

Window aggregation combines individual tweet scores into time windows that show trends. Do not look only at the average score; also store sample size, positive ratio, negative ratio, and high-engagement examples.

1def summarize_window(scored_rows):
2    if not scored_rows:
3        return {
4            "tweet_count": 0,
5            "avg_sentiment": None,
6            "positive_ratio": 0,
7            "negative_ratio": 0,
8            "top_negative_examples": [],
9        }
10
11    tweet_count = len(scored_rows)
12    avg_sentiment = sum(row["sentiment_score"] for row in scored_rows) / tweet_count
13    positive_count = sum(1 for row in scored_rows if row["sentiment_label"] == "positive")
14    negative_count = sum(1 for row in scored_rows if row["sentiment_label"] == "negative")
15
16    top_negative_examples = sorted(
17        scored_rows,
18        key=lambda row: (row["sentiment_score"], -int(row.get("reply_count") or 0)),
19    )[:5]
20
21    return {
22        "tweet_count": tweet_count,
23        "avg_sentiment": round(avg_sentiment, 4),
24        "positive_ratio": round(positive_count / tweet_count, 4),
25        "negative_ratio": round(negative_count / tweet_count, 4),
26        "top_negative_examples": [
27            {
28                "tweet_id": row["tweet_id"],
29                "score": row["sentiment_score"],
30                "text": row["text"][:180],
31            }
32            for row in top_negative_examples
33        ],
34    }
35
36summary = summarize_window(scored_tweets)
37print(summary)

Step 4: Production Expansion

Answer: Step 4: Production Expansion is implemented by calling the TwexAPI endpoint documented in this guide with a Bearer Token; batch or paginated requests reduce overhead to ~10 credits per call at 20+ QPS.

Production expansion splits collection, scoring, aggregation, and alerts into recoverable jobs. You can scale collection with multiple query workers, but the write path should stay strict: dedupe first, score second, and keep every model output tied to the query that produced it.

At minimum, handle these points:

1. Configurable queries: Store query, language, exclusions, refresh interval, and max page count in configuration.
2. Separate cursor and dedupe: next_cursor is for pagination; tweet_id is the dedupe key.
3. Empty-window protection: When there are no new posts, do not calculate fake averages; mark sample count as 0.
4. Model versioning: Scores from changed models or thresholds should not be compared as if they came from the same model.
5. Sentiment decay: Refresh active topics on a schedule so old sentiment does not dominate the dashboard.
6. Human review queue: Posts with high negative scores, high engagement, or key accounts should go to review.

Fields To Store

Answer: Fields To Store means using TwexAPI Bearer APIs on api.twexapi.io for this user case. A typical read uses 10 Credits (about $0.10 per 1,000), and qualifying paid plans publish 20+ QPS. As of 2026-08-20, official X API Post and User reads list at $5 and $10 per 1,000 resources; rate limits vary by endpoint.

Store the minimum evidence needed to trace every score back to the source post, query, and model version.

Alert Rules

Answer: Alert Rules means using TwexAPI Bearer APIs on api.twexapi.io for this user case. A typical read uses 10 Credits (about $0.10 per 1,000), and qualifying paid plans publish 20+ QPS. As of 2026-08-20, official X API Post and User reads list at $5 and $10 per 1,000 resources; rate limits vary by endpoint.

Alert rules should turn sentiment scores into action-ready signals, not page the team for every negative fluctuation. Combine score movement with sample size, engagement weight, and query importance.

Start with three alert types:

• Negative ratio spike: The latest window has a higher negative ratio than the previous window and exceeds your minimum sample count.
• High-engagement negative post: One negative post has replies, reposts, or views far above the window median.
• Critical query hit: Brand, outage, policy, price, safety, or regulatory terms appear in new negative samples.

Each alert should include the query, time window, sample count, average score, negative ratio, representative tweet_id, and model_version. That gives the receiver enough context to decide whether to escalate.

Summary

Answer: Summary means using TwexAPI Bearer APIs on api.twexapi.io for this user case. A typical read uses 10 Credits (about $0.10 per 1,000), and qualifying paid plans publish 20+ QPS. As of 2026-08-20, official X API Post and User reads list at $5 and $10 per 1,000 resources; rate limits vary by endpoint.

Build sentiment analysis as a traceable data pipeline: poll /twitter/advanced_search/page with sortBy: "Latest", page with next_cursor, deduplicate by tweet_id, score new posts, aggregate by time window, and save the raw evidence behind every number.