Starling

The credibility gap

Forums are the least accountable form of investment research. Anyone can post a call, and nothing checks it afterwards.

This creates an information asymmetry in reverse. The experienced trader who’s been consistently accurate for years looks identical to the newcomer making their first wild guess. There’s no reputation system, no track record, no empirical basis for trust.

Score every extracted prediction against what the price did, and credibility becomes a measured quantity rather than a reputation.

Traditional financial credibility comes from credentials — CFA certifications, institutional affiliations, Bloomberg terminals. But retail investor communities operate outside this framework. They’re credential-free zones where an anonymous username can move markets.

INSTITUTIONAL                    RETAIL FORUMS
─────────────                    ─────────────
CFA Certification                Anonymous Username
Track Record (Audited)           Post History (Unverified)
Regulatory Oversight             Community Moderation
Fiduciary Duty                   No Accountability
High Trust Barrier               Zero Trust Barrier

The gap creates opportunity for empirical credibility systems.

This gap isn’t just theoretical. It has real consequences:

Information asymmetry

Experienced traders with genuine insight are drowned out by volume. The signal-to-noise ratio approaches zero.

Manipulation vulnerability

Without accountability, pump-and-dump schemes thrive. Bad actors face no consequences for consistently wrong or misleading predictions.

Wasted attention

Retail investors spend hours parsing forum posts with no way to prioritise whose opinions actually matter.

The extraction pipeline

Building credibility from chaos requires a systematic approach. We process forum data through a four-stage pipeline that transforms unstructured discussion into measurable prediction accuracy.

1. Ingestion — stealth scraping with Playwright. JavaScript rendering. Rate-limited to avoid detection. Thread-level batching for efficiency.
2. Analysis — Perplexity AI extracts sentiment, direction, magnitude estimates, and time horizons from natural language posts.
3. Validation — backtesting against Yahoo Finance data. Multiple time horizons: 7-day, 30-day, 90-day windows.
4. Aggregation — per-user accuracy scores. Ticker-specific performance. Confidence-weighted rankings.

Sentiment extraction

The AI analysis layer processes entire forum threads in single API calls — a cost optimisation that also provides better context. Individual posts are analysed within the conversational flow of the discussion.

The extraction handles ambiguity. Not every post contains a tradeable prediction — many are questions, reactions, or general commentary. The AI distinguishes between explicit predictions (clear directional calls with price targets), implicit sentiment (general bullish/bearish tone without specific targets), and neutral content (questions, news sharing, or factual discussion).

Time horizon buckets

Predictions aren’t binary. A correct 7-day call is different from a correct 90-day thesis. We validate across three distinct windows:

Accuracy metrics

We measure prediction quality across multiple dimensions. Direction accuracy alone isn’t sufficient — magnitude matters.

Direction Accuracy    Did price move in predicted direction?
                      UP / DOWN / NEUTRAL

Magnitude Accuracy    How close was the % change estimate?
                      |predicted - actual| / actual

Composite Score       Weighted combination with confidence factor
                      Score = (0.6 * direction)
                            + (0.3 * magnitude)
                            + (0.1 * confidence_calibration)

A user who correctly predicts direction 80% of the time but wildly overestimates magnitude isn’t as valuable as one who’s accurate on both dimensions. The composite score captures this nuance.

The leaderboard emerges

After processing 6 months of HotCopper data, patterns emerge. The distribution of accuracy is not normal — it’s heavily skewed.

FigureUser accuracy distribution (n=12,847). Most users cluster around 50% — random. The right tail reveals signal.

Most users perform at or below chance level — consistent with noise. But the right tail tells a different story. Roughly 2% of users maintain accuracy above 65% across 50+ predictions. This is statistically significant signal.

User personas

The data reveals distinct user archetypes based on prediction behaviour and accuracy patterns:

The specialist

High accuracy on 2-3 specific tickers. Deep sector knowledge. Fewer predictions but higher conviction. Average 68% direction accuracy in their domain.

The technician

Chart-focused predictions. Strong near-term accuracy, weaker long-term. High volume of predictions. Pattern recognition over fundamentals.

The promoter

Consistently bullish regardless of conditions. High prediction volume, low accuracy. Often first to post on small-cap announcements. Signal value: negative.

System architecture

The technical implementation prioritises cost efficiency and scalability. Processing millions of posts through AI analysis requires careful batching and caching.

HotCopper ──[playwright]──→ Posts DB ──[perplexity]──→ Predictions
                                                            │
                                                            ▼
Yahoo Finance ──[yfinance]──→ Price Data ──[backtest]──→ Outcomes
                                                            │
                                                            ▼
                                              User Rankings ──→ Dashboard

Thread-level batching

One Perplexity API call per forum thread, not per post.

Activity-based scheduling

Active threads scraped every 6 hours, dormant threads weekly.

Response caching

Duplicate content detection avoids redundant AI processing.

Resumable checkpoints

Long-running scrapes can resume from interruption.

The console

The system exposes a Next.js dashboard for exploring credibility data in real-time. Key interfaces include a dashboard overview with real-time metrics and recent activity, user rankings sorted by composite credibility score, extracted predictions with sentiment analysis and validation status, and an AI operations panel for monitoring extraction and analysis pipelines.

Key research questions

• Does historical prediction accuracy predict future accuracy? Is credibility persistent or mean-reverting?
• Can we identify market manipulation patterns through coordinated prediction behaviour?
• What’s the correlation between prediction confidence and actual accuracy across user tiers?
• Can ensemble predictions from top-ranked users outperform individual institutional analysis?

Future applications

Credibility API

Real-time credibility scores for forum users. Integration with trading platforms to surface high-signal predictions.

Multi-platform expansion

Extend beyond HotCopper to Reddit (r/ASX_Bets), Twitter FinTwit, and international equivalents.

Manipulation detection

Pattern recognition for coordinated pump schemes. Early warning system for retail investor protection.