# Starling

Market Credibility Intelligence

Alpha · Published 2025-04-01 · Tags: Financial

1,200 predictions a day, none of them scored.

Australia’s largest retail investor forum: 650,000 users, over two million posts a year, and no record of whose calls came good. A consistently accurate poster is indistinguishable from a first-time guess.

**650K+** Active users

**2M+** Posts/year

**0%** Tracked

## Credibility gap

Forums are the least accountable form of investment research. Anyone can post a call, and nothing checks it afterwards.

Figure: A murmuration of identical marks over a skyline, one bird ringed and labelled with a record

The result is an information asymmetry in reverse: a trader who has been consistently accurate for years looks identical to a newcomer making a first wild guess. There is no reputation system, track record or empirical basis for trust.

> Score every extracted prediction against what the price did, and credibility becomes a measured quantity rather than a reputation.

Traditional financial credibility comes from credentials: CFA certifications, institutional affiliations, Bloomberg terminals. Retail investor communities have none of that, and an anonymous username can still move markets.

```text title="Credibility sources: traditional vs retail"
INSTITUTIONAL                    RETAIL FORUMS
─────────────                    ─────────────
CFA Certification                Anonymous Username
Track Record (Audited)           Post History (Unverified)
Regulatory Oversight             Community Moderation
Fiduciary Duty                   No Accountability
High Trust Barrier               Zero Trust Barrier

Gap: no empirical credibility system on the retail side.
```

The gap has practical consequences:

Experienced traders with genuine insight are drowned out by volume. The signal-to-noise ratio approaches zero.
Without accountability, pump-and-dump schemes thrive. Bad actors face no consequences for consistently wrong or misleading predictions.
Retail investors spend hours parsing forum posts with no way to prioritise whose opinions actually matter.

## Extraction pipeline

Forum data goes through a four-stage pipeline that turns unstructured discussion into measurable prediction accuracy.

1. Ingestion — stealth scraping with Playwright. JavaScript rendering. Rate-limited to avoid detection. Thread-level batching for efficiency.
2. Analysis — Perplexity AI extracts sentiment, direction, magnitude estimates, and time horizons from natural language posts.
3. Validation — backtesting against Yahoo Finance data. Multiple time horizons: 7-day, 30-day, 90-day windows.
4. Aggregation — per-user accuracy scores. Ticker-specific performance. Confidence-weighted rankings.

## Sentiment extraction

The AI analysis layer processes each forum thread in a single API call. That saves cost and gives more context, since individual posts are analysed within the flow of the discussion.

```text title="Sample extraction from a HotCopper post"
INPUT (forum post):
  "LKE looking strong here. Chart forming a cup and handle,
   expecting a breakout above 0.85 within the next few weeks.
   Could see 1.20 by end of month if volume picks up."

OUTPUT (structured prediction):
  ticker:       LKE
  sentiment:    BULLISH
  direction:    UP
  entry_price:  0.85
  target_price: 1.20
  time_horizon: MEDIUM_TERM (8-30 days)
  confidence:   0.72
```

Not every post contains a tradeable prediction; many are questions, reactions or general commentary. The AI separates explicit predictions (clear directional calls with price targets), implicit sentiment (general bullish/bearish tone without specific targets) and neutral content (questions, news sharing, or factual discussion).

## Time horizon buckets

A correct 7-day call is a different thing from a correct 90-day thesis, so predictions are validated across three windows:

| Window      | Range      | Predictions/day | Character                                           |
| ----------- | ---------- | --------------- | --------------------------------------------------- |
| Near-term   | 1-7 days   | 342             | Momentum plays, news reactions, technical breakouts |
| Medium-term | 8-30 days  | 187             | Earnings plays, catalyst events, trend following    |
| Long-term   | 31-90 days | 94              | Fundamental thesis, sector rotation, macro calls    |

## Accuracy metrics

Prediction quality is measured on magnitude as well as direction.

```text title="Accuracy decomposition"
Direction Accuracy    Did price move in predicted direction?
                      UP / DOWN / NEUTRAL

Magnitude Accuracy    How close was the % change estimate?
                      |predicted - actual| / actual

Composite Score       Weighted combination with confidence factor
                      Score = (0.6 * direction)
                            + (0.3 * magnitude)
                            + (0.1 * confidence_calibration)
```

A user who calls direction correctly 80% of the time but wildly overestimates magnitude is worth less than one who is accurate on both, and the composite score reflects that.

## Leaderboard

Figure: A histogram shape: a heap of columns at chance and a thin tail to the right of a line, the tail picked out

Across 6 months of HotCopper data, accuracy is not normally distributed. It is heavily skewed.

User accuracy distribution (n=12,847). Most users cluster around 50%, which is chance. The signal is in the right tail.

Most users perform at or below chance, consistent with noise. Roughly 2% of users hold accuracy above 65% across 50+ predictions, which is statistically significant.

## User personas

Prediction behaviour and accuracy sort users into a few archetypes:

High accuracy on 2-3 specific tickers. Deep sector knowledge. Fewer predictions but higher conviction. Average 68% direction accuracy in their domain.
Chart-focused predictions. Strong near-term accuracy, weaker long-term. High volume of predictions. Pattern recognition over fundamentals.
Consistently bullish regardless of conditions. High prediction volume, low accuracy. Often first to post on small-cap announcements. Signal value: negative.

## System architecture

The implementation is built for cost and scale: processing millions of posts through AI analysis needs careful batching and caching.

```text
HotCopper ──[playwright]──→ Posts DB ──[perplexity]──→ Predictions
                                                            │
                                                            ▼
Yahoo Finance ──[yfinance]──→ Price Data ──[backtest]──→ Outcomes
                                                            │
                                                            ▼
                                              User Rankings ──→ Dashboard
```

One Perplexity API call per forum thread, not per post.
Active threads scraped every 6 hours, dormant threads weekly.
Duplicate content detection avoids redundant AI processing.
Long-running scrapes can resume from interruption.

## Console

A Next.js dashboard shows the credibility data in real time. It has an overview with real-time metrics and recent activity, user rankings sorted by composite credibility score, extracted predictions with sentiment analysis and validation status, and an AI operations panel for monitoring the extraction and analysis pipelines.

## Open research questions

- Does historical prediction accuracy predict future accuracy? Is credibility persistent or mean-reverting?
- Do coordinated prediction patterns identify market manipulation?
- What’s the correlation between prediction confidence and actual accuracy across user tiers?
- Can ensemble predictions from top-ranked users outperform individual institutional analysis?

## Future applications

Real-time credibility scores for forum users. Integration with trading platforms to surface high-signal predictions.
Extend beyond HotCopper to Reddit (r/ASX_Bets), Twitter FinTwit, and international equivalents.
Pattern recognition for coordinated pump schemes. Early warning system for retail investor protection.

This research focuses on the Australian Securities Exchange (ASX) via HotCopper. For collaboration inquiries: <research@drksci.com>