Labs
Starling
Market Credibility Intelligence
- Status
- Alpha
- Stack
- FastAPI, Next.js, PostgreSQL, Playwright, Perplexity AI, Yahoo Finance
1,200 predictions a day, none of them scored.
Australia’s largest retail investor forum: 650,000 users, over two million posts a year, and no record of whose calls came good. A consistently accurate poster is indistinguishable from a first-time guess.
Credibility gap
Forums are the least accountable form of investment research. Anyone can post a call, and nothing checks it afterwards.
The result is an information asymmetry in reverse: a trader who has been consistently accurate for years looks identical to a newcomer making a first wild guess. There is no reputation system, track record or empirical basis for trust.
Score every extracted prediction against what the price did, and credibility becomes a measured quantity rather than a reputation.
Traditional financial credibility comes from credentials: CFA certifications, institutional affiliations, Bloomberg terminals. Retail investor communities have none of that, and an anonymous username can still move markets.
INSTITUTIONAL RETAIL FORUMS
───────────── ─────────────
CFA Certification Anonymous Username
Track Record (Audited) Post History (Unverified)
Regulatory Oversight Community Moderation
Fiduciary Duty No Accountability
High Trust Barrier Zero Trust Barrier
Gap: no empirical credibility system on the retail side.The gap has practical consequences:
- Information asymmetry
- Experienced traders with genuine insight are drowned out by volume. The signal-to-noise ratio approaches zero.
- Manipulation vulnerability
- Without accountability, pump-and-dump schemes thrive. Bad actors face no consequences for consistently wrong or misleading predictions.
- Wasted attention
- Retail investors spend hours parsing forum posts with no way to prioritise whose opinions actually matter.
Extraction pipeline
Forum data goes through a four-stage pipeline that turns unstructured discussion into measurable prediction accuracy.
- Ingestion — stealth scraping with Playwright. JavaScript rendering. Rate-limited to avoid detection. Thread-level batching for efficiency.
- Analysis — Perplexity AI extracts sentiment, direction, magnitude estimates, and time horizons from natural language posts.
- Validation — backtesting against Yahoo Finance data. Multiple time horizons: 7-day, 30-day, 90-day windows.
- Aggregation — per-user accuracy scores. Ticker-specific performance. Confidence-weighted rankings.
Sentiment extraction
The AI analysis layer processes each forum thread in a single API call. That saves cost and gives more context, since individual posts are analysed within the flow of the discussion.
INPUT (forum post):
"LKE looking strong here. Chart forming a cup and handle,
expecting a breakout above 0.85 within the next few weeks.
Could see 1.20 by end of month if volume picks up."
OUTPUT (structured prediction):
ticker: LKE
sentiment: BULLISH
direction: UP
entry_price: 0.85
target_price: 1.20
time_horizon: MEDIUM_TERM (8-30 days)
confidence: 0.72Not every post contains a tradeable prediction; many are questions, reactions or general commentary. The AI separates explicit predictions (clear directional calls with price targets), implicit sentiment (general bullish/bearish tone without specific targets) and neutral content (questions, news sharing, or factual discussion).
Time horizon buckets
A correct 7-day call is a different thing from a correct 90-day thesis, so predictions are validated across three windows:
| Window | Range | Predictions/day | Character |
|---|---|---|---|
| Near-term | 1-7 days | 342 | Momentum plays, news reactions, technical breakouts |
| Medium-term | 8-30 days | 187 | Earnings plays, catalyst events, trend following |
| Long-term | 31-90 days | 94 | Fundamental thesis, sector rotation, macro calls |
Accuracy metrics
Prediction quality is measured on magnitude as well as direction.
Direction Accuracy Did price move in predicted direction?
UP / DOWN / NEUTRAL
Magnitude Accuracy How close was the % change estimate?
|predicted - actual| / actual
Composite Score Weighted combination with confidence factor
Score = (0.6 * direction)
+ (0.3 * magnitude)
+ (0.1 * confidence_calibration)A user who calls direction correctly 80% of the time but wildly overestimates magnitude is worth less than one who is accurate on both, and the composite score reflects that.
Across 6 months of HotCopper data, accuracy is not normally distributed. It is heavily skewed.
Most users perform at or below chance, consistent with noise. Roughly 2% of users hold accuracy above 65% across 50+ predictions, which is statistically significant.
User personas
Prediction behaviour and accuracy sort users into a few archetypes:
- Specialist
- High accuracy on 2-3 specific tickers. Deep sector knowledge. Fewer predictions but higher conviction. Average 68% direction accuracy in their domain.
- Technician
- Chart-focused predictions. Strong near-term accuracy, weaker long-term. High volume of predictions. Pattern recognition over fundamentals.
- Promoter
- Consistently bullish regardless of conditions. High prediction volume, low accuracy. Often first to post on small-cap announcements. Signal value: negative.
System architecture
The implementation is built for cost and scale: processing millions of posts through AI analysis needs careful batching and caching.
HotCopper ──[playwright]──→ Posts DB ──[perplexity]──→ Predictions
│
▼
Yahoo Finance ──[yfinance]──→ Price Data ──[backtest]──→ Outcomes
│
▼
User Rankings ──→ Dashboard- Thread-level batching
- One Perplexity API call per forum thread, not per post.
- Activity-based scheduling
- Active threads scraped every 6 hours, dormant threads weekly.
- Response caching
- Duplicate content detection avoids redundant AI processing.
- Resumable checkpoints
- Long-running scrapes can resume from interruption.
Console
A Next.js dashboard shows the credibility data in real time. It has an overview with real-time metrics and recent activity, user rankings sorted by composite credibility score, extracted predictions with sentiment analysis and validation status, and an AI operations panel for monitoring the extraction and analysis pipelines.
Open research questions
- Does historical prediction accuracy predict future accuracy? Is credibility persistent or mean-reverting?
- Do coordinated prediction patterns identify market manipulation?
- What’s the correlation between prediction confidence and actual accuracy across user tiers?
- Can ensemble predictions from top-ranked users outperform individual institutional analysis?
- Credibility API
- Real-time credibility scores for forum users. Integration with trading platforms to surface high-signal predictions.
- Multi-platform expansion
- Extend beyond HotCopper to Reddit (r/ASX_Bets), Twitter FinTwit, and international equivalents.
- Manipulation detection
- Pattern recognition for coordinated pump schemes. Early warning system for retail investor protection.
This research focuses on the Australian Securities Exchange (ASX) via HotCopper. For collaboration inquiries: research@drksci.com