Catchup Yardstick answers a deceptively simple question: if your game is currently behind another, when will you catch up? The honest answer requires more than a straight-line projection — Roblox CCU growth is rarely linear, ceilings exist, regimes shift, and every claim needs a confidence interval. We solve this with a 10-layer bivariate ML stack we call Cyct. This article walks through each layer, what it does, and why it's there.
The Problem in One Picture
Pick any two Roblox games — a "chaser" (smaller, growing) and a "leader" (larger, the target). Plot their concurrent-user curves on the same axes. The chaser is catching up if and only if its CCU curve eventually intersects the leader's. When that intersection happens depends on growth rates, ceilings, seasonality, audience overlap, and a dozen other signals. A single regression line will be wrong. A single per-game forecast will be overconfident. The right answer is a probability distribution over time, not a date.
Cyct's goal isn't to give you one number. It's to tell you the probability of catchup at every point on a 30-to-730-day horizon, with calibrated intervals and a list of which features are doing the work.
The Ten Layers
The orchestrator composes ten layers in sequence. Each layer's output feeds the next; each layer can return null without breaking the response. The frontend renders whatever's available — so a brand-new pair with no trained models still gets scenarios + Monte Carlo + caveats, just with a low-confidence flag.
Layer 0 — Feature Engineering
We pull 60–90 days of CCU history for both games, plus side metadata (genre, creator type, gamepass count, audience country distribution). From that we compute the input vector: current gap, gap percentage, velocity (30-day), acceleration (delta-velocity), genre Jaccard overlap, normalised audience overlap, and ~20 other features. Missing values are either imputed from genre medians or marked as missing — the downstream models handle both cases.
Layer 1 — Deterministic Scenarios
Before reaching for ML, we fit three explicit growth curves to the chaser's recent CCU: linear OLS (constant velocity),exponential decay (growth slowing toward a ceiling), andGompertz (S-curve with a saturation plateau). These are cheap to compute, easy to explain, and give the user three intuitive "if-then" trajectories. They're also the fallback when the ML layers have no data.
Layer 2: Per-game CCU Forecasts
Each game gets a forecast of its daily average players with an 80% prediction range. The model is trained across thousands of Roblox games at once, so a young game borrows patterns from games that were in the same situation, and it is served only where it beats a weekday-adjusted baseline in weekly backtests; elsewhere the baseline itself is used. The two forecasts are then subtracted to get the gap forecast.
Layer 3 — Quantile Regression of Catchup Day
Trained on ~3,000 historical (chaser, leader) snapshots where we already know the outcome, this layer predicts directly: given the Layer-0 features at time t, what's the expected number of days until catchup? We use quantile regression at the p10 / p50 / p90 levels so the output is a band, not a point. Training is rotated weekly (Sunday 03:30 UTC) on a separate ML node so it doesn't compete with forecast training.
Layer 4 — Random Survival Forest
Survival analysis flips the question: instead of "when?", we ask "what's the probability of catchup by day T?" — for T in {30, 60, 90}. A Random Survival Forest handles right-censored data natively (pairs where the chaser never caught up during the observation window are information, not noise). The 90-day cap is a hard limit set by Roblox's CCU-history retention. Beyond 90 days, we don't claim to know.
| Horizon | What Layer 4 returns |
|---|---|
| 30 days | P(catchup ≤ 30d), 0.0 – 1.0 |
| 60 days | P(catchup ≤ 60d), 0.0 – 1.0 |
| 90 days | P(catchup ≤ 90d), 0.0 – 1.0 |
| > 90 days | null — outside the training horizon |
Layer 5 — Empirical Analogs (k-NN)
The model's bias is one thing; the evidence is another. Layer 5 searches the historical pair database for the five closest matches to the current chaser/leader pair, measured on Layer-0 feature distance. The UI shows those matches with their actual catchup outcomes, so the user can sanity-check the ML's claim against real precedent. If the model says "60% chance in 45 days" but every analog took 180+ days, you should weight the analogs.
Layer 6 — Regime Detection
A game's recent CCU curve falls into one of five regimes: launch(just released, noisy), growth (sustained upward), plateau(flat), decline (sustained downward), or spike(transient burst). We tag both games' regimes and surface them to the user — a catchup forecast where the leader is in "spike" and the chaser is in "growth" is very different from one where the leader is in "decline" and the chaser is in "plateau". Same numbers, different story.
Layer 7 — Saturation Ceiling
We re-use the Gompertz fit from Layer 1 to estimate each game's asymptotic ceiling. If the chaser is already at >85% of its estimated ceiling, no amount of "growth" is going to close the gap — the ceiling is the gap. The UI flags this with a saturation badge. This is the layer that prevents Catchup from producing absurd "this game will pass Adopt Me in 8 months" claims when the math doesn't support it.
Layer 8 — Monte Carlo Trajectory Composer
We combine the Layer-2 forecasts, the Layer-3 regression band, and a noise model (variance estimated from the recent residuals) into 1,000 trajectory simulations. From those simulations we extract the crossover probability density — the chart the user sees showing "probability of catchup on day X" across the horizon. The Monte Carlo layer is the only one guaranteed to return something on a cold-start, because it can fall back to the Layer-1 deterministic scenarios when forecasts or regression are missing.
Layer 9 — Conformal Calibration
Trained models lie about their confidence. A quantile regression thatsays it covers 80% of outcomes in its p10–p90 band typically covers 60–95% depending on the regime. Conformal prediction is a post-hoc calibration step: we hold out 20% of training data, measure the empirical coverage of the model's intervals, and adjust each new prediction's width until the historical coverage matches the claim. After Layer 9, when Catchup says "80% interval", it means it.
Layer 10 — Explainability + Caveats
The final layer assembles three things: feature drivers(which Layer-0 features had the highest contribution to this specific prediction, via SHAP-style attribution), caveats (a list of data-quality warnings — short history, regime mismatch, saturation, audience-overlap mismatch, etc.), and a confidence flag (high / medium / low based on caveat count, coverage of each layer, and analog match quality). The UI's "Caveats" block reads directly from this layer.
Cold-Start Behavior
The first time someone queries a pair, no trained models may exist for that genre, or the pair may be too new to have history. The orchestrator is designed to degrade gracefully:
- Layer 0 always runs (features are derived, not learned).
- Layer 1 always runs (curve fits don't need training data).
- Layer 8 always runs (it can fall back to Layer 1 outputs).
- Layers 2, 3, 4, 5, 9, 10 may return null — the response carries a
cold_start: truemetadata flag, and Layer 10 surfacesconfidence: 'low'.
This means a brand-new pair still gets a usable answer (scenarios + Monte Carlo from deterministic fits + an explicit caveat that the ML layers weren't available). You're never staring at an empty page.
How the Training Pipeline Stays Fresh
Cyct retrains weekly on Sunday at 03:30 UTC. The schedule is fixed off peak so it doesn't compete with the daily aggregation jobs. The training dataset is rebuilt from scratch each run: we take the top 200 games by CCU, generate ~3,000 (chaser, leader) snapshot pairs sampled within-genre by Jaccard overlap, and label each row with "did chaser catch leader within 60 days, sustained for at least N days?" The 60-day forward window is the longest we can construct from Roblox's 90-day CCU retention.
Per-layer retraining tasks also exist (regression, survival, analog index, conformal calibration) so we can surgically refresh individual layers without rebuilding the whole pipeline — useful if one layer's residuals drift before the others.
What This Means for You
When you open Catchup Yardstick, you're seeing the output of all ten layers composed. The chart isn't a forecast — it's a probability distribution over time. The "expected catchup day" number isn't a promise — it's the p50 of the survival curve with conformal-calibrated bounds. The analogs aren't decoration — they let you sanity-check the model against real precedent. And the caveats aren't hidden — Layer 10 surfaces them at the bottom of the page with plain-English explanations.
Use the deterministic scenarios when you want to reason about "if growth stays linear" or "if the leader plateaus". Use the ML estimate when you want the empirical answer. Use the analogs when the ML number feels too aggressive or too conservative. Use the caveats whenever you're about to share a number externally.
Honest Limitations
Cyct is a model. Models are wrong. Three real limits worth knowing:
- 90-day cap on survival predictions. Roblox only exposes ~90 days of CCU history, so Layer 4 can't learn beyond that horizon. Long-range forecasts fall back to Layer 8 Monte Carlo, which inherits all the uncertainty of the underlying scenarios.
- Genre-mismatch pairs underperform. A shooter vs. a tycoon shares almost no audience structure. The model can still forecast crossover, but Layer 5 analogs will be weaker and Layer 10 will flag low confidence.
- Spike events are not predicted. If a creator drops a viral video tomorrow, the curve will move. No model captures exogenous shocks. Catchup is a "given current dynamics" tool — pair it with your own judgment about what's likely to change.
Try it on your own games at /intelligence/catchup — pick a chaser, pick a leader, and read what the ten layers have to say.