Calibration

How a Challenge's bar is placed from real play, and how Challenge keeps it honest once money is on it.

Why the bar is measured

A threshold is not a taste decision. The prize is fixed and Challenge funds it, so the bar sets the pass rate, the pass rate sets the expected payout per attempt, and a bar guessed a little too low turns every attempt into a loss that looks correctly configured. Break-even at the platform's standard fee ratio is a 12.5% pass rate. The default target is 10%.

So a Challenge earns its threshold. It is created free, with no bar, and it cannot charge until one has been placed from a sample big enough to trust.

There are three ways that sample can arrive.

The three paths

All three place the bar with the same arithmetic and the same hardening. What differs is who produced the numbers, and that difference is priced by the trust treatment rather than by the maths.

Observed

The cold-start path. The Challenge is calibrating, players attempt it for free, and when it has collected 1,000 attempts from 200 distinct players the sweep places the bar and takes it live.

Two minimums, because they are bought with different currencies. Attempts are what an attacker can manufacture; players are what they have to recruit. A percentile taken over three people is not a measurement of a population, whatever its sample size says.

Three further rules make a free sample hard to poison:

• One score per player. The bar sits on each player's best result, not on the pool of every attempt. Against a pooled sample, playing badly on purpose buys a softer bar. Against per-player bests it buys nothing: a single good run, even an accidental one, replaces every deliberate miss that player has.
• A 1% share cap. No player may contribute more than 1% of the counted sample, with a floor of 3. Past the cap, extra bad attempts change nothing at all, so the effort curve goes flat.
• Determinism. The kept subset is each player's most recent scores, in a fixed order, so two calibration runs over the same data produce the same bar.

Placing the bar on bests deliberately errs hard rather than soft. A bar that is too hard costs players; a bar that is too soft costs money; only one of those is recoverable.

Both minimums have per-Challenge overrides (min_calibration_n, min_calibration_players). They are platform and operations settings, not partner dials.

Listen-first

The primary path, and the reason Challenge.record() ships before anything chargeable.

Every ordinary round is recorded against the metric, silently: nothing rendered, no session required, no Challenge involved. When a metric has 5,000 rounds from 200 distinct player refs, a Challenge created on that metric is given its bar from them and is live in the same request that created it.

The point is that no player ever meets a calibrating Challenge. The first person to open your game is otherwise shown something that cannot pay, and asked to keep playing so somebody else can win later.

Details worth knowing:

• 5,000 rather than 1,000 because the evidence is weaker. An observed attempt was paid for and adjudicated by Challenge; a listened round is a number your client sent. Five times the sample does not make an unverified number verified, but it does take the distribution out of doubt.
• One player ref may write at most 300 scores per metric in a rolling 24 hours. Over the cap the call still answers success and the score is dropped: a listener that can fail is a listener a partner eventually removes.
• Listened rounds place a bar once, at creation, and only at creation. They are never merged into a later recalibration, because listened rows are keyed on your player_ref and cannot always be attributed to an account, and a recalibration that cannot remove a flagged player's contribution is worse than one with less data.
• If the targets are not met, nothing happens. The Challenge is created calibrating and collects its sample the observed way.

The Developer Portal reports listening progress per metric, and beside it the metric key each of your Challenges is waiting on. That mismatch, between the string your game sends and the string your Challenge listens for, is the one failure mode of this path and it is invisible without that view.

Partner-seeded

Runs on your server. You already have the history and would rather not wait for either sample.

Five percentiles and a sample size. Not rows: that is all a threshold needs, it carries no personal data, and it comes out of one query on your side. The threshold above is the 10th percentile of a lower_is_better metric, interpolated between the p5 and p25 anchors, because letting 10% through a bar that rewards smaller numbers means sitting at the 10th percentile. On a higher_is_better metric the same 10% target sits at the 90th.

The Challenge is live when this returns.

The rules it holds to:

• sample_size must be a whole number of at least 500. A tail estimate from thirty scores is noise dressed as data, and the observed path demands 1,000 adjudicated attempts before it trusts a bar.
• p5 through p95 must ascend. Percentiles that do not ascend are not percentiles.
• The Challenge must be calibrating. A live one is refused because reseeding would move the bar under players mid-flight; a suspended one is refused because a new distribution is not a way to reset the breaker that pulled it.
• A rung cannot be seeded at all. Its bar is measured from the players who claimed the Challenge below it. See The Challenge ladder.
• The bar is interpolated between the anchors you gave, never extrapolated past them. A target pass rate that would need a percentile outside p5 to p95 is refused, with the achievable rate named, rather than silently clamped to the edge anchor.
• Idempotency-Key is required. Seeding takes the Challenge live, so a partner whose request succeeded but whose response was lost has to be able to retry and learn what actually happened.

Provisional, or ramped, never both

Two mechanisms bound the cost of a bar that turns out to be wrong. Which one applies is decided by calibration_source, and no Challenge is ever subject to both.

Provisional answers a doubt about honesty. Neither a listened round nor a supplied percentile was paid for or adjudicated by Challenge, so neither says what the realised pass rate will be once real money is on it. Volume does not close that gap: a listened sample five times the observed floor is still five times as much unverified data. A prize over the provisional ceiling answers challenge_not_available and names the limit.

The ramp answers a doubt about price, not honesty. An observed bar was placed on free play by people with no reason to try, and then met by people who paid a fee, which is a different population. So the first 200 paid attempts run at half the prize while the drift monitor gathers enough results to tell a true 10% pass rate from a 20% one. Attempt 201 is the first at the full prize, and the change is automatic.

The top-up. A win during the ramp is not locked at the half. Once the ramp completes and the drift monitor confirms the bar held, every ramp-window winner is paid the other half of the configured prize automatically, through the same ledger their original prize moved through. If the bar fails instead, the Challenge is suspended and no top-up is paid: the halved window is exactly the exposure the ramp exists to cap. Player-facing surfaces may therefore state the full prize beside the ramped one, phrased as conditional on the bar holding.

During a ramp, the prize field is the reduced amount and ramp.fullPrize is what it will become. Always show prize as the amount on offer now. Stacking the two mechanisms would price a Challenge at a tenth, which is why a source is on one list or the other.

The drift monitor

Calibration is not a one-time event, because populations change. A Challenge calibrated on early adopters meets a broader audience; you patch your game and everyone's scores jump. The realised pass rate moves, and because the prize is fixed, the expected cost per attempt moves with it.

Every live Challenge is checked against its last 500 settled outcomes. Below 30 outcomes the pass rate is noise and no judgement is made.

Above it, two questions are asked separately:

• Fairness. Is the realised pass rate within the band around the target? The band is 5 percentage points by default, so a 10% Challenge is judged against 5% to 15%.
• Solvency. Does the expected payout, the realised pass rate times the prize, exceed the fee? A Challenge can sit inside its band and still lose money.

Suspension

Fail either and the Challenge is set to suspended, immediately, and challenge.suspended fires with the realised rate, the target and the reason. New attempts answer challenge_not_available.

The response is suspension rather than an alert because an alert needs someone awake and a circuit breaker does not. It is deliberately not a silent threshold change: recalibrating on the spot would move the bar under players mid-flight. Attempts already open keep the threshold frozen at their creation, so no result changes retroactively.

A bar that passes far more players than it was priced for is also the symptom of a sandbagged calibration, so suspension triggers one sandbag pass: players whose free scores sit far below their paid ones are flagged, dropped from the sample whole rather than trimmed, and the bar is recomputed without them. That recalibration does not reinstate anything. The Challenge stays suspended until a human disposes of the case, and no challenge.calibrated event fires, because none went live.

Promotion

Inside the band and solvent, an unconfirmed source is promoted to observed.

That is the half that makes the provisional paths worth using: a partner who supplied an accurate distribution, or let Challenge listen to 5,000 honest rounds, would otherwise be permanently capped at a fifth of their tier for having been helpful. On promotion the provisional ceiling lifts on its own, with no request to make and no ticket to open.

A rung_derived bar is promoted on the same evidence, though it had no ceiling to lift. What it gains is the record that its halved target survived contact with a real field.

Events

Related

• Challenges: the status machine and what each status means for your integration
• The Challenge ladder: how a rung's bar is measured and priced
• Programs and keys: tier ceilings, and what the provisional cap is a fraction of
• Sandbox: running a Challenge through calibration without money

---

Source: https://www.withchallenge.com/docs/calibration.md Docs index for AI agents: https://www.withchallenge.com/llms.txt