Docs

There isn't much to learn. That's on purpose.

Quick start

Create an account. Your project comes with its keys ready, and the setup page shows you this code with them already filled in.

<script src="https://harrowsoft.com/thermostat/sdk.js"></script>
<script>
  Thermostat.start('client-your-key', { userID: 'the-signed-in-user-id' }).then((client) => {
    // Is this on for them?
    if (client.flag('new_search')) { showNewSearch(); }

    // Which version of the test are they in?
    const steps = client.experiment('checkout').get('steps', 3);

    // Record what they do.
    client.track('purchase', 24.5);
  });
</script>

That's the whole integration. Events appear on the Events page the moment they arrive.

The five things

Thing	What it is
Experiments	A guess, some variants, one number you're trying to move, and guardrails for what shouldn't get worse. Ends in a verdict.
Flags	On, off, or a set of values, for whoever you choose. Rules run top to bottom, and the first match decides.
Metrics	A number built from an event: share of users who did it, times per user, total per user, average value, or one event per another.
Audiences	A saved group, built from rules or an uploaded list of IDs.
Events	What your app sends. Everything else is computed from these.

Layers (experiments that must not share users) and holdouts (a group kept back from many changes) are there when you need them, and stay out of the way when you don't.

JavaScript SDK

One small file with no dependencies. It works in browsers and in Node 18 or later. If the network fails, it uses the last values it saw, and failing that, your fallbacks. Your app never breaks because of us.

Call	What it does
`Thermostat.start(key, user, options)`	Loads this user's values. Always resolves. Options: `environment`, `timeoutMs`, `api`.
`client.flag(name, fallback)`	True if the flag is on for them.
`client.config(name).get(key, fallback)`	A value from a config flag.
`client.experiment(name).get(key, fallback)`	A value from their variant. They count as exposed from the first value you read. `.variant` is the variant's name.
`client.track(name, value, properties)`	Records something they did. Value and properties are optional.
`client.setUser(user)`	Changes who they are, for example after signing in.
`client.flush()`	Sends anything queued. Happens by itself every few seconds and when the page closes.

A user can have userID, email, country, locale, appVersion, custom for your own attributes, and customIDs for other things you test by, such as a company. You only need one ID. Don't send personal details you don't need; an opaque ID is enough.

Which key goes where

`client-`	Safe in a website or app. Reads this user's values and sends events.
`secret-`	For servers. Keep it secret. Downloads all the rules.
`console-`	For scripts. Keep it secret. Can do anything you can do in the app.

Any other language

Everything is one HTTP request away.

# What does this user get? Send "hash": "none" for readable names.
curl https://harrowsoft.com/thermostat/v1/initialize \
  -H "statsig-api-key: client-your-key" \
  -d '{"user": {"userID": "user-1"}, "hash": "none"}'

# Just one flag or experiment.
curl https://harrowsoft.com/thermostat/v1/check_gate \
  -H "statsig-api-key: secret-your-key" \
  -d '{"user": {"userID": "user-1"}, "gateName": "new_search"}'

# Record something.
curl https://harrowsoft.com/thermostat/v1/log_event \
  -H "statsig-api-key: secret-your-key" \
  -d '{"events": [{"eventName": "purchase", "value": 24.5, "user": {"userID": "user-1"}}]}'

If you log exposures yourself, send a statsig::config_exposure event with metadata: {"config": "checkout", "ruleID": "..."}, using the rule_id you were given. Statsig's own server SDKs work too, as the next section explains.

Move from Statsig

Thermostat speaks the same protocol as Statsig's SDKs, so your code stays as it is.

1. Bring your project over. In Thermostat, go to Settings, then Move from Statsig. Paste a Console API key, plus a server secret key if you can. We use them for that one read and never store them. You'll see how each item will come across before anything is written.

2. Point your SDK at Thermostat. For most SDKs, that's one line.

// Browser
new StatsigClient('client-your-thermostat-key', user, {
  networkConfig: { api: 'https://harrowsoft.com/thermostat/v1' },
});

// Node
await Statsig.initialize('secret-your-thermostat-key', {
  api: 'https://harrowsoft.com/thermostat/v1',
});

# Python
statsig.initialize('secret-your-thermostat-key', StatsigOptions(
    api='https://harrowsoft.com/thermostat/v1',
))

Statsig's newer server core SDKs, such as @statsig/statsig-node-core and statsig-python-core, take three addresses instead of one:

new Statsig('secret-your-thermostat-key', {
  specsUrl: 'https://harrowsoft.com/thermostat/v2/download_config_specs',
  logEventUrl: 'https://harrowsoft.com/thermostat/v1/log_event',
  idListsUrl: 'https://harrowsoft.com/thermostat/v1/get_id_lists',
  enableIdLists: true,
});

We test the browser SDK and both kinds of server SDK in Node and Python, unmodified, before every release. Statsig's SDKs for other languages use the same protocol, but we haven't tested each one yet, so try yours on a staging server first.

3. Roll it out like any deploy, and leave Statsig running until you're happy. You can import again as often as you like; it updates what it brought, and leaves alone anything you made here.

What comes across

In Statsig	Here
Feature gates, dynamic configs, parameter stores	Flags. Overrides come too, and now end after 90 days.
Experiments	Experiments, with their salts, groups and metrics, plus Statsig's last results for history.
Segments	Audiences, including ID lists.
Metrics	Metrics. Funnels and composite metrics need rebuilding, and the report lists them.
Layers and holdouts	Layers and holdouts.
Autotunes	Regular tests, not started. Autopilot ramps on a fixed plan rather than shifting traffic toward the leader.
Audit log	Your history, marked as coming from Statsig.

Keeping users in their variants

With a server key, we bring Statsig's exact compiled rules and salts, and our evaluator uses the same hashing, so every user lands where they did before. We check this by running Statsig's real SDKs against Thermostat and comparing them with our own answers, check by check. Without a server key, anything at full traffic still lands the same way, but partial rollouts and layers may reshuffle, and the import report says which.

Past raw events stay with Statsig, because its API doesn't export them. New results start from your switch-over, and Statsig's last results stay on each experiment. You can also upload past events yourself from the Events page.

The AI

The AI isn't switched on yet. We're connecting it now, so this describes how it will work. Autopilot works today, because it doesn't need the AI.

Press Ctrl K (or ⌘K) anywhere. It reads your experiments, metrics and events through the same tools as the app, and can query your raw data.

Drafts happen straight away, because they affect nobody.
Anything live (starting a test, changing traffic, shipping, flipping a flag, emailing the team) waits for your approval. A project can choose to let it act without asking.
Autopilot is a setting on an experiment, not the AI thinking on its feet. It's a fixed plan: raise traffic through set steps while everything looks healthy, and pause the moment a guardrail is harmed or the traffic split breaks.
Programs are standing goals, such as "keep testing the checkout until purchases are up 10%". It runs one test after another and emails you after each step. Starting one always needs a person's yes.

Everything it does is in the audit log under its own name, and it can only do what the signed-in person could. AI features send the question and the project data needed to answer it to Anthropic. They don't run unless you use them or turn on automation for something.

API

Everything the app does goes through a JSON API, and the same routes accept a console- key, so anything can be scripted.

curl https://harrowsoft.com/thermostat/api/p/PROJECT_ID/experiments \
  -H "Authorization: Bearer console-your-key"

The main ones are /experiments, /flags, /metrics, /audiences, /overrides, /events, /why and /query. Experiments also have /start, /pause, /resume, /traffic and /decide.

How the numbers work

Nothing here is secret. Click any row of a scorecard to see its inputs, and download the raw assignments if you'd like to check it yourself.

The comparison

Each user gets one value per metric. We compare each variant's mean with control's, using each group's own variance (Welch). Percentages are that difference as a share of control. Ratio metrics such as clicks per view use the delta method on per-user numerators and denominators.

Reading results early

Check a normal test every day and you'll find a false winner far more often than the 5% it promises. So a test here splits its 5% between two boundaries. It counts as clear when it crosses either one, and looking doesn't cost anything.

One built around the test's planned length, with 4% of the budget. With t the share of the plan so far and SE the standard error today, the range's half-width is

z(1 - α/4) · SE / √t

It starts wide and reaches about 2.3 standard errors on the last planned day, close to the 1.96 of a test you may only read once. By the reflection principle, the chance of ever crossing it when nothing has changed is at most α, however often you look. If people are arriving slowly, t follows the share of people expected by the end rather than the calendar, because most of the evidence is still to come.

One that holds forever, with the other 1%: a mixture sequential test, with variance V and a typical effect τ of 5% of the metric's mean.

sqrt( V(V+τ²)/τ² · ( 2·ln(1/α) + ln((V+τ²)/V) ) )

This one can stop a test early when the effect is huge, and it's the only one left once a test passes its planned end. From then on the numbers stay as they were on the last planned day, unless the new evidence is overwhelming. Leaving a test running until something turns up doesn't work here.

Decisions are made on the metric's own scale, not on the percentage. A percentage divides by control, which can look far more certain than it is when control happens to include one very large value.

Less noise

What users did in the 14 days before they entered a test explains a lot of what they do during it. We remove that with CUPED, using a coefficient pooled across all variants. Extreme values are capped at the 99.9th percentile so one big account can't decide a test.

Several comparisons

The primary metric's alpha is divided by the number of variants and primary metrics. Guardrails use a looser alpha, because missing harm costs more than a false alarm. Secondary metrics get a Benjamini-Hochberg check, and we warn you when one only looks clear on its own.

Can you trust it?

Traffic split: a chi-square test of observed users against the planned split. Below p = 0.001, results are withheld until it's fixed, because something is dropping or misassigning users.
Groups started equal: we compare the groups' behaviour before the test began.
Bad traffic: known bots, and users who saw more than one variant, are left out. The count is shown.

How we check all this

We simulate it. In each case below nothing really changed, across 2,000 simulated tests looked at 28 times each. The table shows how often a false winner appeared, against the 5% we promise. A shorter version of this simulation is part of the test suite that runs before every release.

What was measured	Thermostat	Ordinary test, same looks
Conversion of 10%, 20,000 people	2.9%	27.1%
Conversion of 0.2%, 20,000 people	2.9%	23.7%
Conversion of 10%, 500 people	3.4%	26.5%
Conversion of 10%, split 90/10	2.9%	29.0%
Conversion of 10%, most people arriving late	3.1%	40.9%
Revenue with very uneven amounts, 2,000 people	1.4%	17.4%
Revenue with very uneven amounts, 20,000 people	2.6%	24.4%
Events per person, 20,000 people	2.9%	29.3%
Events per session, 20,000 people	3.6%	27.7%
Conversion of 10%, looked at 2,000 times	4.0%	53.0%

Being able to look costs little. To have an 80% chance of finding a 10% lift on a 10% conversion rate, a test you may only read once needs about 14,100 people in each group. Here it's about 18,100, and you can read it whenever you like.

We also ran the whole product, verdicts and all, on 300 tests where nothing changed, each read daily for two weeks with 6,000 people. It called a winner 0.3% of the time, a loser 1.0%, and raised a false alarm on a guardrail 3.3%.

The verdict

A variant wins when the primary metric is clearly better, no guardrail is clearly harmed, and the test has covered at least a full week. Harm is called straight away. "No real difference" means the whole range fits inside the smallest change you said would matter.

Your data

Your data is yours. From Settings you can download your whole setup, every raw event and every assignment, in JSON. You can also delete a project or your account there. See the privacy notice.

To erase one person, look them up under Why? and choose Erase this person, or call POST /thermostat/api/p/<project>/people/forget with {"id": "..."} and an API key. Their events, assignments and ID are removed within a minute. Totals in results you've already seen update the next time results refresh.

For rules on country, we look up a browser's country from its IP address and don't store the address. This site or product includes IP2Location LITE data available from https://lite.ip2location.com.

Limits during early access

A project can send up to 20 million events a month and keep up to 10 GB of raw data. Past the monthly limit, flags and experiments keep working and your app is unaffected, but new events aren't stored until the 1st, and we'll tell you. When storage nears 10 GB, the oldest raw events are removed to make room. Results of finished tests are kept, and nothing a running test needs is touched. Settings shows what you've used.