Tests report two kinds of results: lift metrics, which measure how much a piece of tested content moved a metric compared with the control, and direct reaction metrics, which measure how respondents reacted to the content itself. The API currently returns lift metrics only; direct reaction metrics are coming. This page explains how to read results correctly, and what the API does and doesn’t include.
Tests, content and controls
A test is one Swayable study. In a typical test, respondents are randomly assigned to a treatment arm: they see one asset (a piece of content, such as an ad or message) or the control (a placebo with no message). Because the arms are randomized, the difference between them measures the asset’s effect.
Each result covers one metric (such as Brand Love or Purchase Consideration), one asset, and everyone in the test.
Two kinds of metrics
| Lift metrics | Direct reaction metrics | |
|---|---|---|
| What they measure | The asset’s effect on attitudes or intentions, such as brand love, favorability or purchase consideration | How respondents reacted to the asset itself, such as how happy, inspired or informed it made them feel, or whether they found it relevant or worth sharing |
| Compared with the control? | Yes. The result is the asset’s arm minus the control arm. | No. The result is the score in the asset’s treatment arm only. |
| In the API today? | Yes | Coming soon |
A reaction score on its own doesn’t show persuasion. A high “Inspired” score means respondents said the asset inspired them, not that it changed what they think. Use lift metrics to judge effect, and reaction metrics to understand why an asset did or didn’t work.
Confidence intervals
Every result carries a 95% confidence interval, ci95_lo to ci95_hi.
| The interval is… | Read it as… |
|---|---|
| Entirely above zero | A positive effect, significant at about the 95% level |
| Entirely below zero | A negative effect, significant at about the 95% level |
| Spanning zero | Inconclusive. The test couldn’t distinguish the effect from zero. This is not evidence of no effect. |
This table applies to lift metrics. For direct reaction metrics, the interval is the plausible range for the asset’s own score, and there’s no zero line to test against. To compare two assets on a reaction metric, look at whether their intervals overlap: if they overlap substantially, the difference between them may not be meaningful.
Each interval stands alone and isn’t adjusted for the number of comparisons. When you compare many assets, metrics and segments, about 1 in 20 results can look significant by chance. Treat a single significant result among many as a lead to confirm, not a conclusion.
Segments
Today, results cover everyone in the test; each label ends in “(overall)”. Results for the audience segments configured on your test are coming. Segment results are smaller samples than overall results, so their intervals will usually be wider.
Small segments (coming with segment results). Results for groups of 20 or fewer respondents will still be shown, flagged as modeled results. Swayable’s model produces an estimate even for small groups, but treat these results as directional only.
Trackers
Trackers measure the same metrics repeatedly over time. Tracker results over time are coming to the API.
Availability and empty results
Each test reports whether its results are available. Tests that are draft, template, withheld or no_comparison return an empty results list. An empty list means no results are available, not a measured zero.
How current the data is
Swayable publishes your tests and their reviewed results to the API. Automatic updates are coming: new tests will appear on their own, with results usually available within 1 day of a test being finalized. The API returns results as they stand; it doesn’t start new analyses.
The preview
Some tests include a preview: up to 20 fixed example entries that show the kinds of fields the test collected. Every call returns the same entries. The preview is illustrative only. Never use it to estimate population results.
What’s not included
- Respondent-level data, respondent IDs or model scores
- Custom audience cuts
- Other organizations’ tests
Example prompts
Stakeholder update
List my available tests, ask me which to summarize, then draft a 300-word stakeholder update. Include the strongest findings within each test with their 95% intervals, and separate reported results from your interpretation.
Compare assets
For the test I choose, compare the assets within each metric in a compact table: asset, lift, unit and 95% interval. Call a lift significant only if its interval excludes zero.
Inspect the preview
Show the preview for this test and explain each field. Make clear that it describes these example entries only, not the full test.