How to backtest earnings-call signals with StockQL
Start with an audited transcript split, a fixed point-in-time panel, and post-reaction returns. Then let uncertainty—not an attractive story—decide whether a signal survives.
To backtest earnings-call sentiment, first lock the universe and call identity; separate prepared remarks from questions and answers; compute each feature from information available at the call; skip the immediate market reaction; and estimate uncertainty across both time and issuers. The language model or dictionary is only one replaceable part.
We built that research rail around StockQL and tested four plausible features: quarter-over-quarter finance tone, explicit references back to prepared remarks, semantic reuse of prepared remarks, and answer evasion. None earned promotion. That outcome is useful. A reusable pipeline should make a clean “stop” decision as easy to publish as a promising result.
1. Freeze the panel before reading the result
StockQL held 173,795 earnings-call transcripts spanning 2005–2026 in the research snapshot. The archive is the starting material, not the backtest universe. A historical test still needs explicit eligibility, identity, timing, and outcome rules.
The fixed base tone/return panel contained 17,641 adjacent-call pairs from 1,844 tickers across 14 call quarters. It used normalized U.S. common shares, retained inactive names, and required a pre-call 60-session median dollar volume of at least $25 million, a last pre-call close of at least $5, complete forward prices, and immediately adjacent fiscal quarters 45–150 days apart. Feature-specific exclusions then left 17,570 calls for ACR and 17,285 events for evasion; those sample sizes are not interchangeable.
| Layer | Locked rule | Failure it prevents |
|---|---|---|
| Call identity | Fiscal identity, verbatim call date, transcript ID, and content hash | Ticker reuse or revised text silently changing a row |
| Universe | Point-in-time price and liquidity guards; no current-active filter | Survivorship and present-day eligibility leaking backward |
| Pairing | Immediate fiscal-quarter neighbors with a bounded day gap | Comparing unrelated calls because a quarter is missing |
| Availability | Features use call-time text; forward prices enter only as outcomes | The future influencing feature construction |
A ticker by itself is not stable transcript identity. Acquisitions, reused symbols, differing fiscal labels, and provider corrections all make that shortcut unsafe. The separate guide A ticker is not an earnings transcript’s identity covers those lineage checks.
StockQL’s contribution here is composition: transcripts, security identity, exchange-normalized prices, and SPY sessions are queryable in one PostgreSQL database. The research layer still owns the hypothesis and its exclusions. Keeping those jobs separate makes a new signal cheaper to test without pretending the database chose the strategy.
The fixed research panels are local, ignored artifacts—not a downloadable StockQL dataset or a hosted turnkey rail. An entitled self-hosted StockQL user can reproduce or adapt the method against the transcript corpus their data license permits.
2. Audit prepared remarks and Q&A instead of guessing
Prepared remarks and analyst Q&A answer different research questions. Tone in a scripted opening is not interchangeable with language produced under questioning, and a similarity feature needs a defensible boundary between the two.
The source transcripts did not carry a reliable section field, so the rail used a deterministic character-offset splitter and audited it on a fixed 943-call sample. It produced 886 usable splits (94.0%), identified three legitimate calls with no Q&A, and quarantined 54 unresolved calls. Ambiguity became an exclusion, not a model-generated guess.
That gate runs before feature scoring. The base panel admits only ok current and prior splits and records each marker tier and transcript hash; downstream feature stages verify the inherited panel SHA, so a parser or source change cannot silently rewrite the sample. A new signal can require both sections, Q&A only, or prepared remarks only without rebuilding the boundary logic.
3. Measure returns after the call-day reaction
The outcome was a stock’s cumulative close-to-close return divided by SPY’s cumulative return, minus one:
abnormal return = (stock_end / stock_start)
/ (SPY_end / SPY_start) - 1The start was the first SPY session strictly after the call date. Endpoints were sessions 20, 40, and 60. In the primary 60-session test, call day and the first post-call close did not enter the return. This is the common +2-to-+20/40/60 research shape: ask about drift after the immediate reaction rather than relabeling the announcement move as prediction.
Continuous features were ranked into equal-weight quintiles inside each call calendar quarter; explicit ACR compared calls with any reference against calls without one. The report then showed three views of uncertainty:
- the observation-level top-minus-bottom spread;
- the mean of 14 quarterly spreads with Newey–West lag 1; and
- a 2,000-draw ticker-cluster bootstrap interval for repeated issuers.
Those views answer different failure modes. Quarter-level estimates expose time instability. Ticker clustering stops repeated calls from one issuer from masquerading as independent evidence. Horizon and robustness cuts show whether a result survives ordinary design choices.
The two semantic features—implicit ACR and evasion—used one frozen local 256-dimensional static embedding model, minishlab/potion-base-8M, with a pinned snapshot and environment. The tone baseline used a pinned Loughran–McDonald dictionary instead. Neither model nor dictionary received return labels. The three scored pipelines took about 18.3 minutes total on CPU, including roughly 1.50 million chunks embedded for ACR and evasion. “Weekend-scale” is literal here: a new idea can be measured without turning into a new platform.
4. Four attractive signals produced four stop decisions
A pipeline is not validated because it finds a trade. It is validated when identity checks, calibration, uncertainty, and stopping rules remain intact after an appealing number appears.
| Signal | Primary evidence | Decision |
|---|---|---|
| Q&A tone delta | 60-session Q5−Q1: −29.98 bp; ticker-cluster 95% CI [−107.02, +46.26] | No-go. Signs changed across horizons and quintiles were not monotone. |
| Explicit prepared-reference rate | 60-session with-reference minus without: −11.70 bp; CI [−61.47, +41.61] | No-go. Small and null across horizon, split-confidence, and question-toughness cuts. |
| Implicit prepared/answer similarity | 60-session Q5−Q1: −67.26 bp; CI [−143.05, +7.31] | No-go. Suggestive bearish direction, but every interval crossed zero and the sign was ambiguous before observation. |
| Answer evasion | Positive-control rank AUC: 0.450; expected direction > 0.500 | Reject the proxy. Its return result is uninterpretable, not a valid null. |
The implicit-similarity result is the useful temptation. Its length-adjusted 60-session interval ended at +1.69 bp—close enough to zero to invite a “near miss” story. But several related cuts had been run, quarterly evidence was weak, and the economic sign was not uniquely fixed in advance. Promotion would have converted exploration into an after-the-fact hypothesis.
The evasion test shows a different stop. A feature can generate a clean numeric column and still fail to measure its label. Literal refusal and follow-up phrases scored less evasive than unflagged answers in the corpus. The validity failure comes before any return interpretation.
5. Plug a new transcript hypothesis into the existing rail
Within this research implementation, a new hypothesis should replace the feature extractor, not rebuild the corpus, return engine, or inference code. The practical contract is one derived row per eligible call, keyed to the rail’s existing identity and content hash. An entitled self-hosted user would reproduce or adapt the guarded local panel; the public site does not ship the panel or a hosted executor.
| Step | What to lock before looking at returns |
|---|---|
| 1. State the claim | Name the text section, scalar feature, expected direction, economic mechanism, primary horizon, and unit of analysis. |
| 2. Define validity | Choose an independent validation or calibration check appropriate to the feature, such as a labeled set or directional positive control. |
| 3. Freeze extraction | Pin code, dictionary or model snapshot, environment, chunking, missing-value verdicts, and quality exclusions. |
| 4. Preserve lineage | Emit transcript ID, fiscal identity, call date, content hash, split verdict, feature version, numeric value, and quality verdict—never substitute a ticker-only join. |
| 5. Join outcomes | Verify hashes, attach the existing +20/+40/+60 abnormal returns, and keep the primary horizon distinct from robustness checks. |
| 6. Apply promotion gates | Require valid measurement, stable interpretation, issuer/time-aware uncertainty, and a locked future test before calling anything investable. |
For example, a hypothesis that “a sudden rise in supply-chain specificity predicts positive drift” must define specificity before seeing returns. It might use Q&A only, count a fixed vocabulary or score a frozen labeled model, and require agreement on a human-reviewed calibration set. The output joins the existing panel by transcript identity and hash; StockQL supplies prices and benchmark sessions; the same quarterly and ticker-aware evaluation decides whether the idea advances.
If a feature has a calibration requirement and fails it, stop there. If measurement is valid but its return interval crosses zero, publish the bounded null. If an exploratory cut looks promising, write a new pre-registered specification and test it on a non-overlapping future period. Do not rename the exploratory cut “primary.”
The StockQL backtesting guide covers the underlying SQL discipline for universe formation, signal timing, transaction costs, and point-in-time results. The transcript rail adds text lineage, section quality, local feature computation, and issuer-aware inference.
6. Questions quant-curious readers ask
- How do you backtest earnings-call sentiment?
- Freeze a point-in-time universe, identify each call with more than a ticker, derive sentiment from call-time text, delay the outcome past the immediate reaction, compare stocks within the same period, and report uncertainty that accounts for repeated issuers. Lock the dictionary or model before joining returns.
- Does StockQL perform sentiment analysis?
- No. StockQL provides the self-hosted transcript, identity, price, and benchmark plumbing. A dictionary, local model, hosted model, notebook, or agent can produce derived features under the researcher’s own data and model boundaries.
- Why keep a pipeline after four tests did not advance?
- Because the expensive reusable work is the audited split, stable identities, guarded panel, return construction, and inference. Three bounded nulls and one rejected proxy stopped with evidence; the next hypothesis can reuse the method instead of rebuilding weeks of plumbing.
- Is StockQL open source?
- No. StockQL is source-available and self-hosted. You can inspect and run the supplied source on your own infrastructure under its license; “open source” would make a different licensing claim.
- What would justify reopening these signals?
- Implicit prepared/answer similarity needs a separately motivated, pre-registered sign and interaction tested on non-overlapping returns. Evasion needs an independent human-labeled corpus that validates the scorer before either alpha or policy use. Those are entry conditions, not scheduled follow-up.
What StockQL contributes
A transcript hypothesis is easy to describe and surprisingly easy to test badly. StockQL keeps the durable market-data layer local and queryable: long-form calls, fiscal and security identity, exchange-normalized prices, benchmark sessions, and the structured facts needed to define a historical universe.
The research rail composes those pieces with an audited section splitter, frozen feature code, leakage guards, and explicit uncertainty. That composition is the value: a quant-curious researcher can spend a weekend testing the signal rather than a month reconstructing the database—and can keep an honest null when the signal does not survive.