Today’s bulk market-data file may not be final

Bulk endpoints make daily ingestion efficient: one response can replace thousands of per-symbol requests. Their operational risk is equally concentrated. If a once-daily job reads a file while it is still filling, every late row becomes a persistent hole.

HTTP 200, valid CSV, and a plausible row count are transport checks. Completeness is a separate property that has to be observed over the provider’s publication cycle.

1. Treat bulk output as a snapshot

StockQL measured EODHD’s US bulk-last-day endpoint on August 21, 2026. About two hours after the close, at 22:02 UTC, the response contained 19,559 rows. The finalized file addressed by the same trading date later contained 50,247 rows.

The difference was not merely empty catalog padding. Some late rows carried real volume as high as 32.8 million shares. A single early fetch would have left active, liquid securities missing for that date.

The final file also contained roughly 33,000 flat zero-volume placeholder rows. Counting every line as a usable market observation would make the later file look complete while importing data that did not represent a trade. Completeness must be defined over eligible, usable rows—not bytes or CSV lines.

2. Define observable completeness

A production ingest can measure what the provider does not declare:

• Record the requested trading date, fetch time, response digest, total rows, usable rows, and per-venue counts.
• Compare the eligible universe with symbols that received a non-placeholder observation.
• Distinguish a market holiday from a thin or partial file using an audited exchange calendar.
• Alert on cohort-sized absences rather than immediately changing thousands of security lifecycles.
• Keep missingness visible until a later snapshot resolves it or the lifecycle system supplies a different explanation.

No single threshold proves completion across every exchange. The contract is a bounded re-observation policy plus explicit residuals.

3. Recheck explicit dates

StockQL’s daily pipeline performs a bulk-completeness re-pass. It fetches the prior explicit trading date again and can revisit the preceding dates when the stored universe still has a material missing cohort. It inserts only absent usable rows and skips flat placeholders.

This shape is idempotent: a repeated final file does not duplicate bars, while a later-expanded file fills the holes. Because the request names the date explicitly, the re-pass does not accidentally slide to a newer “last day” while the old date remains incomplete.

Late data and delisting logic also stay separate. A symbol missing from one partial snapshot is not a lifecycle event. Only sustained availability evidence, guarded against correlated provider incidents, can move that state.

4. Compare publication systems fairly

FMP’s bulk endpoint has a different shape: one measured response covered roughly 66,000 global rows in 3.8 MB and required client-side filtering for an exchange. Historical dates can also carry adjusted-price semantics that differ from the latest-day file. It is efficient, but it does not make publication timing or field mapping disappear.

StockQL can compare symbol presence across feeds to surface a suspicious cohort, then return to each provider’s own dated endpoint for the serving fact. The second feed is an alarm and an arbiter, not permission to merge unlike rows blindly.

Next: why a ticker is not an earnings transcript’s identity →