The Audit
On April 18, 2026, we ran a comprehensive sync-drift audit comparing our local data warehouse against what was deployed to DigitalOcean production. The findings: massive invisible drift — not corruption, not data loss, but a quiet divergence where production was running on a fraction of the data we thought we'd deployed.
Object Storage Drift
| Bucket | Status | Detail |
|---|---|---|
| latticegraph-derived | In sync | Byte-identical |
| latticegraph-candidate | In sync | Byte-identical |
| latticegraph-raw | Major drift | 1,117 files / ~63 GB exist only locally |
The 1,117 missing files break down by source:
| Source | Files Missing from Production |
|---|---|
| OCP (Open Catalyst Project) | 885 files |
| OMat24 | 100 files |
| LeMat-Traj | 91 files |
| OMol25 | 20 files |
| Other sources | 21 files |
Database Drift
- 16 production-only schemas (15 empty shells created by migrations but never populated)
- 12 local-only tables — data that was built locally and never pushed
- 19 production-only tables — tables created by production migrations that local hadn't run
- 32 common tables with differing row counts — local had +12.6M rows overall
No data corruption was found. Zero content mismatches on shared files and tables. The drift was entirely about coverage — production had less data, not wrong data.
The DigitalOcean Key-Naming Bug
Every object in the DigitalOcean latticegraph-raw Spaces bucket had its source prefix duplicated. Instead of alexandria/file.parquet, the key was alexandria/alexandria/file.parquet. This doubled-prefix bug affected every uploaded file and meant that sync scripts comparing local paths against DO paths would see them as different objects — explaining part of the 1,117-file gap.
License Risk Discovery
The audit also surfaced a license-risk finding: approximately 7.5M rows of data from sources with unvetted licenses were present in the DigitalOcean production Postgres database. The highest urgency: GNoME data (CC-BY-NC-4.0) — non-commercial use only — was loaded into a production database backing a commercial API.
Why This Matters
Sync drift is invisible by default. The application runs, the API returns results, and the dashboard looks fine. But the results are computed against a subset of the data, and decisions made from those results carry hidden confidence bounds that nobody knows about.
- Missing 885 OCP files means catalyst screening runs against incomplete adsorption data.
- Missing 12.6M database rows means cross-source agreement scores are computed from fewer sources than expected.
- A path-prefix bug means sync tooling silently passes when it should fail.
The fix: automated sync-drift checks as part of the deployment pipeline, comparing row counts, file counts, and path structures before marking a deployment as healthy.