One hot copy in GCS, two Filecoin archives, and an on-chain fingerprint every read must match. Tested on Calibration, not yet in production.
A team that buys a training dataset keeps using it long after the transaction closes. By then the download link may have expired and the services behind it may have restarted. The team needs the file back in full, and proof that nobody replaced or damaged it. If nothing checks, the wrong file can be delivered without anyone noticing.
We built and tested a storage path for that case. It would keep each delivered data version in three places: a hot copy in Google Cloud Storage (GCS) and cold archives at two independent Filecoin storage providers. An on-chain fingerprint, the PieceCID, would check every copy before it is returned.
Current status: These results come from test data on the Filecoin Calibration network and controlled fault injection. Humanbased, the first application built on XnY, stores production files in GCS only. Filecoin archiving and authorized decryption-key delivery are not yet a self-service production capability, so customer data does not yet receive this protection automatically. No result here is a production throughput, latency, or capacity figure.
Three layers, three jobs
- GCS Origin: the hot copy. Every read tries it first.²
- Two Filecoin providers: the cold archives. A CID names data by its content, not its location.³ Each copy is bound to an on-chain record, so recovery does not rely on one vendor’s account of what it holds.
- On-chain fingerprint: the chain stores identifiers and hashes, never the file.¹⁵ If one byte changes, the SHA-256 or PieceCID check fails and the copy is rejected.
The fingerprint detects tampering but cannot prevent it. A copy fetched from any location must still match the original hash.⁴ The fingerprint also cannot recreate bytes that are gone, which is why there are two archives.
Packed for recovery
Recovering a file takes more than its bytes: the system must also know which version they belong to and how to reassemble them. Each archive is a package with an outer index, a version snapshot, an encrypted manifest that records chunk order, sizes, and digests, and encrypted data chunks whose count depends on file size. The test package had six components in total, including three data chunks.
Every component goes to both providers, so a complete archive has on-chain storage records equal to twice its component count, or 12 for the test package. One transaction can register several of them in a batch.
Restarts without re-uploading
A provider can accept an upload and the service can exit before saving the result. A restart that trusts only its local database uploads again and pays for a duplicate. Instead, each restart reconciles local progress against on-chain records before it continues. The completion receipt is written only once every component has a copy at both providers, so a package with a missing copy never counts as archived.
Reads fail closed
Reads try the Origin, then the preferred provider, then the fallback provider. No provider byte reaches the user until three checks pass:
- Length: catches truncated or padded files.
- SHA-256: matches every byte to the archived content.
- PieceCID: ties those bytes to the object registered on chain.
A failed check rejects that copy and moves to the next source. When every source fails, the service returns 502. The service returns a file only after it finds and verifies a copy.
Experiment reference
Setup
We connected two providers on the Filecoin Calibration test network and ran the full cycle: write two copies, read each back, verify both. Files came in four sizes: 1 MiB, 16 MiB, 64 MiB, and roughly 1 GiB (1,016 MiB in practice). Samples were chosen to probe size and concurrency limits, not to model production traffic.
A1. Two-copy write and readback
Result: 97 completed runs. Every completed run passed length and SHA-256 checks on both copies, and all 6 runs at 1,016 MiB completed.
A2. Concurrent archives
Result: 8/8 and 16/16, 24 files in all. Both rounds used 16 MiB files, which is why that size has the most runs. One round archived eight distinct files at once, the other sixteen. Every file reached both providers and passed readback from both copies, so simultaneous jobs kept per-file copy and verification behavior under these conditions.
A3. Memory and time for a 1,016 MiB file
Result: 16 m 57.1 s end to end, with the Go heap peaking at 258.1 MiB.
The client reads the file while uploading it and computing its fingerprint. In one single-file run from a local client to Calibration, the Go heap peaked at 258.1 MiB, so the whole file never sat in the heap at once. That figure excludes the rest of process memory, and one run says nothing about larger files or terabyte-scale capacity.
The second copy was stored at 482.0 s. Both on-chain records were confirmed at 762.8 s (12 m 42.8 s). Sequential readback and verification from both providers took another 4 m 14.3 s.
A4. The smallest tail chunk
Result: a 127-byte final chunk archived, and all 12 PieceCIDs matched.
Fixed-size chunking can leave a tiny final chunk, and Filecoin cannot compute a PieceCID for an encrypted object under 127 bytes. One unstorable tail can stop the whole archive from completing. We tested the smallest storable tail: 99 bytes of plaintext plus 28 bytes of fixed encryption overhead. All six components were recovered from the selected copies.
A shorter tail merges into the previous chunk, a path verified only in controlled tests.
A5. Forced restarts
Result: each provider gained exactly 6 Pieces, with 0 duplicates.
Using the recovery logic intended for production, we forced an exit at five points: right after claiming a task, after the upload call returned, after recording the first copy, after finishing one component, and after all 12 copies but before the completion receipt.
In the gap where a provider had accepted the upload but the database had no record, reconciliation found both storage records on chain and reused them instead of uploading again.
A6. Read failover and fault injection
Result: the fallback provider served the file with 200; corrupt or unverifiable copies were refused.
We disconnected the Origin, then the preferred provider, then all three sources. With the Origin down and the preferred provider blocked, the fallback provider’s copy was retrieved, verified, and returned with 200. We also made a provider return errors, truncated content, and oversized content. Each bad copy was rejected, and 502 came back only when every source failed. No Calibration provider returned corrupt data on its own.
The first Origin-disconnected run returned 502 on 18 of 37 checks, though the files were intact. The RPC endpoint alternated between two confirmed chain views about 52 blocks apart, and the older one could not yet see the new storage records. Once both views advanced, the same checks passed. This shows safe failure. It does not measure failover time or production availability.
References
- XnY Protocol Docs — Architecture. Responsibilities of on-chain records and off-chain content. docs.xny.ai/concepts/core-systems/architecture
- Google Cloud Documentation — Cloud Storage overview. The GCS object storage model and common use cases. docs.cloud.google.com/storage/docs/introduction
- Filecoin Docs — Glossary. Official definitions for CIDs, storage providers, and storage proofs. docs.filecoin.io/reference/general/glossary
- XnY Protocol Docs — Storage & Serving. Storage backends, content hashes, retrieval locations, and the roles of GCS and Filecoin. docs.xny.ai/concepts/core-systems/storage-compute-serving
- XnY Protocol Docs — Data Assembly. On-chain commitments for data versions and off-chain storage of the content itself. docs.xny.ai/concepts/core-systems/data-assembly