Object store corruption

TVO-STORE, fsck, and what a hash mismatch means.

Written By Dustin

Last updated 14 minutes ago

tovio fscktovio fsck --strict

Re-hashes objects, verifies refs and the audit chain, and scans for leaked key material. TVO-STORE-004 is the aggregate exit for a scan that found problems — the individual findings are in the output.

What a hash mismatch means

TVO-STORE-001. Every object is re-hashed on read and on receipt. A mismatch means the bytes are not what their address says they are — corruption, or tampering. TOVIO surfaces it rather than accepting it, which is the entire point of content addressing.

What to do

  1. Stop writing to the repository.
  2. Note whether it followed a disk problem, a crash, or a restore.
  3. If you have a good clone elsewhere, that is your recovery — objects are content-addressed, so a healthy clone is authoritative.
  4. If corruption is on a Forge, tell the operator before doing anything.

tovio fsck --repair is not a corruption repair, despite the name. It only reconciles an interrupted commit left pending by a crash against its journal, so a repository that refuses to open can be reopened. It takes the write lock and does nothing about corrupt object bytes — so it is the right tool for "the repository will not open after a crash" and the wrong one for a hash mismatch.

Things that look like corruption and are not

  • TVO-STORE-002 — an obliterated payload. That is typed absence with a signed tombstone, and fsck distinguishes it from damage.
  • TVO-STORE-005 — the store records a layout capability this build does not implement, e.g. a pack format written by a newer version. Your binary is too old, not your data broken. Upgrade.
  • TVO-SYNC-005 — a missing object in a partial clone is a backfill failure, not corruption. A missing object in a complete clone is corruption.

Maintenance that stopped safely

  • TVO-STORE-006 — a durable gc --repack transaction cannot be safely resumed or rolled back, so it stopped rather than guess.
  • TVO-STORE-007 — causal ref maintenance stopped on invalid or unpersistable state.
  • TVO-STORE-010 — a ref was opened under one name and the register there declares a different one, meaning the filesystem resolved two ref names onto one file. This shows up on case-insensitive filesystems. Fail-closed, and worth reporting.

If you suspect tampering

Report privately to security@tovio.dev, never on a public board. Content substitution that is accepted as genuine is explicitly in scope as a security defect.

Reclaiming space is different

tovio gc --dry-run

gc is undo-safe and will not collect what the op-log still needs.

Reporting it

The code, the count of affected objects, and what preceded it (crash, disk, restore). Never object addresses or paths.