Clarification on avenues for file level corruption reversal

[chocmake]

Just had some questions about in practical terms the process of a user detecting corrupted files (on the backup side that is) and then restoring known-good copies back to a Borg archive.

• Does check with --verify-data surface to the user which user files were detected as part of a chunk having a checksum mismatch (ie: corrupted) or does it only surface which chunks in the archive files are affected? Wondering how aware the user can be made about which files they should replace with backup copies.

  • I realize due to deduplication that chunks may represent data across multiple different paths/filenames but even so wondering if Borg surfaces some simple indication of eg. one of the user file paths or something the user can practically take action with.

• If a user were able to determine which files were corrupted and add them back to the source data and then trigger a Borg backup it reads from the docs that --repair can 'self-heal' any chunks zeroed (marked as corrupted) from a prior --repair as Borg keeps the prior known-good checksums for data of those the zeroed chunks.

  • Question I have is how does this behave if --repair hasn't been used previously (and thus hasn't had a chance to zero-out corrupted chunks) but a user has been made aware of corruption via check and added known-good files back to the source data, then runs a backup, and subsequently runs --repair? Does repair recognize and 'heal' the affected corrupted chunks in one operation in such a case?

Edit: replaced use of 'data files' with 'user files' to avoid being confused with an archive's data directory.

[ThomasWaldmann]

Let me answer the 2nd question first:

You will need 2x borg check --repair:

• first one is to detect and delete corrupted chunks and replace references to them by references to all-zero replacement chunks
• (in between, you should try to write valid chunks to the repo, e.g. by at least doing all your usual backups, maybe also by backing up historical versions of affected files)
• second one is to replace the all-zero replacement chunks by present-again valid chunks.

[ThomasWaldmann]

1st question:

borg check --repair has 2 parts:

• repository level check: here it does a pass over all chunks (objects) in the repo an deletes any corrupted ones and also recreates the repo index, so it is corresponding to the segment files. for this part, it only does a simple crc32 check and does not need the borg key. for ssh: repos, it will run on the borg server side.
• verify-data and archives level part: for this part it will need the borg key so it can decrypt stuff. verify-data will do a full pass over all chunks again and this time will do strong, cryptographic verification. it does not know about "archives" or "files" at this time, as it is only looking at the chunks (pieces). after that, it will go over all archives, read all file metadata in these archives and will make sure that all chunks referenced by each file are actually present in the repository. if a chunk is missing, it will be replaced by a all-zero chunk. if a previously missing chunk (that got replaced by a all-zero replacement chunk) is present again, it will be healed. IIRC, it will tell you archive name, file name and offset or chunk id in the log output.

In general, if you are interested in borg giving you a lot of details, use -v so it runs at INFO log level. The default is WARNING.

[chocmake]

Thanks. Ran some tests in a VM with Borg v1.4.0 (standalone binary) to observe the behavior.

Setup:

1. Added a handful of files to a source directory.
2. Initialized a test repo with Borg.
3. Created a backup.
4. Intentionally corrupted one of the larger archive files in the repo using a hex editor to delete some of its content (to simulate unexpected corruption).

Test results (all using the --verbose and --log-json common option arguments):

1. Subsequently ran a backup with create to see what would be reported. It failed early with errors (before it got to outputting any JSON), reporting a Data integrity error: Invalid segment entry size along with a Python traceback and system/CLI command info.
2. Ran check --verify-data but without --repair to see what it would output. It noted there was a Segment entry checksum mismatch and noted the segment number and offset, though at this point no related user file paths are noted.
3. Ran check again but with --repair. This time it noted the full path of the user file associated with the missing chunk, along with noting it had replaced the chunk with an all-zero chunk.
4. To confirm it had zeroed the chunk I mounted the repo and tried to open the affected file. It errored.
5. Unmounted and ran the same repair command again (without any intermediate new snapshot created) since was curious what would occur. It reported the previously 'missing' chunk is still missing.
6. Made a new backup via create.
7. Ran --repair again. This time, as Thomas hinted above, it detected the valid file in the new snapshot and reported the path of the user file that was 'healed'. So yeah, some intermediate snapshot is required to be run after the first repair command for the healing mechanism to work.
8. Confirmed the file was healed by re-mounting and opening the file. All backups containing that file were similarly fixed.

[chocmake]

A drawback, in terms of automatic logging/notifying the user, is no file paths surface until --repair is run, which isn't necessarily desirable to run automatically (that is, forcing a response to the interactive prompt confirmation it asks for).

Should note that I'm basing this risk on the docs warning that repair may not be capable of handling other types of corruption besides missing/corrupted chunks. So seems undesirable to automate.

So the best one could do is log/notify there is corruption but until the user runs the --repair command there wouldn't be insight into which user files are affected.

Edit: is this worth submitting as a feature request? Either for check per se or an optional argument for check that isn't --repair (or even like a --dry-run but for --repair)? Or is there some limitation/technical reason I'm not considering.

[ThomasWaldmann]

If one does not give --repair, borg check won't remove corrupt chunks from repo and from index.

As long as such a chunk is present / in the index, the archive level part of borg check won't know that there is a problem with it.

[chocmake]

Just for clarity, does that mean that without --repair actually modifying the archive files (that is, eg: zeroing corrupted chunks) that such a list of user file paths affected by a corrupted chunk wouldn't be able to be output?

Reason I wondered is it seems Borg maintains a relationship mapping between the representation of user files contained in each Borg backup/snapshot and which chunk they relate to. Which for notification/logging purposes if there were a way to obtain which user files were affected by a corrupted chunk but without having to actually run --repair destructively would be useful (to get a sense of what is damaged in the backup, though --repair would have to be run later anyway obviously).

(And thanks for your time with this)

[ThomasWaldmann]

Not with the borg 1.x code.

To find corrupted chunks, it only does one pass over them, just iterating all chunks in the repo without considering any relationship to files or archives. This is for efficiency to be as fast as possible. For remote repos, that code runs server-side without the borg key, so it can not even decrypt archives.

When processing archives, it only checks whether the referenced chunks exist in the index - if it would read all chunks referenced by each archive it would take ages because each chunk would be read multiple times.

[chocmake]

Since this is a continuation of some related testing just thought I'd note here for anyone interested some unexpected behavior from check's reporting when used without --repair following previously zeroed out chunks.

I found that after --repair had zeroed an intentionally corrupted chunk, then I ran a create backup successfully, that if I next ran check but without --repair it would report as having healed the missing chunks, when it actually hadn't.

Steps:

• Removed the repo hints, index and integrity files as part of this test (was checking how Borg would respond to both missing repo metadata and a corrupted chunk file simultaneously).
• Intentionally corrupted the largest data file in the repo. Since I have a PNG image in the source data larger than other files in the source data these larger data files have always contained the image data for these tests.
• Ran check with --verify-data and --repair. This zeroed out the corrupted chunks.
• Ran a successful backup with create.
• Ran check with --verify-data but without --repair. And this is where it reported the PNG image as being healed in all backups (removed time values from the log):

...
{"type": "log_message", "message": "borg-backup-test-1: home/user/Documents/test-data/test.png: Healed previously missing file chunk! (Byte 0-475076, Chunk acd0521e02a47fde43e172300e6728268517a5676ebe7079007644aff35f1dac).", "levelname": "INFO", "name": "borg.archive"}
{"type": "log_message", "message": "borg-backup-test-1: home/user/Documents/test-data/test.png: Completely healed previously damaged file!", "levelname": "INFO", "name": "borg.archive"}
{"type": "log_message", "message": "Analyzing archive borg-backup-test-2 (5/8)", "levelname": "INFO", "name": "borg.archive"}
{"type": "log_message", "message": "borg-backup-test-2: home/user/Documents/test-data/test.png: Healed previously missing file chunk! (Byte 0-475076, Chunk acd0521e02a47fde43e172300e6728268517a5676ebe7079007644aff35f1dac).", "levelname": "INFO", "name": "borg.archive"}
{"type": "log_message", "message": "borg-backup-test-2: home/user/Documents/test-data/test.png: Completely healed previously damaged file!", "levelname": "INFO", "name": "borg.archive"}
{"type": "log_message", "message": "Analyzing archive borg-backup-test-3 (6/8)", "levelname": "INFO", "name": "borg.archive"}
{"type": "log_message", "message": "borg-backup-test-3: home/user/Documents/test-data/test.png: Healed previously missing file chunk! (Byte 0-475076, Chunk acd0521e02a47fde43e172300e6728268517a5676ebe7079007644aff35f1dac).", "levelname": "INFO", "name": "borg.archive"}
{"type": "log_message", "message": "borg-backup-test-3: home/user/Documents/test-data/test.png: Completely healed previously damaged file!", "levelname": "INFO", "name": "borg.archive"}
{"type": "log_message", "message": "Analyzing archive borg-backup-test-4 (7/8)", "levelname": "INFO", "name": "borg.archive"}
{"type": "log_message", "message": "borg-backup-test-4: home/user/Documents/test-data/test.png: Healed previously missing file chunk! (Byte 0-475076, Chunk acd0521e02a47fde43e172300e6728268517a5676ebe7079007644aff35f1dac).", "levelname": "INFO", "name": "borg.archive"}
{"type": "log_message", "message": "borg-backup-test-4: home/user/Documents/test-data/test.png: Completely healed previously damaged file!", "levelname": "INFO", "name": "borg.archive"}
{"type": "log_message", "message": "Analyzing archive borg-backup-test-5 (8/8)", "levelname": "INFO", "name": "borg.archive"}
{"type": "log_message", "message": "Archive consistency check complete, no problems found.", "levelname": "INFO", "name": "borg.archive"}

• However after using mount to check if it had actually healed I found only the latest backup contained the openable PNG image. While the prior backups all errored when opening it.
• So I ran check with --repair and it reported the heals like the prior output but this time it actually worked for all the backups it reported (confirmed by re-mounting the backups).

[ThomasWaldmann]

Opened #8533 for this.