My repository ran out of space during backups and is now corrupt.

[grifferz]

After thinking everything was just right I left a whole bunch of backup tasks to run across the last 24 hours. Apparently I was incorrect as space on my repo ran out! This shouldn't have happened unless I accidentally backed up a lot of things which should not have been backed up, or some other issue occurred.

While trying to determine what went wrong, I found that check is unhappy:

$ sudo -u restic rustic check                                                           [INFO] using config /etc/rustic/rustic.toml                                             
[INFO] repository local:/srv/restic/repo: password is correct.                          
[INFO] using no cache                                                                   [00:00:00] getting snapshots...           ████████████████████████████████████████      
[00:00:01] reading index...               ████████████████████████████████████████      
  577/577       [00:00:00] listing packs...               ⠁                             
                [WARN] pack e04d6298 not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.                                                   
[WARN] pack e0351889 not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.
[WARN] pack 51cab1d1 not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.                                                                   
[WARN] pack 515975ce not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.                                                                   
[WARN] pack 51942075 not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.

[…many more of this similar message…]

[00:00:00] listing packs...                                                             [00:00:00] checking trees...              ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░      
    0/904       [ERROR] dir "boot" subtree blob 64b7176d is missing in index            
[ERROR] dir "data" subtree blob f614dc41 is missing in index                            
[ERROR] dir "etc" subtree blob 9ff42fd0 is missing in index                             
[ERROR] dir "home" subtree blob f4f1f48e is missing in index                            
error: `rustic_core` experienced an error related to `internal operations`.             
                                                                                        
Message:                                                                                
Tree ID `64b7176d` not found in index                                                   
                                                                                        
                                                                                        
Some additional details ...                                                             
                                                                                        
Backtrace:                                                                              
disabled backtrace (set 'RUST_BACKTRACE="1"' environment variable to enable)

I then issued rustic repair index:

[INFO] using config /etc/rustic/rustic.toml
[INFO] repository local:/srv/restic/repo: password is correct.
[INFO] using no cache
[WARN] error reading pack 5e7121d3 (-> removing from index): Error: Data is too short (less than 16 bytes), cannot decrypt. (kind: related to cryptographic operations)
[WARN] error reading pack 50209f8e (-> removing from index): Error: Data is too short (less than 16 bytes), cannot decrypt. (kind: related to cryptographic operations)

[…many more…]

Those warnings remain no matter how many times I run repair index.

How should I proceed?

Thanks,
Andy

[grifferz]

Oh I should add that when I discovered that the repository had run out of space I did have to give it a lot more space before I could do anything at all, so I did that and there is available disk space right now.

[aawsome]

Did you try to run rustic repair index with the nightly version?
There was a bug which led to not repairing the index at all in this case. This, however, has been fixed in rustic-rs/rustic_core#406 and should be in the latest nightly build.

[grifferz]

Unfortunately I'm having some difficulty here. My backup host is currently Debian 12 (oldstable as of a couple of weeks ago, but still supported by Debian for at least the next 18 months). However your nightly x86_64-unknown-linux-gnu requires a newer version of glibc than is present on my host.

As I may end up having to use this binary for some time, I do not want to use a static musl binary as there can be performance issues.

I tried building both head and commit rev e5500f5 (which I think your nightly was built from) with cargo install --git https://github.com/rustic-rs/rustic.git --rev e5500f57ba7277828499cca7be2351798aff4ad9 rustic-rs but both of these failed with error:

error[E0061]: this method takes 3 arguments but 2 arguments were supplied               
   --> src/config/hooks.rs:54:17
    |                                       
54  |             cmd.run(context, what)?;
    |                 ^^^--------------- argument #3 is missing                         
    |                                       
note: method defined here      
   --> /root/.cargo/git/checkouts/rustic_core-abe509602c0243d0/3790932/crates/core/src/repository/command_input.rs:134:12
    |                                                                                   
134 |     pub fn run<I, K, V>(&self, context: &str, what: &str, env: I) -> RusticResu...
    |            ^^^
help: provide the argument                                                              
    |
54  |             cmd.run(context, what, /* env */)?;                                   
    |                                  +++++++++++ 

I'm not that familiar with cargo install --git. I must have dome something wrong as I assume these nightlies were successfully built by CI. Is there any documentation on how to build one?

[grifferz]

In the interests of making progress here I did download your nightly musl binary. Unfortunately it gives the same warnings every time and appears to make no difference. That is, I get the same warnings as above while running rustic repair index and then the same crash when running rustic check, then if I run rustic repair index again it once more gives the same warnings.

$ sudo -u restic rustic --version
rustic v0.9.5-31-ge5500f5-nightly

[aawsome]

Ah, sorry, this is an old version. The nightly pipeline failed because of a netbsd build problem. I deactivated the netbsd build; the next nightly should have the fix included.
Sorry for the inconvenience!

[aawsome]

nightly should be now of version b6b31ca

[grifferz]

When trying to build the nightly myself I still get the same build error as above by the way. Is cargo install --git https://github.com/rustic-rs/rustic.git --rev b6b31cae01cf3afc82273b0087082f370bd7a843 the correct way to do it?

Trying with a downloaded musl binary…

Yeah exact same behaviour. repair index gives many warnings like this:

[WARN] error reading pack 6ade59f5 (-> removing from index): Error: Data is too short (less than 16 bytes), cannot decrypt. (kind: related to cryptographic operations)
[WARN] error reading pack 9eb6d86a (-> removing from index): Error: Data is too short (less than 16 bytes), cannot decrypt. (kind: related to cryptographic operations)
[WARN] error reading pack 205b0e27 (-> removing from index): Error: Data is too short (less than 16 bytes), cannot decrypt. (kind: related to cryptographic operations)

while check aborts with an error:

$ sudo -u restic rustic check                                                           [INFO] using config /etc/rustic/rustic.toml                                             
[INFO] repository local:/srv/restic/repo: password is correct.                          [INFO] using no cache                                                                   
[00:00:00] getting snapshots...           ████████████████████████████████████████      
  905/905                                                                               [00:00:02] reading index...               ████████████████████████████████████████      
  578/578                                                                               
[00:00:00] listing packs...               ⠉                                             
[WARN] pack e04d6298 not referenced in index. Can be a parallel backup job. To repair: '
rustic repair index'.                                                                   
[WARN] pack e0351889 not referenced in index. Can be a parallel backup job. To repair: '
rustic repair index'.                                                                   
[WARN] pack 51cab1d1 not referenced in index. Can be a parallel backup job. To repair: '
rustic repair index'.

[pages of this]

[WARN] pack 9990b0e0 not referenced in index. Can be a parallel backup job. To repair: '
rustic repair index'.                                                                   
[00:00:00] listing packs...                                                             [00:00:00] checking trees...              ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░      
    0/905                                                                               
[ERROR] dir boot subtree blob 64b7176d is missing in index                              
[ERROR] dir data subtree blob f614dc41 is missing in index                              
[ERROR] dir etc subtree blob 9ff42fd0 is missing in index                               
[ERROR] dir home subtree blob f4f1f48e is missing in index                              
error: `rustic_core` experienced an error related to `internal operations`.             
                                                                                        
Message:                                                                                
Tree ID `64b7176d` not found in index                                                   
                                                                                        
                                                                                        
Some additional details ...                                                             
                                                                                        
Backtrace:                                                                              
disabled backtrace (set 'RUST_BACKTRACE="1"' environment variable to enable)

$ sudo -u restic rustic --version
rustic v0.9.5-43-gb6b31ca-nightly

[aawsome]

It seems like rustic repair index did what it should do: It removed the truncated pack files from the index. So, the next step is to re-run a rustic backup (do this for all sources you have in you snapshots) and see if the errors of check are still there. If yes, the next step would be to run rustic repair snapshots

[grifferz]

I don't understand how repair index can have done its work if it keeps saying the same warnings over and over with check still unable to run as well.

I also don't understand how doing another set of backups is going to make check work. The documentation says that if check is not working then the repository must be fixed before doing any more backups to it.

Also, this happened because my repository ran out of space. I was able to give it more space but I cannot forget --prune things right now. So, is there now data in my repository that I just cannot remove?

Could this be a fatal situation if I was in a position of not actually being able to give it more space (full repo filesystem, but unable to forget --prune anything to free space)?

$ sudo rustic forget --prune 32e7a148 
[INFO] using config /etc/rustic/rustic.toml
[INFO] repository local:/srv/restic/repo: password is correct.
[INFO] using no cache
[00:00:00] getting snapshots...           ████████████████████████████████████████          1/1                                                                                 
[INFO] snapshots:
| ID       | Time       | Host       | Label | Tags | Paths      | Action | Reason     |
|----------|------------|------------|-------|------|------------|--------|------------|
| 32e7a148 | 2025-08-21 |      foo.e |       | auto | data/backu | remove | id         |
|          | 20:24:53   | xample.com |       |      | p          |        | argument   |
|          |            |            |       |      | etc        |        |            |
|          |            |            |       |      | usr/local  |        |            |
|          |            |            |       |      | var        |        |            |
[00:00:00] removing snapshots...          ████████████████████████████████████████          1/1                                                                                 
[INFO] repository local:/srv/restic/repo: password is correct.
[INFO] using no cache
[00:00:02] reading index...               ████████████████████████████████████████        578/578                                                                               
[00:00:00] reading snapshots...           ████████████████████████████████████████        904/904                                                                               
Well, this is embarrassing.

rustic-rs had a problem and crashed. To help us diagnose the problem you can send us a crash report.

We have generated a report file at "/tmp/report-ccff265f-d3b0-4754-8ea8-69881a9bf3b1.toml". Submit an issue or email with the subject of "rustic-rs Crash Report" and include the report as an attachment.

- Homepage: https://rustic.cli.rs/
- Authors: the rustic-rs team

We take privacy seriously, and do not perform any automated error collection. In order to improve the software, we rely on people to submit reports.

Thank you kindly!
Aborted

Do you want me to submit that crash report?

[grifferz]

Unfortunately I need to push on with my evaluation of backup software as my need to stop using rsnaoshot is quite pressing. Purely out of interest I'm going to try using restic to repair this repo and see what happens, then I'm going to delete all 2TiB of it and start over. Thank you for all your help!

[aawsome]

@grifferz Did you try to run rustic backup or rustic repair snapshots?

rustic backup can be run safely as soon as the index only references healthy pack files which should be the case in your situation now.
We know that the documentation can be improved. Help regarding (updating) the docu is very much appreciated.

Besides this, the steps to repair the repo are always the same (and I'm trying to tell you this now for quite a couple of days):

1. Remove the defect pack files from the index. This can be done by optionally removing these files manually from the repo. In an case you have to run rustic repair index to do this.
2. (Optional) Re-run rustic backup if you have snapshots missing data. This can heal the repository as the data of existing snapshots may be added by this step. Especially in your case it should heal the repo as your source did not change.
3. If some snapshots still miss data, either manually remove them or run rustic repair snapshots. But note that in your case your snapshots are "truncated" and do not longer contain the original backuped data.

You can try restic, but the steps to repair the repository are exactly the same there, as they are consequences about how the repository format is defined.

Actually I would suggest you to just remove the whole repository and re-do all you backups if this repairing process takes too much time or is not transparent enough...

[T1MOXA]

Similar case for me
rustic-rs v0.9.5

I even freed some space deleting non repo files.
This is second time I got repo corrupt by no space left error.

# rustic prune --max-repack 0 -P local
[INFO] using config /root/.config/rustic/local.toml
[INFO] repository local:/mnt/hdd/rustic/home: password is correct.
[INFO] using no cache
error: `rustic_core` experienced an error related to `cryptographic operations`.

Message:
Data is too short (less than 16 bytes), cannot decrypt.


Some additional details ...

Backtrace:
disabled backtrace (set 'RUST_BACKTRACE="1"' environment variable to enable)

# rustic check -P local
[INFO] using config /root/.config/rustic/local.toml
[INFO] repository local:/mnt/hdd/rustic/home: password is correct.
[INFO] using no cache
[00:00:02] getting snapshots...           ████████████████████████████████████████        232/232                                                                                                                error: `rustic_core` experienced an error related to `cryptographic operations`.

Message:
Data is too short (less than 16 bytes), cannot decrypt.


Some additional details ...

Backtrace:
disabled backtrace (set 'RUST_BACKTRACE="1"' environment variable to enable)

# rustic backup -P local
[INFO] using config /root/.config/rustic/local.toml
[INFO] repository local:/mnt/hdd/rustic/home: password is correct.
[INFO] using no cache
error: `rustic_core` experienced an error related to `cryptographic operations`.

Message:
Data is too short (less than 16 bytes), cannot decrypt.


Some additional details ...

Backtrace:
disabled backtrace (set 'RUST_BACKTRACE="1"' environment variable to enable)

# rustic repair index -P local
[INFO] using config /root/.config/rustic/local.toml
[INFO] repository local:/mnt/hdd/rustic/home: password is correct.
[INFO] using no cache
[00:00:22] listing packs...                                                                                                                                                                                      error: `rustic_core` experienced an error related to `cryptographic operations`.

Message:
Data is too short (less than 16 bytes), cannot decrypt.


Some additional details ...

Backtrace:
disabled backtrace (set 'RUST_BACKTRACE="1"' environment variable to enable)

# rustic repair snapshots -P local
[INFO] using config /root/.config/rustic/local.toml
[INFO] repository local:/mnt/hdd/rustic/home: password is correct.
[INFO] using no cache
error: `rustic_core` experienced an error related to `cryptographic operations`.

Message:
Data is too short (less than 16 bytes), cannot decrypt.


Some additional details ...

Backtrace:
disabled backtrace (set 'RUST_BACKTRACE="1"' environment variable to enable)

[aawsome]

@T1MOXA Can you please retry the rustic repair index using the latest nightly build?

[T1MOXA]

Replaced package rustic-rs v0.9.5 (https://github.com/rustic-rs/rustic.git#a789b507) with rustic-rs v0.9.5 (https://github.com/rustic-rs/rustic.git#d0e7f0ae)

# rustic repair index -P local
[INFO] using config /root/.config/rustic/local.toml
[INFO] repository local:/mnt/hdd/rustic/home: password is correct.
[INFO] using no cache
[00:00:22] listing packs...                                                                                                                                                            error: `rustic_core` experienced an error related to `cryptographic operations`.

Message:
Data is too short (less than 16 bytes), cannot decrypt.


Some additional details ...

Backtrace:
disabled backtrace (set 'RUST_BACKTRACE="1"' environment variable to enable)

[ble: exit 1][ble: elapsed 580.861s (CPU 26.2%)]

[T1MOXA]

Any ideas how to fix repo ?
Or I should wipe all again ?

UPD.
Wiped repo, and recreated again, now with check of free space left before backup.