Back up of already snapshoted folder in another back up is slow

[enboig]

I have a "dockers" folder where I do backup ups regularly. The structure is:

• /mnt/dockers
• /mnt/dockers/app1
• /mnt/dockers/app2
• /mnt/dockers/app3
• /mnt/dockers/app...

Sometimes, when upgrading an app, I make a backup of "/mnt/dockers/appX"; if everything works fine, after some days I delete this snapshot.

I observed that, despite data already backed up in "/mnt/dockers" snapshots, if there is no previous snapshot of "/mnt/dockers/appX" it takes a long time.

Shouldn't cache avoid this? All files appear as new despite most of them are in

First bakcup:

# time rustic backup "${PWD}" --label=this
[00:00:02] reading index...               ████████████████████████████████████████         83/83                                                                                                                                
[00:00:00] getting latest snapshot...     ████████████████████████████████████████        226/226                                                                                                                               
[01:50:02] backing up...                  ████████████████████████████████████████ 702.85 GiB/702.85 GiB 109.00 MiB/s (ETA -)                                                                                                   Files:       95318 new, 0 changed, 0 unchanged
Dirs:        6380 new, 0 changed, 0 unchanged
Added to the repo: 11.0 MiB (raw: 27.7 MiB)
processed 95318 files, 702.9 GiB
snapshot fc7178fe successfully saved.

real    112m24,578s
user    115m27,341s
sys     13m26,514s

Second snapshot:

# time rustic backup "${PWD}" --label=this
[00:00:02] reading index...               ████████████████████████████████████████         85/85                                                                                                                                
[00:00:00] getting latest snapshot...     ████████████████████████████████████████        227/227                                                                                                                               [00:00:47] backing up...                  ████████████████████████████████████████ 702.86 GiB/702.86 GiB 14.81 GiB/s  (ETA -)                                                                                                   
Files:       0 new, 46 changed, 95272 unchanged
Dirs:        0 new, 15 changed, 6365 unchanged
Added to the repo: 15.0 MiB (raw: 47.6 MiB)
processed 95318 files, 702.9 GiB
snapshot 4849242f successfully saved.

real    3m8,510s
user    0m17,463s
sys     0m9,024s

[aawsome]

@enboig This is the expected behavior:
In the first case, there is no suitable parent snapshot used. This means all data is read and chunked before rustic can realize that all data in fact is already contained in the repo, aka "deduplication". The time is the time needed for reading, chunking (which typically is the bottleneck) and hashing the data.

In the second case, rustic uses a parent snapshot and can therefore determine unchanged files by the metadata like size and timestamps. For unchanged files it directly uses the parent's data without the need for any reading/chunking/hashing.

[aawsome]

@enboig I forgot to tell that you can manually use the latest snapshot of /mnt/dockers as a parent for a new snapshot of /mnt/dockers/appX: If you know the id of the snapshot, use --parent <ID> and it will use this snapshot to detect unchanged files.

[enboig]

Manually setting parent worked. Thanks.

After giving it a second thought, "autodetecting" previuos snapshot would be hard:

1. The path may be shorter
2. You may get more than one possible snapshot
3. It may have different "ignore" rules

[aawsome]

You are right. The fun fact is, that I already started working about this (#493):

• support for multiple parents is actually not hard to add - I already have a working change and will make soon a PR from it.
• the harder part is to search for suitable parents by paths (your case is in fact only a very easy special case of a parent dir) and filter out superseded ones, but I think it is feasable.
• About ignore rules, you are right - but this is also the case with identical paths and is not taken into account there - so I think it should be ok to also ignore it for a better parent support ;-)

[aawsome]

First step for better parent support: rustic-rs/rustic_core#427