Increase the memory allocated to 'BOT_TMPDIR' to 1GB #5078
Conversation
|
/gcbrun |
9640-Akhil
force-pushed
the
no_space_left
branch
from
8eaf0ac to
28e1d2a
Compare
|
/gcbrun |
marktefftech
left a comment
marktefftech
left a comment
There was a problem hiding this comment.
@javanlacerda From our analysis, the largest corpus size needed to download is approximately ~800MB, hence the selection of 1GB as the size.
Please let us know if this works for you.
I think it should be all good. Any inputs here @jonathanmetzman ? |
|
ACtually better than enlarging this, could we put the backup in the same volume as the testcases? |
jonathanmetzman
left a comment
jonathanmetzman
left a comment
There was a problem hiding this comment.
Have you verified this works?
docker/base/setup_clusterfuzz.sh
Outdated
| # set up mounts in this case. | ||
| if [[ -z "$DISABLE_MOUNTS" ]]; then | ||
| # Setup Tmpfs dirs for frequently accessed files to save disk I/O. | ||
| mount -t tmpfs -o size=250M,mode=777 tmpfs $INSTALL_DIRECTORY/clusterfuzz/bot/inputs/fuzzer-testcases/ |
There was a problem hiding this comment.
Let's put it here and increase this size to 1.25GB
There was a problem hiding this comment.
Thank you for the suggestion! I have pushed the requested changes.
Regarding the verification, we have confirmed the change locally with the following steps:
- Built the base docker image.
- Ran the image, ensuring that the local tmpfs directories and configuration are similar to production.
- Ran the start.sh script to verify the new memory allocation for the mounts.
The changes are working as intended.
The 90MB tmpfs mount allocated for 'BOT_TMPDIR' is insufficient for downloading large corpus backup files, leading to the "OSError: [Errno 28] No space left on device" error in Cuttlefish and Linux hosts. This change increases the tmpfs mount size to 1GB to accommodate larger corpus backup files.
9640-Akhil
force-pushed
the
no_space_left
branch
from
004cc8e to
8b57fde
Compare
|
/gcbrun |
|
The sheriff should merge this. I will add it's high risk and has the potential to break fuzzing everywhere. |
5 participants
The 90MB tmpfs mount allocated for 'BOT_TMPDIR' ('/mnt/scratch0/tmp') is insufficient for downloading large corpus backups, leading to "OSError: [Errno 28] No space left on device".
This error occurs during corpus synchronization, when a corpus backup zip file is downloaded to a temporary file within '/mnt/scratch0/tmp' in _sync_corpus_to_disk() before being extracted. If the said corpus' zip file exceeds 90MB, the download attempt fails.
This change modifies _sync_corpus_to_disk() to use the 'clusterfuzz/bot/inputs/fuzzer-testcases' directory instead of 'BOT_TMPDIR', and also increases the size of 'fuzzer-testcases' tmpfs mount to 1.25GB to accommodate for a large corpus backup zip file and prevent this error.