Reduce RAM usage for large CSV exports. #254
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
this reduces the RAM usage significantly when exporting big tables by using a temporary file for storage instead of dumping everything to RAM
mkangia
approved these changes
Nov 7, 2025
| for row in table.rows: | ||
| writer.writerow(row) | ||
|
|
||
| tempfile.flush() # Flush buffer before zip reads tempfile |
Contributor
There was a problem hiding this comment.
is this to ensure the file is written?
Contributor
Author
There was a problem hiding this comment.
Yes, exactly, otherwise the zip file can contain an empty CSV file.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a rebase of PR #243 with an additional commit to ensure data is written to disk before creating the zip archive.
The change uses a named temp file instead of in-memory StringIO when creating the CSV export. (The output is still a zipped CSV file.)
This change is covered by test
tests/test_writers.py::TestWriters::test_CsvTableWriter