-
Notifications
You must be signed in to change notification settings - Fork 28
feat(core): add PackCachingBackend for API-expensive backends #447
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Response to @aawsome's FeedbackThank you @aawsome for your detailed and thoughtful feedback! I completely understand and appreciate your concerns about architecture, memory usage, and the trade-offs involved. 1. Acknowledgment of ConcernsYou're absolutely right about the architectural considerations:
2. Why I Chose This ApproachMy goal was actually broader than just fixing my GDrive issue - I wanted to enable the same performance benefits for all cloud services that OpenDAL supports (GDrive, OneDrive, Dropbox, etc.) where API calls are expensive. I deliberately designed this solution as a transparent backend layer to:
3. Offer to Collaborate on Prune OptimizationI would be very happy to collaborate on a proper prune-specific optimization! Your suggestion of:
...sounds like the right long-term approach. I'm willing to work on this, though I should mention I'm still learning Rust (I come from CNC/robotics programming, Python, C#, etc.) - so I might need some guidance on the rustic codebase architecture. 4. Making PackCachingBackend ConfigurableI propose making the cache configurable to address your concerns: Option A: Configuration-based [pack-cache]
enabled = true # or false to disable completely
max_packs = 128 # configurable cache size
max_memory_mb = 6144 # memory limit (e.g., 6 GB)
backends = ["gdrive", "onedrive"] # which backends to cacheOption B: Prune-only activation
Option C: Both
Which approach would you prefer? I'm happy to implement whichever makes the most sense for rustic's architecture. 5. Practical Reality CheckThe performance difference is dramatic:
For GDrive users, this is the difference between "unusable" and "practical". Even with the memory overhead and extra data transfer, it's a massive improvement. SummaryI see this PR as a pragmatic short-term solution that:
I'm committed to working on the proper prune optimization you suggested - this cache can serve as a bridge until we have that in place. What do you think? Should I add configuration options, or would you prefer to discuss the prune-specific approach first? P.S.: I'm still learning Git workflows - sorry for the noise with the previous PRs. I've learned to use clean branches now! 😅 |
|
Hi @mro68! Thanks a lot for the proposals. I must say, I took to opportunity to check out how the prune rewrite could be optimized directly - and already implemented this in #448. |
|
Closing in favor of #448, which provides an elegant algorithmic solution to the same problem. My testing showed both approaches achieve similar performance for Thanks @aawsome for the alternative solution! 👍 Happy New Year! 🎆 Wishing you a smooth transition into 2026 and a healthy, successful year ahead! |
This PR replaces #445 with a clean branch/commit history.
Summary:
PackCachingBackend(LRU in-memory cache) to reduce API calls for expensive backends like Google Drive by caching pack files.Dependency:
location()+ OpenDAL bump).Once feat(backend/opendal): include scheme in location() and bump opendal to 0.55.0 #446 is merged, I will rebase this PR onto
mainto keep the diff focused.Testing:
cargo fmtcargo clippy --all-features -- -D warningscargo test --all-featuresNotes: