Data freshness npm packages

[rachelcdavies]

On checking some popular npm packages I found that the last synced at date seems to be quite a long time ago.

Example https://packages.ecosyste.ms/registries/npmjs.org/packages/electron, last synced 1 year ago.

Thanks for fixing this issue for react yesterday.

I have not gathered a long list of popular packages that are out of date as I am hoping a general solution could be found for well established packages with a high last synced at value.

Please let me know if you need more examples.

Thanks

[andrew]

I've manually kicked off a load of syncs of npm packages that have not been synced recently, I need to work out why they aren't being picked up by the background jobs correctly so going to leave this open for now.

[rachelcdavies]

Thanks Andrew. I can potentially run some queries at my end to try and see if there's some kind of pattern to the ones that haven't been syncing.

[andrew]

Investigating some potential issues with sidekiq-unique-jobs that might have caused this, deploying this fix: b2dfd4d

Will need to go back through and sync lots of things to ensure everything is up to date.

[andrew]

Orphaned unique job locks were accumulating in Redis because the weekly cleanup task (REDIS.del('uniquejobs:digests')) only deleted the sorted set index but left the individual lock keys behind. Over 1.3 million orphaned locks had built up, causing jobs to silently skip because the gem thought they were still locked.

Fixed the cleanup task to use the gem's proper API which removes all associated keys per digest. Configured the reaper to run every 60s instead of the default 10 minutes, enabled the reaper resurrector so it recovers after hard kills during deploys, and reduced lock expiration from 2 hours to 1 hour. Added rake tasks for listing and manually clearing locks.

This should help avoid this problem in the future.

[rachelcdavies]

Thanks Andrew. Things already looking much better at our end. I'll take a deeper look tomorrow and hopefully we can then close this issue.