Skip to content

Conversation

@uxairibrar
Copy link
Collaborator

@uxairibrar uxairibrar commented Feb 13, 2025

Closes #47

  • Configured Django-Q task scheduling to automate harvesting at defined intervals.
  • Developed unit tests for task scheduling (test_task_scheduling) and duplicate prevention (test_no_duplicates).
  • Created HarvestingEvent model to track harvesting jobs and their statuses.
  • Modified the Publication model to support additional metadata.
  • Added Django Admin actions for manually triggering harvesting from the admin panel.

@uxairibrar uxairibrar changed the title Harvest data from online OJS instances at fixed interval Harvest data from online OJS instances at fixed interval # 47 Feb 13, 2025
Copy link
Member

@nuest nuest left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see any code related to Django-Q in this PR.

Could you please add a few illustrating screenshots how the interaction with the admin UI works? Thanks!

@uxairibrar
Copy link
Collaborator Author

I don't see any code related to Django-Q in this PR.

Could you please add a few illustrating screenshots how the interaction with the admin UI works? Thanks!

So, I wanted to automate this scheduling process, so I put adding scheduler on save in the models file. However, now I have also added a seperate admin action for it.

@uxairibrar uxairibrar marked this pull request as draft March 19, 2025 00:46
@nuest
Copy link
Member

nuest commented Mar 19, 2025

@uxairibrar Tests are working again and branch is updated from main.

@nuest nuest changed the title Harvest data from online OJS instances at fixed interval # 47 Harvest data from online OJS instances at fixed interval Mar 26, 2025
@uxairibrar uxairibrar marked this pull request as ready for review April 4, 2025 13:46
nuest
nuest previously approved these changes Apr 4, 2025
@nuest nuest dismissed their stale review April 4, 2025 15:24

Tests are failing.

Copy link
Member

@nuest nuest left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

@nuest nuest left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll go ahead and merge - thanks for the efforts.

I do think though that we will have to improve the tests over time, as not all conditions are tested, for example:

  • You check for "greater than zero" many publications, when you should know the exact number and test for that.
  • You test only that the DOI starts with "10.", when you know the exact DOI. What if the regex if incorrect towards the end of the DOI?

@nuest nuest merged commit cbf4788 into main Apr 8, 2025
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Harvest data from online OJS instances at fixed interval

3 participants