3895 check against oa start date

[RK206]

• Issue: https://github.com/DOAJ/doajPM/issues/3895

---

Restrict upload of articles before OA start date

This PR checks the publication date of the article and checks against OA start date of the Journal. If the article publication date is before than the OA start date, fail the upload process

This PR...

• has scripts to run
• has migrations to run
• adds new infrastructure
• changes the CI pipeline
• affects the public site
• affects the editorial area
• affects the publisher area
• affects the monitoring

Developer Checklist

Developers should review and confirm each of these items before requesting review

• Code meets acceptance criteria from issue
• Unit tests are written and all pass
• User Test Scripts (if required) are written and have been run through
• Project's coding standards are met
  • No deprecated methods are used
  • No magic strings/numbers - all strings are in constants or messages files
  • ES queries are wrapped in a Query object rather than inlined in the code
  • Where possible our common library functions have been used (e.g. dates manipulated via dates)
  • Cleaned up commented out code, etc
  • Urls are constructed with url_for not hard-coded
• Code documentation and related non-code documentation has all been updated
  • Core model documentation has been added to if needed: https://docs.google.com/spreadsheets/d/1lun2S9vwGbyfy3WjIjgXBm05D-3wWDZ4bp8xiIYfImM/edit
  • Events and consumers documentation has been added if needed: https://docs.google.com/spreadsheets/d/1oIeG5vg-blm2MZCE-7YhwulUlSz6TOUeY8jAftdP9JE/edit
• Migation has been created and tested
• There is a recent merge from develop

Reviewer Checklist

Reviewers should review and confirm each of these items before approval
If there are multiple reviewers, this section should be duplicated for each reviewer

• Code meets acceptance criteria from issue
• Unit tests are written and all pass
• User Test Scripts (if required) are written and have been run through
• Project's coding standards are met
  • No deprecated methods are used
  • No magic strings/numbers - all strings are in constants or messages files
  • ES queries are wrapped in a Query object rather than inlined in the code
  • Where possible our common library functions have been used (e.g. dates manipulated via dates)
  • Cleaned up commented out code, etc
  • Urls are constructed with url_for not hard-coded
• Code documentation and related non-code documentation has all been updated
  • Core model documentation has been added to if needed: https://docs.google.com/spreadsheets/d/1lun2S9vwGbyfy3WjIjgXBm05D-3wWDZ4bp8xiIYfImM/edit
  • Events and consumers documentation has been added if needed: https://docs.google.com/spreadsheets/d/1oIeG5vg-blm2MZCE-7YhwulUlSz6TOUeY8jAftdP9JE/edit
• Migation has been created and tested
• There is a recent merge from develop

Testing

Feature testing:

https://doaj.github.io/doaj-docs/feature/3895_check_against_oa_start_date/testbook/index.html#article_xml_upload/doaj_xml/upload_a_file_before_oa_start_date_of_the_journal

Regression testing:

Article XML upload tests:

https://doaj.github.io/doaj-docs/feature/3895_check_against_oa_start_date/testbook/index.html#article_xml_upload/doaj_xml

Deployment

What deployment considerations are there? (delete any sections you don't need)

Configuration changes

What configuration changes are included in this PR, and do we need to set specific values for production

Scripts

What scripts need to be run from the PR (e.g. if this is a report generating feature), and when (once, regularly, etc).

Migrations

What migrations need to be run to deploy this

Monitoring

What additional monitoring is required of the application as a result of this feature

New Infrastructure

What new infrastructure does this PR require (e.g. new services that need to run on the back-end).

Continuous Integration

What CI changes are required for this

[richard-jones]

@RK206 I had a couple of problems testing this:

• The unit tests don't pass, I uploaded the list of failures for you to look at; looks like a fixtures thing
• I couldn't run through the user test. I tried creating a journal with the right issns using testbook (updated the instructions), and uploading the file, but I got a different error to the one we expected.

[richard-jones]

I ran the whole test suite and got a number of failures, not related to this file, I think there's a general problem with the fixtures for some of the tests:

FAILED test_bll_article_batch_create_article.py::TestBLLArticleBatchCreateArticle::test_01_batch_create_article_033_34 - portality.bll.exceptions.ArticleNotAcceptable: Article(s) 'Article Title' cannot be uploaded before OA start date of the Journal
FAILED test_bll_article_batch_create_article.py::TestBLLArticleBatchCreateArticle::test_01_batch_create_article_089_90 - portality.bll.exceptions.ArticleNotAcceptable: Article(s) 'Article Title' cannot be uploaded before OA start date of the Journal
FAILED api_tests/test_api_crud_article.py::TestCrudArticle::test_02_create_article_success - portality.api.common.Api400Error: Article(s) '{title}' cannot be uploaded before OA start date of the Journal
FAILED api_tests/test_api_crud_article.py::TestCrudArticle::test_08_update_article_success - portality.api.common.Api400Error: Article(s) '{title}' cannot be uploaded before OA start date of the Journal
FAILED api_tests/test_api_crud_article.py::TestCrudArticle::test_09_update_article_fail - portality.api.common.Api400Error: Article(s) '{title}' cannot be uploaded before OA start date of the Journal
FAILED api_tests/test_api_crud_article.py::TestCrudArticle::test_10_delete_article_success - portality.api.common.Api400Error: Article(s) '{title}' cannot be uploaded before OA start date of the Journal
FAILED api_tests/test_api_crud_article.py::TestCrudArticle::test_11_delete_article_fail - portality.api.common.Api400Error: Article(s) '{title}' cannot be uploaded before OA start date of the Journal
FAILED api_tests/test_api_crud_article.py::TestCrudArticle::test_12_too_many_keywords - portality.api.common.Api400Error: Article(s) '{title}' cannot be uploaded before OA start date of the Journal
FAILED api_tests/test_api_bulk_article_class.py::TestBulkArticle::test_01_create_articles_success - portality.api.common.Api400Error: Article(s) 'Article Title' cannot be uploaded before OA start date of the Journal
FAILED api_tests/test_api_crud_article.py::TestCrudArticle::test_14_test_via_endpoint - assert 400 == 201
FAILED api_tests/test_api_crud_article.py::TestCrudArticle::test_15_no_redirects - assert 400 == 201
FAILED api_tests/test_api_bulk_article_class.py::TestBulkArticle::test_04_delete_article_success - portality.api.common.Api400Error: Article(s) 'Article Title' cannot be uploaded before OA start date of the Journal
FAILED api_tests/test_api_bulk_article_class.py::TestBulkArticle::test_05_delete_articles_fail - portality.api.common.Api400Error: Article(s) 'Article Title' cannot be uploaded before OA start date of the Journal
FAILED api_tests/test_api_crud_article.py::TestCrudArticle::test_16_no_redirects - assert 400 == 201
FAILED test_bll_article_create_article.py::TestBLLArticleCreateArticle::test_01_create_article_0_1 - portality.bll.exceptions.ArticleBeforeOAStartDate: Article(s) '{title}' cannot be uploaded before OA start date of the Journal
FAILED test_bll_article_create_article.py::TestBLLArticleCreateArticle::test_01_create_article_2_3 - portality.bll.exceptions.ArticleBeforeOAStartDate: Article(s) '{title}' cannot be uploaded before OA start date of the Journal
FAILED api_tests/test_api_bulk_article_class.py::TestBulkArticle::test_06_test_via_endpoint - TimeoutError: fail to meet the condition within the timeout period.
FAILED test_bll_article_create_article.py::TestBLLArticleCreateArticle::test_01_create_article_8_9 - portality.bll.exceptions.ArticleBeforeOAStartDate: Article(s) '{title}' cannot be uploaded before OA start date of the Journal
FAILED application_processors/test_application_processor_emails.py::TestUpdateRequestReviewEmails::test_01_maned_review_emails - assert 1 == 3
FAILED api_tests/test_api_bulk_article_class.py::TestBulkArticle::test_07_v1_no_redirects - TimeoutError: fail to meet the condition within the timeout period.
FAILED test_paths.py::TestPaths::test_get_project_root - AssertionError: assert 'doaj3' == 'doaj'
FAILED test_create_article.py::TestCreateOrUpdateArticle::test_00_no_doi_and_url_changed - portality.bll.exceptions.ArticleBeforeOAStartDate: Article(s) '{title}' cannot be uploaded before OA start date of the Journal
FAILED api_tests/test_api_bulk_article_class.py::TestBulkArticle::test_08_v2_no_redirects - TimeoutError: fail to meet the condition within the timeout period.
FAILED test_update_article.py::TestCreateOrUpdateArticle::test_00_no_doi_and_url_changed - portality.bll.exceptions.ArticleBeforeOAStartDate: Article(s) '{title}' cannot be uploaded before OA start date of the Journal
FAILED test_create_article.py::TestCreateOrUpdateArticle::test_01_new_doi_new_url - portality.bll.exceptions.ArticleBeforeOAStartDate: Article(s) '{title}' cannot be uploaded before OA start date of the Journal
FAILED test_update_article.py::TestCreateOrUpdateArticle::test_01_new_doi_new_url - portality.bll.exceptions.ArticleBeforeOAStartDate: Article(s) '{title}' cannot be uploaded before OA start date of the Journal
FAILED test_create_article.py::TestCreateOrUpdateArticle::test_04_old_doi_new_url - portality.bll.exceptions.ArticleBeforeOAStartDate: Article(s) '{title}' cannot be uploaded before OA start date of the Journal
FAILED test_update_article.py::TestCreateOrUpdateArticle::test_04_old_doi_new_url - portality.bll.exceptions.ArticleBeforeOAStartDate: Article(s) '{title}' cannot be uploaded before OA start date of the Journal
FAILED test_create_article.py::TestCreateOrUpdateArticle::test_05_new_doi_old_url - portality.bll.exceptions.ArticleBeforeOAStartDate: Article(s) '{title}' cannot be uploaded before OA start date of the Journal
FAILED test_update_article.py::TestCreateOrUpdateArticle::test_05_new_doi_old_url - portality.bll.exceptions.ArticleBeforeOAStartDate: Article(s) '{title}' cannot be uploaded before OA start date of the Journal
FAILED test_admin_edit_metadata.py::TestAdminEditMetadata::test_02_update_article_metadata_no_url_fulltext - portality.bll.exceptions.ArticleBeforeOAStartDate: Article(s) '{title}' cannot be uploaded before OA start date of the Journal
FAILED test_admin_edit_metadata.py::TestAdminEditMetadata::test_03_update_fulltext_valid - portality.bll.exceptions.ArticleBeforeOAStartDate: Article(s) '{title}' cannot be uploaded before OA start date of the Journal
FAILED test_admin_edit_metadata.py::TestAdminEditMetadata::test_05_update_doi_valid - portality.bll.exceptions.ArticleBeforeOAStartDate: Article(s) '{title}' cannot be uploaded before OA start date of the Journal
FAILED api_tests/test_api_crud_returnvalues.py::TestCrudReturnValues::test_03_articles_crud - AssertionError: 400
FAILED event_consumers/test_update_request_publisher_accepted_notify.py::TestUpdateRequestPublisherAcceptedNotify::test_should_consume - assert False
FAILED event_consumers/test_update_request_publisher_assigned_notify.py::TestApplicationPublisherAssignedNotify::test_should_consume - assert False
FAILED event_consumers/test_update_request_publisher_rejected_notify.py::TestUpdateRequestPublisherRejectedNotify::test_should_consume - assert False
FAILED event_consumers/test_update_request_publisher_submitted_notify.py::TestUpdateRequestPublisherSubmittedNotify::test_should_consume - AssertionError: assert False
FAILED test_tasks_harvest.py::TestHarvester::test_harvest - TimeoutError: fail to meet the condition within the timeout period.

[RK206]

Sorry I missed to test other testcases which are dependent on the issue.
I fixed those testcases now. There are few more testcases that are failing but those are not related to this issue. So didn't attempt to fix them.
Ready to test again

[richard-jones]

I made a small adjustment to the error text, otherwise this was all good, ready for test