IA-4360: Make import of users in bulk easier

[edil3ra]

What problem is this PR solving?

Make user import csv in bulk, add optional default configuration for empty columns

Related JIRA tickets

IA-4360

Changes

• iaso/api/profiles/bulk_create_users.py
  • Split create endpoint into 3 phases: csv extraction, fields validations, bulk creation
  • Add bulk_configuration parameter for default values (permissions, projects, user_roles, orgunit, language, organization, teams)
• iaso/tasks/bulk_create_users_email.py
  • New task for sending multiple emails invitations
• iaso/tests/test_create_users_from_csv.py
  • Add tests for bulk_configuration with 3 scenarios
  • Add tests for email invitation behavior (with/without password)

How to test

I think the simplest way to test is to use the frontend branch (still in progress): https://github.com/BLSQ/iaso/tree/IA-4360-make-import-of-users-in-bulk-easier-front

1. Happy Path - IA4360_happy_path.csv

username,password,email,first_name,last_name,orgunit,orgunit__source_ref,profile_language,dhis2_id,organization,permissions,user_roles,projects,teams,phone_number,editable_org_unit_types
IA4360_happy_user1,SecurePass123!,happy1@test.com,Happy,UserOne,IA4360_kinshasa,,en,IA4360_HAPPY001,WHO,iaso_forms,IA4360_manager,IA4360_project_alpha,IA4360_health_team,+243100000001,
IA4360_happy_user2,SecurePass456!,happy2@test.com,Happy,UserTwo,IA4360_hospital_a,,fr,IA4360_HAPPY002,MSF,iaso_submissions,IA4360_fieldworker,IA4360_project_beta,IA4360_beta_team,+243100000002,
IA4360_happy_user3,SecurePass789!,happy3@test.com,Happy,UserThree,467,,en,IA4360_HAPPY003,UNICEF,"iaso_forms,iaso_submissions",IA4360_manager,"IA4360_project_alpha,IA4360_project_beta","IA4360_health_team,IA4360_field_team",+243100000003,48

Expected Response: 200 OK

{"Accounts created": 3}

---

2. Email Invitation Only - IA4360_email_invitation_only.csv

username,password,email,first_name,last_name,orgunit,orgunit__source_ref,profile_language,dhis2_id,organization,permissions,user_roles,projects,teams,phone_number,editable_org_unit_types
IA4360_email_only1,,invite1@test.com,Invite,UserOne,IA4360_kinshasa,,en,IA4360_EMAIL001,WHO,iaso_forms,IA4360_manager,IA4360_project_alpha,,+243300000001,
IA4360_email_only2,,invite2@test.com,Invite,UserTwo,IA4360_hospital_a,,fr,IA4360_EMAIL002,MSF,iaso_submissions,IA4360_fieldworker,IA4360_project_beta,,+243300000002,
IA4360_email_only3,,invite3@test.com,Invite,UserThree,IA4360_kinshasa,,en,IA4360_EMAIL003,UNICEF,,,,,,

Expected Response: 200 OK

{"Accounts created": 3}

(Email invitations sent in background task)

---

3. Validation Failures - [IA4360_validation_failures.csv]https://bluesquare.atlassian.net/issues?jql=assignee%20%3D%20currentUser()%20AND%20project%3D%27IA%27%20AND%20statusCategory%20!%3D%203&selectedIssue=IA-4360

username,password,email,first_name,last_name,orgunit,orgunit__source_ref,profile_language,dhis2_id,organization,permissions,user_roles,projects,teams,phone_number,editable_org_unit_types
IA4360_existing_user,ValidPass123!,unique1@test.com,Existing,Username,IA4360_kinshasa,,en,IA4360_FAIL001,TestOrg,iaso_forms,IA4360_manager,IA4360_project_alpha,,+243200000001,
IA4360_fail_weak_pass,123,weak@test.com,Weak,Password,IA4360_kinshasa,,en,IA4360_FAIL002,TestOrg,iaso_forms,IA4360_manager,IA4360_project_alpha,,+243200000002,
IA4360_fail_bad_email,ValidPass123!,not-an-email,Bad,Email,IA4360_kinshasa,,en,IA4360_FAIL003,TestOrg,iaso_forms,IA4360_manager,IA4360_project_alpha,,+243200000003,
IA4360_fail_bad_perm,ValidPass123!,badperm@test.com,Bad,Permission,IA4360_kinshasa,,en,IA4360_FAIL004,TestOrg,non_existent_permission,IA4360_manager,IA4360_project_alpha,,+243200000004,
IA4360_fail_bad_project,ValidPass123!,badproject@test.com,Bad,Project,IA4360_kinshasa,,en,IA4360_FAIL005,TestOrg,iaso_forms,IA4360_manager,NonExistentProject,,+243200000005,
IA4360_fail_bad_role,ValidPass123!,badrole@test.com,Bad,Role,IA4360_kinshasa,,en,IA4360_FAIL006,TestOrg,iaso_forms,NonExistentRole,IA4360_project_alpha,,+243200000006,
IA4360_fail_dup_dhis2,ValidPass123!,dupdhis2@test.com,Duplicate,DHIS2,IA4360_kinshasa,,en,IA4360_EXISTING_DHIS2,TestOrg,iaso_forms,IA4360_manager,IA4360_project_alpha,,+243200000007,
IA4360_fail_no_auth,,,No,Auth,IA4360_kinshasa,,en,IA4360_FAIL008,TestOrg,iaso_forms,IA4360_manager,IA4360_project_alpha,,+243200000008,
IA4360_fail_multi_perm,ValidPass123!,multiperm@test.com,Multi,BadPerms,IA4360_kinshasa,,en,IA4360_FAIL010,TestOrg,"fake_perm1,fake_perm2,iaso_forms",IA4360_manager,IA4360_project_alpha,,+243200000010,
IA4360_fail_multi_proj,ValidPass123!,multiproj@test.com,Multi,BadProjects,IA4360_kinshasa,,en,IA4360_FAIL011,TestOrg,iaso_forms,IA4360_manager,"FakeProject1,IA4360_project_alpha,FakeProject2",,+243200000011,
,ValidPass123!,nousername@test.com,No,Username,IA4360_kinshasa,,en,IA4360_FAIL016,TestOrg,iaso_forms,IA4360_manager,IA4360_project_alpha,,+243200000016,
IA4360_fail_bad_team,ValidPass123!,badteam@test.com,Bad,Team,IA4360_kinshasa,,en,IA4360_FAIL017,TestOrg,iaso_forms,IA4360_manager,IA4360_project_alpha,NonExistentTeam,+243200000017,
IA4360_fail_bad_org,ValidPass123!,badorg@test.com,Bad,Organization,IA4360_kinshasa,,en,IA4360_FAIL018,,iaso_forms,IA4360_manager,IA4360_project_alpha,IA4360_health_team,+243200000018,

Expected Response: 400 Bad Request

{
  "error": {
    "file": {
      "csv_validation_errors": [
        {"row": 2, "errors": {"password": "This password is too short..."}},
        {"row": 3, "errors": {"email": "Enter a valid email address."}},
        {"row": 4, "errors": {"permissions": "Invalid permissions: non_existent_permission"}},
        {"row": 5, "errors": {"projects": "Invalid projects: NonExistentProject"}},
        {"row": 6, "errors": {"user_roles": "Invalid user roles: NonExistentRole"}},
        {"row": 8, "errors": {"password": "Either password or email is required..."}},
        {"row": 9, "errors": {"permissions": "Invalid permissions: fake_perm1, fake_perm2"}},
        {"row": 10, "errors": {"projects": "Invalid projects: FakeProject1, FakeProject2"}},
        {"row": 11, "errors": {"username": "This field may not be blank."}},
        {"row": 12, "errors": {"teams": "Invalid teams: NonExistentTeam"}}
      ]
    }
  }
}

---

4. Password No Email - IA4360_password_no_email.csv

username,password,email,first_name,last_name,orgunit,orgunit__source_ref,profile_language,dhis2_id,organization,permissions,user_roles,projects,teams,phone_number,editable_org_unit_types
IA4360_pwd_only1,SecurePass123!,,Password,Only1,IA4360_kinshasa,,en,IA4360_PWDONLY001,WHO,iaso_forms,IA4360_manager,IA4360_project_alpha,,+243500000001,
IA4360_pwd_only2,SecurePass456!,,Password,Only2,IA4360_hospital_a,,fr,IA4360_PWDONLY002,MSF,iaso_submissions,IA4360_fieldworker,IA4360_project_beta,,+243500000002,
IA4360_pwd_only3,SecurePass789!,,Password,Only3,IA4360_kinshasa,,en,IA4360_PWDONLY003,UNICEF,"iaso_forms,iaso_submissions",IA4360_manager,"IA4360_project_alpha,IA4360_project_beta",,+243500000003,

Expected Response: 200 OK

{"Accounts created": 3}

[quang-le]

Emails can be reused across users (eg: multi-account users), so this should not be considered an error

[quang-le]

Same as email

[quang-le]

As discussed change the model and serializer to use foreign key/many to one relationships when possible

[edil3ra]

many to many relationship are updated in model: https://github.com/BLSQ/iaso/pull/2701/changes/BASE..986392b5c5ec79998964ca4d208ea55c91e1f4d3#diff-8fb5b58193ea0350e70424248362015e84b5c750ae0193edb9ff2e0401619b01
and serializers receive a list of intgers
the only field that is still a CharField is the organization because it's directly attached to profile model