Skip to content

feat: aggregation of worldpop data#89

Merged
Thuhaa merged 26 commits intomainfrom
feat/aggregation
Jan 20, 2025
Merged

feat: aggregation of worldpop data#89
Thuhaa merged 26 commits intomainfrom
feat/aggregation

Conversation

@Thuhaa
Copy link
Contributor

@Thuhaa Thuhaa commented Dec 13, 2024 •
edited
Loading

The following tasks are done in this PR:

  1. Improvement of the azure storage module
  2. Implementation of core code for aggregation
  3. population sync command
  4. population run-aggregate command

@Thuhaa Thuhaa temporarily deployed to azure container registry December 13, 2024 18:18 — with GitHub Actions Inactive
@Thuhaa Thuhaa temporarily deployed to github container registry December 13, 2024 18:18 — with GitHub Actions Inactive
@Thuhaa Thuhaa temporarily deployed to github container registry December 17, 2024 13:46 — with GitHub Actions Inactive
@Thuhaa Thuhaa temporarily deployed to azure container registry December 17, 2024 13:46 — with GitHub Actions Inactive
@Thuhaa Thuhaa temporarily deployed to github container registry December 17, 2024 19:31 — with GitHub Actions Inactive
@Thuhaa Thuhaa temporarily deployed to azure container registry December 17, 2024 19:31 — with GitHub Actions Inactive
@Thuhaa Thuhaa temporarily deployed to azure container registry December 17, 2024 19:55 — with GitHub Actions Inactive
@Thuhaa Thuhaa temporarily deployed to github container registry December 17, 2024 19:55 — with GitHub Actions Inactive
@Thuhaa Thuhaa temporarily deployed to azure container registry December 17, 2024 20:15 — with GitHub Actions Inactive
@Thuhaa Thuhaa temporarily deployed to github container registry December 17, 2024 20:15 — with GitHub Actions Inactive
@Thuhaa Thuhaa temporarily deployed to github container registry December 18, 2024 07:44 — with GitHub Actions Inactive
@Thuhaa Thuhaa temporarily deployed to azure container registry December 18, 2024 07:44 — with GitHub Actions Inactive
@Thuhaa Thuhaa temporarily deployed to github container registry December 18, 2024 08:10 — with GitHub Actions Inactive
@Thuhaa Thuhaa temporarily deployed to azure container registry December 18, 2024 08:10 — with GitHub Actions Inactive
@Thuhaa Thuhaa temporarily deployed to azure container registry December 20, 2024 12:05 — with GitHub Actions Inactive
@Thuhaa Thuhaa temporarily deployed to github container registry December 20, 2024 12:05 — with GitHub Actions Inactive
@Thuhaa Thuhaa temporarily deployed to github container registry January 16, 2025 13:21 — with GitHub Actions Inactive
@Thuhaa Thuhaa temporarily deployed to azure container registry January 16, 2025 13:21 — with GitHub Actions Inactive
@Thuhaa Thuhaa temporarily deployed to azure container registry January 16, 2025 13:58 — with GitHub Actions Inactive
@Thuhaa Thuhaa temporarily deployed to github container registry January 16, 2025 13:58 — with GitHub Actions Inactive
@Thuhaa Thuhaa temporarily deployed to github container registry January 16, 2025 14:54 — with GitHub Actions Inactive
@Thuhaa Thuhaa temporarily deployed to azure container registry January 16, 2025 14:54 — with GitHub Actions Inactive
@Thuhaa Thuhaa temporarily deployed to azure container registry January 16, 2025 16:59 — with GitHub Actions Inactive
@Thuhaa Thuhaa temporarily deployed to github container registry January 16, 2025 16:59 — with GitHub Actions Inactive
@Thuhaa Thuhaa temporarily deployed to github container registry January 16, 2025 17:32 — with GitHub Actions Inactive
@Thuhaa Thuhaa temporarily deployed to azure container registry January 16, 2025 17:32 — with GitHub Actions Inactive
@Thuhaa Thuhaa temporarily deployed to azure container registry January 16, 2025 17:33 — with GitHub Actions Inactive
@Thuhaa Thuhaa temporarily deployed to github container registry January 16, 2025 17:33 — with GitHub Actions Inactive
@Thuhaa Thuhaa temporarily deployed to azure container registry January 20, 2025 08:36 — with GitHub Actions Inactive
@Thuhaa Thuhaa temporarily deployed to github container registry January 20, 2025 08:36 — with GitHub Actions Inactive
@Thuhaa Thuhaa changed the title fix: cleanup and aggregation feat: aggregation of worlpop data Jan 20, 2025
@Thuhaa Thuhaa changed the title feat: aggregation of worlpop data feat: aggregation of worldpop data Jan 20, 2025
@Thuhaa Thuhaa marked this pull request as ready for review January 20, 2025 08:41
@Thuhaa Thuhaa merged commit d3f2c49 into main Jan 20, 2025
3 checks passed
@Thuhaa Thuhaa deleted the feat/aggregation branch January 20, 2025 08:47
This was referenced Jan 20, 2025
Closed
Closed
@JinIgarashi JinIgarashi linked an issue Jan 24, 2025 that may be closed by this pull request
[Sync command] Duplicate WorldPop source to UNDP Azure #68
Closed
iferencik pushed a commit that referenced this pull request Apr 10, 2025
@Thuhaa
feat: sync and aggregation of worldpop data (#89)
b45bc4a 
* fix: cleanup and aggregation

* WIP: process specific age and sex groups:

* run aggregations

* feat: process all combinations implementation

* fix: run it for all countries with sync command

* remove persistent temp files

* check for if aggregates exist

* fix: update metadata for sizes

* WIP: debugging

* fix: breaking before all blocks are processed

* upload data

* refactor: to upload for all

* refactor: process remaining

* refactor: process remaining

* WIP

* fix: update to use  and  instead of  and

* add download-path argument

* refactor: improve population workflow

* shuffle list

* shuffle age_sex constant

* chunk1

* chunk2

* chunk3

* feat: cleanup
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Sync command] Duplicate WorldPop source to UNDP Azure

1 participant

@Thuhaa