diff --git a/README.md b/README.md index 87dac9c..f07cf1c 100644 --- a/README.md +++ b/README.md @@ -1,272 +1,34 @@ -CommCare Sync -============= +CommCare Data Pipeline +====================== + +CommCare Data Pipeline simplifies the setup and management of your +CommCare data pipeline. + +It is a self-hosted, standalone web application designed to manage a +CommCare “data warehouse” over the command-line +[CommCare Data Export Tool](https://dimagi.atlassian.net/wiki/spaces/commcarepublic/pages/2143955952/CommCare+Data+Export+Tool+DET). -CommCare Sync simplifies the setup and management of your CommCare data -pipeline. It is a self-hosted, standalone web application designed to -manage a CommCare “data warehouse” over the command-line -[CommCare Data Export Tool](https://dimagi.atlassian.net/wiki/spaces/commcarepublic/pages/2143955952/CommCare+Data+Export+Tool+DET?atl_f=PAGETREE). This turnkey solution allows you to export data from CommCare and store it in a local or cloud-based database, including MySQL, PostgreSQL, -Amazon RDS, GCP Cloud SQL, and Azure SQL Database. With CommCare Sync, -you can utilise these key features: +Amazon RDS, GCP Cloud SQL, and Azure SQL Database. With CommCare Data +Pipeline, you can utilise these key features: - **Automated Configuration:** Generate a Data Export Tool (DET) configuration file directly from CommCare. -- **Seamless Integration:** Connect CommCare Sync to your CommCare - project space(s) and database(s). +- **Seamless Integration:** Connect CommCare Data Pipeline to your + CommCare project space(s) and database(s). -- **Scheduled Data Syncs:** Upload a DET configuration file to automate - data transfers from CommCare to your database on a defined +- **Scheduled Data Exports:** Upload a DET configuration file to + automate data transfers from CommCare to your database on a defined schedule. -- **Sync Monitoring:** Track and manage data sync activities through - CommCare Sync’s built-in log feature. - -**This documentation shows you how to set up a self-hosted version of -CommCare Sync using the source code.** For help installing and managing -production environments, you can follow the -[documentation](https://commcare-sync-ansible.readthedocs.io/en/latest/) -of the commcare-sync ansible repository. - - -Production Setup ----------------- - -### Configuration Steps - -To configure CommCare Sync: - -1. In CommCare HQ Data page, create a form or case export - -2. Download the DET config file - -3. Open your Excel DET config file to see the fields from your export - with the option of mapping specific data types. If you aren't - transforming your data, there's no step needed here. - -4. Open CommCare Sync, create a new account (instructions below) - -5. Add a project by pasting your CommCare project space name - -6. Add your database via the Admin Site (can be any available database) - -7. Add an export from your new project, and add your database and your - config file you downloaded in step 2 - -8. Run export. This applies the configuration file to do an initial sync - of all the data from your CommCare project space. - -9. View the log to see more info - like to confirm how much data was - pulled in - -10. Connect your BI tool of choice, and start exploring the data - -### Note for projects syncing data from multiple CommCare project spaces: - -The process described above is for connecting a single CommCare project -space. If you are connecting data from multiple CommCare project -spaces, you need to add each project space as a Project in CommCare -Sync (step 5), and repeat the process of downloading each DET config -file per project space (step 2), to then each be uploaded to CommCare -Sync (steps 7 & 8). (IMPORTANT: there is a new feature release that -will allow applying the same DET config file to multiple project spaces -in the CommCare sync tool). - -### Download your DET config file from CommCare - -- Download or create a - [Data Export Tool](https://dimagi.atlassian.net/wiki/x/8CvKfw) config - file. - -- The easiest way to create these is to start with a normal export - configuration on HQ and have it generated. See here for more details: - [CommCare Data Export Tool (DET) | Creating an Excel Query File in CommCare HQ](https://dimagi.atlassian.net/wiki/spaces/commcarepublic/pages/2143955952/CommCare+Data+Export+Tool+DET#Creating-an-Excel-Query-File-in-CommCare-HQ) - -- Edit the DET file using the “best practices” below - -### Create a CommCare Sync account - -Ask a site admin to create an account for you and share credentials, -then change your password. - -### Sync your data - -To sync data, follow the following steps: - -1. If you haven’t already, add the CommCare project space in - the “CommCare Setup” tab. - -2. If you haven’t already, add a CommCare account that has access to the - project space. - -3. Note: If you have a privileged account, it's recommnded that you - create a service account in the target project space instead with - minimal permissions and use that for data syncing. - -4. Add the export from the “Exports” tab. - -5. On the export details page, click “run”. - -6. When the run completes, view the logs to confirm it ran - successfully. - -Data will be updated for all exports on a schedule (currently every 12 -hours, managed by a system admin). - -### Data Export Tool Best Practices - -Some recommendations for modifying the DET config files downloaded from HQ: - -1. Double check the name of the sheet (tab) in your DET config workbook - to be something specific to your project / case type. The tab's name, - not the .xlsx file name, will be used as the table name in SQL. The - default of “Cases” or “Forms” should not be used, but instead - changed to e.g. “covid_19_index_cases” - -2. Add a “str2date” mapping to any date properties and fields. This will - make it easier to use them in various BI tools. - -3. There is a subtle difference between the 'Extra Arguments' field in a - CommCare Sync Export and a CommCare Data Export Tool parameter. If an - argument in CommCare Sync takes multiple parameters - ('since' and 'until,' for example) it must be formatted - like: --until=2020-09-30 - -### Adding Databases - -Databases can be added by site admins by using the "databases" link in -CommCare Sync sidebar navigation. The database may need to also be -separately created by a system admin on the server. - -### Administration - -System administration is documented in our -[production environment documentation](https://commcare-sync-ansible.readthedocs.io/en/latest/system-administration.html). - - -Developer Setup - Docker ------------------------- - -The easiest way to get up and running is with -[Docker](https://www.docker.com/). - -Just [install Docker](https://www.docker.com/get-started) and -[Docker Compose](https://docs.docker.com/compose/install/) and then -run: - -```shell -make init -``` - -This will spin up a database, web worker, celery worker, and Redis -broker and run your migrations. - -You go to [localhost:8000](http://localhost:8000/) to view the app. - -### Using the Makefile - -You can run `make` to see other helper functions, and you can view the -source of the file in case you need to run any specific commands. - -For example, you can run management commands in containers using the -same method used in the `Makefile`. e.g. - -```shell -docker-compose exec web uv run manage.py createsuperuser -``` - - -Developer Setup - Native ------------------------- - -You can also install/run the app directly on your OS using the -instructions below. - -### Prerequisites - -- PostgreSQL (or other SQL DB, but you'll have to edit the settings if - not Postgres) - -- Redis - -Install uv: -```shell -curl -LsSf https://astral.sh/uv/install.sh | sh -``` - -Create a virtual environment and install dependencies: -```shell -uv sync --dev -``` - -To activate the virtual environment: -```shell -source .venv/bin/activate -``` - -Create a database: - -```shell -psql -U -h localhost -p 5432 -CREATE DATABASE commcare_sync; -\q -./manage.py migrate -``` - -### Running server - -```shell -./manage.py runserver -``` - -### Building front-end - -To build JavaScript and CSS files, first install npm packages: - -```shell -npm install -``` - -Then to build (and watch for changes locally) just run: - -```shell -npm run dev-watch -``` - -To build the files for production run: - -```shell -npm run build -``` - -### Running Celery - -Celery is used to run background tasks, including all the -commcare-export runs as well as the scheduled tasks. To run it you can -use: - -```shell -celery -A commcare_sync worker -l info -``` - -Or to also include periodic tasks to run all exports on a schedule: - -```shell -celery -A commcare_sync worker -l info -B -``` - -### Running Tests - -To run tests: - -```shell -./manage.py test -``` - +- **Export Monitoring:** Track and manage data export activities through + CommCare Data Pipeline’s built-in log feature. -Deployment ----------- +See [the documentation](docs/index.md) for details on installing a +development environment and setting up a test data pipeline. -To set up a production server, see -[commcare-sync-ansible](https://github.com/dimagi/commcare-sync-ansible) +See the +[**commcare-sync-ansible** documentation](https://commcare-sync-ansible.readthedocs.io/en/latest/) +for details on installing and managing a production environment. diff --git a/docs/config.md b/docs/config.md new file mode 100644 index 0000000..3e170eb --- /dev/null +++ b/docs/config.md @@ -0,0 +1,106 @@ +Configuring CommCare Data Pipeline +---------------------------------- + +### Configuration Steps + +To configure CommCare Data Pipeline: + +1. In CommCare HQ Data page, create a form or case export + +2. Download the DET config file + +3. Open your Excel DET config file to see the fields from your export + with the option of mapping specific data types. If you aren't + transforming your data, there's no step needed here. + +4. Open CommCare Data Pipeline, create a new account (instructions below) + +5. Add a project by pasting your CommCare project space name + +6. Add your database via the Admin Site (can be any available database) + +7. Add an export from your new project, and add your database and your + config file you downloaded in step 2 + +8. Run export. This applies the configuration file to do an initial + export of all the data from your CommCare project space. + +9. View the log to see more info - like to confirm how much data was + pulled in + +10. Connect your BI tool of choice, and start exploring the data + +### Note for projects exporting data from multiple CommCare project spaces: + +The process described above is for connecting a single CommCare project +space. If you are connecting data from multiple CommCare project +spaces, you need to add each project space as a Project in CommCare +Data Export (step 5). Repeat the process of downloading each DET +config file per project space (step 2). Then upload them to CommCare +Data Pipeline (steps 7 & 8). + +### Download your DET config file from CommCare + +- Download or create a + [Data Export Tool](https://dimagi.atlassian.net/wiki/x/8CvKfw) + config file. + +- The easiest way to create one is to start with a normal export + configuration on HQ and have it generated. See here for more details: + [CommCare Data Export Tool (DET) | Creating an Excel Query File in CommCare HQ](https://dimagi.atlassian.net/wiki/spaces/commcarepublic/pages/2143955952/CommCare+Data+Export+Tool+DET#Creating-an-Excel-Query-File-in-CommCare-HQ) + +- Edit the DET file using the “best practices” below + +### Create a CommCare Data Pipeline account + +Ask a site admin to create an account for you and share credentials, +then change your password. + +### Export your data + +To export data, follow the following steps: + +1. If you haven’t already, add the CommCare project space in + the “CommCare Setup” tab. + +2. If you haven’t already, add a CommCare account that has access to the + project space. + +3. Note: If you have a privileged account, it's recommended that you + create a service account in the target project space instead with + minimal permissions and use that for data exporting. + +4. Add the export from the “Exports” tab. + +5. On the export details page, click “run”. + +6. When the run completes, view the logs to confirm it ran + successfully. + +Data will be updated for all exports on a schedule (currently every 12 +hours, managed by a system admin). + +### Data Export Tool Best Practices + +Some recommendations for modifying the DET config files downloaded from HQ: + +1. Double check the name of the sheet (tab) in your DET config workbook + to be something specific to your project / case type. The tab's name, + not the .xlsx filename, will be used as the table name in SQL. The + default of “Cases” or “Forms” should not be used, but instead + changed to e.g. “covid_19_index_cases” + +2. Add a “str2date” mapping to any date properties and fields. This will + make it easier to use them in various BI tools. + +3. There is a subtle difference between the 'Extra Arguments' field in a + CommCare Data Pipeline Export and a CommCare Data Export Tool + parameter. If an argument in CommCare Data Pipeline takes multiple + parameters ('since' and 'until', for example) it must be formatted + like: --until=2020-09-30 + +### Adding Databases + +Databases can be added by site admins by using the "databases" link in +CommCare Data Pipeline sidebar navigation. The database may need to also +be separately created by a system admin on the server. diff --git a/docs/deploy_prod.md b/docs/deploy_prod.md new file mode 100644 index 0000000..04c1697 --- /dev/null +++ b/docs/deploy_prod.md @@ -0,0 +1,5 @@ +Deploying code changes +---------------------- + +Managing a production environment is documented in our +[production environment documentation](https://commcare-sync-ansible.readthedocs.io/en/latest/system-administration.html). diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000..a4778e4 --- /dev/null +++ b/docs/index.md @@ -0,0 +1,9 @@ +CommCare Data Pipeline +====================== + +* [Configuring CommCare Data Pipeline](config.md) +* [Setting up a test data pipeline](test_pipeline.md) +* [Installing a development environment with Docker](install_docker.md) +* [Installing natively](install_native.md) +* [Installing a production environment](install_prod.md) +* [Deploying code changes](deploy_prod.md) diff --git a/docs/install_docker.md b/docs/install_docker.md new file mode 100644 index 0000000..d56be86 --- /dev/null +++ b/docs/install_docker.md @@ -0,0 +1,31 @@ +Installing a development environment with Docker +------------------------------------------------ + +The easiest way to get up and running is with +[Docker](https://www.docker.com/). + +Just [install Docker](https://www.docker.com/get-started) and +[Docker Compose](https://docs.docker.com/compose/install/) and then +run: + +```shell +make init +``` + +This will spin up a database, web worker, celery worker, and Redis +broker and run your migrations. + +Go to http://localhost:8000/ to view the app. + + +### Using the Makefile + +You can run `make` to see other helper functions, and you can view the +source of the file in case you need to run any specific commands. + +For example, you can run management commands in containers using the +same method used in the `Makefile`. e.g. + +```shell +docker-compose exec web uv run manage.py createsuperuser +``` diff --git a/docs/install_native.md b/docs/install_native.md new file mode 100644 index 0000000..11bf923 --- /dev/null +++ b/docs/install_native.md @@ -0,0 +1,85 @@ +Installing natively +------------------- + +You can also install CommCare Data Pipeline directly on your operating +system using the instructions below. + +### Prerequisites + +- PostgreSQL (or other SQL database, but you'll have to edit the + settings if not Postgres) + +- Redis + +Install uv: +```shell +curl -LsSf https://astral.sh/uv/install.sh | sh +``` + +Create a virtual environment and install dependencies: +```shell +uv sync --dev +``` + +To activate the virtual environment: +```shell +source .venv/bin/activate +``` + +Create a database: + +```shell +psql -U -h localhost -p 5432 +CREATE DATABASE commcare_sync; +\q +./manage.py migrate +``` + +### Running the server + +```shell +./manage.py runserver +``` + +### Building the front-end + +To build JavaScript and CSS files, first install npm packages: + +```shell +npm install +``` + +Then to build (and watch for changes locally) just run: + +```shell +npm run dev-watch +``` + +To build the files for production run: + +```shell +npm run build +``` + +### Running Celery + +Celery is used to run background tasks and scheduled tasks like export +runs. To run it you can use: + +```shell +celery -A commcare_sync worker -l info +``` + +Or to also include periodic tasks to run all exports on a schedule: + +```shell +celery -A commcare_sync worker -l info -B +``` + +### Running tests + +To run tests: + +```shell +pytest +``` diff --git a/docs/install_prod.md b/docs/install_prod.md new file mode 100644 index 0000000..c3b142e --- /dev/null +++ b/docs/install_prod.md @@ -0,0 +1,5 @@ +Installing a production environment +----------------------------------- + +To set up a production server, see +[commcare-sync-ansible](https://github.com/dimagi/commcare-sync-ansible). diff --git a/docs/test_pipeline.md b/docs/test_pipeline.md new file mode 100644 index 0000000..2261a7b --- /dev/null +++ b/docs/test_pipeline.md @@ -0,0 +1,351 @@ +Setting up a test data pipeline +------------------------------- + +To verify a data pipeline, working from data collection to report +rendering, we should start by determining what data we want to appear in +the report. And for that, we should dive into DHIS2. + +### Credentials for DHIS2 + +Navigate to https://im.dhis2.org/public/instances and select a stable +branch. You will be taken to a login page. Log in with username "admin" +and password "district". + +### Identifying indicators in DHIS2 to collect data for in CommCare + +You will be presented with an Antenatal Care dashboard. To make testing +easier, we want a graph that shows data for last month. Click the +"Dashboards" button in the top left, and choose "Immunization". Scroll +down to "Immunization: OPV 3 coverage last month". That should do. + +From the title we can guess ... + +1. In CommCare we need to collect data on Oral Polio Vaccine + immunizations, associated with a location ("organisation unit" in + DHIS2 terminology) and a date. We will track the number of doses + given to an individual. +2. Using CommCare Data Pipeline, we export this data to a SQL database + like PostgreSQL. +3. In Postgres we aggregate by month using a view. +4. Then we transform aggregated data into a JSON payload for DHIS2's + DataSet API. Indicators will need to be identified by their DHIS2 + IDs. + +So our next task is to find the DHIS2 IDs of the indicators for OPV 3 +coverage. + +Click on the ⋮⋮⋮ (3×3 grid) icon in the top right. Open "Maintenance". + +#### Data element: OPV3 doses given + +Open the "Data Element" tab. Click the "List" icon in the "Data +element" box. + +Search "OPV". We are looking for "OPV3 doses given". Note that "Domain +type" is set to "Aggregate" (as opposed to "Tracker"). This is correct +for the type of integration we are building. + +Note that "Category combination" is "Location and age group". The +CommCare app will need to collect "location" and matching "age group" +categories, and the Postgres aggregation will need to group by them. +They will be included in the API payload, identified by their "category +option combination" IDs. + +Click the ⋮ (vertical ellipsis) "Actions" menu in the row for "OPV3 +doses given" and choose "Show details". Note the value of "Id" is +"vI2csg55S9C". + +#### Category combination: Location and age group + +Next, open the "Category" tab. Click the "List" icon in the "Category +combination" box. + +In the row for "Location and age group", click the ⋮ "Actions" menu and +choose "Show details". Note the value of "Id" is "dzjKKQq0cSO". + +Click on the row to open "Location and age group". Note that its +"Categories" are "Location Fixed/Outreach" and "EPI/nutrition age". + +Click "Cancel". + +#### Category: Location Fixed/Outreach + +Click "Category" in the left panel. + +In the row for "Location Fixed/Outreach", click the ⋮ "Actions" menu and +choose "Show details". Note the value of "Id" is "fMZEcRHuamy". Click on +the row. Note that the "Category options" are "Fixed" and "Outreach". + +Click "Cancel" + +#### Category: EPI/nutrition age + +In the row for "EPI/nutrition age", click the ⋮ "Actions" menu, choose +"Show details", and note that "Id" is "YNZyaJHiHYq". Click on the row, +and note that its "Category options" are "<1y" and ">1y". + +#### Category option: Fixed + +Click "Category option" in the left panel. + +Search for "Fixed". Click the ⋮ "Actions" menu and choose "Show +details". Note the value of "Id" is "qkPbeWaFsnU". + +Do the same for the other category options. + +#### Category option combination: Fixed, <1y + +The next DHIS2 IDs we need are for the four category option +combinations. The DHIS2 API payload will need to identify aggregation +groupings by their combination of category options. + +Click "Category option combination" in the left panel. Search for +"Fixed". Get the IDs of "Fixed, <1y" and "Fixed, >1y". Do the same for +"Outreach, <1y" and "Outreach, >1y". + +#### Location: Ngelehun CHC + +Last, we need at least one location. + +Open the "Organisation Unit" tab, and click the "List" icon in the +"Organisation units" box. + +In the left panel, expand "Bo", then "Badjia", and select "Ngelehun +CHC". Open "Actions", choose "Show details", and note that its ID is +"DiszpKrYNg8". + +#### Summary + +Here are all the IDs we will need: + +| Type | Name | ID | +|-----------------------|-------------------------|-------------| +| Data set | Child Health | BfMAe6Itzgt | +| Data element | OPV3 doses given | vI2csg55S9C | +| Category combo | Location and age group | dzjKKQq0cSO | +| Category | Location Fixed/Outreach | fMZEcRHuamy | +| Category option | Fixed | qkPbeWaFsnU | +| Category option | Outreach | wbrDrL2aYEc | +| Category | EPI/nutrition age | YNZyaJHiHYq | +| Category option | <1y | btOyqprQ9e8 | +| Category option | >1y | GEqzEKCHoGA | +| Category option combo | Fixed, <1y | Prlt0C1RF0s | +| Category option combo | Fixed, >1y | psbwp3CQEhs | +| Category option combo | Outreach, <1y | V6L425pT3A0 | +| Category option combo | Outreach, >1y | hEFKSsPV5et | +| Organisation unit | Ngelehun CHC | DiszpKrYNg8 | + +### Collecting data in CommCare + +As a CommCare app builder, you will have an idea of what forms your app +would need for collecting these indicators. Some values, like +"vaccination_date" + +But for the sake of testing, we will skip the app building, and use a +case import instead. + +In CommCare, using Data Dictionary, create two case types with the +following case properties: + +#### Case type: child + +| Case Property | Data Type | +|---------------|-----------| +| external_id | Plain | +| name | Plain | +| dob | Date | + +#### Case type: vaccination + +| Case Property | Data Type | Values | +|------------------|-----------------|-----------------| +| vaccination_date | Date | | +| child_ext_id | Plain | | +| org_unit_id | Plain | | +| location_fo | Multiple Choice | fixed, outreach | +| opv1_dose_given | Multiple Choice | yes, no | +| opv2_dose_given | Multiple Choice | yes, no | +| opv3_dose_given | Multiple Choice | yes, no | + +> [!Note] +> I have intentionally left DHIS2 IDs out of the data collected in +> CommCare, except for "org_unit_id". This more accurately models a +> real-world scenario, where CommCare app builders will not be using +> DHIS2 IDs for case properties in CommCare. The value for "org_unit_id" +> would come from a custom location field, named something like +> "dhis2_id", on a corresponding "Ngelehun CHC" CommCare location. + +Then create two case import spreadsheets with some test data. In the +"org_unit_id" column, set all values to "DiszpKrYNg8" for "Ngelehun +CHC". Import the spreadsheets into CommCare HQ Staging. + +### An example of a prompt to create a Postgres view + +Using CommCare Data Pipeline, export the cases into Postgres. + +Using yor favorite Postgres client: +* In the "Child" table, add an index for the "external_id" column. +* In the "Vaccination" table, set the "child_ext_id" field as a foreign + key to "Child.external_id" + +Now strike up a conversation with an LLM fluent in SQL. (Gemini, ChatGPT +or Claude are all good options. If you can choose your model, pick the +latest coding model at a medium reasoning level.) Ask it something like: + +> I want to create a view in PostgreSQL to count the number of children +> who have been given an OPV3 vaccination dose. I want to aggregate them +> by organisation unit, and by age group categories "less than or equal +> to 12 months" and "greater than 12 months", and by location categories +> "fixed" and "outreach". I want a sum for each combination of those +> categories. +> +> The data is stored in two tables. The tables look like this: +> +> Child: +> +> | Field | Type | +> |-------------|-------------| +> | external_id | indexed key | +> | name | string | +> | dob | date | +> +> Vaccination: +> +> | Field | Type | Values | +> |------------------|-----------------------|-----------------| +> | child_ext_id | fk: Child.external_id | | +> | vaccination_date | date | | +> | org_unit_id | string | | +> | location_fo | string | fixed, outreach | +> | opv1_dose_given | string | yes, no | +> | opv2_dose_given | string | yes, no | +> | opv3_dose_given | string | yes, no | +> +> * Join the tables on Vaccination.child_ext_id = Child.external_id +> +> * Include only records where Vaccination.opv3_dose_given = 'yes' +> +> * Use Vaccination.vaccination_date - Child.dob to determine the age of +> the child in months. +> +> * Allow the view to be filtered by month of Vaccination.vaccination_date. +> +> Please give me the SQL statement to create this view in PostgreSQL. +> Name it "opv3_coverage_view". + +Copy and paste the SQL into your Postgres client. Test it. Continue the +conversation with the AI assistant to fix anything that isn't working +the way you want. + +> [!Note] +> If you use an AI agent like Claude Code or OpenAI Codex, and a +> Postgres command line client like "psql", then the AI agent can create +> the view itself, test that it is working as you described, and fix it +> if it is not. Windows users can use Ubuntu on WSL to install these: +> * `curl -fsSL https://claude.ai/install.sh | bash` +> * `sudo apt install postgresql-client` +> Tell Claude or Codex that it can use "psql" to execute SQL commands. + +> [!Warning] +> Do not give AI agents access to databases that contain real data! + +### An example of a prompt to create a JSON payload view + +Go back to your AI assistant, and ask it to create Postgres view that +returns a JSON payload. Maybe something like, + +> I want to create a view in PostgreSQL named "opv3_coverage_payload_view". +> The new view will be queried with +> `SELECT json_payload FROM opv3_coverage_payload_view WHERE period = '{YYYYMM}';` +> where "{YYYYMM}" is a month, e.g. "202512" is December 2025. +> +> The "opv3_coverage_payload_view" view must filter "opv3_coverage_view" +> by the month given in the "period" parameter. +> +> When filtered by month, "opv3_coverage_payload_view" must return only +> one row. +> +> The "json_payload" field is a JSON object in the format of an API +> request payload. To understand the required payload format, read the +> documentation on the DHIS2 Data Values API. See the section "Sending +> bulks of data values". Note the JSON format: +> https://docs.dhis2.org/en/develop/using-the-api/dhis-core-version-239/data.html#webapi_sending_bulks_data_values +> +> Values from "opv3_coverage_view" must be transformed into uid values for +> DHIS2. Use the following mapping: +> +> | location_fo | Age (months) | DHIS2 uid | +> |-------------|--------------|-------------| +> | fixed | <= 12 | Prlt0C1RF0s | +> | fixed | > 12 | psbwp3CQEhs | +> | outreach | <= 12 | V6L425pT3A0 | +> | outreach | > 12 | hEFKSsPV5et | +> +> An example value of "json_payload" (DO NOT include comments): +> +> ```json +> { +> "dataValues": [ +> { +> "period": "202512", // month (YYYYMM) +> "orgUnit": "DiszpKrYNg8", // org_unit_id +> +> "dataElement": "vI2csg55S9C", // (constant) OPV3 doses given +> "categoryOptionCombo": "Prlt0C1RF0s", // location_fo = 'fixed', age_mo <= 12 +> +> // The sum of OPV3 doses given to "location_fo = 'fixed', age_mo <= 12" in Dec 2025 +> "value": "12" +> }, +> { +> "period": "202512", +> "orgUnit": "DiszpKrYNg8", +> +> "dataElement": "vI2csg55S9C", +> "categoryOptionCombo": "psbwp3CQEhs", // location_fo = 'fixed', age_mo > 12 +> +> // The sum of OPV3 doses given to "location_fo = 'fixed', age_mo > 12" in Dec 2025 +> "value": "18" +> } +> // ... +> ] +> } +> ``` +> +> Please use the command +> `psql -h postgres.example.com -p 5432 -u myusername -d mydatabase` +> with password "mypassword" to create this view in PostgreSQL. Query the +> view for December 2025 to verify that the value of the "json_payload" +> field is correct. + +(Or just ask for the SQL statement to create the view.) + +### Config for “DB Query Forwarder” to forward to DHIS2 + +The tricky part of configuring a data forwarder in CommCare Data +Pipeline is the query. For PostgreSQL, the following will filter +"period" by last month: + +```sql +SELECT json_payload +FROM opv3_coverage_payload_view +WHERE period = TO_CHAR(CURRENT_DATE - INTERVAL '1 month', 'YYYYMM'); +``` + +(The query will be different for other database engines. For SQL Server +it is: + +```sql +SELECT json_payload +FROM opv3_coverage_payload_view +WHERE period = FORMAT(DATEADD(MONTH, -1, GETDATE()), 'yyyyMM'); +``` +) + +### How to verify whether data arrived in DHIS2 + +Run your data forwarder. Go back to DHIS2. Check whether your graph +shows any data. + +> [!Warning] +> I have not tested this! I don't know whether the graph in DHIS2 is +> actually working!