Prepare repo for assignment

Author: lscholten

.github/CODEOWNERS

@@ -0,0 +1 @@
+* @dbt-labs/dx

.github/workflows/validate.yml

@@ -0,0 +1,18 @@
+name: Validate dbt project
+
+on: push
+
+jobs:
+  validate:
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v4
+      - name: Install uv
+        uses: astral-sh/setup-uv@v5
+        with:
+          enable-cache: true
+      - name: Install the project
+        run: uv sync --all-extras --dev
+      - name: Run DBT
+        run: uv run dbt build

.gitignore

@@ -1,5 +1,11 @@
 
 target/
+dbt_packages/
 dbt_modules/
 logs/
 **/.DS_Store
+.user.yml
+venv/
+env/
+**/*.duckdb
+**/*.duckdb.wal

.sqlfluff

@@ -0,0 +1,3 @@
+[sqlfluff]
+
+dialect = duckdb

data/.gitkeep .sqlfluffignoredata/.gitkeep renamed to .sqlfluffignore

@@ -1,78 +1,102 @@
-## Testing dbt project: `jaffle_shop`
+# Storio - Data Principal Engineer Assignment
 
-`jaffle_shop` is a fictional ecommerce store. This dbt project transforms raw data from an app database into a customers and orders model ready for analytics.
+This repo contains a scuffed version of `jaffle_shop`, a fictional ecommerce store. This project will be used to test your refactoring skills. In a nutshell: This dbt project transforms raw data from an app database into models ready for analytics.
+However, the project is not well-structured, and the code is not very readable. Your task is to refactor the code to make it more readable and maintainable.
+Things to consider:
 
-### What is this repo?
-What this repo _is_:
-- A self-contained playground dbt project, useful for testing out scripts, and communicating some of the core dbt concepts.
+- Warehouse layers
+- Code readability
+- Testing
 
-What this repo _is not_:
-- A tutorial — check out the [Getting Started Tutorial](https://docs.getdbt.com/tutorial/setting-up) for that. Notably, this repo contains some anti-patterns to make it self-contained, namely the use of seeds instead of sources.
-- A demonstration of best practices — check out the [dbt Learn Demo](https://github.com/fishtown-analytics/dbt-learn-demo-v2-archive) repo instead. We want to keep this project as simple as possible. As such, we chose not to implement:
-    - our standard file naming patterns (which make more sense on larger projects, rather than this five-model project)
-    - a pull request flow
-    - CI/CD integrations
-- A demonstration of using dbt for a high-complex project, or a demo of advanced features (e.g. macros, packages, hooks, operations) — we're just trying to keep things simple here!
+The project contains [seeds](https://docs.getdbt.com/docs/building-a-dbt-project/seeds) that includes some (fake) raw data from a fictional app along with some basic dbt [models](https://docs.getdbt.com/docs/building-a-dbt-project/building-models), tests, and docs for this data.
 
-### What's in this repo?
-This repo contains [seeds](https://docs.getdbt.com/docs/building-a-dbt-project/seeds) that includes some (fake) raw data from a fictional app.
+## Running this project
 
-The raw data consists of customers, orders, and payments, with the following entity-relationship diagram:
+Prerequisities: Python >= 3.8
 
-![Jaffle Shop ERD](/etc/jaffle_shop_erd.png)
+1. Install the project in a virtual environment using your favorite python/env management tool
+   - `uv`
+   - `pipenv`
+   - `poetry`
+   - `venv`
+   - ...
+2. (`uv run`) `dbt build`
+3. (`uv run`) `dbt docs generate`
+4. (`uv run`) `dbt docs serve`
 
+## Verifying your environment
 
-### Running this project
-To get up and running with this project:
-1. Install dbt using [these instructions](https://docs.getdbt.com/docs/installation).
+1. Ensure your [profile](https://docs.getdbt.com/reference/profiles.yml) is setup correctly from the command line:
 
-2. Clone this repository.
+   ```shell
+   dbt --version
+   dbt debug
+   ```
 
-3. Change into the `jaffle_shop` directory from the command line:
-```bash
-$ cd jaffle_shop
-```
+2. Load the CSVs with the demo data set, run the models, and test the output of the models using the [dbt build](https://docs.getdbt.com/reference/commands/build) command:
 
-4. Set up a profile called `jaffle_shop` to connect to a data warehouse by following [these instructions](https://docs.getdbt.com/docs/configure-your-profile). If you have access to a data warehouse, you can use those credentials – we recommend setting your [target schema](https://docs.getdbt.com/docs/configure-your-profile#section-populating-your-profile) to be a new schema (dbt will create the schema for you, as long as you have the right privileges). If you don't have access to an existing data warehouse, you can also setup a local postgres database and connect to it in your profile.
+   ```shell
+   dbt build
+   ```
 
-5. Ensure your profile is setup correctly from the command line:
-```bash
-$ dbt debug
-```
+3. Query the data:
 
-6. Load the CSVs with the demo data set. This materializes the CSVs as tables in your target schema. Note that a typical dbt project **does not require this step** since dbt assumes your raw data is already in your warehouse.
-```bash
-$ dbt seed
-```
+   Launch a DuckDB command-line interface (CLI):
 
-7. Run the models:
-```bash
-$ dbt run
-```
+   ```shell
+   duckcli jaffle_shop.duckdb
+   ```
 
-> **NOTE:** If this steps fails, it might mean that you need to make small changes to the SQL in the models folder to adjust for the flavor of SQL of your target database. Definitely consider this if you are using a community-contributed adapter.
+   Run a query at the prompt and exit:
 
-8. Test the output of the models:
-```bash
-$ dbt test
-```
+   ```
+   select * from customers_with_order_info where customer_id = 42;
+   exit;
+   ```
 
-9. Generate documentation for the project:
-```bash
-$ dbt docs generate
-```
+   Alternatively, use a single-liner to perform the query:
 
-10. View the documentation for the project:
-```bash
-$ dbt docs serve
-```
+   ```shell
+   duckcli jaffle_shop.duckdb -e "select * from customers_with_order_info where customer_id = 42"
+   ```
 
-### What is a jaffle?
-A jaffle is a toasted sandwich with crimped, sealed edges. Invented in Bondi in 1949, the humble jaffle is an Australian classic. The sealed edges allow jaffle-eaters to enjoy liquid fillings inside the sandwich, which reach temperatures close to the core of the earth during cooking. Often consumed at home after a night out, the most classic filling is tinned spaghetti, while my personal favourite is leftover beef stew with melted cheese.
+   or:
 
----
-For more information on dbt:
-- Read the [introduction to dbt](https://docs.getdbt.com/docs/introduction).
-- Read the [dbt viewpoint](https://docs.getdbt.com/docs/about/viewpoint).
-- Join the [dbt community](http://community.getdbt.com/).
----
+   ```shell
+   echo 'select * from customers_with_order_info where customer_id = 42' | duckcli jaffle_shop.duckdb
+   ```
+
+4. Generate and view the documentation for the project:
+   ```shell
+   dbt docs generate
+   dbt docs serve
+   ```
+
+## Running `build` steps independently
+
+1. Load the CSVs with the demo data set. This materializes the CSVs as tables in your target schema. Note that a typical dbt project **does not require this step** since dbt assumes your raw data is already in your warehouse.
+
+   ```shell
+   dbt seed
+   ```
+
+2. Run the models:
+
+   ```shell
+   dbt run
+   ```
+
+   > **NOTE:** If you decide to run this project in your own data warehouse (outside of this DuckDB demo) and steps fail, it might mean that you need to make small changes to the SQL in the models folder to adjust for the flavor of SQL of your target database. Definitely consider this if you are using a community-contributed adapter.
+
+3. Test the output of the models using the [test](https://docs.getdbt.com/reference/commands/test) command:
+   ```shell
+   dbt test
+   ```
+
+## Browsing the data
+
+Some options:
+
+- [DuckDB UI](https://duckdb.org/docs/stable/extensions/ui.html)
+- [duckcli](https://pypi.org/project/duckcli/)
+- [DuckDB CLI](https://duckdb.org/docs/installation/?environment=cli)

README.md

@@ -1,23 +1,24 @@
+name: "jaffle_shop"
 
-name: 'jaffle_shop'
-version: '0.1'
-profile: 'jaffle_shop'
 config-version: 2
+version: "0.1"
 
-source-paths: ["models"]
-analysis-paths: ["analysis"]
+profile: "jaffle_shop"
+
+model-paths: ["models"]
+seed-paths: ["seeds"]
 test-paths: ["tests"]
-data-paths: ["data"]
+analysis-paths: ["analysis"]
 macro-paths: ["macros"]
 
 target-path: "target"
 clean-targets:
-    - "target"
-    - "dbt_modules"
-    - "logs"
+  - "target"
+  - "dbt_modules"
+  - "logs"
+
+require-dbt-version: [">=1.0.0", "<2.0.0"]
 
 models:
   jaffle_shop:
-      materialized: table
-      staging:
-        materialized: view
+    materialized: table