[FEATURE]: Parametrization of rules

[mwojtyczka]

Is there an existing issue for this?

• I have searched the existing issues

Problem statement

Currently the rules are defined statically and there is no way to parametrize them.

Proposed Solution

Introduce a way to parametrize rules via variables substitution to make them more generic and scalable so that a rule can be used for many tables and columns.

Parametrization (variable substitiution) could be supported in:

• all string arguments of a function
• name
• filter
• criticality

Example:

checks = [
      {
            "criticality": "{{ criticality }}",
            "name": "{{ param1 }}_greater_than_{{ treshold1 }}",
            "check": {
                "function": "sql_expression",
                "arguments": {"expression": "{{ param1 }} > {{ treshold1 }}", "msg": "{{ param1 }} greater than {{ treshold1 }}"},
            },
            "filter": "{{ param2 }} = 'column2'",
        },
        {
            "criticality": "error",
            "check": {
                "function": "is_not_null",
                "arguments": {"column": "{{ param2 }}"},
            },
        }
]

# Pass params
variables = {"param1": "column1", "treshold": 10m, "param2": "column2", "criticality":"error"}

valid_and_quarantine_df = engine.apply_checks(df, checks, ref_dfs=ref_dfs, variables=variables)

Additional Context

Requested by multiple customers.