[CI] Refactor GHA workflow for cog unit tests (#673)

* [CI] Rework cog unit test to be aware of cog-specific deps

This creates a new GH Actions workflow that replace the existing
`cogUnitTests.yml`. It basically generates a `tox.ini` with
`testenv`s corresponding to all cogs, with cog-specific deps, and
run all tests within their directories. The workflow file also
allows for easy Red bot (upstream) version anchoring, as well as
having only tests for specific cogs run on `workflow_dispatch`.

* [CI] Fix formatting for `gen_tox_ini.py`

* [CI] Fix `Red-DiscordBot` tag in `cog_unit_test.yaml`

There is no `v` prefix.

* [CI] Make workflow cog unit test look at `cogs` dir

This commit adds great flexibility to generator scripts and adjust the
`cog_unit_test.yaml` workflow file to specify the cogs dir ("cogs") and
INI template file paths.

* [CI] Use `cog_` prefix for `tox` envs in cog unit test

* [CI] Add a step in cog unit test workflow to show generated configs

This helps with debugging.

* [CI] Empty `requirements.txt` if `requirements` is not in `info.json`

* [CI] Set name for the "set matrix output" step in cog unit test

* [CI] Account for cogs without `info.json` in cog unit test

* [CI] Let `tox` install Red deps directly via `git`

* [CI] Fix `tox` Git dep path in cog unit test `tox.ini` template

It is not supposed to be prefixed with `-r`.

* [CI] Simplify and use absolute path for cogs dir

This commit renders templates redundant. Tox env var substitution is
enough to get the job done.

* [CI] Correct INI paths in cog unit test workflow

* [CI] Install Red dev deps in cog unit test workflow

This is done just to get Red's development dependencies `pytest` and
`tox`.

* [CI] Remove `#egg` in package URL and install `tox`

It turns out that `tox` is not included when installing
`red-discordbot[dev]`.

Also, `#egg` is deprecated.
See pypa/pip#13157 for more details.

* [CI] Use absolute paths for all paths in env in cog unit test workflow

* [CI] Allow cogs with no tests to pass in cog unit test workflow

* [CI] Ignore Pytest's exit code 5 in cog unit test workflow

This exit code is used when Pytest does not find any tests.

* [CI] Use extra target `test` instead of `dev` from `red-discordbot`

`dev` will try to drag in `sphinx-prompt` 1.7.0 which is not compatible
with Python 3.11.

* [CI] Let tox allow bash and sh in cog unit test workflow

* [CI] Use absolute paths in tox's `allowlist_externals`

* [CI] Skip installing `red-discordbot` in cog unit test workflow

Only `tox` is needed, and then `tox`, upon installing
`red-discordbot[test]`, will use that environment to run `pytest`.

* [CI] Simplify `workflow_dispatch` for cog unit test workflow

If the ability to input cog names for manual runs is wanted later, it
can then be implemented.

Also, this commit cleans up the `discover` job and removes the
installing of `jq` in `test` job. `jq` is available already.

* [CI] Make `jq` output JSON list in one line in cog unit test workflow

Without this, GitHub will run into this error:
"Unable to process file command 'output' successfully."

* [CI] Inline env vars in cog unit test workflow

Except for `COGS_PATH`, all other environment variables can be inlined.

* [CI] Exclude the cogs dir passed into `find` in cog unit test workflow

This is a `find` gotcha. The flag `-mindepth 1` has to be there to fix
this.

* [CI] Extract and use Red version from `setup.py`

* [CI] Clean up env vars in cog unit test workflow

This commit also utilizes `COGS_REL_PATH` in `hashFiles` so that there
is no longer the hardcoded `cogs` dir in `hashFiles` anymore.

* [CI] Remove the "empty" comment in `jq` in cog unit test workflow

* [CI] Collect only testful cogs in cog unit test workflow

This commit relies on Pytest's test collection logic to extract the
names of the cogs with tests. Tox is used to keep the
`red-discordbot[test]` base dependency central in one INI file. Cogs
with no tests will not be included in the `cogs` output of the
`discover` job, meaning they will not get recognized by the `test` job.

* [CI] Fix the `run` field of `collect-testful-cogs`

* [CI] Correct `--cogs-dir` to `--cogs-path` in `cog_unit_test_tox.ini`

* [CI] Rename `SCRIPT_PATH` to `SCRIPT_PY` in `cog_unit_test_tox.ini`

It is simply because other envs and flags (like `PYTEST_INI`) end with
their known extensions.

Author: quachtridat

.github/workflows/cogUnitTests.yml

@@ -0,0 +1,110 @@
+name: Cog Unit Tests
+
+on:
+  push:
+    branches:
+      - V3/develop
+      - V3/testing
+      - V3/prod
+  pull_request:
+    branches:
+      - V3/develop
+      - V3/testing
+      - V3/prod
+  workflow_dispatch:
+
+env:
+  COGS_REL_PATH: "cogs"
+
+jobs:
+  discover:
+    runs-on: ubuntu-latest
+    outputs:
+      cogs: ${{ steps.collect-testful-cogs.outputs.cogs }}
+      red-version: ${{ steps.extract-red-version.outputs.version }}
+    env:
+      # Keep this in sync with Red-DiscordBot's minimum supported version.
+      # This is defined in various places, but extracting any of them is non-trivial.
+      PYTHON_VERSION: "3.8"
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Extract Red version
+        id: extract-red-version
+        run: |
+          version=$(python "${{ github.workspace }}/setup.py" --version 2> /dev/null)
+          echo "version=${version}" >> $GITHUB_OUTPUT
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ env.PYTHON_VERSION }}
+
+      - name: Cache pip + tox
+        uses: actions/cache@v4
+        with:
+          path: |
+            ~/.cache/pip
+            .tox
+          key: "${{ runner.os }}-py${{ env.PYTHON_VERSION }}-red${{ steps.extract-red-version.outputs.version }}"
+
+      - name: Install tox
+        run: pip install "tox"
+
+      - name: Collect testful cogs
+        id: collect-testful-cogs
+        env:
+          COGS_PATH: "${{ github.workspace }}/${{ env.COGS_REL_PATH }}"
+          OUTPUT_FILE: "/tmp/cog_unit_test_collect.json"
+          PYTEST_INI: "${{ github.workspace }}/.github/workflows/configs/cog_unit_test_pytest.ini"
+          SCRIPT_PY: "${{ github.workspace }}/.github/workflows/scripts/collect_testful_cogs.py"
+        run: |
+          tox run -c "${{ github.workspace }}/.github/workflows/configs/cog_unit_test_tox.ini" -e "collect"
+          echo "cogs=$(jq -c '.' '/tmp/cog_unit_test_collect.json')" >> $GITHUB_OUTPUT
+
+  test:
+    needs: discover
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        # Keep this in sync with Red-DiscordBot's supported versions.
+        # They are defined in setup.py and pyproject.toml, but extracting them is non-trivial.
+        python-version: ["3.8", "3.9", "3.10", "3.11"]
+        cog: ${{ fromJson(needs.discover.outputs.cogs) }}
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Extract cog requirements
+        env:
+          COG_PATH: "${{ github.workspace }}/${{ env.COGS_REL_PATH }}/${{ matrix.cog }}"
+        run: |
+          if [ -f "${COG_PATH}/info.json" ]; then
+            jq -r ".requirements[]?" "${COG_PATH}/info.json" > "${COG_PATH}/.requirements.txt"
+          else
+            touch "${COG_PATH}/.requirements.txt"
+          fi
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+
+      - name: Cache pip + tox
+        uses: actions/cache@v4
+        with:
+          path: |
+            ~/.cache/pip
+            .tox
+          key: "${{ runner.os }}-py${{ matrix.python-version }}-red${{ needs.discover.outputs.red-version }}-${{ matrix.cog }}-${{ hashFiles(format('{0}/{1}/.requirements.txt', env.COGS_REL_PATH, matrix.cog)) }}"
+          restore-keys: "${{ runner.os }}-py${{ matrix.python-version }}-red${{ needs.discover.outputs.red-version }}-${{ matrix.cog }}-"
+
+      - name: Install tox
+        run: pip install "tox"
+
+      - name: Run tox
+        env:
+          COG_PATH: "${{ github.workspace }}/${{ env.COGS_REL_PATH }}/${{ matrix.cog }}"
+          COG_REQUIREMENTS_TXT: "${{ github.workspace }}/${{ env.COGS_REL_PATH }}/${{ matrix.cog }}/.requirements.txt"
+          PYTEST_INI: "${{ github.workspace }}/.github/workflows/configs/cog_unit_test_pytest.ini"
+          RED_DISCORDBOT_BRANCH: "${{ needs.discover.outputs.red-version }}"
+        run: tox run -c "${{ github.workspace }}/.github/workflows/configs/cog_unit_test_tox.ini" -e "test"

.github/workflows/cog_unit_test.yaml

@@ -0,0 +1,2 @@
+[pytest]
+python_files = test*.py

.github/workflows/configs/cog_unit_test_pytest.ini

@@ -0,0 +1,18 @@
+[tox]
+isolated_build = true
+skipsdist = true
+
+[testenv]
+deps =
+    red-discordbot[test] @ git+https://github.com/Cog-Creators/Red-DiscordBot@{env:RED_DISCORDBOT_BRANCH:V3/develop}
+
+[testenv:collect]
+commands =
+    python "{env:SCRIPT_PY:collect.py}" --cogs-path "{env:COGS_PATH:cogs}" --pytest-ini "{env:PYTEST_INI:pytest.ini}" --output-file "{env:OUTPUT_FILE:collect.json}" {posargs}
+
+[testenv:test]
+deps =
+    {[testenv]deps}
+    --requirement "{env:COG_REQUIREMENTS_TXT:requirements.txt}"
+commands =
+    pytest --config-file "{env:PYTEST_INI:pytest.ini}" "{env:COG_PATH}" {posargs}

.github/workflows/configs/cog_unit_test_tox.ini

@@ -0,0 +1,60 @@
+#!/usr/bin/env python3
+import argparse
+import json
+from pathlib import Path
+import sys
+from typing import List, Set, Union
+
+import pytest
+
+
+class CogCollector:
+    """Pytest plugin to collect cogs relative to a cogs directory."""
+
+    def __init__(
+        self,
+        cogs_path: Union[str, Path],
+    ) -> None:
+        self.cogs_path = Path(cogs_path).resolve()
+        self.collected_cogs: Set[str] = set()
+
+    def pytest_collection_modifyitems(
+        self,
+        session: pytest.Session,
+        config: pytest.Config,
+        items: List[pytest.Item],
+    ) -> None:
+        for item in items:
+            rel_path = item.path.relative_to(self.cogs_path)
+            cog_name = rel_path.parts[0]
+            self.collected_cogs.add(cog_name)
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--cogs-path", type=str, default="cogs", help="Path to cogs directory")
+    parser.add_argument("--output-file", type=str, default=None, help="File to write output JSON")
+    parser.add_argument("--pytest-ini", type=str, default=None, help="Path to pytest.ini file")
+
+    args = parser.parse_args()
+    cogs_path: str = args.cogs_path
+
+    pytest_args = ["--collect-only", "--quiet"]
+    if args.pytest_ini:
+        pytest_args.extend(["--config-file", args.pytest_ini])
+    pytest_args.append(str(Path(cogs_path).resolve()))
+
+    cog_collector_plugin = CogCollector(cogs_path)
+    pytest_plugins = [cog_collector_plugin]
+
+    pytest.main(pytest_args, plugins=pytest_plugins)
+
+    if args.output_file:
+        with open(file=args.output_file, mode="w") as f:
+            json.dump(sorted(cog_collector_plugin.collected_cogs), f)
+    else:
+        json.dump(sorted(cog_collector_plugin.collected_cogs), sys.stdout)
+
+
+if __name__ == "__main__":
+    main()