Define directory structure for tests

[furtib]

We should discuss a better structure for our tests

[nettle]

First draft to start discussion, just thinkin out loud:

.github/
├── ISSUE_TEMPLATE/
│   └── bug_report.md
├── actions/
│   └── setup.yaml
└── workflows/
    └── ci.yaml
src/
test/
├── foss/
│   ├── yaml-cpp/
│   │   ├── bazel.patch
│   │   └── test.sh
│   └── zlib/
│       ├── BUILD
│       ├── WORKSPACE
│       └── test.sh
├── unit/
│   ├── avoidance/
│   ├── config/
│   ├── ctu/
│   ├── generated/
│   ├── platforms/
│   ├── simple/
│   │   ├── inc/
│   │   ├── src/
│   │   └── BUILD
│   ├── transitive/
│   └── virtual-includes/
│       ├── inc/
│       └── src/
├── test_unit.py
├── test_foss.py
└── test.sh

FOSS = Free Open Source Software

[furtib]

I like the idea of moving the FOSS projects into the test directory!

I had an idea that we have a lot of repeating code in the initialization (formerly patch) scripts, so we could place these repeating elements into a folder somewhere!

My idea for the FOSS part would be something like this:

foss/
├─ yaml-cpp/
│  ├─ init.sh
├─ zlib/
│  ├─ init.sh
├─ templates/
│  ├─ WORKSPACE.template
│  ├─ .bazelversion

What do you think?

[Szelethus]

As we grow our testing infrastructure, I'd also be very much in favour of integrating pytest. To me, after switching to it in CodeChecker itself it emits a very easy-to-browse output.

[nettle]

As we grow our testing infrastructure, I'd also be very much in favour of integrating pytest. To me, after switching to it in CodeChecker itself it emits a very easy-to-browse output.

Pytest is good, but I would say only as an optional tool since it is one more external dependency.
I would like to keep compatibility with standard unittest

[nettle]

I had an idea that we have a lot of repeating code in the initialization (formerly patch) scripts, so we could place these repeating elements into a folder somewhere!

foss/
├─ yaml-cpp/
│  ├─ init.sh
├─ zlib/
│  ├─ init.sh
├─ templates/
│  ├─ WORKSPACE.template
│  ├─ .bazelversion

Yes, WORKSPACE.template is definitely good idea! I guess we can also add MODULE.template later.
I'm not sure about .bazelversion because it might be easier to generate, but that's not that important.

BTW, why init.sh? What's your idea? And how do we run test(s)?

[furtib]

The idea, with init.sh is that since we already created a folder for each project, there is no point in naming the script the project.
I named it after what it does. And since they are basically just clones the repositories, and apply our rules to them, I found init.sh a fitting name.
I would place the actual running of test in the foss_test.py (or just a foss_test.sh) to avoid having to repeat Bazel commands in each project's files.
Running a test would entail: (content of foss_test.py)

• Going into each folder
• Running the init.sh script (if necessary)
• Run bazel clean (if necessary)
• Run bazel build for both rules.

As for .bazelversion I just needed a central location for it so I don't have to hard code it individually in every init.sh. (I assume that most will use bazelisk, as we do in CI, this is why we need it at all)

[furtib]

There are tests for compiling the analyzed targets. What should we do with them? (Maybe get rid of them?) It doesn't make sense to just copy them to a separate directory, and it is also irrational to build all the cc_binary targets in every test directory.