[edgeai-modelmaker] cannot generate artifacts

[MattMak0402]

Describe the bug
I followed the edgeai-modelmaker instructions step by step, the model can be trained and converted into onnx file but there are some issues generating artifacts. The artifacts folder under

/local_data/home/mattmak/edgeai-tensorlab/edgeai-modelmaker/data/projects/htt-warehouse-20250313/run/20250314--141431/yolox_s_lite/compilation/AM62A/work/od-8220

is empty. The run.log is attached, please look at it. Seems like the problem is related TIDLCompilationProvider.

run.log

According to run.log, the TIDL (Texas Instruments Deep Learning) provider, which is required for hardware acceleration on the AM62A target, could not be initialized. As a result, the process falls back to the CPUExecutionProvider.

# All necessary imports at the beginning
Package                  Version              Editable project location
------------------------ -------------------- ----------------------------------------------------------------------------------
absl-py                  2.1.0
addict                   2.4.0
aenum                    3.1.15
aliyun-python-sdk-core   2.15.2
aliyun-python-sdk-kms    2.16.5
attrs                    24.2.0
autocfg                  0.0.8
autocommand              2.2.2
backports.tarfile        1.2.0
cachetools               5.5.0
caffe2onnx               1.0.2
certifi                  2024.8.30
cffi                     1.17.1
charset-normalizer       3.3.2
click                    8.1.7
cloudpickle              3.0.0
colorama                 0.4.6
colored                  2.2.4
coloredlogs              15.0.1
contourpy                1.3.0
crcmod                   1.7
cryptography             43.0.1
cycler                   0.12.1
Cython                   3.0.11
dataclasses              0.6
debugpy                  1.8.6
decorator                5.1.1
dill                     0.3.8
distro                   1.9.0
dlr                      1.13.0
edgeai_benchmark         10.0.0+0cab45c       /local_data/home/mattmak/edgeai-tensorlab/edgeai-benchmark
edgeai_modelmaker        10.0.0+0cab45c       /local_data/home/mattmak/edgeai-tensorlab/edgeai-modelmaker
edgeai-torchmodelopt     10.0.0               /local_data/home/mattmak/edgeai-tensorlab/edgeai-modeloptimization/torchmodelopt
edgeai_xvision           9.0.0                /local_data/home/mattmak/edgeai-tensorlab/edgeai-torchvision/references/edgeailite
exceptiongroup           1.2.2
filelock                 3.14.0
flatbuffers              1.12
fonttools                4.54.1
fsspec                   2024.2.0
gluoncv                  0.10.5.post0
google-auth              2.35.0
google-auth-oauthlib     0.4.6
graphviz                 0.20.3
grpcio                   1.66.1
h5py                     3.12.1
huggingface-hub          0.29.1
humanfriendly            10.0
idna                     3.10
imageio                  2.37.0
importlib_metadata       8.5.0
importlib_resources      6.4.0
inflect                  7.3.1
iniconfig                2.0.0
jaraco.collections       5.1.0
jaraco.context           5.3.0
jaraco.functools         4.0.1
jaraco.text              3.12.1
Jinja2                   3.1.3
jmespath                 0.10.0
joblib                   1.4.2
json-tricks              3.17.3
kiwisolver               1.4.7
lazy_loader              0.4
loguru                   0.7.3
Markdown                 3.7
markdown-it-py           3.0.0
MarkupSafe               2.1.5
matplotlib               3.9.2
mdurl                    0.1.2
ml_dtypes                0.5.1
mmcv                     2.2.0
mmdeploy                 1.3.1                /local_data/home/mattmak/edgeai-tensorlab/edgeai-mmdeploy
mmdet                    3.3.0                /local_data/home/mattmak/edgeai-tensorlab/edgeai-mmdetection
mmengine                 0.10.5
model-index              0.1.11
more-itertools           10.3.0
mpmath                   1.3.0
multiprocess             0.70.16
munkres                  1.1.4
networkx                 3.2.1
ninja                    1.11.1.1
numpy                    1.23.0
nvidia-cublas-cu11       11.11.3.6
nvidia-cuda-cupti-cu11   11.8.87
nvidia-cuda-nvrtc-cu11   11.8.89
nvidia-cuda-runtime-cu11 11.8.89
nvidia-cudnn-cu11        9.1.0.70
nvidia-cufft-cu11        10.9.0.58
nvidia-curand-cu11       10.3.0.86
nvidia-cusolver-cu11     11.4.1.48
nvidia-cusparse-cu11     11.7.5.86
nvidia-nccl-cu11         2.20.5
nvidia-nvtx-cu11         11.8.86
oauthlib                 3.2.2
onnx                     1.14.0
onnx_graphsurgeon        0.3.26
onnx-simplifier          0.4.36
onnxruntime              1.15.0
onnxruntime-tidl         1.14.0+10000005
onnxscript               0.2.0
onnxsim                  0.4.35
opencv-python            4.10.0.84
opencv-python-headless   4.10.0.84
opendatalab              0.0.10
openmim                  0.3.9
openxlab                 0.1.2
ordered-set              4.1.0
osrt_model_tools         1.2
oss2                     2.17.0
packaging                24.1
pandas                   2.2.3
pillow                   11.1.0
Pillow-SIMD              9.5.0.post2
pip                      25.0.1
platformdirs             4.3.6
pluggy                   1.5.0
plyfile                  1.1
portalocker              3.1.1
prettytable              3.11.0
progiter                 2.0.0
progressbar              2.5
protobuf                 3.20.3
psutil                   6.0.0
pyasn1                   0.6.1
pyasn1_modules           0.4.1
pybind11                 2.13.6
pybind11_global          2.13.6
pycocotools              2.0.8
pycparser                2.22
pycryptodome             3.20.0
pydot                    3.0.2
Pygments                 2.18.0
pyparsing                3.1.4
pytest                   8.3.3
python-dateutil          2.9.0.post0
pytz                     2023.4
PyYAML                   6.0.2
requests                 2.28.2
requests-oauthlib        2.0.0
rich                     13.4.2
rsa                      4.9
safetensors              0.5.2
scikit-image             0.24.0
scikit-learn             1.5.2
scipy                    1.13.1
seaborn                  0.13.2
setuptools               76.0.0
shapely                  2.0.6
six                      1.16.0
sympy                    1.13.1
tabulate                 0.9.0
tensorboard              2.11.2
tensorboard-data-server  0.6.1
tensorboard-plugin-wit   1.8.1
termcolor                2.4.0
terminaltables           3.1.10
tflite                   2.10.0
tflite-runtime           2.12.0
thop                     0.1.1.post2209072238
threadpoolctl            3.5.0
tifffile                 2025.1.10
timm                     1.0.15
tomli                    2.0.1
torch                    2.4.0+cu118
torchaudio               2.4.0+cu118
torchinfo                1.8.0
torchvision              0.19.0+cu118
tornado                  6.4.1
tqdm                     4.65.2
triton                   3.0.0
tvm                      0.12.0
typeguard                4.3.0
typing_extensions        4.12.2
tzdata                   2024.2
urllib3                  1.26.20
wcwidth                  0.2.13
Werkzeug                 3.0.4
wheel                    0.44.0
wurlitzer                3.1.1
yacs                     0.1.8
yapf                     0.40.2
yolox                    0.1.0                /local_data/home/mattmak/edgeai-tensorlab/edgeai-yolox
zipp                     3.20.2

**Expected behavior and Actual Behaviour**
Artifacts should be generated. But now the generated artifacts folder is empty.

**Additional context**
In the previous version of edgeai-tensorlab (r10.0), there were indeed some bugs but I fixed all of them and everything is fine. However, for version r10.1, I encounter the problem mentioned here, which I had never had before.

[AbdulghaniAltaweel]

The same problem using version r9.2 .. ! has it been solved ?
Thanks

[MattMak0402]

Yep, I solved it, but I am not sure if this works with r9.2.

Could you send out your run.log file under your model compilation folder? It should state out the root cause of the error.

Anyways, in my case, I dragged all libraries and dependencies from tidl_tools under edgeai_tidl_tools (make sure to run the setup.sh first) to /edgeai-tensorlab/edgeai-benchmark/tools/AM62A/tidl_tools. Then reboot the pc, and now it works. For more details, please check out

https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1488000/sk-am62a-lp-edgeai-modelmaker-cannot-generate-artifacts

I used another account (steve chim) to ask the same question there and hope the forum helps.

[AbdulghaniAltaweel]

Hello,

in the run.log I had the same error msg

EP Error Unknown Provider Type: TIDLCompilationProvider when using ['TIDLCompilationProvider', 'CPUExecutionProvider']
Falling back to ['CPUExecutionProvider'] and retrying.

In my case recreating the python venv and reinstalling the project using ./setup_all (without copying tidl_tools from edgeai-tidl-tools) did the job. Reboot is a little bit difficult because I am working on server.
Next I will try to run the binaries on target.

Thanks for help and have a nice day.