esp_delta_ota fails with error "esp_delta_ota: Error while applying patch: Function not implemented." (IEC-113)

[ntremble-sst]

Answers checklist.

• I have read the documentation of the component in question and the issue is not addressed there.
• I have searched the issue tracker for a similar issue and not found a similar issue.

General issue report

Hi,

I created the esp_delta_ota example project and had the HTTP sample work correctly on the ESP32-S3 target.

I then attempted to incorporate that code into my project using the same IDF 5.1.2 as I used for the example, but it does not complete. The primary differences are:

• the binaries are much larger (5MB). My test has only a small change, but the patch comes out at 89KB heatshrink compressed.
• the patch is being read from an eMMC card, having been received via CAN at some earlier point. Standard POSIX file i/o.

When I run the code it commences feeding the first byte after the 64 byte header (magic+SHA256+reserved). It gets in about a dozen bytes, then the detools function dfpatch_size() returns 'not implemented' with a size value of 656.

I have tried different sized blocks (64 bytes, 1024 bytes, 2048 bytes) but always the same error.

This log shows what is happening. I have added some logging to detools, so the line numbers will be a little out compared to the pure code, but only by a few lines. I can fix that if it helps.

I found it necessary to modify the python script somewhat to get it to run on Windows, but I did try running the original on Linux and got the same output, so pretty sure that's not the problem.

esp_delta_ota 1.10 direct from the component library as a managed component.
detools 0.53 (part of the same component)
heatshrink 0.4.1 (part of the same component)
esp-idf 5.1.2
python 3.11.2
ESP32-S3-WROOM-1-N16R8

Thank you.

d (67496) OTA: enUpdateFirmware(): feeding block to patch updater
detools process_init() line 1221: entry
detools chunk_get_no_check() line 87: data: 0x04
detools chunk_get_no_check() line 87: data: 0x80
detools chunk_get_no_check() line 87: data: 0xA7
detools chunk_get_no_check() line 87: data: 0xAC
detools chunk_get_no_check() line 87: data: 0x05
detools process_dfpatch_size() line 1249: entry
detools patch_reader_decompress() line 1051: call algorithm specific decompress()
detools chunk_get_no_check() line 87: data: 0x44
detools chunk_get_no_check() line 87: data: 0xC8
detools chunk_get_no_check() line 87: data: 0x42
detools patch_reader_decompress() line 1051: call algorithm specific decompress()
detools chunk_get_no_check() line 87: data: 0xA0
detools process_dfpatch_size() line 1259: size 656 not implemented
E (67549) esp_delta_ota: Error while applying patch: Function not implemented.

[ntremble-sst]

The following files are attached as a single zip file:

• a log from the patch generation, including detools info output.
• a putty log of a failed patch application. The actual patch application commences at line 91, failure occurs at line 283.
• my modified detools.c/h, redone such that line numbers match the original unmodified file.

I've also included the patch file as a separate zip in case it is of use.

Note I have set up a Kconfig to control the definitions that were in the detools CMakeLists.txt (now commented out), you will see the changes related to that at the top of detools.h. They are working because (a) I can see the included/excluded code regions in my IDE, and (b) I did set breakpoints to confirm the correct [heatshrink] code was being reached.

The putty log includes a couple of dumps, being the header (magic+SHA256+reserved=64 bytes) and the first block of data from the actual patch as generated by detools.

As a slight aside, how hard would it be to add support for better compression? My test show that e.g. LZMA compression performs far better, yielding patch files in the hundreds of bytes range, but unfortunately, while provided for by the code, it is not actually implemented. For our application the size of heatshrink compressed patches is likely to be a real issue as our updates will be transferred over a cellular link. We have plenty of storage, but bigger files have more chance of being interrupted during transfer, as well as the data cost considerations.

my-files.zip
patch-95-96.zip

[ntremble-sst]

I just tried feeding our patch file through the Espressif example code. Same result. See attached image.

[ntremble-sst]

I found an issue in my code, and now it appears to be working. I will close this issue and reopen/recreate if necessary.

[ntremble-sst]

I have determined the issue is in the patch generation i.e. the python script.

I would expect that given the same two input files the patch generation script should produce identical output on every run, so I should be able to take the sample old and new binaries and produce the identical patch file.

Using the sample binary and patch files provided with the https_delta_ota example (part of the esp_delta_ota v1.10 component) I am unable to regenerate the patch file using a fresh ESP-IDF v5.1.2 installation on a Debian 12.5.0 virtual machine (or on Windows 10).

The screen shot below shows my problem. If I use the sample binaries and attempt to recreate the identical patch file I get a python error, and the patch generation fails. That says to me that the python script as supplied is not the one that was used to create that sample patch file - or perhaps it was, but the environment has changed such that it no longer works.

I hit this same issue on Windows, where I modified the script to make it work, but it still does not generate the same patch file given the same input files. Note I specified --chip esp32s3 as that is what I require, but I did also try with --chip esp32, that was no better.

This leads to my earlier issue, as the patch I am generating is in some manner faulty, causing the patch process to fail.

[hmalpani]

Hello @ntremble-sst
The python script esp_delta_ota_patch_gen.py is used to create patch file for ESP-IDF application binaries. It parses some data in the old binary which is used in header of the patch file created. Therefore that python script cannot be used for other files. I recommend you directly use detools command as follows: detools create_patch -c heatshrink <base_binary> <new_binary> <patch_file_name>

I think we can update the documentation so that such confusion doesn't occur in future.

[ntremble-sst]

Hello @hmalpani,

Yes, I spent quite some time working out what that script does, and I am using a cut down version of it to generate the header (magic+SHA256+28 bytes reserved) for my own binaries. There does seem to be an issue with the way the python script calls detools however, so I wound up using python to create the header and a direct call to detools to create the patch, followed by a concatenation step. I am now successfully generating patch files for my own binaries with the same structure, but obviously with different hashes in the header.

It would be good if the documentation was updated a bit.

What would be really good would be if the other compression methods were supported by the IDF for decompression. For our large binaries heatshrink does not do a particularly good job. My tests indicate LZMA would be much better. Unfortunately, although the conditionals are there, some code is missing.

Thank you for your response.

[ntremble-sst]

Hello @hmalpani,

Just to let you know I have it working now for any pair of binary files I care to feed it.

I will close this issue.

[mjgci]

Hello everyone,

I am having package installation issues or executing the script successfully.
I have the standard Windows installation setup for ESP-IDF and for Visual Studio Build Tools.

I try to use the ESP-IDF paths for detools installation however, I am unable to proceed.

The errors after installing the requirements with the detools package results in these:

"C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.44.35207\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -IC:\Espressif\v5.4.2\tools\Espressif\python_env\idf5.4_py3.11_env\include -IC:\Espressif\v5.4.2\tools\Espressif\tools\idf-python\3.11.2\include -IC:\Espressif\v5.4.2\tools\Espressif\tools\idf-python\3.11.2\Include "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.44.35207\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.44.35207\ATLMFC\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\VS\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.26100.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.26100.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.26100.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.26100.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.26100.0\cppwinrt" /Tcdetools/libdivsufsort/divsufsort.c /Fobuild\temp.win-amd64-cpython-311\Release\detools\libdivsufsort\divsufsort.obj
divsufsort.c
"C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.44.35207\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -IC:\Espressif\v5.4.2\tools\Espressif\python_env\idf5.4_py3.11_env\include -IC:\Espressif\v5.4.2\tools\Espressif\tools\idf-python\3.11.2\include -IC:\Espressif\v5.4.2\tools\Espressif\tools\idf-python\3.11.2\Include "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.44.35207\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.44.35207\ATLMFC\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\VS\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.26100.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.26100.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.26100.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.26100.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.26100.0\cppwinrt" /Tcdetools/sais/sais.c /Fobuild\temp.win-amd64-cpython-311\Release\detools\sais\sais.obj
sais.c
detools/sais/sais.c(31): fatal error C1083: Cannot open include file: 'Python.h': No such file or directory
error: command 'C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.44.35207\bin\HostX86\x64\cl.exe' failed with exit code 2

I have posted to the ESP32 Forums as well. The link is: Esp delta ota + Secure Boot + Flash Encryption

I found a similar issue discussed over MS Visual Studio Forums: How to fix fatal error C1083: Cannot open include file: 'stdio.h': No such file or directory

@ntremble-sst @hmalpani Your setup work fine and are able to execute the script esp_delta_ota_patch_gen.py with the correct headers for the patch.
Any help for the above issue will be appreciated. Thanks.

[ntremble-sst]

@mjgci I do not use Visual Studio for this work, so I really don't think I'm able to help, other than to say it looks like an environment error.

Unfortunately I think you will just need to go through it step by step and work out just what is missing. When you have it properly sorted you should be able to build identical files from either the CLI or the IDE.