Unexpected TINF_DATA_ERROR when using uzlib_uncompress with small input and output buffers

[RyanLindemanCAE]

I am working on an embedded project where RAM is very limited. The firmware update image is compressed (and encrypted with AES) using the Deflate algorithm. My embedded processor doesn't have enough RAM to uncompress the entire image so I need to do it piece mill using uzlib. When I attempt to make multiple calls to uzlib_uncompress it results in receiving a TINF_DATA_ERROR at some point before the end of the image has been reached.
To reproduce the exact scenario I did the following steps:

• Create an image sufficiently large enough to trigger the error (I did 10000 bytes):
  openssl rand -out orig.bin 10000
• Modify tgzip to allow for creating a compressed image with only the compressed bytes:

diff --git a/examples/tgzip/tgzip.c b/examples/tgzip/tgzip.c
index d869d38..33eb24d 100644
--- a/examples/tgzip/tgzip.c
+++ b/examples/tgzip/tgzip.c
@@ -99,21 +99,7 @@ int main(int argc, char *argv[])

     /* -- write output -- */

-    putc(0x1f, fout);
-    putc(0x8b, fout);
-    putc(0x08, fout);
-    putc(0x00, fout); // FLG
-    int mtime = 0;
-    fwrite(&mtime, sizeof(mtime), 1, fout);
-    putc(0x04, fout); // XFL
-    putc(0x03, fout); // OS
-
     fwrite(comp.outbuf, 1, comp.outlen, fout);
-
-    unsigned crc = ~uzlib_crc32(source, len, ~0);
-    fwrite(&crc, sizeof(crc), 1, fout);
-    fwrite(&len, sizeof(len), 1, fout);
-
     fclose(fout);

     return 0;

./tgzip orig.bin compressed.bin

• Use the attached decomp.c example to try to decompress the above file:
  decomp.txt

Instructions for compiling it provided (I use WSL on Windows).

The goal is to decompress a series of bytes to feed into the firmware upgrade of another processor over a serial connection so each uzlib_uncompress call would be done before sending the uncompressed bytes over the serial connection.
Thank you for any help you can provide in troubleshooting what is going wrong.

From my actual image, you can see that with a buffer size of 128, I only get slightly into the 2nd call before it fails:

Started new block: type=2 final=0
codelen counts: 7 0 2 2 2 3 1 2 0 0 0 0 0 0 0 0
codelen offsets: 0 0 0 2 4 6 9 10 12 12 12 12 12 12 12 12
lit code lengths (275): 5 6 7 7 6 7 7 8 7 8 8 8 8 8 8 8 7 8 9 8 8 8 9 8 7 9 8 8 8 8 9 9 6 7 7 7 8 8 9 9 7 9 8 7 8 8 10 9 7 9 8 8 9 9 9 10 9 10 9 8 9 10 10 10 7 9 7 8 8 9 6 8 7 8 8 7 8 9 9 8 9 10 10 10 9 10 11 11 10 10 10 9 10 10 11 10 7 10 10 9 9 10 10 10 7 9 10 10 10 10 11 11 7 10 10 9 9 9 10 12 7 9 10 9 9 10 10 9 8 9 9 9 10 10 10 10 9 11 10 10 10 9 11 10 8 9 9 8 9 9 11 10 9 9 9 9 10 10 11 11 9 11 10 9 10 11 11 10 9 9 10 9 10 10 10 9 8 7 8 9 9 8 10 11 10 8 9 9 10 8 10 7 9 10 9 9 10 10 10 10 10 10 10 10 10 9 11 11 7 7 9 9 10 10 11 11 8 9 9 10 10 10 10 9 8 10 10 10 10 10 9 6 9 9 10 9 10 10 10 11 6 8 9 9 9 9 9 6 6 7 7 7 7 7 7 6 13 4 5 5 7 7 7 7 7 7 10 11 9 7 9 9 12 12 13
dist code lengths (30): 7 0 0 5 6 6 6 6 5 6 6 5 4 4 4 4 4 5 4 5 4 5 4 4 5 6 5 5 5 7
codelen counts: 0 0 0 0 1 3 9 38 43 77 79 20 3 2 0 0
codelen offsets: 0 0 0 0 0 1 4 13 51 94 171 250 270 273 275 275
codelen counts: 2 0 0 0 9 10 7 2 0 0 0 0 0 0 0 0
codelen offsets: 0 0 0 0 0 9 19 26 28 28 28 28 28 28 28 28
03b40148019001bdd9a2040000000000
ffffffffffffffffffffffffffffffff
ffffffffffffffffffffffffffffffff
4150503279c63b75b3c3dcd21db6064d
0002000000000400e02b010000000120
4ff080430122c3f87825704708b500f0
8ff8fff7f5ff00f029f808bd00f01f02
40090128044b02f5e07218bf4ff0a043
032143f82210704700030050

Inflated bytes (result=-3, len=140)

The failure appears to be this comparison:
offs > d->dest - d->destStart

[pfalcon]

Thanks for the report. Unfortunately, currently I don't have time for the active uzlib maintenance. Sorry for the bot reply though, I made sure the ticket stays open until better times.

[jomar-honsi]

I have encountered the same issue. The challenge is the offset that is computed based on the table dist_base at the start of tinflate.c. This is an offset into the entire file indexing backwards from the current processed part. Thus, for small buffers it simply doesn't work. In the file that I use (generated by Linux gzip), the offset grows to nearly 0x8000 so the buffer needs to be at least this big.

In the example by @RyanLindemanCAE the buffer size is 128 but you can see already from the codelen offsets dump that there is a problem.

I have not yet figured out a way to make inflate work with smaller buffers, so any feedback is appreciated. I wonder if anything can be done on the compression side ? Any feedback is appreciated.

[IllestDillest]

Any feedback is appreciated.

Based on what I've been able to read, the gzip compression algorithm expects a sliding window size of 32kBytes, and it appears that it is not adjustable either up or down.

If memory constrained, this may be useful, I haven't tried it yet, but expect that I may need to soon.