Compression implementation improvement ideas

[pfalcon]

• genlz77: as DEFLATE is limited to 32K dictionary size, can store uint16_t offsets in hashtable, instead of current uint8_t*
• genlz77: should accept HASH_BITS as param, hashtable ptr as param, MAX_OFFSET (i.e. dict size) as param
• defl_static: probably can cut size of mirrorbytes
• defl_static: coderecord: no need to store max
• defl_static: coderecord: min can be uint16_t
• defl_static: coderecord: code can be uint8_t
• defl_static: coderecord: extrabits can be uint8_t

[TerryE]

• genlz77 dictionary should have a runtime sizing option,
• Don't statically allocate RAM for tables used per call or at least have configuration options to allocate on heap and built them dynamically with init / close bookends. This is essential for use the library on IoT devices.
• Consider the option of using an N-way hash e.g. N×4 as this will improve compression ratios for a small runtime penalty. 1024×4 performs better than 4096×1.

PS Here are some initial results on one of my test images for % size reduction compared to the 2048/1 case.

hash/way   1       2       4       8
128     14.63%   6.05%   1.05%  -2.63%
256     10.10%   2.87%  -1.60%  -4.15%
512      4.40%  -0.97%  -4.36%  -5.69%
1024     2.09%  -2.85%  -4.90%  -5.86%
2048     0.00%  -4.04%  -5.37%  -6.06%

The time (real) varied from 4mSec on the 128/1 case up to 5mSec on the 2048/8. By way of comparison my M×N brute force aglo achieved a -17.60% size reduction at a (real) runtime of 125mSec. Not sure if you think that it's worth it.

[TerryE]

@pfalcon, one specific suggestion: this line where you realloc the output buffer every 64 bytes dominates your run time. I use a simple iterative allocator where I initially a guess compression factor (6x) to prealloc the output buffer based on the input file size. If and when this is exhausted, I then use the actual compression factor so far to extrapolate the final size and do the next realloc. OK this might alloc the few extra Kb, but a few reallocs is a lot better than 4K reallocs on a 256K file.