Skip to content

optimize hdiffz -m memory; multi-thread patch base; #431

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
sisong merged 48 commits into from
Sep 19, 2025
Merged

optimize hdiffz -m memory; multi-thread patch base; #431

sisong merged 48 commits into from
Sep 19, 2025

Conversation

@sisong
Copy link
Owner

@sisong sisong commented Sep 19, 2025 •
edited
Loading

  • optimize $hdiffz -m required memory size when newData similar to oldData;
  • patch_single_stream support pre-read some old data for optimize disk read, when patch with large cache memory;
  • now, opened xxh3 & xxh128 ChecksumPlugin when dir-diff, checksum fast; (NOTE: need v4.12 patcher);
  • cmdline hpatchz support show progress;
  • cmdline hpatchz support multi-thread for single compressed diff (create by $hdiffz -SD);
    multi-thread used for I/O & decompress, maybe help when patch;
  • recode C++ multi-thread code to C code base, so remove _IS_USED_CPP11THREAD define.
  • update min NDK version from r16b to r23c; ( android 14 API to 16; remove .so for armeabi )
  • Android .so makefile build support directory patch (default closed, need set DIR=1);
  • when hdiffz test diffFile, all check-patcher run with stream mode(slower than run in memory);
  • printf some long time diff process info;

close #166 close #227
ref #427 ref #396

sisong added 30 commits July 31, 2025 17:53
@sisong
recode _writeToNewOrDiff_by() clearer state;
be6d717
@sisong
recode: translate chinese comments into english comments;
26f238a
@sisong
android .so add support directory patch (default closed, need set DIR…
d7a49df 
…=1);
@sisong
remove _IS_USED_CPP11THREAD (for used MT in C);
b7abc22
@sisong
recode CPP MT API, added C API;
1927bcc
@sisong
recode parallel_import.cpp to parallel_import_c.c; add struct hpatch_…
e3d7ce0 
…mt_t,hinput_mt_t,houtput_mt_t;
@sisong
add struct hcache_old_mt_t;
fe4fc25
@sisong
rename mt code files;
c7996f1
@sisong
struct hinput_mt_t support decompress stream as a uncompressed stream;
887d376
@sisong
recode struct houtput_mt_t API as a hpatch_TStreamOutput;
78accc5
@sisong
recode added _patch_cache_all_old() for $hpatchz -m support single co…
0080a3a 
…mpressed diff;
@sisong
struct _TOutStreamCache support 0 cache size (all write to stream);
14fea94
@sisong
patch MT api support single compressed diff;
a5b50e5
@sisong
recode hpatch mt, add struct hpatch_mt_base_t;
0eec80d
@sisong
add hpatch_mt_registeCondvar(), when onError or onFinish, got broadcast
71fa313
@sisong
Merge branch 'master' into dev/patch-by-mt
60bab6c 
# Conflicts:
#	Makefile
#	libParallel/parallel_channel.h
#	libParallel/parallel_import.cpp
#	libParallel/parallel_import.h
@sisong
$hpatchz support multi-thread for single compressed diff (create by $…
95d657d 
…hdiffz -SD);
@sisong
update multi-thread (threadNum>1) patching notes, ...
bd2238c
@sisong
now patch_single_stream support _IS_NEED_CACHE_OLD_BY_COVERS optimize…
c5be1d5 
… disk read; (MT mode todo)
@sisong
recode cache part of old data logic;
a3d6690
@sisong
update unit_test for mt patch;
f4ddff3
@sisong
recode dir_patch's root path code;
6ee43ef
@sisong
recode MT code;
360ce7c
@sisong
update vesion; patch thread default 1;
17e6f30
@sisong
android .so default closed multi-thread;
a65e98c
@sisong
recode bufList & bufSize init for MT;
c18fcc0
@sisong
move hpatch_mt_base_t func code;
feb71ea
@sisong
add C_ATOMIC API;
8f22d68
@sisong
recode wait threads end;
e264e64
@sisong
hpatch mt support onFinishThread(finishThreadListener);
f7f967b
sisong added 18 commits September 5, 2025 11:13
@sisong
recode wait all thread end code to struct _hthreads_waiter_t;
16a4ae4
@sisong
add c_channel_t C API;
5e2563d
@sisong
reopen vbuf for c file;
e1aa3f3
@sisong
update min NDK version from r16b to r23c; ( android 14 API to 16; rem…
5aca25c 
…ove .so for armeabi )
@sisong
remove C_ATOMIC API; struct hpatch_mt_t not use hpatch_mt_base_t;
c1aaa6c
@sisong
when diff -t , all check-patcher run with stream; increase check-patc…
362e043 
…her & checksum ... buf size;
@sisong
move code file "match_inplace.h" path;
775ad91
@sisong
recode struct TMatchBlock to TMatchBlockBase & TMatchBlockMem & TMatc…
e34db39 
…hBlockStream; recode struct TDiffData;
@sisong
support map_streams_befor_serialize optimize serialize speed;support …
28bfe25 
…map_streams_befor_serialize optimize serialize speed;
@sisong
recode serialize_compressed_diff() & stream_serialize_compressed_diff…
2659a4b 
…() to one; recode TCoversOptimMB<TMatchBlockMem> to TCoversOptimMem, TCoversOptimMB<TMatchBlockStream> to TCoversOptimStream;
@sisong
recode diff stream by block, optimize mem size; remove hdiff_in_mem()…
c603ba7 
…, used hdiff_by_stream();
@sisong
patch support show progress;
c86cf4f
@sisong
now, opened xxh3 & xxh128 ChecksumPlugin when dir-diff, checksum fast…
eb2bb30 
…; (NOTE: need v4.12 patcher);
@sisong
add hpatch_TProgressStreamOutput read_writed (vcd patcher needed)
d9e1eea
@sisong
Merge branch 'master' into v5.0-dev/op-diff-mem
31477c7
@sisong
when stdout redirected to a file, not show progress
672af3f
@sisong
printf some long time diff process info; remove not used readFileAll();
3600cd4 
ref #396
@sisong
add printf a long time diff process info;
78e0647
@sisong sisong merged commit d65858b into master Sep 19, 2025
18 of 20 checks passed
@sisong sisong deleted the v5.0-dev/op-diff-mem branch September 19, 2025 13:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Progress when applying a patch 加个进度变量?

2 participants

@sisong