When using with HTTP Chunked Transfer Encoding, 6.7MB becomes 10.5MB due to chunking overheads

[SevereOverfl0w]

This is a big problem for me. I am using transit-json on a Web endpoint, and I am streaming the result of encoding. The stream is using HTTP chunked transfer encoding. This essentially indicates the overall size is unknown, and informs you of how many bytes are about to come repeatedly until it is finished.

Due to the flushes from Transit, these chunks end up very small. Usually just a string and some open/close parens. This overhead causes the 6.7MB of transit to transport as 10.5MB of HTTP. That is, it gets significantly bigger when transferred this way. For this application and expected timing, this directly impacts 10s of load time.

transit-clj seems to flush after every write (https://github.com/cognitect/transit-java/blob/cff7111c2081fc8415cd9bd6c6b2ba518680d660/src/main/java/com/cognitect/transit/impl/AbstractEmitter.java#L189). I would guess this is related to #43.

[SevereOverfl0w]

Additionally, these chunking overheads seem to drastically limit the ability of gzip to function.

[computeremotion]

https://clojure.org/community/contributing

https://ask.clojure.org/

[puredanger]

Those links are primarily for Clojure itself or the Clojure contrib libraries. transit-clj is not a Clojure core project, it is managed by Cognitect (admittedly this is mostly the same people, but they have different licenses and contribution processes).

This is the proper place to log this issue for this project.

[computeremotion]

sry. my bad.

[zeitstein]

Reporting from Slack. Saw this exact behaviour when using transit-body-interceptor from pedestal, which uses .flush. Using the strategy mentioned by @borkdude in #43 improves matters significantly.