v3.3.2

🟢 The maintainer is looking for a job! CV: https://marshal.dev

What's Changed

• Fix DASL testing CI by @MarshalX in #99
• Fix leading zero handling of CIDs in DAG-CBOR by @MarshalX in #98

Full Changelog: v3.3.1...v3.3.2

v3.3.1

🟢 The maintainer is looking for a job! CV: https://marshal.dev

What's Changed

• Add DASL testing harnesses to CI workflow by @MarshalX in #97
• Fix error handling in array and map length checks by @MarshalX in #95
• Fix error handling in CID and ULEB128 processing functions by @MarshalX in #96

Full Changelog: v3.3.0...v3.3.1

v3.3.0

🟢 The maintainer is looking for a job! CV: https://marshal.dev

What's Changed

❗ Almost x2 speed for CAR decoding (Bluesky firehose), ~25-35% faster DAG-CBOR decoding, and ~17% faster CID decoding!

I do have more ideas on what to try to speed it up even further! For example, applying BOLT on top of the existing PGO.

⚡ Optimizations

• Huge performance gain for DAG-CBOR decoding and CID by replacing deprecated Rust crate libipld with cbor4ii by @vmx in #80 and even contributed upstream!
• Optimize DAG-CBOR encoding for arrays and maps by @MarshalX in #88
• Optimize CAR decoding by reducing allocations by @MarshalX in #90
• Optimize executable size (panic = "abort") by @MarshalX in #86

Other

• Add pytest-benchmark GitHub Actions workflow by @MarshalX in #82
• Add lint and format check GHA workflow by @MarshalX in #83
• Add CONTRIBUTING.md by @MarshalX in #85
• Update CodSpeed by @MarshalX in #81
• Fix formatting in profiling by @MarshalX in #84

🏎️ Performance comparison. Before (v3.2.0) VS After (v3.3.0):

You can find more benchmark results here: #92

New Contributors

• @vmx made their first contribution in #80

Full Changelog: v3.2.0...v3.3.0

v3.2.0

What's Changed

❗ Possible breaking change for you: dropped support for PyPy 3.9 and PyPy 3.10 (both no longer supported upstream) in #77. CPython is still supported from 3.8 to 3.14.

• Add Python 3.14 support by @moreati in #74
• Update Rust to 1.90 by @MarshalX in #75
• Update PyO3 to 0.27.1 by @MarshalX in #75
• Fix repository URL specification in pyproject.toml file by @moreati in #73
• Fix clippy warnings by @MarshalX in #76
• Fix CodSpeed by @MarshalX in #78

New Contributors

• @moreati made their first contribution in #73

Full Changelog: v3.1.1...v3.2.0

v3.1.1

What's Changed

• Bump rust to 1.87.0 by @MarshalX in #67
• Specify Rust version using env variable in workflows by @MarshalX in #67
• Fix bignum encoding by @MarshalX in #66

Full Changelog: v3.1.0...v3.1.1

v3.1.0

What's Changed

• Add type stubs by @zzstoatzz in #56
• Add PyPy 3.11 support by @MarshalX in #63
• Update PyO3 to 0.25.1 with performance optimizations by @MarshalX in #63
• Update README by @MarshalX in #64
• Fix panic on encoding map with integer keys by @MarshalX in #60
• Fix panic on non UTF-8 string by @MarshalX in #59
• Fix smallest and largest CBOR integer roundtrip by @MarshalX in #58
• Fix decoding of infinities and NaNs according to specs by @MarshalX in #62
• Fix handling of concatenated objects / CBOR sequence by @MarshalX in #65

New Contributors

• @zzstoatzz made their first contribution in #56
• @makew0rld reported a lot of bugs in #57 during the work on test suite for DASL

Full Changelog: v3.0.1...v3.1.0

v3.0.1

What's Changed

• Update PyO3 to 0.23.4 by @MarshalX in #54
• Specify the dynamic version in pyproject.toml by @fxcoudert in #53
• Fix stack overflow when decoding deeply nested structures like arrays and maps by @MarshalX in #51

Possible breaking change for you: dropped support for PyPy 3.8

New Contributors

• @fxcoudert made their first contribution in #53

Full Changelog: v3.0.0...v3.0.1

v3.0.0

What's Changed

❗ Breaking changes

• Drop Python 3.7 support by @MarshalX in #50

🆕 Features

• Add Python 3.13 support by @MarshalX in #50

Other

• Update PyO3 to 0.22.5 by @MarshalX in #50
• Update decode_car signature, bytes CIDs instead of str in README by @snarfed in #49

New Contributors

• @snarfed made their first contribution in #49

Full Changelog: v2.0.0...v3.0.0

v2.0.0

What's Changed

❗ Breaking changes

CID in CAR block keys and CID DAG-CBOR are now in raw byte form instead of stringified one!

• Encode and decode CID as bytes (major 2) instead of string (major 3) by @MarshalX in #36

⚡ Bunch of new optimizations

• Optimize duplicate check by @MarshalX in #34
• Optimize type checking by @MarshalX in #37
• Optimize keys duplication check by @MarshalX in #39
• Optimize array creation by @MarshalX in #40
• Optimize string encoding by @MarshalX in #41

🆕 Features

• Add CID encoding (encode_cid(bytes) -> str) by @MarshalX in #38
• Add DAG-CBOR profiling by @MarshalX in #35

Other

• Fix upload to PyPI by @MarshalX in #33
• Update CodSpeed to v3 by @MarshalX in #47
• Update PyO3 to 0.22.2 by @MarshalX in #48
• Lock rust toolchain on v1.76.0 because higher versions segfaults during PGO gather by @MarshalX in #46

🏎️ Performance comparison. Before (v1.2.3) VS After (v2.0.0):

Raw bench results

Before (v1.2.3):

Hello World Decode:
===================
libipld  : 144 ns

Hello World Encode:
===================
libipld  : 150 ns

Realistic Decode Tests:
=======================
canada.json.dagcbor            libipld  : 5.11 ms (197.31 MB/s)
citm_catalog.json.dagcbor      libipld  : 3.43 ms (95.31 MB/s)
github.json.dagcbor            libipld  : 0.17 ms (270.36 MB/s)
twitter.json.dagcbor           libipld  : 2.15 ms (178.96 MB/s)

Realistic Decode Car Tests:
===========================
atproto_repo.car               libipld  : 119.97 ms (154.22 MB/s)

Realistic Encode Tests:
=======================
canada.json.dagcbor            libipld  : 1.19 ms (848.65 MB/s)
citm_catalog.json.dagcbor      libipld  : 1.66 ms (196.42 MB/s)
github.json.dagcbor            libipld  : 0.11 ms (429.15 MB/s)
twitter.json.dagcbor           libipld  : 0.87 ms (443.77 MB/s)

After (v2.0.0):

Hello World Decode:
===================
libipld  : 124 ns

Hello World Encode:
===================
libipld  : 116 ns

Realistic Decode Tests:
=======================
canada.json.dagcbor            libipld  : 4.65 ms (216.57 MB/s)
citm_catalog.json.dagcbor      libipld  : 2.96 ms (110.37 MB/s)
github.json.dagcbor            libipld  : 0.15 ms (311.00 MB/s)
twitter.json.dagcbor           libipld  : 1.75 ms (219.74 MB/s)

Realistic Decode Car Tests:
===========================
atproto_repo.car               libipld  : 92.44 ms (200.13 MB/s)

Realistic Encode Tests:
=======================
canada.json.dagcbor            libipld  : 1.00 ms (1011.47 MB/s)
citm_catalog.json.dagcbor      libipld  : 1.60 ms (203.57 MB/s)
github.json.dagcbor            libipld  : 0.06 ms (733.96 MB/s)
twitter.json.dagcbor           libipld  : 0.56 ms (686.31 MB/s)

Test stand: M1 Pro, Python 3.12.

I highly recommend using Python 3.12. The lower version will be slower.

P.S. Breaking change not only gives a performance boost but also aligns implementations with other languages.

Full Changelog: v1.2.3...v2.0.0

v1.2.3

What's Changed

⚡ Thanks to PGO and new Bound API in pyo3 v0.21 I achieved huge performance boost ones again! The pyo3 team did an awesome work! Many thanks to all contributors! Also, I implemented own CAR reader that gives +11%

• Migrate to pyo3 v0.21 Bound API by @MarshalX in #23
• Add PGO-optimized wheels by @MarshalX in #30, #21
• Add own implementation of CAR reader by @MarshalX in #28
• Add IPLD Codec Fixtures by @MarshalX in #19
• Add CAR and CID benchmarks by @MarshalX in #20
• Improve work with numbers by @MarshalX in #25
• Use refs in args and Bound in return type where possible by @MarshalX in #31

🏎️ Performance comparison. Before (v1.2.2) VS After (v1.2.3):

Raw bench results

Before (v1.2.2):

Hello World Decode:
===================
libipld  : 275 ns

Hello World Encode:
===================
libipld  : 237 ns

Realistic Decode Tests:
=======================
canada.json.dagcbor            libipld  : 7.51 ms (134.15 MB/s)
citm_catalog.json.dagcbor      libipld  : 4.97 ms (65.71 MB/s)
github.json.dagcbor            libipld  : 0.23 ms (201.43 MB/s)
twitter.json.dagcbor           libipld  : 2.70 ms (142.51 MB/s)

Realistic Decode Car Tests:
===========================
atproto_repo.car               libipld  : 170.09 ms (108.77 MB/s)

Realistic Encode Tests:
=======================
canada.json.dagcbor            libipld  : 1.89 ms (533.01 MB/s)
citm_catalog.json.dagcbor      libipld  : 2.86 ms (114.02 MB/s)
github.json.dagcbor            libipld  : 0.14 ms (335.64 MB/s)
twitter.json.dagcbor           libipld  : 1.07 ms (358.30 MB/s)

After (v1.2.3):

Hello World Decode:
===================
libipld  : 130 ns

Hello World Encode:
===================
libipld  : 143 ns

Realistic Decode Tests:
=======================
canada.json.dagcbor            libipld  : 5.24 ms (192.11 MB/s)
citm_catalog.json.dagcbor      libipld  : 3.49 ms (93.51 MB/s)
github.json.dagcbor            libipld  : 0.18 ms (257.51 MB/s)
twitter.json.dagcbor           libipld  : 2.11 ms (182.31 MB/s)

Realistic Decode Car Tests:
===========================
atproto_repo.car               libipld  : 123.72 ms (149.55 MB/s)

Realistic Encode Tests:
=======================
canada.json.dagcbor            libipld  : 1.22 ms (823.65 MB/s)
citm_catalog.json.dagcbor      libipld  : 1.69 ms (192.80 MB/s)
github.json.dagcbor            libipld  : 0.11 ms (410.94 MB/s)
twitter.json.dagcbor           libipld  : 0.80 ms (481.72 MB/s)

Test stand: M1 Pro, Python 3.12.

I highly recommend using Python 3.12. The lower version will be slower.

Full Changelog: v1.2.2...v1.2.3