Skip to content

Commit 85eed17

Browse files
grtlrgrtlr
authored
Move Arrow transforms to re_arrow_combinators crate (#11805)
### Related * #11635 * #11789 ### What Allows us to factor out reusable patterns (with Rerun semantic information) without polluting `re_arrow_util` with a dependency to `re_types`.
1 parent 2137249 commit 85eed17

19 files changed

+542
-495
lines changed

‎ARCHITECTURE.md‎

Lines changed: 31 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -99,11 +99,11 @@ Of course, this will only take us so far. In the future we plan on caching queri
9999
Here is an overview of the crates included in the project:
100100

101101
<picture>
102-
<img src="https://static.rerun.io/crates/6c9a072dd87630c7fb0085b09d46168c5fd4249c/full.png" alt="">
103-
<source media="(max-width: 480px)" srcset="https://static.rerun.io/crates/6c9a072dd87630c7fb0085b09d46168c5fd4249c/480w.png">
104-
<source media="(max-width: 768px)" srcset="https://static.rerun.io/crates/6c9a072dd87630c7fb0085b09d46168c5fd4249c/768w.png">
105-
<source media="(max-width: 1024px)" srcset="https://static.rerun.io/crates/6c9a072dd87630c7fb0085b09d46168c5fd4249c/1024w.png">
106-
<source media="(max-width: 1200px)" srcset="https://static.rerun.io/crates/6c9a072dd87630c7fb0085b09d46168c5fd4249c/1200w.png">
102+
<img src="https://static.rerun.io/crates/9ff35e6610b4636dcb4a20c856f12e9b2161cb6d/full.png" alt="">
103+
<source media="(max-width: 480px)" srcset="https://static.rerun.io/crates/9ff35e6610b4636dcb4a20c856f12e9b2161cb6d/480w.png">
104+
<source media="(max-width: 768px)" srcset="https://static.rerun.io/crates/9ff35e6610b4636dcb4a20c856f12e9b2161cb6d/768w.png">
105+
<source media="(max-width: 1024px)" srcset="https://static.rerun.io/crates/9ff35e6610b4636dcb4a20c856f12e9b2161cb6d/1024w.png">
106+
<source media="(max-width: 1200px)" srcset="https://static.rerun.io/crates/9ff35e6610b4636dcb4a20c856f12e9b2161cb6d/1200w.png">
107107
</picture>
108108

109109
<!-- !!! IMPORTANT!!!
@@ -197,7 +197,7 @@ Update instructions:
197197
### Data flow
198198

199199
| Crate | Description |
200-
|----------------------|-------------------------------------------------------------------|
200+
|--------------------- |-------------------------------------------------------------------|
201201
| re_redap_client | Official client for the Rerun Data Protocol |
202202
| re_redap_tests | Official test suite for the Rerun Data Protocol |
203203
| re_data_loader | Handles loading of Rerun data from file using data loader plugins |
@@ -218,30 +218,31 @@ Update instructions:
218218

219219
### Utilities
220220

221-
| Crate | Description |
222-
| ------------------ | ------------------------------------------------------------------------------------ |
223-
| re_analytics | Rerun's analytics SDK |
224-
| re_arrow_util | Helpers for working with arrow |
225-
| re_auth | Authentication and authorization helpers |
226-
| re_byte_size | Calculate the heap-allocated size of values at runtime |
227-
| re_capabilities | Capability tokens |
228-
| re_case | Case conversions, the way Rerun likes them |
229-
| re_crash_handler | Detect panics and signals, logging them and optionally sending them to analytics. |
230-
| re_error | Helpers for handling errors. |
231-
| re_format | Miscellaneous tools to format and parse numbers, durations, etc. |
232-
| re_int_histogram | A histogram with `i64` keys and `u32` counts, supporting both sparse and dense uses. |
233-
| re_log | Helpers for setting up and doing text logging in the Rerun crates. |
234-
| re_mcap | Convert MCAP into Rerun-compatible data. |
235-
| re_memory | Run-time memory tracking and profiling. |
236-
| re_perf_telemetry | In and out of process performance profiling utilities for Rerun & Redap |
237-
| re_ros_msg | Parsing and deserializing ROS messages |
238-
| re_smart_channel | A channel that keeps track of latency and queue length. |
239-
| re_span | An integer range that always has a non-negative length |
240-
| re_string_interner | Yet another string interning library |
241-
| re_tracing | Helpers for tracing/spans/flamegraphs and such. |
242-
| re_tuid | 128-bit Time-based Unique Identifier |
243-
| re_uri | Parsing and constructing of Rerun URIs |
244-
| re_video | Video decoding library |
221+
| Crate | Description |
222+
| -------------------- | ------------------------------------------------------------------------------------ |
223+
| re_analytics | Rerun's analytics SDK |
224+
| re_arrow_combinators | Type-safe, composable transformations for Arrow arrays |
225+
| re_arrow_util | Helpers for working with arrow |
226+
| re_auth | Authentication and authorization helpers |
227+
| re_byte_size | Calculate the heap-allocated size of values at runtime |
228+
| re_capabilities | Capability tokens |
229+
| re_case | Case conversions, the way Rerun likes them |
230+
| re_crash_handler | Detect panics and signals, logging them and optionally sending them to analytics. |
231+
| re_error | Helpers for handling errors. |
232+
| re_format | Miscellaneous tools to format and parse numbers, durations, etc. |
233+
| re_int_histogram | A histogram with `i64` keys and `u32` counts, supporting both sparse and dense uses. |
234+
| re_log | Helpers for setting up and doing text logging in the Rerun crates. |
235+
| re_mcap | Convert MCAP into Rerun-compatible data. |
236+
| re_memory | Run-time memory tracking and profiling. |
237+
| re_perf_telemetry | In and out of process performance profiling utilities for Rerun & Redap |
238+
| re_ros_msg | Parsing and deserializing ROS messages |
239+
| re_smart_channel | A channel that keeps track of latency and queue length. |
240+
| re_span | An integer range that always has a non-negative length |
241+
| re_string_interner | Yet another string interning library |
242+
| re_tracing | Helpers for tracing/spans/flamegraphs and such. |
243+
| re_tuid | 128-bit Time-based Unique Identifier |
244+
| re_uri | Parsing and constructing of Rerun URIs |
245+
| re_video | Video decoding library |
245246

246247
### Test crates
247248

‎Cargo.lock‎

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8358,6 +8358,16 @@ dependencies = [
83588358
"web-sys",
83598359
]
83608360

8361+
[[package]]
8362+
name = "re_arrow_combinators"
8363+
version = "0.27.0-alpha.8+dev"
8364+
dependencies = [
8365+
"arrow",
8366+
"insta",
8367+
"re_arrow_util",
8368+
"thiserror 2.0.17",
8369+
]
8370+
83618371
[[package]]
83628372
name = "re_arrow_ui"
83638373
version = "0.27.0-alpha.8+dev"
@@ -8390,7 +8400,6 @@ dependencies = [
83908400
"re_tuid",
83918401
"re_types_core",
83928402
"serde_json",
8393-
"thiserror 2.0.17",
83948403
]
83958404

83968405
[[package]]
@@ -9462,7 +9471,7 @@ dependencies = [
94629471
"nohash-hasher",
94639472
"parking_lot",
94649473
"percent-encoding",
9465-
"re_arrow_util",
9474+
"re_arrow_combinators",
94669475
"re_build_info",
94679476
"re_build_tools",
94689477
"re_byte_size",

‎Cargo.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -93,6 +93,7 @@ rerun-cli = { path = "crates/top/rerun-cli", version = "=0.27.0-alpha.8", defaul
9393

9494
# crates/utils:
9595
re_analytics = { path = "crates/utils/re_analytics", version = "=0.27.0-alpha.8", default-features = false }
96+
re_arrow_combinators = { path = "crates/utils/re_arrow_combinators", version = "=0.27.0-alpha.8", default-features = false }
9697
re_arrow_util = { path = "crates/utils/re_arrow_util", version = "=0.27.0-alpha.8", default-features = false }
9798
re_auth = { path = "crates/utils/re_auth", version = "=0.27.0-alpha.8", default-features = false }
9899
re_byte_size = { path = "crates/utils/re_byte_size", version = "=0.27.0-alpha.8", default-features = false }

‎crates/top/re_sdk/Cargo.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,7 @@ server = ["dep:re_smart_channel", "dep:tokio"]
4545

4646

4747
[dependencies]
48-
re_arrow_util.workspace = true
48+
re_arrow_combinators.workspace = true
4949
re_build_info.workspace = true
5050
re_byte_size.workspace = true
5151
re_chunk.workspace = true

‎crates/top/re_sdk/src/lenses/ast.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ use arrow::{
1111
datatypes::DataType,
1212
};
1313

14-
use re_arrow_util::transform::{self, Transform as _};
14+
use re_arrow_combinators::{self as transform, Transform as _};
1515
use re_chunk::{
1616
ArrowArray as _, Chunk, ChunkComponents, ChunkId, ComponentIdentifier, EntityPath, Timeline,
1717
TimelineName,

‎crates/top/re_sdk/src/lenses/error.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
#[derive(Debug, thiserror::Error)]
44
pub enum Error {
55
#[error(transparent)]
6-
Transform(#[from] re_arrow_util::transform::Error),
6+
Transform(#[from] re_arrow_combinators::Error),
77

88
#[error(transparent)]
99
Arrow(#[from] arrow::error::ArrowError),

‎crates/top/re_sdk/src/lenses/op.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66

77
use std::sync::Arc;
88

9-
use re_arrow_util::transform::{self, Transform as _};
9+
use re_arrow_combinators::{self as transform, Transform as _};
1010
use re_chunk::{
1111
ArrowArray as _,
1212
external::arrow::{

‎crates/utils/re_arrow_combinators/Cargo.toml‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
[package]
2+
name = "re_arrow_combinators"
3+
authors.workspace = true
4+
description = "Type-safe, composable transformations for Arrow arrays."
5+
edition.workspace = true
6+
homepage.workspace = true
7+
include.workspace = true
8+
license.workspace = true
9+
publish = true
10+
readme = "README.md"
11+
repository.workspace = true
12+
rust-version.workspace = true
13+
version.workspace = true
14+
15+
[lints]
16+
workspace = true
17+
18+
[package.metadata.docs.rs]
19+
all-features = true
20+
21+
22+
[dependencies]
23+
re_arrow_util.workspace = true
24+
25+
arrow.workspace = true
26+
thiserror.workspace = true
27+
28+
[dev-dependencies]
29+
insta.workspace = true

‎crates/utils/re_arrow_combinators/README.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
# re_arrow_combinators
2+
3+
Part of the [`rerun`](https://github.com/rerun-io/rerun) family of crates.
4+
5+
[![Latest version](https://img.shields.io/crates/v/re_arrow_combinators.svg)](https://crates.io/crates/re_arrow_combinators)
6+
[![Documentation](https://docs.rs/re_arrow_combinators/badge.svg)](https://docs.rs/re_arrow_combinators?speculative-link)
7+
![MIT](https://img.shields.io/badge/license-MIT-blue.svg)
8+
![Apache](https://img.shields.io/badge/license-Apache-blue.svg)
9+
10+
Type-safe, composable transformations for Arrow arrays.
11+
12+
Provides building blocks for constructing complex data transformations through composition.
13+
These transformations are designed to be used as primitives for user-defined functions (UDFs)
14+
in query engines like DataFusion, as well as in SDK features like lenses.

0 commit comments

Comments
 (0)