Tests
476
Throughput
4.2 GB/s
Documentation
15.9K lines
Published
PyPI + crates.io
What is rypipe
rypipe is a reusable ingestion framework, not a single-format parser. It separates format-specific parsing from format-agnostic execution. Add a new format by implementing two small traits: Splitter and RecordParser.
rypipe itself does not ship parsers. Those live in separate adapter packages. Install the engine plus the adapters you need. The framework handles parallel scheduling, memory-bounded execution, query pushdown, and a chainable pipeline API.
Problem
Parsing row-oriented byte streams into columnar formats (Apache Arrow) usually requires writing a full engine per format. Each format gets its own parallel scheduler, memory manager, and execution infrastructure. The result is duplicated effort and inconsistent performance characteristics across formats.
Why it's hard
The difficulty is separating format-specific parsing (two small traits) from format-agnostic execution (parallel scheduling, bounded memory, query pushdown, zero-copy Arrow export) while maintaining throughput at memory bandwidth. The adapter contract test suite verifies that any adapter produces identical output across single-thread, parallel, and bounded execution modes.
Constraints
- Bounded memory: cannot load entire files into RAM
- Parallel execution must scale linearly with cores
- Pipeline stages must fuse into the parse loop (no Python round-trip)
- Adapter API must be minimal: two traits, not a full engine
Key decisions
Two-trait adapter model
Adding a new format requires implementing Splitter and RecordParser. The engine handles everything else.
Predicate-first evaluation
Filters push down into the Rust parse loop. Rows that fail are rejected before any Python object is created.
Layout prediction via memcmp
Predicts row structure from byte patterns, skipping column discovery when the schema is known.
Zero-copy Arrow export
Data moves from parse buffers directly into Arrow arrays without intermediate copies.
Origin
rypipe was originally developed as the ingestion engine for crxml, a Crystal Reports XML adapter. The engine design, performance characteristics, and API were shaped by production use with crxml. crxml is the reference adapter that proved the model: complex nested schemas, large files, parallel processing, and advanced filtering.
Tradeoffs
- Adapters must be written per format (not a universal parser)
- Rust core adds build complexity vs. pure Python
- Schema must be known upfront for maximum throughput