Epic: Compression & batched-render correctness (LZ4 ceiling, payload header, render_list parity) #5

New issue

Closed

opened 2026-06-29 06:00:07 +00:00 by p13marc · 0 comments

p13marc commented

2026-06-29 06:00:07 +00:00

(Migrated from github.com)

Summary

Compression and the batched-render path have correctness bugs that silently corrupt or drop data.

Why now (0.5.0)

These cause hard failures on large frames and silent data loss for batched buffers — unacceptable for a media transport.

Tasks

LZ4 16 MB ceiling. decompress() hardcodes MAX_DECOMPRESSED_SIZE = 16 * 1024 * 1024 (compression.rs:178) because LZ4 raw blocks don't carry the decompressed size — any frame decompressing to >16 MB fails. Prepend a u32/varint length header to LZ4 payloads (zstd/gzip are self-describing and fine).
Robust negotiation header. Add a magic/length header to compressed payloads so a receiver built without compression features errors clearly instead of forwarding compressed garbage downstream. Stop relying on smuggling gst.compression through the user.* attachment namespace.
Fix inverted LZ4 level semantics vs. the documented "higher = better compression" (LZ4 higher = faster/less).
render_list() parity. It currently sends b.as_slice() directly (zenohsink/imp.rs:1337), bypassing compression and send-buffer-meta — batched buffers silently lose timing metadata and aren't compressed. Route list items through the same encode path as render(), or refuse render_list when those features are enabled.
Dead dropped stat. Statistics.dropped is exposed as a property but never incremented. Wire it up (e.g. drop-mode congestion / ring-buffer eviction — see Epic 7) or remove it.

Acceptance criteria

A >16 MB raw frame round-trips through LZ4 compression.
A non-compression receiver fed compressed input produces a clear error (not garbage downstream).
Batched buffers preserve compression + buffer metadata identically to single-buffer render.
dropped reflects reality or is removed.

References

Report §4.7, §4.9.
src/compression.rs:178 (LZ4 ceiling), level handling.
src/zenohsink/imp.rs:1337 (render_list raw send).

Risks / breaking changes

Adding a header to compressed payloads breaks 0.4.x compression wire compatibility — intentional and versioned (breaking-change).

## Summary Compression and the batched-render path have correctness bugs that silently corrupt or drop data. ## Why now (0.5.0) These cause hard failures on large frames and silent data loss for batched buffers — unacceptable for a media transport. ## Tasks - [ ] **LZ4 16 MB ceiling.** `decompress()` hardcodes `MAX_DECOMPRESSED_SIZE = 16 * 1024 * 1024` (`compression.rs:178`) because LZ4 raw blocks don't carry the decompressed size — any frame decompressing to >16 MB **fails**. Prepend a `u32`/varint length header to LZ4 payloads (zstd/gzip are self-describing and fine). - [ ] **Robust negotiation header.** Add a magic/length header to compressed payloads so a receiver built **without** compression features errors clearly instead of forwarding compressed garbage downstream. Stop relying on smuggling `gst.compression` through the `user.*` attachment namespace. - [ ] **Fix inverted LZ4 level semantics** vs. the documented "higher = better compression" (LZ4 higher = faster/less). - [ ] **`render_list()` parity.** It currently sends `b.as_slice()` directly (`zenohsink/imp.rs:1337`), **bypassing compression and `send-buffer-meta`** — batched buffers silently lose timing metadata and aren't compressed. Route list items through the same encode path as `render()`, or refuse `render_list` when those features are enabled. - [ ] **Dead `dropped` stat.** `Statistics.dropped` is exposed as a property but never incremented. Wire it up (e.g. drop-mode congestion / ring-buffer eviction — see Epic 7) or remove it. ## Acceptance criteria - A >16 MB raw frame round-trips through LZ4 compression. - A non-compression receiver fed compressed input produces a clear error (not garbage downstream). - Batched buffers preserve compression + buffer metadata identically to single-buffer render. - `dropped` reflects reality or is removed. ## References - Report §4.7, §4.9. - `src/compression.rs:178` (LZ4 ceiling), level handling. - `src/zenohsink/imp.rs:1337` (`render_list` raw send). ## Risks / breaking changes Adding a header to compressed payloads **breaks 0.4.x compression wire compatibility** — intentional and versioned (`breaking-change`).