marcpardo/netring
1
0
Fork 0
Code Issues 9 Pull requests 1 Projects Releases 6 Packages Wiki Activity Actions
6 releases 25 tags
  • 0.29.0 7226b474e4

    0.29.0 — flowscope 0.22, threat/detection redesign & the observability surface Stable

    p13marc released this 2026-07-04 10:19:23 +00:00 | 0 commits to master since this release

    0.29.0 — 2026-07-04 — flowscope 0.22, threat/detection redesign & the observability surface

    Depends on flowscope 0.22. A breaking release (still pre-1.0). Migration:
    docs/MIGRATING_0.28_TO_0.29.md.

    Fixed

    • AsyncXdpCapture lost wakeup under spaced-out traffic (issue
      #131) — the AF_XDP async
      poller cleared AsyncFd readiness and returned Pending without re-arming the
      waker, so a packet arriving during an idle gap could hang until the next
      unrelated wakeup. poll_read_ready / poll_drain_views now re-poll after
      clear_ready() until either the ring has data or Pending registers a fresh
      waker. Affected the AF_XDP flow/L7 streams and XdpMq since 0.26.

    Changed

    • flowscope 0.20 → 0.22 (issue
      #132) — adopts the typed
      DetectorKind (with attack_technique() → MITRE ATT&CK IDs), the
      Detector/DetectorRegistry trait surface, app_proto classification,
      NameMap, BandwidthByKey/Attribution, WindowedQuantiles, and the
      IpFragmentReassembler. The flow feature now pulls flowscope/analysis.
    • Breaking — IOC surface retyped onto flowscope's threat backbone (issue
      #124) — monitor::ioc::IocSet
      is now flowscope's detect::ioc::IocSet (with IocKind/IocMatch); the
      netring-local risk types were removed. New FlowAnalyzer-based flow-risk
      scoring: flow_analysis() / flow_analysis_with(cfg) / on_analyzed_flow(..)
      emit one flow_risk anomaly per flow at its max severity. flow_risk() is now
      deprecated sugar for flow_analysis().

    Added

    • Detector registry + MITRE ATT&CK tagging (issue
      #127) — .detector(impl Detector) / .detectors(registry) run flowscope NDR detectors in-Monitor;
      ATT&CK technique IDs ride as an attack_technique observation label
      (auto-appended from kind.attack_technique()), lifted into native fields by
      the OCSF and EVE sinks. No AnomalySink::write break.
    • SSH (HASSH) + QUIC fingerprints, TLS PQ key-share (issue
      #128) — SshFingerprint +
      on_ssh_fingerprint (ssh), QuicFingerprint + on_quic_fingerprint
      (quic+tls), and TlsFingerprint.pq_key_share.
    • YARA hot-reload (issue
      #53) — ReloadHandle::set_yara
      / has_yara swap a running monitor's YARA-X rules via arc-swap (load_full
      inside the flush, zero hot-path cost). ReloadHandle's rustdoc gains the
      non-reloadable-surface table, closing the hot-reload epic.
    • Community ID accessors (issue
      #123) — community_id() on
      the flow events and Ctx::community_id(); EVE tls + anomaly records carry it.
    • Application-protocol classification + encrypted-DNS hook (issue
      #133) —
      TlsFingerprint.app_protocol, on_encrypted_dns(..) over DoT/DoH/DoQ,
      known-resolver detection, EVE tls app_proto.
    • Structured DNS handler + passive IP→name map (issue
      #120) — on_dns(..) over a
      DnsView, name_map() / on_name(..) fed by DNS responses (+ TLS SNI
      claims), and Ctx::names(ip).
    • Owner-attributed bandwidth (issue
      #130) —
      with_flow_attribution(..) + owner_bandwidth() / on_owner_bandwidth(..)
      track throughput by a caller-defined owner (tenant / container / subscriber)
      with an unattributed bucket; owner_bandwidth() without an attribution hook is
      a build error.
    • Traffic aggregation (issue
      #121) — aggregate() /
      on_aggregate(..) compute top talkers, a host-pair traffic matrix, and top DNS
      names / TLS SNI over rolling windows, each dimension individually toggleable.
    • RED metrics (issue
      #122) — red() /
      on_red(..) compute request Rate, Error rate and Duration quantiles per
      protocol (DNS by rcode + timeouts with query→response latency; flows by end
      reason with connection lifetime).
    • Rotating + triggered pcap writers (issue
      #125) — RotatingPcapWriter
      (size / duration rotation + file-count / total-bytes retention) and a
      TriggeredPcapWriter + PreTriggerRing + TriggerHandle pre-trigger ring
      buffer for capturing the lead-up to an event.
    • Network-namespace capture (issue
      #126) — a NetNs handle
      (from_name / from_pid / from_path / current) with a scoped-thread
      run_in(..), and CaptureBuilder::netns(&NetNs) to build an AF_PACKET capture
      inside a namespace. Requires CAP_SYS_ADMIN.
    • IP-fragment reassembly (issue
      #134) —
      reassemble_ip_fragments() / reassemble_ip_fragments_with(IpFragConfig)
      (off by default) feed each IPv4 fragment to flowscope's
      IpFragmentReassembler before the flow tracker parses it, synthesizing a
      reassembled frame (L2/VLAN prefix preserved, IPv4 header re-checksummed) once a
      datagram completes — so a payload split across fragments reaches the L7 parsers
      whole. Only the tracker input is reassembled; taps / packet subs / p0f keep the
      raw wire fragments. Overlapping fragments (teardrop / RFC 5722) are dropped
      (evasion signal). Validated by a replay test: a fragmented DNS response parses
      only when reassembly is armed.
    Downloads
  • 0.28.0 7b53723c65

    0.28.0 — flowscope 0.20, AF_XDP maturity & the pre-1.0 API sweep Stable

    p13marc released this 2026-06-29 21:44:05 +00:00 | 27 commits to master since this release

    Added

    • Hot-reload of a packet-tier .expr() filter (issue
      #53) —
      ReloadHandle::set_packet_filter(index, expr) swaps the filter predicate of a
      running monitor's packet subscription (registration order) without dropping
      packets, completing the userspace hot-reload legs alongside set_ioc /
      set_sigma. The predicate lives behind a lock-free arc-swap cell read per
      frame with an allocation-free load() (the zero-copy drain stays Δ 0); a
      frame in flight reads the old-or-new filter, never a torn one. The expr is
      parsed and validated before the swap, so a bad string returns Err and leaves
      the live filter untouched. ReloadHandle::packet_filter_count() reports how
      many packet subs can be reloaded. Caveat (live capture only): the kernel
      cBPF/XDP prefilter is the build-time union of the original predicates, so a
      reload can narrow freely but widening past that union drops the newly-wanted
      frames in-kernel until rebuild (offline replay reloads fully).
    • Load-shedding policy under overload (issue
      #54, action half) — a
      monitor::overload::LoadShedder couples the existing hysteresis
      OverloadDetector with a ShedPolicy { Observe, ShedNewFlows, SampleFlows { keep } } and honest SheddingStats { admitted, shed }. Fed the windowed
      drop_rate via on_capture_stats, it sheds new flows at the dispatch
      boundary while Emergency holds — admit_new_flow(flow_hash) returns
      ShedDecision::{Admit,Shed}, deterministic in a direction-invariant key
      hash so both legs of a flow share one verdict, and counts every decision
      (never a silent drop). Already-admitted flows keep flowing (elephant-friendly).
      netring owns the mechanism + policy + counters; the heuristic (keep rate,
      what's sheddable) is the app's. Boundary: userspace dispatch-level shedding
      only — the kernel-ring (Block/DropOldest/DropNewest) levers and the XDP
      elephant-shunt (Suricata bypass) need the eBPF pre-filter map + a real NIC and
      stay the hardware-gated follow-up under #44. New example:
      monitor_load_shedding.
    • AF_XDP flow + L7 streams and multi-interface fan-in (issue
      #104, part of the flowscope
      0.20 adoption #108) —
      AsyncXdpCapture::flow_stream(extractor) is the AF_XDP analogue of
      AsyncCapture::flow_stream, and AsyncXdpMultiCapture::open([...]) fans N
      (each internally multi-queue) NICs into one XdpMultiFlowStream yielding
      TaggedEvent { source_idx, event } — the motivating shape for a TX/RX-split
      tap across two NICs. SessionStream / DatagramStream are now source-agnostic,
      so xdp_cap.flow_stream(ext).session_stream(parser) / .datagram_stream(parser)
      work over AF_XDP too, fanned in as XdpMultiSessionStream /
      XdpMultiDatagramStream. The dedup / pcap-tap / monotonic-clamp / track logic
      is shared verbatim with AF_PACKET via a new AsyncFlowSource trait.
    • Declarative Backend::Auto capture facade (issue
      #106) —
      MonitorBuilder::capture(iface, Backend::Auto) probes the interface and picks
      the backend (AF_XDP native → SKB → AF_PACKET), logs the chosen plan, and
      surfaces it via Monitor::resolved_capture_plan().
    • Canonical orientation + opt-in infer_tcp_initiator on netring flow
      events (flowscope #118 / #122). FlowStarted<P>, FlowPacket, and
      SessionEvent::{Started,Application} gained orientation: flowscope::Orientation
      (Forward / Reverse) — the deterministic, address-sorted direction axis that
      survives a tap-merge / multi-queue arrival race (the right key for biflow dedup
      and Community ID ordering). MonitorBuilder::infer_tcp_initiator(true) /
      MultiStreamConfig::with_infer_tcp_initiator(true) enable SYN-based initiator
      inference for split-capture topologies (a no-op for a single tap).
    • Per-direction capture-leg binding (issue
      #105, flowscope #120) —
      source_idx now survives the monotonic-timestamp clamp (flowscope #69), so a
      shared/merged tracker surfaces FlowStats::source_idx_{forward,reverse} +
      capture_leg_inconsistent (the tap-miswire / asymmetric-routing IOC) on a
      flow's stats, exposed through merged_flow_stream on AF_PACKET and AF_XDP.
    • FlowRecord::to_ipfix_record() (feature ipfix, issue
      #33) — view a netring
      export::FlowRecord as flowscope's canonical, IANA-IE-keyed
      ipfix::FlowRecord, the shape every flowscope emitter (IPFIX wire / CSV / Zeek
      conn.log / NDJSON) renders from. netring keeps its own ergonomic FlowRecord
      as the stable public type (the core API stays decoupled from flowscope's IE
      registry); this is the opt-in bridge for code that wants the canonical record.
      The mapping now also preserves netring's full 8-variant EndReason in
      flowscope's original_end_reason shadow field — IE 136 flowEndReason
      collapses it to 5 RFC states, so the un-collapsed reason (Fin vs Rst vs
      ParseError vs Evicted vs …) survives for consumers reading the IE record
      directly. (tcpControlBits stays 0 — flowscope's FlowStats doesn't
      accumulate per-flow TCP flags, so there is no source for it; a flowscope
      dependency, not a netring gap.)
    • NIC RX flow steering (issue
      #15) — netring::xdp::steer
      programs the NIC's ethtool RX-NFC rules so multi-queue / sharded AF_XDP capture
      can pin chosen flows to chosen RX queues, closing the AF_XDP↔DPDK gap. A typed,
      validated FlowRule::tcp().dst_port(443).to_queue(3) packs into
      ethtool_rx_flow_spec (IPv4/IPv6 family-checked, kernel mask polarity handled);
      RxSteer::{open, insert, remove, rule_count} rides SIOCETHTOOL /
      ETHTOOL_SRXCLSRL* (the same plumbing as queue_count); SteerGuard
      (RxSteer::guarded) removes the rules on drop with all-or-nothing rollback, like
      PromiscGuard. Byte-packing + family validation are unit-tested cap-free and the
      lo insert degrades cleanly (-EOPNOTSUPP/-EPERM, no panic). Hardware-gated
      (ntuple-capable NIC + CAP_NET_ADMIN). Follow-ups left open under #15: FLOW_RSS
      contexts, rule enumeration (GRXCLSRLALL), XdpCaptureBuilder::steer convenience.
    • AF_XDP RX hardware metadata (issue
      #13) — surface NIC RX timestamp /
      RSS hash / VLAN tag (kernel 6.3+ XDP-hints kfuncs) through the AF_XDP capture path
      into flowscope::RxMetadata. afxdp::metadata defines the fixed 32-byte
      XdpRxMeta BPF↔userspace contract (a magic+version word gates the not-zeroed
      headroom; per-field validity flags). UMEM headroom is reserved opt-in via
      XdpSocketBuilder::rx_metadata(true) — off by default, so zero overhead and no
      behaviour change      . XdpPacket::{timestamp, rx_metadata} surface the parsed
      values and the Monitor's AF_XDP arms feed PacketView::with_rx_metadata,
      preferring the hardware timestamp automatically. The companion redirect_meta.bpf.c
      is committed as source only — the .o needs the maintainer's BPF toolchain +
      a real NIC to compile and validate (the redirect_all.bpf.o convention), so real
      timestamps stay hardware-gated (ice/mlx5/gve); loopback / generic XDP exercise the
      degrade path (no metadata → software timestamp). The userspace contract is fully
      unit-tested incl. the degrade and out-of-bounds-descriptor cases.

    Changed (breaking — flowscope 0.20 adoption, #108)

    • SessionEvent is now a netring type (netring::flow::SessionEvent, also in
      the prelude). flowscope 0.20 retired its public SessionEvent (it became a
      crate-private engine carrier); netring now owns its session-stream event type.
      It is a drop-in with the same variants (Started / Application / Closed /
      FlowAnomaly / TrackerAnomaly) and carries the new additive orientation
      field. The Stream::Item of SessionStream / DatagramStream /
      PcapSessionStream / PcapDatagramStream (+ the TaggedEvent-wrapped multi
      variants) now carries it. Migration: use flowscope::SessionEvent →
      use netring::flow::SessionEvent. flowscope::FlowEvent is unchanged.

    • Offline pcap L7 drivers removed. flowscope 0.20 deleted the per-parser
      FlowSessionDriver / FlowDatagramDriver engines; AsyncPcapSource::{sessions, datagrams} and PcapFlowStream::{session_stream,datagram_stream} are unchanged
      at the call site but now drive a FlowTracker directly (same translation as the
      live SessionStream, so live/offline L7 are equivalent and in-flight flow state
      is preserved across session_stream/datagram_stream). PcapSessionStream:: driver() / PcapDatagramStream::driver() are removed — use .tracker().

    • Typed ParserKind replaces &'static str (flowscope #109) on
      ParserClosed<P>::parser_kind (field + ::new arg), SessionEvent::Application,
      and EventStream::parser_kind(). .as_str() recovers the original slug, so
      emitted JSON / metric labels are byte-for-byte unchanged. The Protocol::NAME
      and flowscope::parser_kinds::* &str constants are unchanged.

    • EVE flow_hash → community_id (flowscope #88). The eve-sink feature now
      pulls flowscope/community-id, so EVE output carries the standard Corelight
      Community ID (Zeek/Suricata/Arkime-interoperable) instead of the proprietary
      FNV-1a flow_hash. Re-key any dashboard/correlation joined on flow_hash;
      the in-process FNV hash stays available as KeyFields::stable_hash().

    • export::FlowRecord gains community_id: Option<String>, loses Copy
      (issue #33). The same Community
      ID rides the flow-export record (NDJSON gains a "community_id" key; the
      canonical IE-keyed IPFIX record carries it). Records are handed to exporters by
      &, so the hot path is unaffected; only code that copied a record out of a
      callback needs .clone().

    • FlowStream's first type parameter is now the capture source (issue
      #104) so one tracking loop is
      shared across AF_PACKET and AF_XDP. Only code that named FlowStream<S, …> /
      SessionStream<S, E, F> explicitly is affected (FlowStream<AsyncCapture<S>, E>
      / FlowStream<AsyncXdpCapture, E>); cap.flow_stream(…) call chains are not.

    • flowscope wire types are #[non_exhaustive] (flowscope #78) — affects only
      struct-literal construction (FiveTupleKey { .. } → FiveTupleKey::new(..),
      likewise DnsRecord::new / HttpRequest::new / Extracted::new / …) or
      exhaustive enum matches (add a _ arm). The free flowscope::<proto>::parse*()
      functions also moved Option<T> → Result<T, ParseError> (flowscope #85);
      netring handles its own L2/L3 parse sites internally.

      See docs/MIGRATING_0.27_TO_0.28.md
      for the full recipe-style migration guide.

    Changed (breaking — pre-1.0 API sweep, #37 §D/§F)

    • Sealed the deep-internal subscription-plumbing traits Subscribable and
      subscription::L7Fields (§D) via a private Sealed supertrait — they are now
      implemented only by netring. These are internal seams of the typed
      subscription engine (the install-onto-builder hook, and the L7-field accessors
      over flowscope's message types); neither is constructible/usable externally
      (the subscription tiers and SessionFields are pub(crate)), so sealing
      closes nothing a caller could reach and lets them gain methods before 1.0
      without a major bump. Sealing now is the conservative 1.0 move: removing a
      seal later is non-breaking, adding one isn't. subscription::FieldSource is
      intentionally left open       — implementing it is the supported way to evaluate a
      parsed .expr() Predicate (parse_expr) against your own data.
    • Renamed config::BuildError → config::BpfBuildError (and the crate-root
      re-export netring::BuildError → netring::BpfBuildError). There were two
      unrelated public BuildError enums — the BPF-filter compiler's and the
      Monitor builder's (error::BuildError) — and only the BPF one was re-exported
      at the crate root, so netring::BuildError silently meant "BPF compile error"
      while the Monitor one was reachable only as netring::error::BuildError. The
      rename removes that trap; the Monitor builder error keeps the name BuildError
      (now unambiguous). Migration: netring::BuildError → netring::BpfBuildError
      (or netring::config::BpfBuildError); the Error::Bpf(_) variant is unchanged.
    • ProtocolInitError's public tuple field is now private — construct it with
      ProtocolInitError::new(impl Into<String>) and read it with .message(). This
      lets the error carry more context later without another break. Migration:
      ProtocolInitError(s) → ProtocolInitError::new(s); read e.0 → e.message().

    Changed

    • IPFIX export now delegates the binary wire encoder to
      flowscope::ipfix::wire       (issue
      #33) — IpfixExporter maps each
      completed FlowRecord onto flowscope's IANA-IE-keyed FlowRecord and encodes it
      with flowscope::ipfix::wire::MessageBuilder over the canonical
      FLOWSCOPE_TEMPLATE_FLOW_IPV4/_IPV6 templates (IDs 256/257). flowscope now owns
      the IE registry + template lifecycle; netring keeps the I/O / transport rails
      (W: Write + Send, resend_templates_every, the async exporter). This deletes
      ~250 LoC of hand-rolled encoder that had drifted from the registry (the old
      template dropped the octetTotalCount/packetTotalCount and tcpControlBits
      IEs), and replaces the latent per-message sequenceNumber increment with the
      builder's per-Data-Record bookkeeping (RFC 7011 §3.1). The emitted records now
      carry the per-direction delta counts (IE 1/2, initiator) and the both-
      directions totals (IE 85/86); tcpControlBits (IE 6) is emitted as 0 until
      FlowRecord carries cumulative TCP flags (tracked in the deferred #33 re-key).
      The ipfix feature now pulls flowscope/ipfix-export. No public-API change
      — export::FlowRecord and IpfixExporter's surface are unchanged; only the bytes
      on the wire become the canonical, richer template.
    Downloads
  • 0.27.0 28eae8fd8e

    0.27.0 — 1.0 API sweep, threat-intel, ML features & Tier-2 protocols Stable

    p13marc released this 2026-06-24 20:33:04 +00:00 | 42 commits to master since this release

    The pre-1.0 stabilization release. Mostly additive — a large network-security-monitoring stack lands as opt-in features — with three breaking changes from the 1.0 API sweep. See docs/MIGRATING_0.26_TO_0.27.md and the full CHANGELOG.

    Breaking (each has a one-line fix — see the migration guide)

    • Capture::packets() is now a lending iterator (next_packet()/for_each()) — closes a .collect() use-after-free (#35).
    • #[non_exhaustive] on the growing public enums and library-produced output structs (#37).
    • The L7 capability markers HasSni/HasHttpHost/HasQname are sealed (#37).

    Added (all opt-in)

    • Threat intel & detection: IOC matching, YARA-X payload scanning, Sigma rule evaluation, live hot-reload of IOC/Sigma sets (reload_handle()), nDPI-style flow-risk scoring, RITA beacon detection.
    • Protocols: QUIC Initial SNI/ALPN, Tier-2 (SSH/HASSH, FTP, SMTP, NTP, SNMP, TFTP, RADIUS, Modbus, DNP3, STUN, WireGuard), Active Directory (SMB/Kerberos/LDAP/RDP), IPv6 NDP watch.
    • Asset & ML: passive asset::Inventory (ARP/NDP/LLDP/CDP/DHCP/SSDP/mDNS/NBNS), p0f OS fingerprinting, nPrint + CICFlowMeter feature export.
    • Output & ops: OCSF Detection-Finding sink, overload/backpressure detection, honest per-source DropStats, HW-timestamp fix, pcapng writer, symmetric RSS for fanout coherence.
    • 1.0 hardening: miri over the unsafe surface + 5 cargo-fuzz targets with a nightly deep-fuzz workflow.
    • Companion crate netring-exporters 0.3.0 adds Parquet + OTLP metrics.

    Depends on flowscope 0.19.

    Downloads
  • 0.26.0 775ce2b117

    netring 0.26.0 — AF_XDP multi-queue capture & promiscuous mode Stable

    p13marc released this 2026-06-16 09:29:26 +00:00 | 92 commits to master since this release

    0.26.0 — 2026-06-16 — AF_XDP multi-queue capture & promiscuous mode

    Completes the AF_XDP capture story (issues #4 + #6): promiscuous mode, the
    high-level XdpCapture (one socket per RX queue, Queues::Auto via ethtool),
    the Monitor xdp_queues single-reactor tier that removes the silent
    single-queue under-capture footgun, and the XdpShardedRunner line-rate tier
    (one Monitor per queue, busy-poll). Additive over 0.25 (existing code compiles
    unchanged). Depends on flowscope 0.16. Companion netring-exporters 0.1.1.

    Added

    • AF_XDP hardening (#6, M5 finish).

      • B1: netring::xdp::default_program(max_queues) now honors its argument —
        the XSKMAP is sized via aya's EbpfLoader::set_max_entries (was ignored; map
        fixed at 256). XdpCapture sizes the map to exactly its queue set.
      • F3: XdpCaptureBuilder::numa_auto() / XdpShardedRunner::numa_auto(true)
        bind every queue's UMEM to the NIC's NUMA node, read from
        /sys/class/net/<iface>/device/numa_node (new netring::xdp::interface_numa_node).
      • F1: XdpSocketBuilder::shared_umem documented as expert-only with the
        per-CPU FILL-queue race caveat; per-socket UMEM (the XdpCapture default)
        stays the blessed multi-queue path. (No shared-UMEM opt-in added to
        XdpCapture — it would be a footgun.)

    • Per-queue sharded AF_XDP capture: XdpShardedRunner (#6, M5 Tier 2).
      The line-rate multi-queue model — one Monitor (worker thread) per RX queue,
      the AF_XDP analogue of ShardedRunner (which shards AF_PACKET via
      PACKET_FANOUT). It builds one shared XdpCapture (one attached program, one
      socket per queue, one promiscuous guard) and hands each shard its socket via a
      new injection seam, so every queue gets full flow tracking on its own core
      (Suricata's threads: auto). Per-queue busy-poll (SO_BUSY_POLL /
      SO_PREFER_BUSY_POLL / SO_BUSY_POLL_BUDGET, also new on XdpCapture) + CPU
      pinning are the performance levers. Use MonitorBuilder::xdp_queues for the
      single-reactor (one-core) tier; XdpShardedRunner when one core can't keep up.
      Example: examples/xdp/xdp_sharded.rs.

    • Monitor multi-queue AF_XDP + async capture (#6, M3/M4).
      MonitorBuilder::xdp_queues(Queues) (monitor-wide, mirroring promiscuous)
      makes a self-loading AF_XDP interface capture every RX queue, not just
      queue 0 — removing the silent single-queue under-capture footgun on multi-queue
      NICs. Queues::Auto opens one socket per queue behind a single program and
      drains them through a unified round-robin (AnyBackend::XdpMq); the default
      stays Queues::Single(0) (no behavior change). New netring::AsyncXdpCapture
      is the tokio front for XdpCapture (per-queue AsyncFds, unified
      readable().await + owned recv(), into_parts() for the worker-per-queue
      model). This is the single-reactor tier; the sharded worker-per-queue tier
      (line rate) is still to come.

    • High-level multi-queue AF_XDP capture (#6).
      netring::xdp::XdpCapture opens one socket per RX queue (the only way to
      capture a whole multi-queue NIC — RSS spreads traffic across queues, which a
      single socket can't see even in promiscuous mode), loading one redirect
      program, registering each socket in its XSKMAP, attaching once, and draining
      them through a unified round-robin next_batch() / next_batch_blocking().
      Each socket gets its own UMEM (the safe default — sharing a UMEM across
      per-CPU sockets races on the FILL ring). Queue selection via
      Queues::{Single, Range, Auto}; Auto auto-detects the RSS queue count with
      netring::xdp::queue_count() (ETHTOOL_GCHANNELS) and falls back to queue 0.
      into_parts() hands out the per-queue sockets + a guard for the
      worker-per-queue model. XdpSocket::is_zerocopy() / XdpCapture::is_zerocopy()
      surface the bind mode (was log-only). This is phases M1–M2 of the 0.26 plan;
      the Monitor xdp_queues(...) integration that removes the single-queue
      under-capture footgun and an async wrapper follow.

    • Promiscuous mode for AF_XDP (#4).
      XdpSocketBuilder::promiscuous(bool) (per-socket) and a backend-agnostic
      MonitorBuilder::promiscuous(bool) (monitor-wide; applies to both AF_PACKET
      and AF_XDP capture interfaces) put the interface into promiscuous mode for
      the capture's lifetime, so AF_XDP can see traffic not addressed to the local
      MAC (SPAN/mirror ports, passive sniffing). Promiscuity is a netdev property —
      AF_XDP has no socket knob for it — so netring holds it through an auxiliary
      AF_PACKET socket joined to PACKET_MR_PROMISC (the same mechanism the
      AF_PACKET path uses). The kernel reference-counts dev->promiscuity and
      releases it automatically when the socket is dropped, including on crash; no
      manual restore, no leak. Default off. Two documented caveats:
      PACKET_MR_PROMISC does not raise the user-visible IFF_PROMISC flag, and on
      a multi-queue NIC a single XSK still only sees its bound queue's RSS share —
      full capture wants one socket per queue (examples/xdp/xdp_multiqueue.rs) or a
      single queue (ethtool -L <iface> combined 1).

    Downloads
  • 0.25.0 1dc5db409b

    netring 0.25.0 — subscriptions, async effects, performance & TX Stable

    p13marc released this 2026-06-15 13:47:06 +00:00 | 109 commits to master since this release

    0.25.0 — 2026-06-15 — subscriptions, async effects, performance & TX

    The complete capability release on the 0.24 keystone: typed 3-tier
    subscriptions + kernel filter pushdown, async read+effect handlers, perf &
    scaling (CPU pinning, dispatch-throughput numbers), the symmetric TX stack,
    the in-Monitor AF_XDP loader, UMEM hugepages/NUMA, and the netring-exporters
    companion crate. Additive over 0.24 (existing code compiles unchanged).
    Depends on flowscope 0.16.

    JA4S licensing fix + JA3/JA4 now actually populate

    • JA4S moved behind the opt-in ja4plus feature (off by default; excluded
      from the monitor / all-parsers umbrellas). JA4S is part of the JA4+ suite
      and is FoxIO License 1.1 (non-commercial; patent pending), not MIT/Apache.
      TlsFingerprint.ja4s only exists under ja4plus, which pulls flowscope's
      ja4plus (≥ 0.16). The default TLS fingerprint surface (JA3 + JA4 client)
      stays royalty-free / BSD. Commercial use of ja4plus requires a FoxIO OEM
      license — see docs/FINGERPRINTS.md. Depends on flowscope 0.16, which
      did the same gating upstream (LICENSE-FoxIO-1.1 + NOTICE).
    • Fix: the tls feature now enables flowscope/tls-fingerprints, so JA3 +
      JA4 client fingerprints actually compute (they were silently always-None
      before — the passthrough was missing). The monitor_ja4_fingerprint example
      now requires ja4plus (it demonstrates JA4S).

    Runtime filter strings — .expr() (Phase A4)

    • packet()/flow::<P>()/session::<P>().expr("tcp and dst port 443") — a small,
      dependency-free recursive-descent parser from a Wireshark-ish filter
      string to the same Predicate AST the typed combinators produce. One AST,
      two frontends: packet().expr("tcp and dst port 443") and
      packet().tcp().dst_port(443) are identical (a test pins this), so a runtime
      string lowers to the same userspace eval and kernel pushdown. This is the
      path for filters from config / CLI / a control plane.
    • Grammar: and/or/not (+ &&/||/!), parens, tcp/udp/icmp,
      [src|dst] port|host|net, vlan, bytes > N / packets > N, and
      tls.sni/http.host/dns.qname ~ GLOB. We deliberately do not depend on
      the dead wirefilter-engine crate (0.6.1/2019). parse() returns a
      ParseError (no panics) on malformed input.

    Subscription engine — typed tiers + filter predicates (Phase A1)

    The new front door (additive; on::<E> unaffected). A subscription pairs
    a strongly-typed tier with a filter [Predicate] and a handler:

    • Three tier constructors in netring::monitor::subscription: packet()
      (every frame), flow::<P: FlowProtocol>(), session::<P: MessageProtocol>().
      Invalid combinations are compile errors — flow::<Http>() and
      session::<Tcp>() don't compile; L7 glob filters are gated per protocol
      (session::<Tls>().sni_glob, session::<Dns>().qname_glob).
    • Typed filter combinators AND into one Predicate AST (proto / ports /
      host / net / vlan — kernel-pushable — plus byte/packet counts and L7
      sni/host/qname globs). The AST is shared by userspace evaluation
      (Predicate::eval over a FieldSource) and, in A2/A3, kernel pushdown;
      Atom::is_kernel_pushable() is the split classifier.
    • Packet tier wired end-to-end: MonitorBuilder::subscribe(packet()…​.to(h))
      runs the handler on every matching frame as a borrowed PacketView,
      synchronously inside the zero-copy drain before flow tracking — so the
      handler sees raw frames pre-tracking with no copy. Works on live capture and
      pcap replay. Monitors with no packet subs keep the track_into-only hot
      loop (dhat stays Δ 0); the run-loop future stays Send.
    • IpNet::contains(&IpAddr) (v4 + v6); dependency-free case-insensitive Glob.

    Kernel filter split + cBPF compiler (Phase A2 / A3a)

    • Predicate::kernel_approx() — the conservative split: a predicate over
      only kernel-pushable atoms that is a superset of the original (every frame
      the filter wants still passes). Userspace atoms relax to Always; Not is
      pushed only when fully kernel-pushable. The full predicate stays the
      userspace filter; the kernel side is a pure prefilter.
    • A classic-BPF compiler lowers a kernel-approx predicate to a BpfFilter
      (DNF → conjunctions of L2–L4 atoms, OR-unioned across packet subs), with a
      safe fallback to "no filter" for shapes it can't express (negations, etc.).
      Verified in-sandbox via the BpfFilter::matches software interpreter.

    Safe automatic kernel pushdown (Phase S1 / S2)

    The kernel prefilter is now computed from every consumer's traffic interest
    and auto-applied to the AF_PACKET capture — safely.

    • Each consumer declares its traffic interest: handlers via a new
      Event::traffic_class() (→ the protocol's Dispatch, default Any),
      protocol parsers via their Dispatch, packet subs via their filter. The
      Monitor folds them into the OR-union kernel_prefilter().
    • Because the union is a superset of every consumer's interest, the pushed
      filter can never drop a frame any consumer wants — starvation-free by
      construction      . It is fail-open: any "wants everything" consumer (a broad
      handler, exporter, tick/report, broadcast, bandwidth), or a union exceeding
      the cBPF clause budget, collapses it to "capture all" (no filter).
    • The run loop applies the union via set_filter on each AF_PACKET socket at
      start. A narrow monitor (protocol::<Tls>()) pushes tcp port 443/8443;
      adding on::<FlowStarted<Udp>> widens it to also pass UDP; adding
      on::<FlowPacket> collapses it to capture-all. Verified end-to-end via the
      BpfFilter::matches interpreter (tests/monitor_kernel_prefilter.rs).
      kernel_prefilter() stays public for inspection. dhat Δ 0 on the hot path.

    Flow & session tier dispatch (Phase S3)

    The flow and session tiers now deliver, at each tier's natural completion
    point     (Retina on_terminate semantics):

    • flow::<P>().bytes_over(N).to(handler) — fires once per flow, at FlowEnded,
      with the accumulated stats (so byte/packet-count filters are meaningful).
    • session::<P>().sni_glob("*.bank").to(handler) — fires with each parsed
      P::Message whose L7 fields (SNI / HTTP host / DNS qname, via a new
      L7Fields trait per message type) and flow 5-tuple match the filter.

    Both are sugar over the existing typed dispatch: .to() installs a
    predicate-gated on::<…> handler. A new Subscribable trait lets the one
    MonitorBuilder::subscribe accept any tier (packet / flow / session). The
    tier's traffic interest is recorded automatically (a superset of the filter),
    so it composes with the S1/S2 kernel-prefilter union safely. Verified end-to-end
    via pcap replay (replay_flow_tier_delivers_once_at_flow_end_gated_by_stats,
    replay_session_tier_dns_qname_glob_gates_delivery). dhat Δ 0.

    The table-driven AF_XDP map program (A3c, hardware-gated) and the runtime
    .expr() string frontend (A4) follow. (Design:
    plans/netring-0.25-subscription-engine-design.md.)

    Async read + effect handlers (Phase B1)

    • New MonitorBuilder::on_effect::<E>(handler) — an async handler that
      reads the Ctx synchronously (Fn(&E::Payload, &Ctx<'_>)) and returns
      a 'static future resolving to an Effects value: a deferred, owned
      description of the writes to apply (today: Effects::emit(anomaly) /
      and_emit; set_state/counter/enqueue are additive follow-ups). The
      run loop awaits the future, then applies the effects to the sink under a
      short &mut Ctx write phase. Because the handler never holds &mut Ctx
      across .await, the run-loop future stays Send — unlike a
      hypothetical Fn(&mut Ctx) -> Future shape. This closes the gap between
      on_async (payload-only, can .await but can't read Ctx) and sync on
      (reads Ctx but can't .await).
    • Effect handlers fire after the sync and async passes for the same
      lifecycle event, in registration order. Monitors that register no effect
      handlers pay nothing: the run loop gates the effect pass on
      effect_handler_count() > 0, so the per-event Ctx-rebuilding translation
      is skipped entirely. dhat steady state stays Δ 0 / 0.

    Dispatcher (Phase B2)

    • The per-monitor distinct-event-type cap is lifted (was a hard 16): the
      lookup table is inline (no hashing) for the first 16 types and spills to
      a hash map beyond, so there's no practical ceiling. dhat stays Δ 0.
    • Debug-only dispatcher type-tag asserts the TypeId → slot mapping stays
      consistent (turns a silent type-erasure desync into a loud test panic;
      zero release cost).

    In-Monitor AF_XDP loader + table-driven filter map (W1a)

    • MonitorBuilder::xdp_interface_loaded(iface) (feature xdp-loader): the
      Monitor attaches the built-in redirect-all XDP program and registers its
      socket on the XSKMAP itself — one-call AF_XDP capture, no external loader.
      (xdp_interface(iface) remains the bring-your-own-program form.)
    • Table-driven filter_redirect.bpf program + loader::filter_program() +
      XdpProgram::set_filter(proto, port, on): a BPF_MAP_TYPE_HASH-driven
      XDP program that redirects only {proto, port}-matching frames (the
      kernel-side early-shed), populated from the subscription union.
    • Fix: the loader now embeds bytecode with aya::include_bytes_aligned!
      instead of include_bytes!. The zero-copy ELF parse requires alignment;
      the plain macro loaded only when the static happened to land aligned and
      failed ("error parsing ELF data") in any build that also pulls tokio
      (i.e. every Monitor build) — so redirect-all was already broken for
      Monitor-on-AF_XDP. Guarded by a cap-free vendored_programs_parse_under_aya
      test.
    • Fix: XdpSocketBuilder::force_replace(true) no longer crashes with
      EINVAL — XDP_FLAGS_REPLACE is netlink-only and rejected by the link API
      on kernel ≥ 5.9, so it's no longer OR'd into the link-create flags; a failed
      attach returns an actionable error instead.

    Active-timeout flow export (W1c)

    • MonitorBuilder::export_active_timeout(period): emits interim FlowRecords
      for long-lived flows every period (NetFlow/IPFIX active-timeout semantics),
      not just one at FlowEnded. Breaking (additive type change):
      FlowRecord.reason is now Option<EndReason> — None marks an ongoing
      active-timeout snapshot (FlowRecord::is_ongoing()); IPFIX maps it to
      flowEndReason 0x02 (active timeout).

    EVE event_type:"tls" records (W1d)

    • netring::anomaly::{EveTlsSink, eve_tls_record} (features eve-sink + tls):
      Suricata-compatible event_type:"tls" EVE records (sni, ja3.hash, ja4,
      ja4s under ja4plus, alpn, 5-tuple, ISO-8601 timestamp) — the protocol-record
      companion to EveSink's event_type:"anomaly". Wire it via on_fingerprint.

    Resilience: backend Reopen + handler panic catching (W1e)

    • BackendErrorPolicy::Reopen: rebuilds a failed capture source in place (same
      kind + filter) so a transient fault (interface flap, driver reset) self-heals.
    • MonitorBuilder::catch_handler_panics(true): wraps sync handlers in
      catch_unwind, converting a panic into Error::HandlerPanic routed through
      the configured HandlerErrorPolicy (pair with Isolate to log + count +
      continue). Off by default; async handlers/effects are not covered.

    Performance & scaling (Phase C)

    • ShardedRunner::pin_cpus(true): pins each shard's OS thread to its core via
      sched_setaffinity (keeps flow state + RX ring + worker core-local).
    • benches/dispatch_throughput.rs: a cap-free userspace pps proxy
      (~4.7 Melem/s/core flow tracking on the dev box), run in CI. docs/PERFORMANCE.md
      documents the capture-vs-dispatch split, the dhat-Δ0 enforced gate, tuning
      levers (pushdown, sharding+pinning, fanout + symmetric-RSS pitfall, AF_XDP,
      busy-poll, hugepages), and an honest real-NIC-pending methodology.
    • monitor/tracing_json example: structured JSON logging of anomalies +
      telemetry via tracing-subscriber.

    TX symmetry (Phase D)

    • AsyncInjector::send_stream(stream, Option<TxPacer>): transmits every frame
      from a Stream<Item = impl AsRef<[u8]>>, optionally paced. TxPacer is a
      token-bucket pacer (packets_per_second / bits_per_second).
    • TX egress timestamping: InjectorBuilder::tx_timestamps(true) enables
      SO_TIMESTAMPING; Injector::read_tx_timestamp() /
      AsyncInjector::read_tx_timestamp() read the SCM_TIMESTAMPING egress
      timestamp off the error queue (hardware-preferred-else-software).

    AF_XDP UMEM hugepages + NUMA (W4)

    • XdpSocketBuilder::hugepages(bool) (MAP_HUGETLB, graceful fallback) +
      numa_node(u32) (mbind/MPOL_BIND, best-effort). After bind, a
      getsockopt(XDP_OPTIONS) check warns when the kernel fell back to COPY mode.

    netring-exporters companion crate (W5)

    • New workspace crate keeping heavy OTLP/Kafka deps out of core.
      OtlpAnomalySink (feature otlp, OTLP/HTTP-JSON logs via blocking ureq)
      and KafkaSink (feature kafka, rdkafka/librdkafka) both implement
      AnomalySink, so they drop into MonitorBuilder::sink(...).
    Downloads
  • 0.21.0 32879eb256

    netring 0.21.0 Stable

    p13marc released this 2026-06-12 07:29:29 +00:00 | 238 commits to master since this release

    Highlights

    • Send Monitor — flowscope 0.13's Driver<E>: Send + Sync unconditional. Plain #[tokio::main] (multi-thread) works without ceremony.
    • Per-CPU sharding via ShardedRunner::new(iface, FanoutMode::Cpu, group_id, n, build_shard) over PACKET_FANOUT.
    • Streaming subscribers — MonitorBuilder::with_broadcast::<P>() + Monitor::subscribe::<P>() -> EventStream<P::Message> (futures_core::Stream + Unpin).
    • Offline pcap replay — MonitorBuilder::pcap_source(path) + pcap_speed_factor(f) + Monitor::replay().
    • Graceful drain via MonitorBuilder::drain_timeout(Duration) + new Monitor::run_until_idle(window) stop condition.
    • Per-flow state — MonitorBuilder::flow_state::<T>(idle_timeout) + ctx.flow_state_mut::<T>().
    • pattern_detector! macro wrapping Arc<Mutex<D: DetectorScore>> (port_scan / beacon_detector / dga_query examples).
    • New sinks — EveSink (feature eve-sink, Suricata EVE JSON), MetricsSink (feature metrics, Prometheus facade), Sha256Sink (feature file-hash).
    • Build-time validation — BuildError::HandlerForUnregisteredProtocol, CounterNotRegistered, ProtocolNotBroadcast, PcapSourceRequired.
    • monitor-quickstart umbrella feature.
    • flowscope dep — 0.11.1 → 0.13.0 (skipping 0.12, absorbed in one bump).

    Backward-compat breaks

    Break Mitigation
    AnomalySink::write key narrows &dyn Debug → &dyn Key Custom sink impls update one method signature
    Monitor: Send propagates #[tokio::main(flavor = "current_thread")] becomes #[tokio::main]
    Legacy ProtocolMonitor / AnomalyMonitor / AnomalyRule gain #[deprecated] Removal in 0.22 — one-cycle deprecation window
    correlate::TimeBucketedCounter deleted; re-export from flowscope ::new(window, bucket) → ::new_unbounded(window, bucket)
    ProtocolSlot: Send supertrait No impact on user code

    Full migration recipes: docs/MIGRATING_0.20_TO_0.21.md

    Full changelog: CHANGELOG.md

    Caveat on Send Monitor

    The Monitor value is Send, but the future returned by Monitor::run_for / run_until_signal stays !Send because AsyncCapture borrows the !Sync mmap ring across awaits. Use tokio::select! to multiplex on the main task; ship anomalies to a spawned worker via ChannelSink when work needs to cross task boundaries.

    Downloads