Networked Audio · GPS Time · Real-Time Linux

GPS-Disciplined AVB & Realtime Remote Audio Exchange

A Raspberry Pi 5 implementation streaming pro-audio between California cities over the public internet: Campbell ↔ Saratoga ↔ Los Angeles.

Jeff Koftinoff <jeff.koftinoff@statusbar.com> · Ford Motor Company. Enthusiastically involved with real-time audio, music, networking, time synchronization, and open source for almost all my life! A personal experiment, presented with Ford Motor Company's gracious support.

Connect on LinkedIn: linkedin.com/in/jdkoftinoff

Motivation · The Dream

A personal experiment: GPS as a global word clock

Can a low-cost Raspberry Pi 5 run a full AVB stack — and can GPS time be the media clock for the whole planet?

GPS-locked gPTP as a shared media clock across the world
Measure unidirectional network latency (GPS on both ends)
Connect AVB streams across non-AVB networks with no sample-rate converters
Low-latency jamming between residences: multichannel 96 kHz audio

The Gear

Three residential sites

MOTU AVB switch or Luminex GigaCore AVB switch
MOTU or Meyer Sound AVB audio device for I/O
TimeMachines TM2000B GPS receiver
Raspberry Pi 5 running linuxptp, locked to gPTP
Decent fibre internet link

The Setup

The easy plan

TM2000B feeds gPTP to the network; the Pi 5 syncs to GPS via gPTP
The Pi 5 emits a media-clock stream locked to GPS/gPTP time
AVB devices lock to that global media clock; streams flow to/from the Pi 5
Two Pi 5s in different cities exchange UDP-encapsulated AVB AAF, buffered long enough to absorb worst-case internet latency

The Preparations

But what is the worst-case latency?

You can't size the buffer until you've measured the network performance.

GPS on each end ⇒ measure latency in each direction independently
Need a tool that locks to GPS time and sends GPS-timestamped UDP to the peer
⇒ the One-Way Latency Measurement tool (owlm-tool)

The Hard Parts

Tricky stuff for owlm-tool

linuxptp stability, and owlm integration with it
Linux PREEMPT_RT kernel on the Pi 5
Real-time timers: accurate 125 µs wakeups
Isolate a CPU core to keep other cores from interfering
Direct IPv6 / IPv4-with-NAT punch-through; STUN for NAT traversal
Temporally redundant messages

The Hard Part — Real-Time Constraints

Can we do real work without missing the next 125 µs?

Can we even hit 125 µs wakeups, repeatably?
Can we do anything useful in a tick without blowing the next one?
When a deadline is missed we need stats, tripwires, and LTTng to root-cause the interference…
…but the introspection tooling itself causes interference. (So make it cheap and lock-free.)

The Hard Part — Interference, and Avoiding It

Own two cores completely

Where the jitter comes from

Interrupts & high-priority drivers
CLOCK_MONOTONIC perturbed by NTP
Thermal throttling
Kernel MMU page work for low-prio tasks blocks high-prio tasks

How we shut it out

PREEMPT_RT · HZ=1000 · NO_HZ_FULL · RCU_NOCB_CPU · IRQ_FORCED_THREADING

isolcpus=2-3 nohz_full=2-3 rcu_nocbs=2-3 irqaffinity=0-1

Cores 2–3 leave the scheduler entirely; cores 0–1 take every IRQ and all of Linux.

MMU interference is unavoidable if any syscall is needed.

Architecture — Core Partitioning

Two cores, one job each

Core 3 — Network I/O

Time-triggered SCHED_FIFO task
All packet TX / RX
Traffic shaping, separation, prioritization

Core 2 — Audio

Owns the media-clock domain
Compensation (playout) buffers
Schedules every audio packet at the correct gPTP time

Cores 2 and 3 communicate exclusively through lock-free SPSC (single-producer / single-consumer) queues with no locks, no syscalls, and no shared scheduler. Cores 0–1 run Linux + all IRQs.

Architecture — Traffic Shaping

The Pi has just enough Ethernet to do AVB

The BCM2712 MAC hardware-timestamps, but it has no separate priority queues, no packet launch-time control, and no traffic shaping.

AVB/TSN determinism needs 802.1Qav credit-based shaping + priority queuing
So the core-3 task does it in software: stream packets spread evenly across every 125 µs class-measurement interval
Inter-packet gap met by construction; priority bandwidth allocated within each 125 µs observation interval

Architecture — Jitter Removal

Inter-thread presentation-time compensation buffer

One buffer keyed by presentation (gPTP/TAI) time sits between the cores: written by core 3 (network), read by core 2 at the exact playout time.

Ingress AVB audio

Local AVTP stream arrives with LAN jitter
Re-clocked to the media clock and played out on time

Ingress inter-city audio

WAN tunnel buffered to the worst-case path (WCL)
Temporally-redundant copies carry the same presentation time — a late/lost primary is filled by its redundant; the buffer simply keeps whichever lands first

Evidence — Campbell ↔ Saratoga (near)

⬇ Full report (PDF) ~20 km apart · one-way ≈ 2 ms baseline. Tap graph to zoom · drag box edge to resize.

Campbell↔Saratoga owlm latency-vs-wall-time (both directions)

Evidence — Campbell ↔ Los Angeles (far)

⬇ Full report (PDF) ~500 km · one-way ≈ 8 ms baseline — distance adds only a few ms. Tap graph to zoom.

The last mile is where the primary latency is. The worst-case tail that sizes your buffer is residential-uplink bufferbloat — not the 10× distance.

Campbell↔Los Angeles owlm latency-vs-wall-time (both directions)

The Audio Packets

What's in each UDP packet

WAN tunnel datagram · AAF v1 / Annex J

Channels	8
Samples / channel	44 (352 total)
Format	int32 · 96 kHz
Audio per packet	458.3 µs
Datagram rate	~2181/s (×2 redundant)

PCM payload	44×8×4 = 1408 B
+ AAF header	44 B
UDP payload	1452 B · one MTU
Grandmaster ID	70:B3:D5:FF:FE:ED:CF:FF

Every packet's ptp_grandmaster_identity is a sentinel I defined in my own OUI-36 range (70-B3-D5-ED-C, 36 bits): this value declares the media clock is TAI disciplined by GPS/GNSS — one shared Earth clock domain. It's the only clock-domain field AAF-over-UDP carries, so two cities with independent GPS receivers know they share one timescale.

The Audio Packets

Local AVB vs. the WAN tunnel

	Smp/pkt	Pkt/s
Native AVB (LAN)	12	8000
UDP tunnel (WAN)	44	~2181

Local format follows the device: MOTU 8A → AM824 (IEC 61883-6), Meyer Galaxy → AAF. The tunnel carries AAF-v1 over UDP either way — 12-sample AVB packets re-framed up to one MTU for the internet, then back at the far end. Both 8 ch / 96 kHz / int32.

The Hard Part — Media-Clock Surprise

GPS clock quality wasn't good enough

AVB audio devices rejected gPTP derived from GPS.

GPS clock quality depends on receiver price, antenna quality, and satellite visibility
The resulting gPTP was too jittery for the devices' media-clock recovery to accept

The Hard Part — Solution

Split frequency from phase

Jitter hurts more locally than between sites, so use a different clock domain at each layer.

Local (low jitter)

Luminex/MOTU switch = gPTP Grand Master
linuxptp locks to that gPTP
GPS-via-NTP drives CRF / AM824 media-clock streams the devices lock to

Between sites (absolute time)

TM2000B = plain NTP server; chrony locks to it
Inter-site UDP timestamped in GPS TAI
Kalman filters reject NTP jitter → ±1 ms cross-site, clean media-clock rate

A model is always wrong but sometimes useful: pick the clock domain that's accurate enough for each requirement.

The Result

8 channels of 96 kHz audio, city to city, over the open internet

Usable links between all three cities at a fixed 25 ms worst-case latency, most of which was the “last mile”, not the distance
Meyer Galaxy locks to our Milan CRF media clock
Both tunnel directions flat & low-loss over a full hour; the full RT path fits even a 1 GB Pi 5
No ALSA / JACK / PipeWire — the entire isochronous path is custom real-time C++

How I built it · LLM (1 of 2)

What I leveraged Claude (Opus 4.7/4.8) for

Modernizing my old C/C++ AVB code into C++23 suitable for bare metal
Clang toolchain & static analysis: clang + libc++, clang-format, clang-tidy, -Werror
Sanitizers: ASan (address), UBSan (undefined behaviour), TSan (thread)
Fuzzing harnesses (libFuzzer) and test line-coverage (llvm-cov)
Deterministic Debian (.deb) builds in podman containers (RPi5 / Debian Trixie); CMake management
Cross-validating my code against the actual AVB standards, header definitions, and state machines

All under tight guardrails (see next slide).

LLM (2 of 2) — How to use it well

Build reusable, tested tools

Where it's good

Strong, explicit acceptance criteria up front
Test-driven: tests first, short functions, testable code
Clean code, best practices, clang-tidy as a gate
Make reusable, tested tools for analysis/refactor
Once the submodules, components & test harnesses exist, agents connect them like building blocks

Where it's bad

Plausible-but-wrong without acceptance criteria + tests
One-shot "do the thing" rots; tools compound
Needs the human to own the physics & the spec

Proof from this project: owlm_analyze, the masked re-analyzer, the colbin renderers, the test harnesses — reusable and tested, written with the model, not by it. Write code that's easy for an LLM to reuse — the components shown here are meant to be used exactly this way.

Open Source

All MIT-licensed

codeberg.org/statusbar
source code

abc.statusbar.com
AVB bandwidth calculator

avb.statusbar.com
AVB presentations

linkedin.com/in/jdkoftinoff
connect on LinkedIn

repos & caveats in the appendix → Jeff Koftinoff <jeff.koftinoff@statusbar.com>

Appendix / Backup

Appendix

Reference material : future work and repositories.

Appendix — Repositories & Caveats

Where the code lives

codeberg.org/statusbar

core	net (raw + POSIX sockets, reactor) · realtime (RT timers, SCHED_FIFO) · itc (lock-free SPSC) · colbin (capture format) · tsn · ieee (Ethernet) · test · sg14 · buffer · config / toml / csv
crypto	AES-128/256 · AES-CBC / GCM / SIV · CMAC · HKDF · SHA-256 / 512 · Ed25519 · ECDSA · X25519 (standard encrypted AVB)
audio	dsp (SIMD, oscillators, complex-coefficient filters) · osc (Open Sound Control) · engine · midi · ltc (linear timecode)
avb	gptp (802.1AS) · ptpclient (linuxptp) · srp (SRP / MSRP / MVRP) · atdecc + nanoavb (AVDECC / 1722.1) · avtp (AAF / AM824 / CRF) · avb_entity · udptun (inter-site tunnel + redundancy) · owlm · stun (NAT)

Caveats: personal experiment, not a product; a low-cost Pi 5 is not a certified AVB endpoint. MIT-licensed, as-is.

Appendix — Future Work

A network TAP device

Intended next step: ‘make the RT core own the Ethernet port’ — non-real-time tasks reach the network only through a virtual TAP it controls.

The real-time core owns all real-time traffic on the Ethernet port directly
Best-effort / non-real-time traffic goes out a TAP device instead of the NIC
The RT core traffic-shapes that best-effort traffic so it can't interfere with the real-time streams
Reuses the core-3 I/O engine + lock-free capture into colbin

`→SpacePgDn`	Next slide
`←PgUp`	Previous slide
`+` / `−` / `0`	Zoom slide in / out / reset
`↑` `↓`	Scroll a zoomed slide
`f`	Fullscreen the browser
`t`	Toggle dark / light theme
`Home` / `End`	First / last slide
On the PDF slides	Click ⤢ Fullscreen PDF, then use the PDF viewer's own zoom & scroll
`?` / `Esc`	Toggle / close this help