Audio

Software mixer + platform device I/O, selected like every other driver. Game code never mutates the mixer — it posts commands; a dedicated worker owns decode and voice state; the device callback only mixes.

Maturity / remaining work: backlog/audio.

Architecture

Game / session thread              Audio worker                    Device callback
─────────────────────              ────────────                    ───────────────
reserveVoice + post play/stop      drain SPSC command ring
setListener / mute / bus   ──cmd──►  decode cue → f32 cache
                                     start / stop voice (hold/loop)
                                     apply spatial / gain / pitch
                                                                   ┌─────────────┐
                                     mixer voice table ──try lock────► mix f32    │
                                                                   │ → device    │
                                                                   └─────────────┘

Contract: shared code never imports platform audio types. Selection is only through backend.Registry (drivers.audio in DriverRecipe). Default names: native (macOS/Windows), noop (Linux scaffolding).

Link modes — audio is a first-class peer of window/input/render/physics/scripting/ui:

Bump backend/module.zig engine_contract_version when the audio factory/vtable layout changes.

Threading model

Three roles, strict ownership:

Commands and samples cross different boundaries. Only the worker decodes; the device callback reads prepared audio and never waits for it:

flowchart TD
    client["Game / session"] -->|Commands| queue["SPSC ring · 256 slots"]
    queue --> worker["Audio worker"]
    worker -->|Decode + publish| mixer["Prepared PCM / stream rings"]
    device["OS device callback"] --> lock{"Try mixer lock once"}
    lock -->|Acquired| mix["Mix active voices"]
    mixer -->|Prepared samples| mix
    lock -->|Contended| silence["Silence + skipped-frame count"]
    silence -.-> catchup["Next lock owner catches up"]
    mix --> output["Device output"]
    silence --> output
    class output accent

Stop / steal lifetime (detach-then-free):

stop:  under mixer lock → null Voice.stream / samples, clear active (generation unchanged)
steal: under mixer lock → claimLocked bumps generation, nulls stream/samples, reserved
then:  worker frees Streamer (+ ring) only after the mixer no longer points at the ring

Client posts only; never frees rings. Worker owns decode cache + streamers. Device: mix only.

The device callback tries the mixer lock once for the whole callback; it never spins or waits for a controller. On contention it outputs silence, atomically records skipped frames and increments callback_dropouts. The next lock owner catches up voice transport and fades in one bounded pass over the voice table before applying further edits. Paused voices stay paused, scheduled starts retain their offsets, and stream rings discard late PCM. Reverb history is cleared after a dropout and Master ramps back from silence. This is nonblocking fallback, not a lock-free/callback-owned voice architecture: contention still causes an audible dropout, now measurable rather than an unbounded wait. Detach-before-free remains mandatory.

Command queue

• Fixed 256-slot SPSC ring (game produces, worker consumes). Lock-free atomics; wake via condition variable.
• Commands are fixed-size PODs (path inlined, max 240 bytes). No heap on the post path.
• Client APIs (playCue, stopVoice, stopAllVoices, updateVoice, setListener / trySetListener, setMasterMute, bus gain/mute, invalidateCue) only post. Overflow on one-shot posts spins briefly; playCue returns error.QueueFull and releases the reserved slot. Last-wins per-tick state (trySetListener, updateVoice) drops instead of spinning — the caller must not record the value as posted so the next tick re-arms.
• Worker processes in order — play then same-frame update is well-defined.

Play lifecycle

1. Client playCue → mixer.reserveVoice (short spin lock) → post play with path + opts → return VoiceHandle immediately.
2. Worker decodes (or hits cache) → startVoice (samples + active + hold/loop flags). Failure → detach voice, publish decode_failed, and log.
3. isPlaying is true while the slot is reserved or active with matching generation (so async play does not look “dead” for one frame).
4. Device mixes only active voices; reserved slots are silent until start.

Handles use index + generation so stale stops after steal/reuse are no-ops. Protect long-lived handles with loop and/or hold: true (see Steal policy).

Synchronization

• Worker ↔ device: one mutex protects the mixer. Controllers acquire it normally; the callback tries once and uses silence/catch-up on contention. Critical sections stay alloc-free.
• Game ↔ worker: command ring only (plus isVoiceLive under the same short mixer lock for handle queries).
• Master mute UI: atomic snapshot updated on post so the toolbar can read without waiting for the worker.

Profiler

When Recording is on (-Dprofiler-timing + runtime Recording). Sections flip per drain / device callback (same model as physics), not for the whole thread lifetime.

High-frequency cmds (update, set_listener, mute/bus) are counted inside audio.drain only — no per-cmd zones (would flood the ring).

The device profiler channel is owned by AudioSystem, registered before the device starts, and removed after callbacks stop. Its ring has a stable address independent of the OS callback thread, so CoreAudio thread rotation across suspend/resume neither consumes new registry slots nor takes the process-registry mutex on the real-time path.

Diagnostics

Build with --profiler-audio (-Dprofiler-audio) to populate the editor right-dock Audio tab. Independent of CPU profiler Recording (--profiler-timing) and of --profiler-residency; defaults off. Without the flag the tab stays present and explains the rebuild option; sampler storage is zero-sized.

Same publish-copy model as --profiler-physics. See UI and editor — Audio tab.

Playback features

• Device pulls interleaved f32 mono or stereo (macOS: AudioQueue; Windows: WASAPI shared).
• Mixer: fixed 64-voice pool; per-bus caps music 8 / voice 8 / sfx 48, plus an enforced aggregate max_voices budget counting active and reserved slots. See Steal policy.
• Playable buses are music / sfx / voice. Master gain/mute are dedicated controls, not a playable bus. Mute, zero gain, and complete spatial attenuation advance source position and fades; explicit pause freezes both.
• Resample: worker maps cue rate → device rate before mix (static cache key path#<rate>; streamers resample into the ring). invalidateCue(path) drops every rate variant. Pitch remains a separate playback-speed control.
• Fades: fade_in_ms on play; fadeStop / crossfade post worker cmds; mix path steps fade gain alloc-free (including under mute / zero bus gain). crossfade(from, to, opts, fade_ms) fade-outs from and starts to with fade_in_ms = fade_ms (when fade_ms == 0, hard-stops from and keeps any opts.fade_in_ms). Reserving to uses reserveVoiceExcluding(..., from.index) so a full bus never steals from (generation bump would no-op the fade-out).
• Ducking: while any voice-bus voice is active, music/sfx bus gains are multiplied by duck amounts (defaults 0.25 / 0.5; attack ~50 ms, release ~200 ms). Voice bus itself is unducked. Tunable via setDuckAmounts (clamped 0..1 on worker).
• Play vs Edit: auto play_on_start only in Play. Edit is silent; Stop/reset posts stop_all and clears component handles.
• Spatial (v2): distance atten × cone × occlusion × constant-power pan, computed once per active spatial voice per buffer (mixer/spatial.zig). Shared stereo reverb send (Schroeder comb+allpass on wet bus; mixer/reverb.zig). No HRTF yet.
• Cues: cooked .shinaudio via Shinra → worker retainAudioCue. Short/non-stream cues decode to a mono/stereo f32 cache preserving the source channel count; streaming_hint cues keep a worker-owned ring filled from a payload cursor (file window or memory). Opus decodes on the worker via static libopus (never in the device dylib).

Steal policy

When a bus or the aggregate budget is full, reserveVoice may steal an active same-bus voice (generation bump → old handle becomes a no-op). Rules:

Long-lived handles: use loop: true and/or hold: true. Legitimate steals of non-held oneshots bump generation — stale stop / fadeStop are no-ops (by design). Soft fail: playCue returns invalid / error.NoFreeVoice when nothing stealable remains.

Mix strip vs toolbar mute

Toolbar chrome soft-syncs master mute from the atomic every frame so strip and toolbar never desync.

Scene JSON

"audio": {
  "cue": "asset://./audio/glass_break",
  "volume": 1.0,
  "pitch": 1.0,
  "loop": false,
  "spatial": true,
  "min_distance": 1.0,
  "max_distance": 50.0,
  "inner_cone_deg": 360,
  "outer_cone_deg": 360,
  "outer_gain": 0,
  "occlusion": 0,
  "occlusion_gain_factor": 1,
  "reverb_send": 0,
  "bus": "sfx",
  "play_on_start": true,
  "enabled": true
}

Engine builtin archetype _audio (components.audio = true) hosts the component — same idea as _script / _asset, not a game package entity. Drop cooked/source audio into the viewport to spawn it with audio.cue set. Music beds typically use spatial: false, bus: "music", loop: true.

Spatial model

Cone model: angle between source forward and (listener − position). If angle ≤ inner/2 → factor 1; ≥ outer/2 → outer_gain; else linear blend. Outer clamped ≥ inner when both < 360. Session tickAudio posts entity world forward from transform into Spatial.forward.

Occlusion: game/session posts the factor (physics integration is a residual). Default 0 = clear path.

Reverb: dry mix as before; samples also accumulate into a preallocated wet scratch (max 4096 frames stereo). After all voices, one Schroeder network processes the wet bus and adds wet * reverb_wet_mix to the output. Master mute still steps fades and decays reverb with silence (no hang / no infinite wet). No alloc on the device path.

No HRTF in this pass.

Assets (Shinra)

Optional <file>.shinmeta.json → audio section: normalize, sample rate, loop, streaming hint, Pcm16Le / PcmF32Le / OpusOgg. Sample project assets: src/games/example/assets/audio/.

Ogg codec: Vorbis sources decode to PCM then cook. Opus sources with format: OpusOgg pass through into the cooked Opus payload (no re-encode). Opus + PCM output format is rejected — use WAV or Vorbis when you need PCM.

streaming_hint: true → worker Streamer + StreamRing with a payload cursor:

Worker may block briefly on pread while refilling a modest window (default ~128 KiB); the device callback never does file I/O — only mixes the ring. Loop rewinds reset PCM frame / recreate the Opus demux+decoder at payload byte 0 (cursor is random-access readAt; no separate seek object).

false/omitted → full decode into the f32 cache (payload always memory-resident).

Editor

• Viewport (show-flag Audio): speaker billboard; spatial sources draw min + max distance rings (great-circle trios) matching mixer falloff, and when both inner_cone_deg and outer_cone_deg are < 360 wire outer / inner aim cones along actor forward (local −Z; full angles → half-angles like the mixer). Either cone ≥ 360 (omni, including defaults) is rings-only. Non-spatial: speaker only (no range rings). Show-flag Audio gates wires; Icons gates the speaker billboard.
• Hierarchy / inspector: audio section, cue picker, bus/volume/spatial fields (including cone / occlusion / reverb send).
• Inspector Preview / Stop preview on a selected audio source with a cue: posts playCue with authored volume / pitch / bus / spatial (entity world position when available). Does not set component started or the permanent play_on_start handle. Only one preview voice at a time; stops on deselect, asset-inspector mode, Play/Stop transitions, and session destroy. Master mute still silences mix output (preview still posts — same as toolbar mute; mute zeros the mix while voices keep stepping).
• Edit mode stays silent for automatic play_on_start; preview is the explicit listen path in Edit.
• Drop .wav/.ogg (or cooked audio) into the viewport → _audio actor with audio.cue.
• Toolbar mute posts AudioSystem.setMasterMute (accent when muted). Toolbar chrome soft-syncs from the master-mute atomic every frame so the Audio tab Mix strip and toolbar never desync; no feedback loop (setIcon/setStyle no-op when unchanged).
• Audio tab Mix strip (always available when session.audio is live; independent of --profiler-audio): stable retained controls for master mute/gain and per-bus gain/mute (Music / SFX / Voice). Edits post only via AudioSystem.setMasterGain / setBusGain / setBusMute / setMasterMute — never touch the mixer under lock. Master mute soft-syncs from the atomic when the panel has no pending local edit. Leaving the tab clears the “live” latch so the next select re-posts strip locals (covers a replaced AudioSystem). Reset mix restores gains to 1.0 and clears bus mutes; master mute is left as-is (toolbar/session). Diagnostics below the strip still require --profiler-audio and rebuild on a timer without destroying Mix widgets.
• Asset browser tiles: when a cooked .shinaudio header is readable, the subtitle badge adds format (PCM / OPUS), optional duration, and STREAM when streaming_hint is set (64-byte header only; no full retain).

Live output monitor

The Audio tab's Live output card works without --profiler-audio. Music, SFX, Voice and Master each show a two-second min/max waveform, peak/RMS meters (−60 to 0 dBFS), one-second peak hold and clip indication. Click a row to expand it; Master starts expanded. Stereo channels have separate lanes; mono Master shows the folded output.

Bus signals are post bus gain/mute, ducking, voice gain, occlusion filtering and spatial attenuation, but pre-master and dry only. Master includes reverb and master gain/mute, measured immediately before the limiter (after mono folding). Bus activity can therefore remain visible while Master is muted. Clip indicators report pre-limiter excursions, not device-side measurements. The Master row's GR is the minimum limiter gain in the captured bucket/block, expressed as positive dB reduction; the waveform intentionally retains the overload that caused it.

audio/output_monitor.zig owns preallocated bus scratch and a bounded single-producer/single-consumer queue. The callback publishes 200 min/max/RMS buckets per second; the retained editor/panels/audio_output.zig element drains at about 30 Hz. A full queue drops telemetry without waiting or affecting playback time. Capture adds no callback allocation, decoding or extra locks. Standalone builds compile out the tap and its storage.

Selecting the tab enables capture; hiding, closing, unmounting or suspending disables it. Disabled editor callbacks perform one atomic gate read and no signal scanning, accumulation or publication; the hidden panel does not drain or redraw telemetry. A callback already in flight may finish, but its old-epoch data is discarded on reopening. Panel teardown disables capture before the audio system is destroyed.

CPU-only comparison (does not start the editor, game or audio device), from src/hikari:

zig build audio-monitor-bench -Dgame-src=/absolute/path/to/src/games/example

This measures callback work with capture hidden/visible, not UI rendering or end-to-end device latency.

Session

SessionCore allocates AudioSystem, creates the device with renderCallback, bindDriver (starts the worker), starts the device after the asset store is configured, and each tickLogic posts listener + source updates (never touches the mixer directly). Asset hot-reload of .shinaudio posts invalidateCue so the worker drops the decode cache and stops voices still holding that buffer. Matching looping + play_on_start scene sources then clear voice / started so the next Play-mode tickAudio re-posts the bed (one-shots with started stay finished and do not mid-fire restart).

Residency: cooked cues are store-retained while decoded; f32 PCM is freed when no voice still samples that buffer (sweepIdleDecoded + releaseAudioCue). Details: Asset residency — audio.

Play mode play_on_start:

• Posts play when the handle is invalid; keeps the handle while reserved/playing.
• Marks started once isPlaying is true (worker applied).
• Soft-pending assets: retry until live or asset state is failed.
• resetAudioSources / leave Play: stop_all + clear handles and started flags.
• Hot-reload invalidate of a cue path: clear voice + started only for looping + play_on_start sources whose cue stem matches (see runtime_session/audio_reload.zig).

Game API

Two ways to play (both post into the same command queue):

Zig (HostApi subtable)

Nullable: hi.audio() is null when the device/worker failed to start.

const audio = hi.audio() orelse return;
const h = audio.playCue(hi.AssetRef.must(.audio, "asset://./audio/glass_break"), .{
    .volume = 1,
    .spatial = true,
    .position = pos,
    .forward = .{ 0, 0, -1 },
    .inner_cone_deg = 90,
    .outer_cone_deg = 180,
    .outer_gain = 0.05,
    .occlusion = 0,
    .reverb_send = 0.2,
    .bus = .sfx,
    .priority = 0,
    .fade_in_ms = 0,
    // .hold = true, // long-lived non-loop; never stolen
});
if (audio.isPlaying(h)) { /* reserved or live */ }
audio.fadeStop(h, 250);
// audio.setReverbMix(0.4);
// audio.stop(h); // hard stop

PlayCueOpts: volume, pitch, loop, hold, bus, spatial v2 fields (position / min/max / cone / forward / occlusion / reverb_send), priority, fade_in_ms, paused, start_frame. Looping or hold: true → never a steal victim.

Bump: root host_api_version_current when the subtable pointer is added/removed; audio_api_version_current on table breaks (additive opts fields included, so games recompile with the editor). Host install: host_bind/audio_api.zig → AudioSystem posts only.

Scene scripts (Audio.*)

Ops are VM-agnostic (audio/script_ops.zig); Kawa only marshals (scene/scripting/kawa_host/audio.zig) and registers through SceneScriptBackend.registerNative from the application session (keeps audio out of the scripting dylib). Same post-only path as HostApi.

const h = Audio.play("asset://./audio/glass_break", {
    volume = 1,
    spatial = true,
    position = Actor.get_position(),
    forward = Actor.get_forward(),
    inner_cone_deg = 90,
    outer_cone_deg = 180,
    outer_gain = 0.05,
    occlusion = 0,
    reverb_send = 0.2,
    bus = "sfx",
    priority = 0,
    fade_in_ms = 0,
    // hold = true, // long-lived non-loop; never stolen
});
if (Audio.is_playing(h)) {}
Audio.fade_stop(h, 250);
const next = Audio.crossfade(h, "asset://./audio/soft_fog_lines", 500, { bus = "music", loop = true });
Audio.set_master_mute(false);
Audio.set_master_gain(0.8); // linear; clamped ≥ 0 on worker
Audio.set_bus_gain("music", 0.5); // or index 0..2: music/sfx/voice
Audio.set_bus_mute("sfx", false);
Audio.set_duck_amounts(0.25, 0.5); // music/sfx targets while voice-bus active; clamped 0..1 on worker
Audio.set_reverb_mix(0.35); // global wet scale

Handle is {i, g} or nil. Same soft-fail rules as Zig. Play opts include hold, cone/occlusion/reverb_send (v5). Mix controls, set_duck_amounts, and set_reverb_mix post only (parity with HostApi).

Sample

• Zig: src/games/example/src/entities/sfx_oneshot_entity.zig (archetype sfx_oneshot, Space / move_up)
• Kawa: src/games/example/assets/scripts/sfx_oneshot.kawa
• Scene: scenes/audio_demo.json places the Zig entity

Manual smoke (Play mode): music bed loops; Space oneshots over it; toolbar mute/unmute; Stop Play cleans up. Open maturity items: backlog/audio.

Transport and synchronization

Audio API v6 exposes pause(handle), resumeVoice(handle), seek(handle, seconds), transport(handle), position(handle), and clock(). Kawa equivalents are Audio.pause, Audio.resume, Audio.seek, Audio.position, Audio.state, and Audio.clock. Mutations queue to the worker; their boolean result means accepted, not completed. Queries take a short mixer lock. Stale/completed handles report stopped and position zero; isPlaying retains its live-handle meaning, including preparing, scheduled, and paused voices.

transport reports preparing, scheduled, playing, paused, or stopped and source position in seconds (pitch changes position advancement; loops wrap). seek accepts finite seconds from 0 through 86400 and clamps to the last source frame. The worker pauses streams under the mixer lock, moves the source decoder in source-rate frames, clears/refills the ring, reanchors the mixer in device-rate frames, then restores pause state. Failed stream seeks stop the voice. Decoding and I/O never run while holding the mixer lock.

clock() returns { frame, sample_rate }: a monotonically advancing count of mixed output frames, including silent buffers. It is not a hardware presentation clock and does not compensate for OS/device buffering. It belongs to one AudioSystem lifetime and does not reset on Play/Stop. Source transport pauses independently of this clock.

Set PlayCueOpts.start_frame (Kawa start_frame) to an absolute future clock frame; zero requests immediate playback. The worker prepares PCM/ring immediately, and the callback starts at the exact offset within its buffer. Multiple prepared voices sharing a deadline start on the same sample. Allow decode/queue lead time: an already-missed deadline is rejected, and a voice not prepared by its deadline becomes stopped rather than starting late. Scheduled voices still consume budget; use hold to prevent steals. paused = true prepares without advancing; resume after a scheduled deadline starts from the held source position at the next callback. Crossfade is immediate-only and rejects paused/scheduled destination options; fade-stop on a paused or not-yet-started voice stops immediately.

const audio = hi.audio() orelse return;
const now = audio.clock();
const start = now.frame + now.sample_rate; // one second preparation lead
const a = audio.playCue(cue_a, .{ .start_frame = start, .hold = true });
const b = audio.playCue(cue_b, .{ .start_frame = start, .hold = true });
_ = audio.pause(a);
_ = audio.seek(a, 2.5);
_ = audio.resumeVoice(a);
_ = b;

configs/audio.json → audio.output_channels is 1 (mono) or 2 (stereo, default), applied at device creation. Mono output folds the complete stereo dry+wet mix as (L + R) / 2 before clipping. Non-spatial mono cues duplicate to both stereo channels; spatial cues are point sources (stereo sources fold to mono before constant-power panning). Stream rings remain stereo staging buffers. Stream underruns output silence but advance source time; late PCM is discarded when the worker catches up.

Streaming and Opus

What “streaming” means here: the worker pulls PCM/Opus into a ring; the device never decodes. Loose/data/ cues use a file-backed payload cursor (pread sliding window, default ~128 KiB) so full payload need not stay resident; pack members still keep a memory-resident payload (see Assets). Residual vs UE-style async disk residency: fills are synchronous worker pread, not an async IO queue.

Lifetime: stop / stop_all / destroy detach Voice.stream under the mixer lock first, then free the Streamer (and ring). Never free-then-stop.

Worker pacing: fill only when a ring is below low-water (~1/4 capacity). Healthy rings → short sleep (~3 ms), not a spin. Idle (no streams, no cmds) → condition wait.

Pitch: stream and static voices both use a fractional cursor (same PlayOpts.pitch behaviour).

Policy (content): short SFX → PCM16, no stream; music/ambience → format: OpusOgg + streaming_hint: true in .shinmeta.json (see soft_fog_lines).

Profiler: audio.decode.opus, audio.stream.fill on the worker.

Build: application links static libopus staged from thin prebuilts 3rd-party/opus/<os>-<arch>/static/ + include/ (Kaji opus unit copies → bin/opus/…). Official Xiph ships source only; prebuilts are built from the pinned tarball. Not Homebrew. Not in audio device dylibs.

Signal conditioning and playback events

Master and bus gain/mute transitions ramp over 5 ms in sample time, including unmute and changes to zero gain. Boot values apply directly before the first render. Muting still advances transport; only explicit pause freezes it. Bus envelopes also affect reverb sends. Master gain is applied after the shared reverb, so changing it scales the complete dry/wet mix.

The final output uses a stereo-linked sample-peak limiter: instantaneous attack, exponential release, no lookahead and no added buffering latency. Mono folds before limiting. This is not an oversampled true-peak limiter; it does not guarantee intersample peak protection. Non-finite final samples become silence; a hard clamp remains as a last safety bound. Startup configuration in configs/audio.json:

{
  "audio": {
    "limiter_enabled": true,
    "limiter_ceiling": 0.98,
    "limiter_release_ms": 100,
    "occlusion_cutoff_hz": 800
  }
}

Spatial voices use a one-pole low-pass filter driven by the existing posted occlusion value. The cutoff moves logarithmically from the clear bandwidth toward occlusion_cutoff_hz; coefficients ramp over 5 ms. At zero occlusion the filter returns to exact bypass. Set cutoff to 0 to disable filtering. Existing occlusion_gain_factor still controls volume loss independently; full occlusion with its default 1 remains fully silent. Filter state resets on slot reuse and seek. No physics queries run on the callback.

Audio API v7 adds pollEvent() and droppedEvents(). The game-thread consumer receives { voice, reason, frame } records for finished, stopped, stolen, decode_failed, missed_start, and invalidated. Voice handles retain the ending generation. Frame is the mixer observation boundary (block end for rendered completions), not an exact source-end or device-presentation timestamp. Preparation/stream/seek failures share decode_failed; detailed errors remain in logs. The fixed 256-event queue drops the oldest entry on overflow and counts losses. It is one consumable session queue, not a broadcast: coordinate consumers, including scripts and the two editor Worlds.

while (audio.pollEvent()) |event| {
    switch (event.reason) {
        .finished => {}, // advance dialogue/music sequencing
        else => {},
    }
}

Kawa equivalents: Audio.poll_event() returns { voice, reason, frame } or nil, and Audio.dropped_events() returns the loss count. The profiler-enabled Audio panel also shows callback dropouts and event losses.

Residuals

Implemented caveats (not a design list). Open work lives in backlog/audio.

Extending

New device backend: platform AudioBackend + factory on DriverModule.audio. Keep the render callback mix-only.

Related