131 lines
8.5 KiB
Markdown
131 lines
8.5 KiB
Markdown
# Audio-Visualizer Architecture Improvements
|
|
|
|
This document outlines a strategy for refactoring the `audio-visualizer` codebase to improve modularity, simplify the creation of new visualizers, and extract highly valuable components currently locked within specific implementations.
|
|
|
|
## 0. Prioritization & Risk Assessment
|
|
|
|
Not all three sections carry the same value-to-risk ratio. Recommended order:
|
|
|
|
| Priority | Section | Verdict |
|
|
|----------|---------|---------|
|
|
| **Do first** | §3 Extracting reusable components | High value, low risk. Pure logic, no GPU, mechanical move + re-export. |
|
|
| **Maybe** | §3 `MorphState<T>` | Worthwhile, but limited — see caveat below. |
|
|
| **Defer** | §1 `Visualizer` trait | Lateral, not a net win — see caveat. Worth it only if several new visualizers are planned. |
|
|
| **Decide first** | §2 generic `ShaderPipeline<U>` | Contradicts a stated architecture policy. Do not start without an explicit decision. |
|
|
|
|
### Risk callouts (must survive any refactor)
|
|
|
|
- **Determinism / `--render` bit-reproducibility.** Any moved component that integrates state (`Spring`, the `Attr` RK4 attractors) must keep the *advance-only-in-`update`* discipline. This is the same rule that holds today; it just has to survive relocation. Verify by diffing a `--render` of the same input before/after the move — output must be bit-identical.
|
|
- **Central analysis invariant.** `Analyzer` remains the only place STFT/AGC/onset math lives (live ≡ offline). `Structure`/`Arch` is a viz-side *consumer* of `Bands`, so moving it is safe — but if it becomes shared, keep its state mutation confined to `update` for the same determinism reason.
|
|
- **`breakcore::NP` (64) Rust↔WGSL coupling + hard-capped march steps.** A generic shader base (§2) must *not* abstract these guardrails away — a runaway march = GPU device-lost / machine hang. naga only validates the WGSL at pipeline-create on a real GPU, so a bad layout cannot be caught in the sandbox.
|
|
|
|
### Caveat on §1 (`Visualizer` trait)
|
|
|
|
The `is_gpu()` branch is a *real* dichotomy, not accidental complexity: `Draw`-based visualizers (`Sigil`/`Scope`) share the chromatic-aberration channel passes dispatched in the bin, whereas `Breakcore` presents/captures its own raymarch target directly. A trait with optional `draw` / `render_gpu` methods does not delete this branch — the caller still switches on `is_gpu_driven()`. The trait *relocates* the branch and standardizes the contract; it does not remove the structural split. Net positive only if multiple new visualizers are actually on the roadmap.
|
|
|
|
### Caveat on §2 (generic `ShaderPipeline<U>`)
|
|
|
|
`CLAUDE.md` deliberately designates `breakcore.rs` as *"the lone, walled-off exception"* to the "no hand-written wgpu pipelines" rule (`post.rs` and the rest stay on nannou's validated renderer). A reusable `ShaderPipeline<U>` base is, by design, an invitation to add more raw wgpu pipelines — i.e. it reverses that policy. This may well be desirable, but it is a deliberate architectural decision that must be made explicitly *before* the extraction, not a side effect of it. If adopted, the base must still preserve the `NP`/UBO-layout coupling and the march-step caps as non-negotiable guardrails.
|
|
|
|
### Caveat on `MorphState<T>`
|
|
|
|
`scope` and `breakcore` share the *shape* of the morph (hold a seeded state, lerp `current`→`target` over a timer), but their *triggers* differ fundamentally: scope morphs on a cooldown-gated rising-flux transient or a 12 s idle; breakcore cross-fades on a long-vs-short loudness-EMA section state machine. A generic `MorphState<T>` can own only the `current`/`target`/`t` bookkeeping and the lerp/timing; the trigger logic stays per-visualizer. Useful, but scoped — it does not unify the interesting (musical) part.
|
|
|
|
## 1. Modularizing the Visualizer System
|
|
|
|
Currently, the `Visual` type in `src/bin/sigil.rs` is an `enum` hardcoding the available visualizers (`Sigil`, `Scope`, `Breakcore`). It also uses an awkward branching mechanism (`is_gpu()`) to handle the divide between `nannou::Draw`-based visualizers and raw `wgpu` pipelines. This violates the Open-Closed Principle and makes adding new visualizers cumbersome.
|
|
|
|
**Recommendation: The `Visualizer` Trait**
|
|
Refactor the enum into a dynamic trait object system. This cleans up `sigil.rs` and standardizes the contract for all visualizers.
|
|
|
|
```rust
|
|
pub struct RenderContext<'a> {
|
|
pub pal: &'a Palette,
|
|
pub scale: f32,
|
|
pub warp: f32,
|
|
pub glow: bool,
|
|
pub tint: [f32; 3],
|
|
// ... other shared gains/tunables
|
|
}
|
|
|
|
pub trait Visualizer {
|
|
fn name(&self) -> &'static str;
|
|
fn seed(&self) -> u64;
|
|
fn reseed(&mut self, seed: u64);
|
|
fn update(&mut self, bands: &Bands, dt: f32);
|
|
fn element_count(&self) -> usize;
|
|
|
|
/// Indicates if this visualizer manages its own wgpu pipeline.
|
|
fn is_gpu_driven(&self) -> bool { false }
|
|
|
|
/// Used by CPU-driven visualizers (nannou::Draw)
|
|
fn draw(&self, draw: &Draw, ctx: &RenderContext) {}
|
|
|
|
/// Used by GPU-driven visualizers to return their raymarch/compute target
|
|
fn render_gpu(&mut self, device: &wgpu::Device, queue: &wgpu::Queue, ctx: &RenderContext) -> Option<&wgpu::Texture> { None }
|
|
}
|
|
```
|
|
|
|
## 2. Designing a New Visualizer Base
|
|
|
|
To support rapid development of completely new visualizer modes (especially WGSL-based ones like `Breakcore`), we should abstract the boilerplate into unified bases.
|
|
|
|
### The `ShaderPipeline` Base
|
|
The `Gpu` struct inside `breakcore.rs` is a fantastic, self-contained raw `wgpu` pipeline. It should be extracted into a generic `viz::shader::ShaderPipeline<U>` where `U` is a `bytemuck::Pod` trait bound for the Uniform Buffer Object.
|
|
|
|
This base would automatically handle:
|
|
- WGSL compilation and wgpu layout boilerplate.
|
|
- Dual-texture ping-ponging for frame feedback.
|
|
- UBO layout mapping and writing.
|
|
|
|
```rust
|
|
pub struct ShaderPipeline<U> {
|
|
// pipeline, bindings, and ping-pong textures
|
|
}
|
|
|
|
impl<U: bytemuck::Pod> ShaderPipeline<U> {
|
|
pub fn new(device: &wgpu::Device, wgsl_source: &str) -> Self { ... }
|
|
pub fn render(&mut self, queue: &wgpu::Queue, ubo: &U) -> &wgpu::Texture { ... }
|
|
}
|
|
```
|
|
A new GPU-based visualizer would then only need to define its UBO struct, its `.wgsl` shader, and implement `Visualizer::render_gpu`.
|
|
|
|
## 3. Extracting Reusable Components
|
|
|
|
Several brilliant components are currently tightly coupled to `breakcore.rs` and `scope.rs`. They should be hoisted to common modules so *any* new visualizer can use them.
|
|
|
|
### From `breakcore.rs`
|
|
1. **`Structure` & `Arch` (Musical Archetype Classifier):**
|
|
The slow feature-vector classifier that detects `Arch` (Ambient, Build, Drop, Breakdown, Groove) and manages `tension`/`release` is incredibly powerful. It converts raw FFT data into *musical context*.
|
|
- **Action:** Move this to `src/viz/structure.rs` (or even `src/audio.rs`). Any new visualizer should be able to query `structure.tension` to drive camera FOV, or trigger a massive event on `Arch::Drop`.
|
|
|
|
2. **Critically Damped `Spring`:**
|
|
The `Spring` struct applies physics-based smoothing to visual targets. It prevents jitter while maintaining immediate snap on transient hits.
|
|
- **Action:** Move to a new `src/viz/math.rs`. This is the perfect primitive for mapping raw audio bands to visual scales/rotations across all visualizers.
|
|
|
|
3. **RK4 Attractors & Knots:**
|
|
The `Attr` (Lorenz/Rössler) and `Knot` geometry generators create stable, chaotic 3D paths.
|
|
- **Action:** Move to a new `src/viz/geometry.rs`.
|
|
|
|
### From `scope.rs`
|
|
1. **3D Vector Figures:**
|
|
The `Figure` generator (Gielis supershapes, 3D Lissajous, Harmonographs) is excellent for generating deterministic, organic geometry.
|
|
- **Action:** Move to `src/viz/geometry.rs` alongside the attractors.
|
|
|
|
2. **The Morphing State Machine:**
|
|
Both `scope` and `breakcore` implement a pattern of holding a seed, tracking an `idle` timer or waiting for a transient, and smoothly `morph`ing to a new state.
|
|
- **Action:** Abstract this into a `MorphState<T>` primitive that wraps a `current` and `target` state, handling the lerp/transition logic automatically.
|
|
|
|
## Summary of Proposed Directory Structure
|
|
```text
|
|
src/viz/
|
|
├── mod.rs
|
|
├── math.rs <-- (NEW) Spring, smoothing, interpolations
|
|
├── geometry.rs <-- (NEW) Figures, Knots, Attractors
|
|
├── structure.rs <-- (NEW) Arch, Structure classifier
|
|
├── shader.rs <-- (NEW) Generic wgpu ShaderPipeline
|
|
├── core.rs <-- (NEW) Visualizer trait and RenderContext
|
|
├── breakcore.rs (Refactored to use new bases)
|
|
├── scope.rs (Refactored to use new bases)
|
|
└── sigil.rs
|
|
``` |