trail mix: Open tempo, key, and waveform analysis

Let's start with Cueport

Cueport is a local-first music player for people who own their music.

Cueport is local-first

The desktop and phone pair directly on a local network or hotspot.
The QR code carries connection details; data moves both ways after pairing.

Cueport v1 workflow

Cueport v1 uses Serato as the source of truth.
The desktop database imports audio and metadata from Serato.
The mobile database is a cached replica.

Problem: DJ tool dependence

DJ tools offer tempo, key, and beat analysis for DJ workflows.
Mainstream tools need their own analysis layer.

Solution: trail mix

trail mix accepts an audio file and returns tempo, key, beat positions, and waveform data as versioned JSON.

Decode and normalization

Audio files become a mono sample buffer along with its sample rate

• Decoding: an audio file becomes evenly spaced amplitude values
• Downmixing: sample buffers combine into one mono signal
• f32: a 32-bit floating-point number for storing those amplitudes
• u32: a non-negative 32-bit integer for storing the sample rate

Why Rust for trail mix

trail mix is a Rust library because Cueport is a Rust app and the analysis has to run on-device with the same result every time.

Why we built our own

The existing libraries in this space are solid, but no single one gave us a permissive, in-process API that also detects mid-track tempo and key changes and emits a versioned format.

MIR vocabulary

Four helpful terms relevant to trail mix.

Onset detection

An onset is where a new sound starts, like a drum hit. It shows up as a sudden jump in energy, so we track the energy frame by frame and add up only the rises. That running total is the onset-strength curve, and its peaks are the candidate beats.

Tempo estimation and beat tracking

Tempo estimation finds the beat period that best matches the repeating peaks in the onset-strength curve. To find a beat switch, the same estimator runs on rolling sections of the track.

Beat tracking with dynamic programming (DP)

Beat tracking uses the estimated BPM to place the actual beat timestamps. Tempo estimation gives BPM, but it does not choose the beat grid offset, that is where those beats start. DP helps choose which grid is most plausible across the whole track. It can keep the sequence regular when one beat is weak or missing, and it can skip a loud off-beat onset.

Beat switches

A track can contain more than one meaningful tempo. Beat Salad reports the dominant BPM and the alternate tempo when the second section is long enough to matter.

Threshold: alternate tempo must cover at least 25% of the track.

Key detection from chroma

Take the 12-note chroma and compare it to a template for each key. The closest match, across all 24 major and minor keys, is the detected key.

\[\operatorname{score}(k) = \frac{\displaystyle\sum_{p=0}^{11}(c_p - \bar{c})\,(T_{p,k} - \bar{T}_k)}{\sqrt{\displaystyle\sum_{p=0}^{11}(c_p - \bar{c})^2 \;\cdot\; \sum_{p=0}^{11}(T_{p,k} - \bar{T}_k)^2}}\]

• The formula: Pearson correlation; how closely the chroma matches a key template, after mean-centering and normalizing both
• score(k): correlation for key k; the highest of 24 keys (12 major + 12 minor) wins
• c_p: chroma energy at pitch class p (0-11, one per semitone)
• T_{p,k}: expected energy for pitch class p in key template k (K-K, Temperley, EDMA, or an ML model)

Key changes

Key Lime reruns chroma-to-template scoring in rolling windows, groups adjacent windows with the same confident key, and reports an alternate key when it covers at least 25% of the track.

Waveform summarization

The min, max, and RMS columns that draw a track lane.

Waveform implementation

The samples are split into up to 1,500 equally sized buckets. Each bucket becomes one column, summarized by three values.

let slice = &samples[start..end];
let min = slice.iter().copied().fold(f32::INFINITY, f32::min);
let max = slice.iter().copied().fold(f32::NEG_INFINITY, f32::max);
let sum_sq: f32 = slice.iter().map(|s| s * s).sum();
let rms = (sum_sq / slice.len() as f32).sqrt();

• slice: the samples in this bucket (track length / up to 1,500 buckets)
• min: lowest sample value; bottom edge of the waveform bar
• max: highest sample value; top edge of the waveform bar
• rms: root mean square energy; renderers can draw it as the filled region inside the bar

Evaluation metrics

How to read the accuracy numbers on the next slide.

Results and gaps

Tempo leads the hand-written DSP baselines and trails the ML models. Key is the weakest analyzer.

Future work and collaboration

What's next for trail mix, and where help from others would make the biggest difference.

Built with open source

trail mix builds on a number of Rust libraries, plus open datasets and the Rust toolchain.

Further reading

Papers, methods, and datasets behind trail mix.

The analyze() function

One analyze() call runs the three analyzers over the same sample buffer and returns one versioned result.

pub fn analyze(audio: AudioBuffer<'_>) -> Analysis {
    let AudioBuffer { samples, sample_rate } = audio;
    Analysis {
        version: 1,
        beat: beat_salad::analyze(samples, sample_rate),
        key: key_lime::analyze(samples, sample_rate),
        waveform: sampler_platter::generate_overview(samples, sample_rate),
    }
}

• AudioBuffer: the input, mono samples plus the sample rate
• Analysis: the output, one versioned result with beat, key, and waveform
• beat_salad::analyze: the analyze in the beat_salad crate, a different function from this one

Onset detection: spectral flux

The onset-strength curve is spectral flux, the summed rise in band energy from one frame to the next.

\[\operatorname{flux}(t) = \sum_{b=1}^{24} \max\!\bigl(0,\; E_b(t) - E_b(t-1)\bigr)\]

• flux(t): onset strength at frame t; its peaks are the candidate beats
• E_b(t): energy in band b at frame t, from 24 pitch-spaced Goertzel filters
• max(0, ...): half-wave rectification, so only rising energy counts, not fading
• sum over b: fold the rises across all 24 bands into one number per frame

Tempo estimation

Autocorrelation of the onset curve, together with an inter-onset interval histogram, locates the beat period. An octave prior then nudges the pick toward mid-range tempos to break the half and double time tie.

\[ \begin{aligned} A(\tau) &= \frac{\sum_t O(t)\,O(t-\tau)}{\sum_t O(t)^2} \\ \operatorname{prior}(\mathrm{bpm}) &= \exp\!\left(-\frac{\log_2(\mathrm{bpm}/120)^2}{8}\right) \end{aligned} \]

• A(τ): normalized autocorrelation of the onset curve at lag τ; peaks mark repeating pulses
• bpm(τ): tempo for lag τ, \(\mathrm{bpm}(\tau) = 60\,f/\tau\) with onset frame rate f
• prior(bpm): soft preference near mid-range tempo, applied as \(0.75 + 0.25\,\operatorname{prior}\)
• pick: the lag with the highest combined score sets the tempo

Beat tracking optimization

Ellis dynamic programming scores a whole beat sequence at once, rewarding strong onsets while keeping regular beat intervals.

\[C(i) = O(i) + \max_{j \lt i}\left[\,C(j) + w\,\exp\!\left(-\frac{\big((i-j)-P\big)^2}{2\sigma^2}\right)\right]\]

\[J(b_1,\ldots,b_m)=\sum_t O(b_t)+0.5\sum_{t>1}\exp\!\left(-\frac{\big((b_t-b_{t-1})-P\big)^2}{2\sigma^2}\right),\quad b_t-b_{t-1}\in[P-0.25P,\;P+0.25P]\]

• C(i): best cumulative score for a beat ending at frame i
• O(i): onset strength at frame i
• P: target beat interval, \(P = 60\,f / \mathrm{BPM}\) with onset frame rate f

Beat tracking inner loop

The forward pass in Rust: for each frame, keep the best predecessor, then add the frame's own onset strength.

// beat interval in onset frames, and how far it may drift
let period = 60.0 * envelope_rate / bpm;
let denom = 2.0 * (period * 0.5).powi(2);
// for a beat at frame i, try each earlier frame j before it
let deviation = (i - j) as f32 - period;
let penalty = (-(deviation * deviation) / denom).exp();
let path = previous_score + penalty * 0.5;
// keep the best predecessor, then add i's own onset strength
let score = onset[i] + best_path;

• period: ideal beat interval in onset frames, from the estimated BPM
• penalty: Gaussian weight for a predecessor, highest when the gap i - j is one period
• path: that predecessor's cumulative score plus the weighted interval reward
• score: frame i's onset strength plus the best predecessor path

Beat switch: full output

The complete beat object Beat Salad returned for the 120-to-90 BPM click track on the demo slide.

"beat": {
  "global_bpm": 120.08,
  "multi_tempo": true,
  "alternate_bpm": 90.03,
  "alternate_coverage": 0.45,
  "tempo_segments": [
    { "start_seconds": 0.0,  "end_seconds": 33.0, "bpm": 120.09 },
    { "start_seconds": 33.0, "end_seconds": 60.0, "bpm": 90.03 }
  ]
}

• global_bpm 120.08: the dominant tempo across the whole file
• alternate_bpm 90.03, alternate_coverage 0.45: the second tempo and the share of the file it covers
• tempo_segments: the two detected sections, split near 33 s against a true 30 s

Key change: full output

The complete key object Key Lime returned for the A-major to C-minor chord track on the demo slide.

"key": {
  "version": 6,
  "key": { "tonic": "C", "mode": "Minor" },
  "confidence": 0.021617552,
  "chroma": [0.0881601, 0.14320727, 0.0827591, 0.08654915, 0.14292027, 0.06208938, 0.048048906, 0.045049556, 0.06878138, 0.15347181, 0.050051074, 0.02891198],
  "segments": [
    { "start_seconds": 0.0, "end_seconds": 27.0, "key": { "tonic": "A", "mode": "Major" }, "confidence": 0.010805504 },
    { "start_seconds": 27.0, "end_seconds": 33.0, "key": { "tonic": "CSharp", "mode": "Minor" }, "confidence": 0.26016137 },
    { "start_seconds": 33.0, "end_seconds": 60.0, "key": { "tonic": "C", "mode": "Minor" }, "confidence": 0.0045011365 }
  ],
  "multi_key": true,
  "alternate_key": { "tonic": "A", "mode": "Major" },
  "alternate_coverage": 0.45
}