trail mix: DSP analysis with a compact transformer key reranker

Cueport

Cueport is a multi-device workstation for DJs that can import Serato libraries or construct local libraries directly from audio files.

Local pairing

The desktop and mobile applications exchange selected library data over a local network or hotspot after QR-based pairing.
The QR code carries connection details; data moves both ways after pairing.

Serato integration

Cueport v1 reads tempo, key, cues, and beat-grid metadata from an existing Serato library on disk.
The desktop database imports audio and metadata from Serato.
The mobile database is a cached replica.

Cueport Direct

Cueport Direct constructs a working music library from local folders without depending on an external DJ database.

• Pick one or more local music folders
• Cueport builds a cueport-direct.db working library
• Browse, prep, play, and edit without a Serato library
• trail mix analyzes audio locally for BPM, key, beats, and waveforms

trail mix

trail mix is a Rust toolkit that decodes audio into mono samples and runs three analyzers behind one versioned JSON output.

Signal input and normalization

Audio decoding normalizes supported file formats into a mono sample buffer and sample rate.

• Decoding: an audio file becomes evenly spaced amplitude values
• Downmixing: sample buffers combine into one mono signal
• f32: a 32-bit floating-point number for storing those amplitudes
• u32: a non-negative 32-bit integer for storing the sample rate

Why build a local analyzer?

Cueport Direct imports ordinary folders on a user's machine. Analysis has to run in process, finish during library import, produce reproducible metadata, and avoid requiring a server.

Onset detection

Onset detection measures frame-to-frame increases in band energy and uses peaks in the resulting onset-strength curve as beat candidates.

Onset detection formula

Spectral flux turns band-energy rises into one onset-strength value per frame.

\[\operatorname{flux}(t) = \sum_{b=1}^{24} \max\!\bigl(0,\; E_b(t) - E_b(t-1)\bigr)\]

• E_b(t): energy in band b at frame t
• max(0, change): count rises and ignore decays
• sum over b: combine all 24 band changes into the onset-strength curve

Tempo estimation

Tempo estimation finds the repeated spacing between onset peaks and converts that spacing into BPM. Beat tracking is the next step.

Beat tracking with dynamic programming (DP)

Tempo gives the beat spacing. DP chooses the beat grid offset, skips loud off-beat peaks, and keeps the grid regular when a beat is weak.

Beat tracking optimization

DP chooses the beat sequence with the best total score: strong onset peaks plus regular spacing near the estimated BPM.

\[ \hat{b}_{1:m}=\arg\max_{b_{1:m}} \left[ \sum_t O(b_t) + \lambda\sum_{t>1} R(b_t-b_{t-1}, P) \right] \]

• b_t: candidate beat time t
• O(b_t): onset strength at that beat time
• P: target beat spacing from the estimated BPM
• R(gap, P): spacing reward; highest when the gap is close to P
• DP job: find the full beat sequence with the highest total score

Beat switches

beat salad reports a dominant BPM and an alternate tempo when a secondary tempo covers at least 25% of the track.

Beat-switch decision rule

Multi-tempo detection reruns tempo estimation in rolling windows and reports a secondary tempo only when it covers enough of the track.

\[ \begin{aligned} \widehat{b}_w &= \operatorname{tempo}(w) \\ b_r &= \operatorname*{mode}_{w \in r}\,\widehat{b}_w \\ q_r &= \frac{\operatorname{duration}(r)}{\operatorname{duration}(\mathrm{track})} \end{aligned} \]

• w: a 12-second analysis window, advanced every 6 seconds
• r: a contiguous run made by grouping neighboring windows whose local BPM estimates agree
• b_r: representative BPM for that run of windows
• Decision rule: report multi_tempo when \(|b_r - b_{\mathrm{global}}| > \delta\) and \(q_r \ge 0.25\)

Tempo evaluation metrics

Tempo uses Acc1 and Acc2. The next slide reports both on GiantSteps Tempo.

Tempo diagnostic results

On GiantSteps Tempo, beat salad leads the DSP systems on exact-tempo Acc1 and trails stratum-dsp by 0.02 on Acc2, which accepts half-time and double-time matches. Beat This learns beat activations instead of using a fixed spectral-flux onset curve.

Waveform summarization

Key vocabulary

These terms define the musical object that key lime predicts.

Key estimation as 24-way classification

Chroma measures energy in 12 pitch classes. The classifier predicts which pitch class is the tonic and whether the mode is major or minor.

DSP key-estimation pipeline

Outputs by stage: windowed audio frames → pitch-class evidence → averaged 12-bin chroma → 24 key scores → global key with optional local segments.

Framing audio for key estimation

key lime analyzes short overlapping slices of audio. A Hann window fades each slice at the edges before frequency analysis.

• Frame: a short slice of the audio signal
• H: hop size between frames
• Hann window: fades the frame edges so boundaries contribute less energy
• Output: windowed audio frames for Goertzel and CQT pitch analysis

Measuring pitch energy in each frame

Goal: turn each windowed audio frame into note-energy estimates. The next step folds those estimates into 12 chroma values.

• Goertzel: ask a few chosen questions, such as "how much A is in this frame?"
• CQT: build a broader note-spaced energy map across the pitch range
• Output: note energy estimates for the next chroma-folding step

Output: 12-value chroma

This is the step where the pipeline first creates chroma: 12 values, one for each pitch class.

• Input: pitch-frequency energy from Goertzel and CQT
• Fold octaves: C2, C3, and C4 all add to the C pitch-class bin
• Output: 12 chroma values, one for each pitch class
• Next step: compare the chroma pattern with 24 key templates

Key templates and scoring

A key template is a 12-value chroma pattern for one key. Compare the song chroma with 24 templates and pick the highest score.

• c: the song chroma, 12 values for pitch-class energy
• T_k: the key template for candidate key k, 12 expected chroma values
• corr: Pearson correlation, high when the two 12-value shapes match
• Selection: the highest of 24 template scores becomes the predicted key

Key evaluation metric

For each track, MIREX gives credit based on how close the predicted key is to the reference key, then averages those credits from 0 to 100.

\[s_i=\text{credit for track }i\text{ from the table},\qquad \text{MIREX}=\frac{s_1+s_2+\cdots+s_N}{N}\]

Reranker data splits

KL means key lime. FMAK means FMA Key, a key-labeled Free Music Archive benchmark; FMAKv2 is the version-2 evaluation set.

Why add a reranker?

On the 1,000-track FMAKv2 holdout, the DSP key baseline scores 61.08 MIREX. Local madmom scores 74.16 MIREX on the same files.

DSP-guided transformer reranker

Reranking means the DSP scorer proposes 24 key scores, then the learned model adjusts those scores using chroma-frame context.

• 1. DSP ranks candidates: produce 24 scores, one for each tonic-mode label
• 2. Chroma gives context: show how pitch-class energy changes across the first 256 frames
• 3. Learned model adjusts scores: add a learned correction to the DSP ranking
• 4. Highest adjusted score wins: choose the global key; local key segments remain DSP-based

Reranker architecture and parameter budget

The chroma branch makes 24 learned score corrections. The DSP branch carries the 24 DSP scores. Add them, then choose the largest score.

How the reranker is trained

Training teaches the reranker when chroma context should change the DSP ranking.

FMAKv2 key-estimation results

Only compare rows with the same run type. The sealed-holdout rows are fair comparisons; published full-FMAKv2 rows are context, not win/loss claims.

Which input features help?

This asks whether the gain comes from reranking alone or from giving the reranker chroma-frame evidence.

New model footprint

This table breaks installed size into analyzer code, model weights, and runtime so the total shows what an app ships.

Accuracy versus installed footprint

The paper footprint figure uses the ONNX package measurement; the Cueport release uses the same weights in native Rust.

Key runtime measurements

KL-FMAK uses the same full DSP analysis as KL-DSP, then adds a sub-millisecond reranker pass.

trail mix next steps

Next work should cover beat analysis, stronger key benchmarks, stronger beat benchmarks, and compact model experiments.

Cueport next steps

Follow Cueport development at usecueport.com.

Selected bibliography

A short reference map for the methods, learned key baselines, tempo baselines, and datasets used in the talk.