Script for Giant Steps benchmarking & improved BPM inference (Octave Correction)

[Paulllux]

Hi @fosfrancesco and team,

I've been using beat_this and achieved excellent results. I wanted to share a Python script I wrote that helps benchmark the model against the Giant Steps dataset.

It implements a custom post-processing routine that:

Corrects Octave Errors: Uses a specific heuristic (78-185 BPM range) to fix the common double/half-time errors found in Trap/DnB.

Calculates "Advanced BPM": Uses a phase-aware circular mean and linear regression on the beat grid to derive a highly precise stable BPM value from the raw beat times.

Runs in Parallel: Processes the dataset using ProcessPoolExecutor for speed.

I thought this might be useful for the community or for future reproducibility tests.

final.py

These are the results i got with the giant steps database ============================================================
FINAL RESULTS

Tracks: 664
Time: 364.4s

Acc1 (Strict 4%): 89.31%
Acc2 (Octave Safe): 95.03%

i got even higher when trying the small models in ensemble

============================================================
FINAL RESULTS (Ensemble)

Time: 790.1s

Acc1 (Strict): 90.92%
Acc2 (Octave): 96.37%

final_ensemble.py

[f0k]

Thank you for sharing! Finally, a task where it helps that we uploaded all three runs per model 🎉 Ensembling directly on the logits level does not work well according to our experiments, but taking the median across multiple tempo predictions is very reasonable.
The Acc1 looks very good. Interesting technique, I can see that filtering out the beginning and end as well as strong deviations from the median beat interval make sense, but are there examples where the linear regression gives a different output than just returning 60 / avg_interval straight away?

[Paulllux]

I'll look into that later. I can still get it higher if i could find a better way to get rid of 1/3 octave errors. Do you have tips to improve that ?

[fosfrancesco]

Thank you for sharing!

The 1/3 octave error is tricky. Even in the training data, we have a messy situation about that.
For example, all datasets cleaned by Böck, e.g., Ballroom, Hainsworth, GTZAN, etc... plus some others like ASAP, will beat a compound meter every 3 eight notes. For example, in a 6/8 time signature, only 2 beats are considered, corresponding to the 1 and 4 eighth notes.
In other datasets, like tapcorrect, harmonix, a 6/8 measure contains 6 beats instead, one for every 8th note.

If you are mainly interested in electronic music, I would expect things to be simpler and to never have this sort of error. Did you find some during your experimentations?

[Paulllux]

It scores excellent on electronic music, it's just the difficult jazz / polyrythmic stuff that is a challenge, but so far there is nobody that nails that.

Seems like it's good enough and I should continue porting it to ONNX so I can implement it in the plugin i'm making. :)

[Paulllux]

I got it a little bit higher on the giant steps database, converted the models to core ml for speed, and ended up with this result (base) paululrix@pauls-MacBook-Pro BeatThisBenchmark % python3 winner6.py /Users/paululrix/Documents/giantsteps-tempo-dataset/benchmark_data.csv
--- SOTA Core ML Benchmark (Tuned Hybrid) ---
Tracks: 664
Processed: 660/664 | Acc1: 92.1%

Done in 154.7s
Acc1 (Strict): 92.02%
Acc2 (Octave): 97.74%

winner6.py

What is strange is that the small models score better than the big ones.

[fosfrancesco]

mmm, interesting. A possible explanation (that is hard to prove unfortunately) is that the model is using a lot of capacity to put the beats in the correct phase, and to handle hard cases like changes of tempo and time signature.
So since your case is "simplified" as you expect a mostly constant tempo and no time signature change, the small model is good enough, and even a bit better.

[f0k]

@Paulllux: Just out of curiosity, how easy was the conversion with coremltools and how much did it improve performance? From the prints, it looks like you ran three small models in 154.7s, which took 790.1s before? Is everything else the same or did you also add multiprocessing or use a different device?

[Paulllux]

I'll sent you the conversion script later. I had to make 4 models. One for the mel spectogram and 3 for the small models.
The performance improved so much I thought at first something went wrong with the benchmark.

Multiprocessing doesn't help, it's all done in the macbook pro m4 neural engine.

I got it working now for mac in my juce plugin with core ml, for windows I still need to test the onnx version.

[Inviz]

I'm interested in the model conversion script too. Also in the bpm computations in general.

[Paulllux]

convert_to_coreml.py

export_preprocessor.py I used this , with this test script

verify.py

[f0k]

Thanks for sharing the scripts! Interesting gotchas about precomputing the RoPE frequencies and reflection padding in the spectrogram. I guess the correct STFT scale factor should be sqrt(fft_length), and is applied by PyTorch when calling torch.stft with normalized=True.