Official repository for TransferBench — a reproducible benchmark for ensemble-based black-box transfer attacks. Accepted to NeurIPS 2025 (Datasets & Benchmarks Track). For citation and paper details see the Paper section below.

〶ransferBench

TransferBench is a Python package designed for evaluating black-box transfer attacks using one or more surrogate models. It provides a flexible and streamlined interface for testing attack effectiveness across a variety of scenarios involving different victim models, surrogate models, and datasets. Optional robust victim models are also supported.

News 📰✨

• (2025-09) 🎉 TransferBench has been accepted to NeurIPS — Datasets & Benchmarks Track! 🚀 See the Paper section below or the OpenReview link for details. 🙏 Thanks to all contributors and collaborators!

Features ✅

• Effortless evaluation of black-box attacks on predefined scenarios.
• Fully customizable via user-defined evaluation scenarios, models, datasets.
• Lightweight core dependencies; extended scenarios available via optional extras.
• Connection with Weight & Biases for inspecting running, missing, and finished evaluations.

Installation ⚙️

To install the standard version:

pip install git+https://[email protected]/pralab/transfer-bench.git

To enable evaluation on robust scenarios or CIFAR datasets, install the additional dependencies:

pip install "git+https://[email protected]/pralab/transfer-bench.git#egg=transferbench[robust]"

or

pip install "git+https://[email protected]/pralab/transfer-bench.git#egg=transferbench[cifar]"

Quickstart 📌

Here's a minimal example to evaluate an attack using the default settings:

from transferbench import AttackEval
from transferbench.attacks_zoo import NaiveAvg

evaluator = AttackEval(NaiveAvg)
print(evaluator.scenarios)  # Display default scenarios
result = evaluator.run(batch_size=4, device="cuda:1")  # Run evaluation
print(result)

For more advanced examples and customization options, see the tutorial notebook.

Evaluation Scenarios 🎯

Attack evaluations are grouped into campaigns, each defining a different set of victim-surrogate model configurations:

• etero: heterogeneous surrogates
• omeo: homogeneous surrogates
• robust: robust victim models (optional)

Scenarios are stored in the directory transferbench/config/scenarios where also scenarios involved in the original papers have been included for comparison.

Scenario information is aggregated in a YAML file as follows

etero-imagenet-inf:
  - hp:
      maximum_queries: 50
      p: "inf"
      eps: 0.062745  # 16/255
    victim_model: "vgg19"
    surrogate_models: ["resnet50", "resnext50_32x4d", "densenet121", "swin_b", "swin_t", "vit_b_32"]  # CNNPool
    dataset: "ImageNetT"

  - hp:
      maximum_queries: 50
      p: "inf"
      eps: 0.062745
    victim_model: "resnext101_32x8d"
    surrogate_models: ["inception_v3", "convnext_base", "vgg16",  "swin_b", "swin_t", "vit_b_32"]  # ResPool
    dataset: "ImageNetT"

  - hp:
      maximum_queries: 50
      p: "inf"
      eps: 0.062745
    victim_model: "vit_b_16"
    surrogate_models: ["inception_v3", "convnext_base", "vgg16", "resnet50", "resnext50_32x4d", "densenet121"]  # ViTPool
    dataset: "ImageNetT"

Other included scenarios:

• omeo-imagenet-inf
• robust-imagenet-inf (optional, requires [robust] installation)
• omeo-cifar10-inf (optional, requires [cifar] installation)
• etero-cifar10-inf (optional, requires [cifar] installation)
• robust-cifar10-inf (optional, requires [cifar,robust] installation)

Command-Line Interface: trbench 🧪

For full pipeline control, use the trbench CLI script. It helps manage experiment runs, tracks progress, and saves results automatically.

See the trbench guide for more details.

Contributing to the Attack Zoo 🤝

We welcome contributions! To contribute to the attacks_zoo or other components, please read our contribution guide.

Implemented Attacks

Implemented attacks, all of which allow batch-wise computation.

Attack	Venue	m	Heterogenous	Robust	Targeted	p	ε	ASR [%]	𝑞̄
SubSpace Guo et al., 2019	NeurIPS	3	Yes	No	No	∞	13/255	98.9%	462
SimbaODS Tashiro et al., 2020	NeurIPS	4	No	No	Yes	∞	13/255	92.0%	985
GFCS Lord et al., 2022	ICLR	4	No	No	Yes	2	$\sqrt{0.001d}$¹	60.0%	20
BASES Cai et al., 2022	NeurIPS	20	No	No	Yes	∞	16/255	99.7%	1.8
GAA Yang et al., 2024	PR	4	No	No	Yes	∞	16/255	46.0%	3.9
DSWEA Hu et al., 2025	PR	10	No	No	Yes	∞	16/255	96.6%	2.7

---

¹ Images included in the experiments have $d=3\cdot 299\cdot299$ pixels, from which $\varepsilon\approx16.37$

Paper 📄 Benchmarking Ensemble-based Black-box Transfer Attacks

TransferBench: Benchmarking Ensemble-based Black-box Transfer Attacks
Fabio Brau, Maura Pintor, Antonio Emanuele Cinà, Raffaele Mura, Luca Scionis, Luca Oneto, Fabio Roli, Battista Biggio

Accepted to NeurIPS 2025 — Datasets and Benchmarks Track.
OpenReview: https://openreview.net/forum?id=uT0A1pjBqu

Bibliography (BibTeX)

@inproceedings{brau2025transferbench,
  title={TransferBench: Benchmarking Ensemble-based Black-box Transfer Attacks},
  author={Brau, Fabio and Pintor, Maura and Cin{\`a}, Antonio Emanuele and Mura, Raffaele and Scionis, Luca and Oneto, Luca and Roli, Fabio and Biggio, Battista},
  booktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
  year = {2025},
  url = {https://openreview.net/forum?id=uT0A1pjBqu}
}

License 📜

TransferBench is released under the CC BY-NC-ND 4.0, see the LICENSE file for full details.