[MAEB] Mark downsampled dataset

[Samoed]

We have some downsampled tasks, e.g. CommonVoice17A2TRetrieval, but we don't mark them as downsampled or modified. I think we can rename them as MiniCommonVoice17A2TRetrieval or something similar and keep full tasks versions too, but without including to benchmarks

@AdnanElAssadi56 @isaac-chung

[isaac-chung]

Please check existing maeb tasks. I think we put "mini" at the end of the name?

[Samoed]

Not to all of them. As I mention in description we have commonvoice that have this only in dataset repository name, not in task name. Probably there are more tasks like this

mteb/mteb/tasks/audio/any_2_any_retrieval/common_voice.py

Lines 186 to 193 in edf141a

	class CommonVoice17A2TRetrieval(AbsTaskRetrieval):
	metadata = TaskMetadata(
	name="CommonVoice17A2TRetrieval",
	description="Speech recordings with corresponding text transcriptions from CommonVoice dataset.",
	reference="https://commonvoice.mozilla.org/",
	dataset={
	"path": "mteb/common_voice_17_0_mini",
	"revision": "62c6f7bb73d8cdb684868c14620ce241448e471b",

This creates inconsistency when we're adding new dataset, e.g. #3904 (comment)

[isaac-chung]

I understand. Just meant that for those we added, it was added at the end, not at the beginning like in your example. We should unify the naming.

[Samoed]

What do you mean in the end? In load_data/dataset_transform? If yes, then most of the time we're just downsampling classifications tasks with stratified_subsampling which is fine I think, but if we're doing something with retrieval, then we need to state this

[diffunity]

@Samoed I think he means the convention for the datasets is "some-dataset-mini" instead of "mini-some-dataset"

Other tasks that are "-mini" prepended in the dataset name but not indicated in the task name:

mteb/commonlanguage-age-mini
mteb/commonlanguage-gender-mini
mteb/commonlanguage-lang-mini
mteb/speech-commands-mini
mteb/voxpopuli-accent-mini
mteb/sib-fleurs-multilingual-mini
mteb/voxpopuli-mini
mteb/birdclef25-mini
mteb/nsynth-mini
mteb/tut-acoustic-scenes-mini

[AdnanElAssadi56]

For all datasets that I subsampled, I added the “mini” suffix to the dataset name. I wasn’t sure whether we also wanted to include it in the task name, especially since we didn’t add “mini” when we previously did stratified subsampling (which is effectively the same task wise). I did include “mini” in the Hugging Face path to avoid misleading users into thinking this is the full original split, and I also noted in the task description of tasks I added that the datasets were subsampled.

I’m not sure how others handled dataset additions. If we agree on a convention, we can do a full pass to unify the structure, and to check if anything was missed.

[Samoed]

When we downsamoled datasets for MMTEB we added HardNegative prefix for retrieval tasks. I agree that we can keep classification tasks as is, but for other types we should add something (at least for common voice)