How to speedup the evaluations on Clustering and Retrieval datasets from MTEB(eng, v1)

[ipsitmantri]

I am evaluating Mistral-7B (in fp32) on the original 56 task MTEB english benchmark. For datasets with more than 1M sentences, on NVIDIA A100 80GB, the evaluation is taking 3-4 days of runtime. I set the device to CUDA to compute the embeddings. Can you suggest how to speedup?

Thank you

[Samoed]

I don't think you can speed up evaluation

[KennethEnevoldsen]

I moved this over to a discussion @ipsitmantri.

We document a few tricks for speeding up evaluations here. The best trick here is probably to reduce precision, and or if you see that the GPU is idling, then make sure that you have the latest version of datasets installed to ensure that you use xet.

Sometimes issues related to speed are discovered and fixed (e.g. #3518), so using the latest version of (or at least mteb>=2.2.0) might help you a bit (Updated docs to also include this point #3634).

Generally though the original MTEB (MTEB(eng, v1) as we denote it now) is frustratingly big, which is why we recommend using MTEB(eng, v2), which resolves a lot of the issues in MTEB(eng, v1) (a few bugs, dataset issues such as removed duplicates and empty documents and removed datasets frequently used for training). v2 also comes with better implementation of many tasks e.g. the clustering tasks which are statistically more stable, while using notably fewer samples.

[KennethEnevoldsen]

Generally, though it will take a long time, e.g. this paper reports:

Full evaluation on MTEB takes more than 40h for Mistral-7B on 8x A100 GPUs.

They also show that unadapted LLM models (e.g. mistral) are generally quite poor. However, for mistral you can enable bidirectional attention and gain see decent improvements for Mistral specifically (they e.g. show this here)