Support open source LLMs with mlx-lm

[lopuhin]

Unfortunately they return logprobs in a different format, which require loading a tokenizer to work.

[Copilot]

Copilot reviewed 3 out of 5 changed files in this pull request and generated no comments.

Files not reviewed (2)

• docs/source/_notebooks/explain_llm_logprobs.rst: Language not supported
• tox.ini: Language not supported

Comments suppressed due to low confidence (1)

eli5/llm/explain_prediction.py:127

• Using assert here may be unsafe in production if Python is run in optimized mode. Consider replacing it with an explicit validation that raises a ValueError if the lengths do not match.

assert len(logprobs.token_logprobs) == len(logprobs.tokens)

[lopuhin]

Rendered tutorial: https://eli5.readthedocs.io/en/latest/tutorials/explain_llm_logprobs.html#open-source-and-other-models