Integrating custom models for Exo #937

nightguarder · 2025-12-20T13:02:01Z

Custom MLX Models Support - Issue #918

Motivation

Fixes Issue #918: Enable users to run custom MLX based models from mlx-community on Hugging Face without manual code updates.

What Changed

1. Frontend UI for Custom Models

Commit: “Add custom models to dashboard"

Added "Custom Models" section to downloads page with HuggingFace model ID input
Implemented "Download Model" button triggering model Download into the exo HOME folder
SAFE registration of the custom model via separate /custom_models API

2. Tests

Commit: “Integrate new test”
@pytest are available in src/exo/worker/tests/test_custom_model.py folder
with future possibility to run them on CI/CD Pipeline on Github file (.github/workflows/custom_models.yml)

3. Persistent Storage & Model Registration

Commit: "Persist storage for custom models"

Fixed resolve_model_meta() to check both short_id keys and full model_id values
Enabled custom model registration to ~/.exo/custom_models.json during download
Models reload automatically on EXO restart from persistent storage
Prettified name for custom models in pretty_name key

4. Safe Downloading logic

Commit: “SAFE model registration”

Implemented "lazy loading" logic. Runners are now skipped for download_only instances until a task is received, while model downloads continue in the background.
Added automatic model registration. Custom models are now registered in ~/.exo/custom_models.json immediately when the download starts, ensuring they are recognized by the system right away.

Why It Works

This implementation enables dynamic custom model loading without requiring manual modifications to model_cards.

Users can:

Download any mlx-community based model via the dashboard from HuggingFace
Have models persist across restarts
Test out the their model once it loads

Known Issues

1. Missing `chat_template.jinja` for Some Models

Some mlx-community models don't include a chat template, causing the model to output its Instructions instead of formatted chat responses. This is a model-specific issue with mlx-community models, not a bug in our implementation.

Workaround: Use models that include proper chat templates (e.g., mlx-community/Qwen2.5-0.5B-Instruct-4bit) or add a chat template.jinja yourself.

Testing

Note that manual rebuild of dashboard cd dashboard && npm run build is needed

Manual Testing

Hardware: MacBook Pro (M4 Pro)
Tested with mlx-community/Qwen2.5-14B-Instruct-8bit
Verified:
- Model appears in downloads list with correct size
- Download progress bar updates in real-time
- Model persists in ~/.exo/custom_models.json
- Prettified name in [pretty_name] key entry
- Model is available after restart of exo
- Chat inference works correctly

Automated Testing

Integration test: src/exo/worker/tests/test_custom_model.py
CI workflow: .github/workflows/test_custom_models.yml

Files Modified

src/exo/master/api.py - Model resolution & API response
src/exo/shared/models/model_cards.py - Persistence logic
src/exo/worker/download/impl_shard_downloader.py - Registration on download
src/exo/worker/plan.py - Scheduler lazy loading logic
dashboard/src/routes/downloads/+page.svelte - Custom models UI
dashboard/src/lib/stores/app.svelte.ts - API integration

nightguarder · 2025-12-20T15:03:20Z

Hi, I have successfully added a new Feature: Testing custom MLX models

Can Someone please clone & run my fork to verify downloading a larger model like mlx-community/gpt-oss-20b-MXFP4-Q8? I don’t have enough RAM :/

nightguarder · 2025-12-20T15:03:55Z

I hope this is something we wanted. Currently only for testing purposes.

nightguarder · 2025-12-20T15:37:29Z

Not sure why my VSCode Prettier auto prettified all the files I’ve changed.

I will probably create a new clean PR where I only change the required code blocks, to keep it clean, if it’s needed to Approve this feature request.

Evanev7 · 2025-12-20T15:37:57Z

Looks good! I wonder if we should directly add the model to the model cards instead of a separate KNOWN_MODELS but there's wider questions to be answered in there.

Evanev7 · 2025-12-20T15:38:38Z

As for prettier, I don't believe our current formatter extends to the dashboard so I don't particularly mind atm

nightguarder · 2025-12-20T15:44:43Z

Looks good! I wonder if we should directly add the model to the model cards instead of a separate KNOWN_MODELS but there's wider questions to be answered in there.

My idea was after the users tests it and verify, then we add it model_cards as official supported model. but yeah, can be skipped.

Evanev7 · 2025-12-20T16:10:37Z

Ok - gpt-oss-20b-MXFP4-Q8 did not work, but the download was completely fine, seems like an upstream problem.

nightguarder · 2025-12-20T18:56:19Z

Ok - gpt-oss-20b-MXFP4-Q8 did not work, but the download was completely fine, seems like an upstream problem.

Yes I see the erorr. this might be more difficult than I thought. Runner 4e13d976-5262-43eb-b513-e9678e673e59 crashed with critical exception Quantized SDPA does not support attention sinks

Evanev7 · 2025-12-20T19:30:53Z

This isn't an issue for this PR - we need to bump mlx versions and test afaik.

nightguarder · 2025-12-21T09:11:58Z

Ok it’s working. GPT-OSS- model loaded. However I had to adedd TEMPORARY overrides as in my commit: 2e446ab Not ideal, we need to wait for official mlx support version.

nightguarder · 2025-12-21T09:22:34Z

GPT-oss-20b has no chat_template.jinja resulting in artifacts and instructions appearing in chat:

QUERY
Hello

EXO
09:25:43
TTFT 555ms•70.7 tok/s
<|channel|>analysis<|message|>We need to be helpful, concise, no reasoning inside answer. Respond "Hello". Maybe ask how to help.<|end|><|start|>assistant<|channel|>final<|message|>Hello! How can I help you today?

Evanev7 · 2025-12-21T11:42:28Z

appreciate the enthusiasm but can we keep this pr down to custom models? the gpt-oss fix is a separate issue.

This reverts commit 2e446ab.

gj-aazoo · 2026-01-02T15:24:22Z

Tested this branch and ran into this, would be useful to be also download models that you converted yourself.

Model ID must start with mlx-community/

nightguarder · 2026-01-02T20:09:27Z

Tested this branch and ran into this, would be useful to be also download models that you converted yourself.

Model ID must start with mlx-community/

You want to run models outside of mlx-community? they are not optimized for exo.

2350d01a9.

gj-aazoo · 2026-01-02T20:34:17Z

Yes, it is a MLX converted model. Sent from Android device Op 2 jan 2026 21:10 schreef Nightguarder ***@***.***>: [https://avatars.githubusercontent.com/u/73370044?s=20&v=4]nightguarder left a comment (exo-explore/exo#937)<#937 (comment)> Tested this branch and ran into this, would be useful to be also download models that you converted yourself. Model ID must start with mlx-community/ You want to run models outside of mlx-community? they are not optimized for exo. — Reply to this email directly, view it on GitHub<#937 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AIIULKDF5AHHSXTJJFWMD3D4E3GA3AVCNFSM6AAAAACPT3UOTGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTOMBWGE2DQMRRG4>. You are receiving this because you commented.Message ID: ***@***.***>

nightguarder · 2026-01-03T19:17:14Z

Yes, it is a MLX converted model. Sent from Android device Op 2 jan 2026 21:10 schreef Nightguarder @.***>:

Ok, You can now try your custom model. Please let me know if it works.

Evanev7 · 2026-01-04T16:58:25Z

Let me know when a good time for re-review is, I'm keen to get this integrated next week.

gj-aazoo · 2026-01-04T20:30:20Z

I'll test it tomorrow. Sent from Android device Op 4 jan 2026 17:58 schreef Evan Quiney ***@***.***>: [https://avatars.githubusercontent.com/u/13599445?s=20&v=4]Evanev7 left a comment (exo-explore/exo#937)<#937 (comment)> Let me know when a good time for re-review is, I'm keen to get this integrated next week. — Reply to this email directly, view it on GitHub<#937 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AIIULKG4IROVZ6IS6JGOWBT4FFBEPAVCNFSM6AAAAACPT3UOTGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTOMBYGI2DQOBSGU>. You are receiving this because you commented.Message ID: ***@***.***>

gj-aazoo · 2026-01-05T12:08:43Z

Hi, I checked out the latest code but I still receive this issue after uv rebuilding: DOWNLOAD Note: Downloading a custom model will automatically place it on available nodes. Model ID must start with mlx-community/ Verzonden vanaf Outlook voor Mac Van: Gert-Jan de Boer ***@***.***> Datum: zondag, 4 januari 2026 om 21:30 Aan: exo-explore/exo ***@***.***> CC: exo-explore/exo ***@***.***>, Comment ***@***.***> Onderwerp: Re: [exo-explore/exo] Integrating custom models for Exo (PR #937) I'll test it tomorrow. Sent from Android device Op 4 jan 2026 17:58 schreef Evan Quiney ***@***.***>: [https://avatars.githubusercontent.com/u/13599445?s=20&v=4]Evanev7 left a comment (exo-explore/exo#937)<#937 (comment)> Let me know when a good time for re-review is, I'm keen to get this integrated next week. — Reply to this email directly, view it on GitHub<#937 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AIIULKG4IROVZ6IS6JGOWBT4FFBEPAVCNFSM6AAAAACPT3UOTGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTOMBYGI2DQOBSGU>. You are receiving this because you commented.

gj-aazoo · 2026-01-05T13:44:42Z

Turns out I needed to rebuild the dashboard, my bad. The download works fine now. [image.png] The model runs fine on our Mac Studio. Thank you! Verzonden vanaf Outlook voor Mac Van: Gert-Jan de Boer ***@***.***> Datum: maandag, 5 januari 2026 om 13:08 Aan: exo-explore/exo ***@***.***> CC: exo-explore/exo ***@***.***>, Comment ***@***.***> Onderwerp: Re: [exo-explore/exo] Integrating custom models for Exo (PR #937) Hi, I checked out the latest code but I still receive this issue after uv rebuilding: DOWNLOAD Note: Downloading a custom model will automatically place it on available nodes. Model ID must start with mlx-community/ Verzonden vanaf Outlook voor Mac Van: Gert-Jan de Boer ***@***.***> Datum: zondag, 4 januari 2026 om 21:30 Aan: exo-explore/exo ***@***.***> CC: exo-explore/exo ***@***.***>, Comment ***@***.***> Onderwerp: Re: [exo-explore/exo] Integrating custom models for Exo (PR #937) I'll test it tomorrow. Sent from Android device Op 4 jan 2026 17:58 schreef Evan Quiney ***@***.***>: [https://avatars.githubusercontent.com/u/13599445?s=20&v=4]Evanev7 left a comment (exo-explore/exo#937)<#937 (comment)> Let me know when a good time for re-review is, I'm keen to get this integrated next week. — Reply to this email directly, view it on GitHub<#937 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AIIULKG4IROVZ6IS6JGOWBT4FFBEPAVCNFSM6AAAAACPT3UOTGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTOMBYGI2DQOBSGU>. You are receiving this because you commented.

nightguarder · 2026-01-05T15:21:24Z

Turns out I needed to rebuild the dashboard, my bad. The download works fine now. [image.png] The model runs fine now.

Great News! Yes you need to update the frontend via cd dashboard && npm run build. What model did you use? No problems with unknown/weird <tokens> appearing in chat? Thanks

gj-aazoo · 2026-01-05T15:30:14Z

https://huggingface.co/gjdeboer/Foundation-Sec-1.1-8B-Instruct-mlx-4Bit Runs just as well as in mlx-gui. Verzonden vanaf Outlook voor Mac Van: Nightguarder ***@***.***> Datum: maandag, 5 januari 2026 om 16:21 Aan: exo-explore/exo ***@***.***> CC: Gert-Jan de Boer ***@***.***>, Comment ***@***.***> Onderwerp: Re: [exo-explore/exo] Integrating custom models for Exo (PR #937) [https://avatars.githubusercontent.com/u/73370044?s=20&v=4]nightguarder left a comment (exo-explore/exo#937)<#937 (comment)> Turns out I needed to rebuild the dashboard, my bad. The download works fine now. [image.png] The model runs fine now. Great News! Yes you need to update the frontend via cd dashboard && npm run build. What model did you use? No problems with unknown messages appearing in chat? Thanks — Reply to this email directly, view it on GitHub<#937 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AIIULKBMFC6CLGOJ3RCIRB34FJ6QVAVCNFSM6AAAAACPT3UOTGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTOMJQHA3TANBRHE>. You are receiving this because you commented.

nightguarder · 2026-01-06T21:14:02Z

@Evanev7
I think we are now ready to do a code review since another user reported successful running a custom model.

However I had some issues after pulling the latest changes after your commit 1ec550d. Mainly with placement.py.

Regarding this commit do you plan to fix the download model status? I find it rather disturbing to show all the default models not being downloaded. Just hide them under a tab Not Downloaded.

Evanev7 · 2026-01-07T13:04:58Z

It's not my favourite, but it's not really my code. WIP I think.
I would really much rather we only checked for models we had actually downloaded instead of trying all of them.

nightguarder · 2026-01-10T19:54:57Z

It's not my favourite, but it's not really my code. WIP I think. I would really much rather we only checked for models we had actually downloaded instead of trying all of them.

@Evanev7 I would like to fix it but this PR already has a lot of changes and we should not complicate things further.

Evanev7 · 2026-01-12T11:30:56Z

Agreed - let's get this merged and we can iterate

JakeHillion

A few things before we can consider merging this:

There are merge conflicts in model_cards and the Svelte file. I am able to fix the model card ones quite easily, but the Svelte changes were significantly overlapping. Please push a merge commit for that.
There are several unrelated changes in here. Please run git rm -r .vscode/, git checkout main dashboard/package-lock.json and commit at least.
Run nix fmt and commit it.
There are lots of unrelated changes spread throughout this. I left a comment on a specific one. After the above steps, please push this PR and take a look through the "Files Changed" on GitHub. If there are any changes which aren't adding integration for custom models, please remove them from the PR so they don't show up anymore.

Thanks for the submission, I look forward to reviewing it in detail once it's merged & cleaned up!

JakeHillion · 2026-01-15T16:46:35Z

dashboard/src/routes/downloads/+page.svelte

+		topologyData,
+		type DownloadProgress,
+		placeInstance,
+	} from "$lib/stores/app.svelte";


These quote changes, for example, shouldn't be in this PR.

integration test done

64b9e28

nightguarder marked this pull request as draft December 20, 2025 13:02

nightguarder mentioned this pull request Dec 20, 2025

Run custom MLX models #918

Open

Evanev7 linked an issue Dec 20, 2025 that may be closed by this pull request

Run custom MLX models #918

Open

nightguarder added 5 commits December 20, 2025 14:17

add logs to gitignore

c656c5d

add new action workflow for integration test

a9c862d

integrate new test

e3048d8

Add custom models to dashboard

979d122

vscode to git ignore

ad891ad

nightguarder added 3 commits December 20, 2025 20:35

prevent users downloading non-mlx-community models

e15e268

override memory check TEMPORARY

2e446ab

Persist storage for custom models.

71b5cc5

nightguarder marked this pull request as ready for review December 21, 2025 09:27

integration tests workflow

842fd01

Evanev7 linked an issue Dec 21, 2025 that may be closed by this pull request

Any Plans To Add Importing A Custom Model Not From Huggingface #747

Open

Evanev7 removed a link to an issue Dec 21, 2025

Any Plans To Add Importing A Custom Model Not From Huggingface #747

Open

Evanev7 linked an issue Dec 22, 2025 that may be closed by this pull request

Is it possible for importing model in to exo node? #801

Open

Revert "override memory check TEMPORARY"

a88fad2

This reverts commit 2e446ab.

Evanev7 linked an issue Dec 26, 2025 that may be closed by this pull request

Allow custom Hugging Face model downloads in dashboard (especially MLX-Community) #1011

Open

AlexCheema mentioned this pull request Dec 31, 2025

How to add more models to exo? #1064

Closed

nightguarder changed the title ~~Integrating custom mlx models~~ Integrating custom models for Exo Jan 2, 2026

nightguarder added 2 commits January 2, 2026 21:17

use pytests

e5e2c0f

Revert "prevent users downloading non-mlx-community models"

85bbc40

2350d01a9.

nightguarder added 4 commits January 3, 2026 20:25

SAFE model registration

c891a74

pretty_name for custom models

fdc3c7d

prevent download issues

b109c51

Merge branch 'main' into feature/issue-918-custom-mlx-models

fdbf1d5

nightguarder requested a review from Evanev7 January 3, 2026 21:06

nightguarder added 3 commits January 3, 2026 22:16

resolve merge conflicts

112688c

downloading custom_models work after merge

990d60b

standardize the code for model registration

5389b6b

Merge branch 'main' into feature/issue-918-custom-mlx-models

3b41474

Merge branch 'main' into feature/issue-918-custom-mlx-models

69e47dc

JakeHillion requested changes Jan 15, 2026

View reviewed changes

Integrating custom models for Exo #937

Are you sure you want to change the base?

Integrating custom models for Exo #937

Conversation

nightguarder commented Dec 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Custom MLX Models Support - Issue #918

Motivation

What Changed

1. Frontend UI for Custom Models

2. Tests

3. Persistent Storage & Model Registration

4. Safe Downloading logic

Why It Works

Users can:

Known Issues

1. Missing chat_template.jinja for Some Models

Testing

Manual Testing

Automated Testing

Files Modified

Uh oh!

nightguarder commented Dec 20, 2025

Uh oh!

nightguarder commented Dec 20, 2025

Uh oh!

nightguarder commented Dec 20, 2025

Uh oh!

Evanev7 commented Dec 20, 2025

Uh oh!

Evanev7 commented Dec 20, 2025

Uh oh!

nightguarder commented Dec 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Evanev7 commented Dec 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nightguarder commented Dec 20, 2025

Uh oh!

Evanev7 commented Dec 20, 2025

Uh oh!

nightguarder commented Dec 21, 2025

Uh oh!

nightguarder commented Dec 21, 2025

Uh oh!

Evanev7 commented Dec 21, 2025

Uh oh!

gj-aazoo commented Jan 2, 2026

Uh oh!

nightguarder commented Jan 2, 2026

Uh oh!

gj-aazoo commented Jan 2, 2026 via email

Uh oh!

nightguarder commented Jan 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Evanev7 commented Jan 4, 2026

Uh oh!

gj-aazoo commented Jan 4, 2026 via email

Uh oh!

gj-aazoo commented Jan 5, 2026 via email

Uh oh!

gj-aazoo commented Jan 5, 2026 via email

Uh oh!

nightguarder commented Jan 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gj-aazoo commented Jan 5, 2026 via email

Uh oh!

nightguarder commented Jan 6, 2026

Uh oh!

Evanev7 commented Jan 7, 2026

Uh oh!

nightguarder commented Jan 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Evanev7 commented Jan 12, 2026

Uh oh!

JakeHillion left a comment

nightguarder commented Dec 20, 2025 •

edited

Loading

1. Missing `chat_template.jinja` for Some Models

nightguarder commented Dec 20, 2025 •

edited

Loading

Evanev7 commented Dec 20, 2025 •

edited

Loading

nightguarder commented Jan 3, 2026 •

edited

Loading

nightguarder commented Jan 5, 2026 •

edited

Loading

nightguarder commented Jan 10, 2026 •

edited

Loading