add Devstral demo as code assistant #3911
Conversation
There was a problem hiding this comment.
Pull request overview
This PR adds support for the Devstral model as a code assistant option in the demo documentation. The changes enable users to deploy and configure the unsloth/Devstral-Small-2507 model for local code assistance tasks.
Changes:
- Added
devstralas a supported tool parser in the LLM reference documentation - Introduced deployment instructions for unsloth/Devstral-Small-2507 model on both Linux and Windows platforms
- Updated existing model deployment commands to include missing parameters and use the
weeklyDocker image tag
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| docs/llm/reference.md | Added devstral to the list of supported tool parsers and updated documentation to reference it |
| demos/code_local_assistant/README.md | Added complete deployment and configuration instructions for Devstral model across Linux and Windows platforms, plus fixes to existing model deployment commands |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
demos/code_local_assistant/README.md
Outdated
| --model_name nsloth/Devstral-Small-2507 \ | ||
| --model_path nsloth/Devstral-Small-2507 |
There was a problem hiding this comment.
The model_name and model_path use 'nsloth' instead of 'unsloth', which is inconsistent with the source_model parameter on line 75. This appears to be a typo and should be 'unsloth' to match the actual model identifier.
| --model_name nsloth/Devstral-Small-2507 \ | |
| --model_path nsloth/Devstral-Small-2507 | |
| --model_name unsloth/Devstral-Small-2507 \ | |
| --model_path unsloth/Devstral-Small-2507 |
| openvino/model_server:weekly \ | ||
| --pull \ |
There was a problem hiding this comment.
The command structure is incorrect. The Docker image name should be followed by container configuration flags (like -d, --rm, -v), not command arguments like --pull. The --pull flag appears to be intended for the ovms executable inside the container, but the command structure doesn't properly separate Docker flags from OVMS command arguments.
| openvino/model_server:weekly \ | ||
| --pull \ |
There was a problem hiding this comment.
The command structure is incorrect. Similar to the previous issue, the Docker image name should be followed by container configuration flags, not command arguments. The --pull flag and subsequent arguments should be passed to the ovms executable inside the container.
| > **Note:** For deployment, the model requires ~16GB disk space and recommended 16GB+ of VRAM on the GPU. For conversion, the original model will be pulled and quantization will require the amount of RAM of the model size. | ||
There was a problem hiding this comment.
| > **Note:** For deployment, the model requires ~16GB disk space and recommended 16GB+ of VRAM on the GPU. For conversion, the original model will be pulled and quantization will require the amount of RAM of the model size. | |
| > **Note:** For deployment, the model requires ~16GB disk space and recommended 16GB+ of VRAM on the GPU. For conversion, the original model will be pulled and quantization will be applied. It requires the amount of RAM equal to the model size <how much?> | |
please fill "how much"
is it 150gb?
| --model_name unsloth/Devstral-Small-2507 \ | ||
| --model_path unsloth/Devstral-Small-2507 | ||
| ``` | ||
| > **Note:** This model requires ~13GB disk space and recommended 16GB+ of VRAM on the GPU for deployment. For conversion, the original model will be pulled and quantization will require the amount of RAM of the model size. |
There was a problem hiding this comment.
| > **Note:** This model requires ~13GB disk space and recommended 16GB+ of VRAM on the GPU for deployment. For conversion, the original model will be pulled and quantization will require the amount of RAM of the model size. | |
| > **Note:** This model requires ~13GB disk space and recommended 16GB+ of VRAM on the GPU for deployment. For conversion, the original model will be pulled and quantization will be applied. It requires the amount of RAM equal to the model size <how much?> |
please fill "how much"
There was a problem hiding this comment.
Also what about VRAM information of other models? We are missing it for qwen3 and qwen2.5
| ovms.exe --add_to_config --config_path models/config_all.json --model_name openai/gpt-oss-20b --model_path openai/gpt-oss-20b | ||
| ``` | ||
| > **Note:** This model requires ~13GB disk space and same amount of VRAM on the GPU for deployment. For conversion, the original model will be pulled and quantization will require the amount of RAM of the model size. | ||
| > **Note:** This model requires ~12GB disk space and recommended 16GB+ of VRAM on the GPU for deployment. For conversion, the original model will be pulled and quantization will require the amount of RAM of the model size. |
There was a problem hiding this comment.
| > **Note:** This model requires ~12GB disk space and recommended 16GB+ of VRAM on the GPU for deployment. For conversion, the original model will be pulled and quantization will require the amount of RAM of the model size. | |
| > **Note:** This model requires ~12GB disk space and recommended 16GB+ of VRAM on the GPU for deployment. For conversion, the original model will be pulled and quantization will be applied. It requires the amount of RAM equal to the model size <how much?> |
| ovms.exe --add_to_config --config_path models/config_all.json --model_name unsloth/Devstral-Small-2507 --model_path unsloth/Devstral-Small-2507 | ||
| ``` | ||
| > **Note:** This model requires ~13GB disk space and recommended 16GB+ of VRAM on the GPU for deployment. For conversion, the original model will be pulled and quantization will require the amount of RAM of the model size. |
There was a problem hiding this comment.
| > **Note:** This model requires ~13GB disk space and recommended 16GB+ of VRAM on the GPU for deployment. For conversion, the original model will be pulled and quantization will require the amount of RAM of the model size. | |
| > **Note:** This model requires ~13GB disk space and recommended 16GB+ of VRAM on the GPU for deployment. For conversion, the original model will be pulled and quantization will be applied. It requires the amount of RAM equal to the model size <how much?> |
🛠 Summary
🧪 Checklist
``