Mistral Medium 3.5 is Mistral AI's latest open-weight model, a 128-billion parameter dense architecture that competes with frontier models at a fraction of the cost. For teams processing millions of tokens daily, self-hosting eliminates per-token API fees and puts you in full control of your inference stack. Data never leaves your infrastructure, latency is predictable, and you can tune every parameter to match your workload.

The model ships under a Modified MIT License with open weights on HuggingFace, making it one of the most capable openly available models for commercial deployment. With native support in vLLM, SGLang, Ollama, and NVIDIA NIM, there are multiple paths to production depending on your scale and infrastructure preferences.

This guide covers everything from GPU sizing and inference engine selection to cost breakeven analysis and production hardening. Whether you're setting up a development environment on a single machine or architecting a multi-node cluster for enterprise workloads, you'll find the practical details here.

1Why Self-Host Mistral Medium 3.5?

The Mistral API works well for prototyping and low-volume workloads. But as usage scales, four factors consistently push teams toward self-hosting:

2Hardware Requirements

Mistral Medium 3.5 is a 128B dense model, meaning all parameters are active during inference (unlike MoE architectures). This makes VRAM requirements straightforward to calculate based on precision.

VRAM Requirements by Precision

Recommended GPU Configurations

3vLLM Deployment

vLLM is the recommended inference engine for Mistral Medium 3.5. It provides PagedAttention for efficient memory management, continuous batching for high throughput, and an OpenAI-compatible API out of the box. Mistral Medium 3.5 requires mistral_common >= 1.11.1 and transformers >= 5.4.0.

Step 1: Install vLLM (Nightly Build Recommended)

Step 2: Launch the Server

Key flags explained:

• --tensor-parallel-size 8 - splits the model across 8 GPUs. Use 4 for FP8 on 4×H100
• --tool-call-parser mistral - enables native Mistral tool-calling format for function calling
• --enable-auto-tool-choice - lets the model decide when to invoke tools automatically
• --reasoning-parser mistral - activates the Mistral reasoning parser for chain-of-thought
• --gpu_memory_utilization 0.8 - reserves 80% of VRAM for the model, leaving headroom for KV cache spikes

EAGLE Speculative Decoding

Mistral provides an EAGLE model for speculative decoding, which can significantly speed up token generation. The EAGLE draft model predicts multiple tokens ahead, and the main model verifies them in parallel, reducing the number of forward passes needed.

4SGLang Deployment

SGLang is a strong alternative to vLLM, with day-zero support for Mistral Medium 3.5 and optimized Docker images for both Hopper (H100) and Blackwell (B200) GPU architectures. It excels at structured generation, constrained decoding, and multi-turn conversation patterns.

Launch with SGLang

Docker Images

SGLang provides pre-built Docker images optimized for specific GPU architectures:

• Hopper (H100/H200) - use the standard SGLang Docker image with CUDA 12.x support for optimal tensor core utilization
• Blackwell (B200/GB200) - SGLang offers dedicated Blackwell images with day-zero support, taking advantage of the newer FP4 and FP8 capabilities

SGLang advantages for Mistral Medium 3.5:

• RadixAttention - caches KV states across conversation turns, reducing latency for multi-turn chat and agentic workflows
• Structured Generation - native support for constrained JSON output and regex-guided decoding, ideal for tool-calling patterns
• OpenAI-Compatible API - exposes /v1/chat/completions for seamless integration with existing clients

5Ollama for Development

For local development and testing, Ollama provides the simplest path to running Mistral Medium 3.5. It handles model downloading, quantization, and serving with a single command. GGUF quantized versions from Unsloth are available, making it possible to run the model on consumer hardware with reduced quality.

Quick Start

Ollama automatically selects the best quantization level for your available hardware. For machines with limited VRAM, it will use smaller GGUF quantizations (Q4_K_M, Q5_K_M) that trade some quality for reduced memory usage.

Vibe CLI Integration

You can point local development tools at your self-hosted vLLM instance. For example, configure ~/.vibe/config.toml to use a local vLLM provider:

6NVIDIA NIM Containers

NVIDIA NIM (NVIDIA Inference Microservices) provides enterprise-grade containerized inference for Mistral Medium 3.5. NIM containers are available on build.nvidia.com and come pre-optimized with TensorRT-LLM for maximum throughput on NVIDIA hardware.

Key advantages of NIM containers:

• Pre-optimized - TensorRT-LLM compilation is done for you, eliminating the lengthy model compilation step
• Enterprise support - backed by NVIDIA AI Enterprise licensing with SLA guarantees
• Kubernetes-native - designed for deployment on Kubernetes with GPU operator integration
• OpenAI-compatible API - standard /v1/chat/completions endpoint for drop-in replacement

NIM is the best choice for enterprises already invested in the NVIDIA ecosystem, particularly those running on DGX systems or NVIDIA-managed Kubernetes clusters. For teams that prefer open-source tooling and more configuration flexibility, vLLM or SGLang remain the better fit.

7Configuration & Optimization

Mistral Medium 3.5 supports several configuration options that let you balance quality, speed, and resource usage for your specific workload.

Reasoning Effort Settings

The model supports a reasoning_effort parameter that controls how much compute the model spends on chain-of-thought reasoning before producing a final answer:

Temperature Tuning

For deterministic outputs (code generation, structured data extraction), use a temperature of 0.1-0.3. For creative tasks and diverse completions, use 0.7-1.0. The default of 0.7 works well for most general-purpose workloads.

Context Length Management

Mistral Medium 3.5 supports long context windows, but longer contexts consume more VRAM for KV cache storage. Use the auto_compact_threshold setting to automatically compact conversation history when it exceeds a specified token count. This prevents OOM errors during long multi-turn conversations while preserving the most relevant context.

Batch Size Tuning

The --max_num_batched_tokens and --max_num_seqs flags in vLLM control how many tokens and sequences are processed in parallel. Higher values increase throughput but require more VRAM. Start with 16384 batched tokens and 128 max sequences, then adjust based on your GPU memory headroom and latency requirements.

8Cost Analysis: Self-Host vs API

Mistral API pricing for Medium 3.5 is $1.50 per 1M input tokens and $7.50 per 1M output tokens. Self-hosting has a fixed monthly GPU cost regardless of token volume. Here's how the economics compare:

9Production Deployment Checklist

Running Mistral Medium 3.5 in production requires more than launching a server. Here are the critical operational patterns to get right before going live:

Monitoring

• Tokens per second (TPS) - track both prefill and decode throughput separately to identify bottlenecks
• Time to first token (TTFT) - critical for interactive applications, should stay under 2 seconds for good UX
• GPU utilization & memory - use nvidia-smi or Prometheus exporters to track per-GPU metrics
• Queue depth - monitor pending request count to detect capacity bottlenecks before they impact latency
• Error rates - track OOM errors, timeout errors, and malformed response rates

Health Checks

Both vLLM and SGLang expose /health endpoints. Configure your load balancer to poll every 10-15 seconds and remove instances that fail three consecutive checks. Dense 128B models can experience GPU memory pressure under heavy load, so health checks catch OOM-related failures early.

Load Balancing

Run multiple vLLM or SGLang instances behind an NGINX or HAProxy load balancer. Use least-connections routing rather than round-robin. LLM requests have highly variable processing times, and least-connections prevents hot-spotting on instances handling long context requests.

Scaling

For cloud deployments, configure auto-scaling based on queue depth or GPU utilization thresholds. Scale up when queue depth exceeds 10 pending requests or GPU utilization stays above 90% for 5+ minutes. Scale down during off-peak hours to reduce costs. Kubernetes with GPU node pools provides the most automated orchestration.

Model Updates

Plan for rolling model updates. When Mistral releases new weights or patches, use blue-green deployment to swap models without downtime. Keep the previous model version available for rollback. Test new weights against your evaluation suite before promoting to production.

10Why Lushbinary for AI Infrastructure

Self-hosting a 128B dense model is a serious infrastructure undertaking. GPU provisioning, multi-GPU networking, precision tuning, load balancing, monitoring, and auto-scaling all require deep infrastructure expertise. At Lushbinary, we handle the full deployment pipeline so your team can focus on building products, not managing GPU clusters.

We've deployed large language models for production workloads across healthcare, fintech, and enterprise SaaS. Whether you need a 4×H100 FP8 setup for cost-efficient serving or an 8×H100 cluster with EAGLE speculative decoding for maximum throughput, we'll architect, deploy, and maintain it.

❓ Frequently Asked Questions

📚 Sources

• HuggingFace - mistralai/Mistral-Medium-3.5-128B Model Weights
• Mistral AI Documentation - API Reference & Model Details
• vLLM Documentation - Deployment & Configuration Guide

Content was rephrased for compliance with licensing restrictions. Hardware recommendations and cost estimates are based on publicly available cloud GPU pricing as of July 2025. Actual costs may vary by provider and region. Always verify on the vendor's website.