Sovereign Open-Source LLMs — from Fine-Tuning to On-Prem GPUs
Choose and size open-source models, run GPUs on Proxmox, OpenStack, Kubernetes and OpenShift, serve models with vLLM, build sovereign RAG and agents, adapt a model through LoRA / QLoRA fine-tuning, then operate and monitor the platform in compliance.
Training objectives
- Choose an open-source model according to use case, language, size and licence.
- Size VRAM and GPU hardware, and choose a quantisation level.
- Run GPUs on Proxmox VE, OpenStack, Kubernetes and OpenShift.
- Serve models in production with vLLM and an OpenAI-compatible API.
- Build sovereign RAG and agents (MCP) on internal data.
- Adapt a model through LoRA / QLoRA fine-tuning and evaluate it.
- Operate, secure and monitor a compliant hybrid AI platform.
Target audience
- MLOps and AI engineers.
- Cloud architects.
- Infrastructure and DevOps engineers.
- Platform teams.
Prerequisites
- Linux, containers and Kubernetes.
- Working knowledge of Python.
- ECI-IA-210 or ECI-IA-301 recommended.
Detailed program
Day 1 — Models and sizing
- Transformers and open-source LLMs: tokenizer, attention, pre-training and instruct models, families of open-weight models, licences and terms of commercial use, reading benchmarks with caution and evaluating on your own cases
- Local inference and formats: Hugging Face, transformers and safetensors, Ollama, llama.cpp and the GGUF format, tokens per second and time to first token, what is realistic on CPU and on GPU. Lab 1: three models running locally, quality and speed comparison
- GPU sizing: estimating VRAM (parameters, precision, cache and headroom), FP16, INT8 and INT4 quantisation, concurrency, context length and throughput, total cost compared with an API. Lab 2: sizing calculator and hardware selection
Day 2 — GPU infrastructure
- GPUs on Proxmox VE and OpenStack: IOMMU and PCI passthrough, vGPU and GPU sharing, NVIDIA drivers and checks, GPU nodes in an OpenStack private cloud. Lab 3: GPU VM with passthrough on Proxmox VE
- GPUs on Kubernetes and OpenShift: NVIDIA GPU Operator, device plugin and containerised drivers, sharing through MIG and time-slicing, quotas, affinities and dedicated nodes, overview of Red Hat OpenShift AI. Lab 4: GPU Operator and a test GPU pod
- Serving models in production: vLLM and its OpenAI-compatible API, Kubernetes deployment, KServe and scaling, model storage on Ceph (RGW S3, CephFS), load testing and tuning. Labs 5 and 6: vLLM on Kubernetes with models stored on Ceph, load test and throughput tuning
Day 3 — RAG, agents and fine-tuning
- Sovereign RAG: ingestion of PDFs and office documents, chunking and metadata, open-source embeddings and vector database, multimodal RAG, reranking, citations and quality evaluation. Labs 7 and 8: RAG on an internal document base, RAG evaluation
- Agents and MCP: tool calling with open-source models, internal MCP servers (ERP, ticketing, monitoring), orchestration and guardrails, least privilege and human validation. Lab 9: tool-using agent through MCP on a local model
- Fine-tuning: when fine-tuning is useful and when RAG is enough, LoRA, QLoRA and PEFT, supervised training, data preparation and quality, before and after evaluation, merging adapters. Labs 10 and 11: QLoRA fine-tuning on a business dataset, evaluating and serving the adapted model
Day 4 — Production, security and compliance
- AI gateway and security: OpenAI-compatible gateway (authentication, quotas, routing), classification and masking of personal data, hybrid routing between local models and an external API, prompt injection, logging and audit. Lab 12: AI gateway with masking and hybrid routing
- Observability and operations: GPU metrics, Prometheus and Grafana, throughput, latency, error rate and cost per request, model updates (versioning, canary, rollback), continuous quality evaluation. Lab 13: AI platform dashboard
- Reference architecture and compliance: sovereign hybrid reference architecture, Law 09-08 and CNDP, Law 2013-450 and ARTCI, sector requirements, capacity plan and roadmap. Labs 14 and 15: complete end-to-end platform, final project
Format and assessment
- On site or live online, 70% hands-on on a dedicated GPU lab; in-house sessions on request
- Delivered in French
- 15 labs, 10 scenarios and 1 graded final project
- Assessment: quiz (10%), validated labs (40%), scenarios (10%), final project (40%)
- ECINTELLIGENCE training attestation awarded from a score of 60%
Materials provided to participants
- ECINTELLIGENCE course manual and lab workbook (PDF)
- VRAM and cost sizing calculator
- Kubernetes and OpenShift manifests: GPU Operator, vLLM, Ceph storage
- Reference code: RAG, MCP agent, QLoRA fine-tuning, AI gateway
- Observability dashboard
- Sovereign hybrid reference architecture
- 10 scenarios with model answers
- Dated state-of-the-art appendix (models, versions, licences)
Next steps
- ECI-IA-301 — Claude for Cloud and DevOps Engineers
- ECI-IA-001 — AI Landscape for Decision-Makers (for your senior management)
- Cloud and infrastructure courses in the catalogue: Proxmox VE, Ceph, OpenShift, OpenStack
- ECINTELLIGENCE consulting: design and deployment of sovereign AI platforms
As models, versions and features evolve quickly, the content is updated before each session. Product names mentioned (PyTorch, OpenShift, Kubernetes, Proxmox, NVIDIA, Hugging Face…) are trademarks of their respective owners.
Certification
At the end of this training, you will receive a certificate of participation issued by squint.
Other training courses that might interest you
AI Landscape for Decision-Makers
Présentiel et distancielA jargon-free day for decision-makers: an overview of predictive, generative and agentic AI, what makes a project succeed, how to …
Claude for Professionals
Présentiel et distancielA hands-on course for every profession: writing effective requests, choosing the right model, making the most of projects, documents, search …
Building with the Claude API and Agents
Présentiel et distancielA developer course: Messages API, streaming, errors and limits, cost control (batch, prompt caching), structured outputs, tools, MCP, Agent SDK, …
Ready to develop your skills?
Join hundreds of professionals who have trusted squint for their skills.
View all our training courses