Thank you for sending your enquiry! One of our team members will contact you shortly.
Thank you for sending your booking! One of our team members will contact you shortly.
Course Outline
AI Sovereignty and Local LLM Deployment
- The risks associated with cloud LLMs, including data retention, input usage for training, and foreign jurisdiction issues.
- Overview of the Ollama architecture, covering the model server, registry, and OpenAI-compatible API.
- Comparison of Ollama with alternatives such as vLLM, llama.cpp, and Text Generation Inference.
- Review of model licensing terms for Llama, Mistral, Qwen, and Gemma.
Installation and Hardware Configuration
- Installing Ollama on Linux with support for CUDA and ROCm.
- Implementing CPU-only fallback strategies and AVX/AVX2 optimizations.
- Deploying via Docker and managing persistent volume mappings.
- Configuring multi-GPU setups and optimizing VRAM allocation.
Model Management
- Retrieving models from the Ollama registry using commands like 'ollama pull llama3'.
- Importing GGUF models from HuggingFace and TheBloke.
- Evaluating tradeoffs across quantization levels such as Q4_K_M, Q5_K_M, and Q8_0.
- Managing model switching and understanding concurrent loading limits.
Custom Modelfiles
- Mastering Modelfile syntax, including FROM, PARAMETER, SYSTEM, and TEMPLATE directives.
- Tuning parameters such as temperature, top_p, and repeat_penalty.
- Engineering system prompts to enforce role-specific behaviors.
- Creating and publishing custom models to a local registry.
API Integration
- Utilizing the OpenAI-compatible /v1/chat/completions endpoint.
- Handling streaming responses and enabling JSON mode.
- Integrating with frameworks like LangChain, LlamaIndex, and custom applications.
- Implementing authentication and rate limiting using reverse proxies.
Performance Optimization
- Managing context window sizing and KV cache efficiency.
- Optimizing batch inference and parallel request handling.
- Allocating CPU threads with NUMA awareness.
- Monitoring GPU utilization and managing memory pressure.
Security and Compliance
- Isolating network access for model serving endpoints.
- Implementing input filtering and output moderation pipelines.
- Maintaining audit logs for all prompts and completions.
- Verifying model provenance through hash validation.
Requirements
- Intermediate-level proficiency in Linux administration and container management.
- A high-level understanding of machine learning principles and transformer architectures.
- Practical familiarity with REST APIs and JSON data formats.
Target Audience
- AI engineers and developers seeking to migrate away from cloud-based LLM APIs.
- Organizations with strict data privacy requirements that preclude the use of external cloud models.
- Government and defense teams that require air-gapped, on-premise language model capabilities.
14 Hours
