Thank you for sending your enquiry! One of our team members will contact you shortly.
Thank you for sending your booking! One of our team members will contact you shortly.
Duration 14 hours
Course Outline
Deployment Strategies
- Trade-offs between hosted APIs and self-hosting
- Patterns for on-premises, edge, and air-gapped environments
- Single model vs. model portfolio approaches
- Cost-per-decision analysis
Hardware Sizing and Capacity
- Memory footprint relative to parameter count
- CPU-only vs. GPU acceleration
- Planning for concurrency and throughput
- Reserving headroom for increasing task lengths
Serving Stacks
- Options including vLLM and SGLang
- Execution via llama.cpp and GGUF
- Using Ollama for local development
- Selection criteria based on decision workload characteristics
Quantization While Preserving Calibration
- Precision formats and memory optimization
- Assessing quality and calibration degradation
- Per-tensor and importance-aware quantization variants
- Conducting acceptance tests post-quantization
Latency Optimization
- Median vs. tail latency considerations
- Latency impact of context length
- Utilizing warm starts and model caching
- Batching multiple decision queries
Edge and Constrained Environments
- Running small decision models on CPU
- Deployment on Apple Silicon and embedded systems
- Offline operation and update protocols
- Distributing weights in air-gapped settings
Containerization and Scaling
- Designing images for model serving
- Managing volumes and caches
- Horizontal scaling and traffic routing
- Setting resource limits and sharing GPUs
Observability
- Tracking request, latency, and error metrics
- Monitoring confidence distributions
- Recording decision outcomes and escalations
- Setting alerts for calibration drift
Versioning and Deployment
- Versioning models and adapters
- Implementing shadow and canary releases
- Pinning model files for audit compliance
- Executing rollback procedures
Security and Multi-Tenancy
- Implementing authentication and rate limiting
- Isolating tenant prompts and logs
- Managing network isolation and egress control
- Logging decisions for audit purposes
Requirements
- Experience with Linux administration and containerization.
- Knowledge of model inference servers and GPU management.
- Fundamental understanding of REST APIs, load balancing, and monitoring.
Target Audience
- MLOps and platform engineers managing inference infrastructure
- SRE teams supporting AI services
- AI engineers deploying decision models on-premises
- Architects designing air-gapped or regulated AI deployments
