Performance Per Dollar
A 70-billion parameter model at full FP16 precision requires 140 GB of GPU memory and generates tokens at a rate determined by memory bandwidth. Optimization techniques reduce memory requirements, increase throughput, and improve quality on your specific tasks, often simultaneously. The goal is not to run the biggest model possible but to run the most capable model for your use case at the lowest infrastructure cost.
Quantization
Compress model weights from FP16 to INT8, INT4, or lower precision. Reduces memory by 2-4x with minimal quality loss for most enterprise tasks. Enables running larger models on smaller GPUs.
Pruning
Remove redundant weights and attention heads that contribute little to output quality. Structured pruning reduces both memory and compute requirements. Unstructured pruning with sparse kernels improves throughput.
Fine-Tuning
Adapt a general-purpose model to your domain with LoRA, QLoRA, or full fine-tuning. A fine-tuned 7B model often outperforms a general 70B model on domain-specific tasks at 1/10th the compute cost.
Benchmarking
Measure actual quality and throughput on your specific tasks, not generic benchmarks. Automated evaluation pipelines compare model variants against ground truth to find the optimal configuration.
Optimization Pipeline
Baseline
Benchmark unoptimized model
Quantize
Compress weights to target precision
Fine-Tune
Adapt to domain-specific tasks
Validate
Quality and throughput verification
Baseline
Benchmark unoptimized model
Quantize
Compress weights to target precision
Fine-Tune
Adapt to domain-specific tasks
Validate
Quality and throughput verification
Model Optimization Pipeline
Quantization Methods
Different quantization approaches offer different tradeoffs between compression ratio, inference speed, and quality preservation. We select the method based on your accuracy requirements and hardware constraints.
GPTQ (GPU-optimized). Post-training quantization using calibration data to minimize quantization error. 4-bit GPTQ reduces a 70B model from 140 GB to 35 GB while preserving 98-99% of full-precision quality on typical enterprise tasks. Fast inference on NVIDIA GPUs with ExLlama or AutoGPTQ kernels.
AWQ (Activation-Aware). Identifies and preserves the most important weight channels during quantization. Consistently outperforms GPTQ on reasoning and code generation tasks at the same bit width. Supported by vLLM and TensorRT-LLM for production serving.
GGUF (CPU+GPU hybrid). Supports mixed quantization levels per layer. Enables CPU offloading when GPU VRAM is insufficient. Q4_K_M and Q5_K_M presets provide the best quality-to-compression ratio for llama.cpp inference. Ideal for edge deployments and cost-constrained environments.
FP8 (H100 native). Hopper Transformer Engine on H100 GPUs runs FP8 inference natively with hardware-accelerated dequantization. 2x throughput over FP16 with negligible quality loss. The simplest optimization if you are deploying on H100 hardware.
Fine-Tuning for Domain Performance
General-purpose models are trained on internet-scale data. Your business has domain-specific vocabulary, formats, and evaluation criteria that general models handle imperfectly. Fine-tuning bridges this gap.
LoRA and QLoRA. Low-Rank Adaptation fine-tunes a small number of adapter weights (0.1-1% of total parameters) while keeping the base model frozen. QLoRA combines LoRA with 4-bit quantization to fine-tune a 70B model on a single GPU. Training takes hours, not days.
Domain adaptation datasets. We help you curate training data from your existing documents, support tickets, and internal knowledge bases. A few hundred high-quality examples dramatically improve performance on your specific extraction, classification, and generation tasks.
Who This Is For
Model optimization is relevant for any organization deploying AI on private infrastructure. Whether you are trying to fit a larger model on existing hardware, reduce inference costs, or improve quality on domain-specific tasks, optimization is the lever that delivers the most ROI per engineering hour invested.
Contact us at ben@oakenai.tech