Production Examples

# Production Examples Real-world production deployments using Megatron-Core and Megatron-LM. ## Meta LLaMA 3 ### Overview - **Models**: 8B, 70B, 405B parameters - **Training Infrastructure**: Two 24,000 H100 GPU clusters - **Total Investment**: Massive scale, months of training - **Training Data**: 15 trillion tokens for 405B model - **Status**: Production deployed (llama.meta.com) ### LLaMA 3.1 405B - Largest Open Model **Architecture**: ```yaml Parameters: 405 billion Layers: 126 Hidden size: 16384 Attention heads: 128 Query groups: 8 (GQA) FFN size: 53248 Vocabulary: 128,256 tokens Max context: 128K tokens (supports up to) Position encoding: RoPE Activation: SwiGLU Normalization: RMSNorm ``` **Training Configuration**: ```bash # 1024 H100 GPUs (128 nodes × 8 GPUs) Tensor Parallel (TP): 8 # Within node Pipeline Parallel (PP): 8 # Across nodes Context Parallel (CP): 2 # For long sequences Data Parallel (DP): 8 # Remaining dimension Total GPUs: 8 × 8 × 2 × 8 = 1024 Effective batch size: 2048 Micro-batch per GPU: 1 Sequence length: 4096 tokens ``` **Performance Metrics**: - **Sustained throughput**: 400 TFlops/GPU - **MFU**: ~46% on H100 - **Uptime**: 95%+ over months - **Efficiency improvement**: 3× vs LLaMA 2 training **Training Duration**: - 15 trillion tokens total - ~54 days on 16,384 H100 GPUs - Or ~6 months on 1,024 H100 GPUs **Key Optimizations Used**: ```bash --use-mcore-models --transformer-impl transformer_engine --sequence-parallel --context-parallel-size 2 --use-distributed-optimizer --overlap-grad-reduce --overlap-param-gather --use-flash-attn-v2 --bf16 ``` **Production Serving**: - Deployed on llama.meta.com - Available via API and download - Used in Meta products (Instagram, Facebook, WhatsApp) ### LLaMA 3 70B **Training Configuration**: ```bash # 64 H100 GPUs (8 nodes × 8 GPUs) TP=4, PP=4, CP=2, DP=2 torchrun --nproc_per_node=8 --nnodes=8 pretrain_gpt.py --num-layers 80 --hidden-size 8192 --num-attention-heads 64 --num-query-groups 8 --seq-length 4096 --micro-batch-size 1 --global-batch-size 1024 --tensor-model-parallel-size 4 --pipeline-model-parallel-size 4 --context-parallel-size 2 --bf16 --use-mcore-models ``` **Memory per GPU**: - Model parameters: 140GB / 4 (TP) / 4 (PP) = 8.75GB - Optimizer states: ~17.5GB - Activations: ~3GB - **Total**: ~30GB per H100 (fits in 80GB) ## NVIDIA Nemotron-4 340B ### Overview - **Organization**: NVIDIA - **Parameters**: 340 billion - **Framework**: NeMo (built on Megatron-Core) - **Purpose**: Enterprise AI foundation model - **Status**: Commercial deployment **Key Features**: - Mixture of Experts architecture - Optimized for enterprise use cases - NeMo framework integration - Production-ready deployment **Architecture**: ```yaml Type: Mixture of Experts (MoE) Total parameters: 340B Active parameters per token: ~40B Experts: 8 Router: Top-2 Context length: 4096 ``` **Training Infrastructure**: - NVIDIA DGX H100 systems - Megatron-Core + NeMo - Multi-node training - Enterprise-grade fault tolerance **Production Features**: - NeMo Guardrails integration - Enterprise support - Customization options - On-premise deployment available ## Microsoft & NVIDIA Megatron-Turing NLG 530B ### Overview - **Organization**: Microsoft + NVIDIA collaboration - **Parameters**: 530 billion (largest dense model when released) - **Year**: 2021 - **Framework**: DeepSpeed ZeRO-3 + Megatron tensor/pipeline parallelism - **Hardware**: 560 NVIDIA A100 80GB GPUs **Architecture**: ```yaml Parameters: 530 billion Layers: 105 Hidden size: 20480 Attention heads: 128 Vocabulary: 51,200 tokens Sequence length: 2048 ``` **Training Configuration**: ```bash # 560 A100 80GB GPUs Tensor Parallel: 8 Pipeline Parallel: 35 Data Parallel: 2 Total: 8 × 35 × 2 = 560 DeepSpeed ZeRO Stage 3: - Full parameter sharding - Gradient sharding - Optimizer state sharding ``` **Innovations**: - First to combine DeepSpeed ZeRO-3 with Megatron parallelism - Demonstrated training at 500B+ scale - Proved viability of extreme parallelism **Performance**: - Trained on 339 billion tokens - Multiple months of training - Achieved state-of-the-art results in 2021 ## BigScience BLOOM 176B ### Overview - **Organization**: BigScience (1000+ researchers) - **Parameters**: 176 billion - **Year**: 2022 - **Framework**: Megatron-DeepSpeed - **Hardware**: 384 NVIDIA A100 80GB GPUs - **Training Duration**: 46 days **Architecture**: ```yaml Parameters: 176 billion Layers: 70 Hidden size: 14336 Attention heads: 112 Vocabulary: 250,680 tokens (multilingual) Sequence length: 2048 Languages: 46 natural languages + 13 programming languages ``` **Training Configuration**: ```bash # 384 A100 80GB GPUs on Jean Zay supercomputer Tensor Parallel: 4 Pipeline Parallel: 12 Data Parallel: 8 Total: 4 × 12 × 8 = 384 Global batch size: 2048 Micro-batch size: 4 Learning rate: 6e-5 Optimizer: Adam (β1=0.9, β2=0.95) ``` **Training Data**: - 366 billion tokens (1.6TB) - ROOTS corpus (custom multilingual dataset) - 46 natural languages - 13 programming languages **Key Achievements**: - Largest multilingual open-source model at release - Trained on public supercomputer (Jean Zay) - Fully documented training process - Open-source model and training code **Public Impact**: - Downloaded 100,000+ times - Used in hundreds of research papers - Enabled multilingual AI research - Demonstrated open science at scale ## DeepSeek-V3 ### Overview - **Organization**: DeepSeek - **Parameters**: 671 billion total, 37B active per token - **Type**: Mixture of Experts (MoE) - **Year**: 2024-2025 - **Framework**: Megatron-Core **Architecture**: ```yaml Type: Mixture of Experts Total parameters: 671B Active parameters per token: 37B Layers: 61 Hidden size: 7168 Attention heads: 128 Query groups: 16 Experts: 256 (massive MoE) Router top-k: 8 (Multi-head Latent Attention) Shared expert size: 18432 ``` **Training Configuration**: ```bash # 1024 H100 GPUs Tensor Parallel (TP): 2 Pipeline Parallel (PP): 16 Expert Parallel (EP): 64 Context Parallel (CP): 1 Total: 2 × 16 × 64 = 2048 slots # Uses overlapping parallelism Global batch size: 4096 Sequence length: 4096 Training tokens: 14.8 trillion ``` **Innovations**: - Multi-head Latent Attention (MLA) router - Shared experts + routed experts - Ultra-large expert count (256) - Advanced load balancing **Performance**: - Competitive with GPT-4 - 37B active params rivals 70B+ dense models - Efficient inference (only 37B active) ## OpenAI GPT-3 175B (2020) ### Overview - **Organization**: OpenAI - **Parameters**: 175 billion - **Year**: 2020 - **Framework**: Megatron-inspired custom implementation - **Hardware**: Thousands of NVIDIA V100 GPUs **Architecture**: ```yaml Parameters: 175 billion Layers: 96 Hidden size: 12288 Attention heads: 96 FFN size: 49152 Vocabulary: 50,257 tokens (GPT-2 BPE) Sequence length: 2048 Context window: 2048 tokens ``` **Training Configuration**: ```bash # Estimated configuration Tensor Parallel: 4-8 Pipeline Parallel: 8-16 Data Parallel: Remaining GPUs Global batch size: 1536 Learning rate: 6e-5 Training tokens: 300 billion ``` **Training Compute**: - 3.14 × 10^23 FLOPs - Equivalent to ~355 GPU-years on V100 - Estimated cost: $4-12 million **Impact**: - Launched modern era of large language models - Demonstrated few-shot learning - Foundation for ChatGPT ## Stability AI StableLM ### Overview - **Organization**: Stability AI - **Framework**: GPT-NeoX (Megatron + DeepSpeed) - **Hardware**: Training on supercomputers - **Status**: Open-source **Models**: - StableLM-Base-Alpha: 3B, 7B - StableLM-Tuned-Alpha: Fine-tuned versions - StableCode: Code-specialized **Training Configuration**: ```yaml Framework: GPT-NeoX Parallelism: Megatron TP/PP + DeepSpeed ZeRO GPUs: A100 clusters Training data: 1.5 trillion tokens (The Pile) ``` **Key Features**: - Fully open-source (Apache 2.0) - GPT-NeoX framework - Trained on The Pile dataset - Multiple model sizes ## Common Production Patterns ### Fault Tolerance **Checkpoint Strategy**: ```bash --save-interval 500 # Save every 500 iterations --save /checkpoints/model_name # Checkpoint directory --load /checkpoints/model_name # Auto-resume from latest ``` **Monitoring**: ```python # Check in progress.txt Job throughput: 45.2 TFLOPs/GPU Cumulative throughput: 44.8 TFLOPs/GPU Memory usage: 68.2 GB / 80 GB Loss: 2.143 ``` ### Data Pipeline **Preprocessing**: ```bash python tools/preprocess_data.py --input data.jsonl --output-prefix /data/processed --vocab-file vocab.json --merge-file merges.txt --tokenizer-type GPT2BPETokenizer --append-eod --workers 64 ``` **Training with Preprocessed Data**: ```bash --data-path /data/processed_text_document --split 969,30,1 # Train/valid/test split ``` ### Monitoring & Logging **Key Metrics to Track**: ```bash # Training metrics - Loss (should steadily decrease) - Learning rate (follows schedule) - Gradient norm (watch for spikes) - Throughput (TFlops/GPU) - MFU percentage # System metrics - GPU utilization (>90%) - Memory usage (<95% of capacity) - Network bandwidth (saturated for TP) - Data loading time (should be minimal) ``` **Production Monitoring Tools**: - TensorBoard for loss curves - Weights & Biases for experiment tracking - Prometheus + Grafana for system metrics - Custom scripts for MFU calculation ### Multi-Datacenter Training **Challenges**: - Higher latency between datacenters - Network bandwidth limitations - Fault isolation **Solutions**: ```bash # Keep TP within datacenter --tensor-model-parallel-size 8 # Single node only # Use PP across datacenters --pipeline-model-parallel-size 16 # Across sites # Data parallel across everything # Automatic from remaining GPUs ``` ## Lessons from Production 1. **Fault Tolerance is Critical** - Save checkpoints frequently (every 500-1000 steps) - Test checkpoint recovery regularly - Monitor for GPU failures 2. **Data Quality Matters More Than Quantity** - LLaMA 3: Carefully curated 15T tokens - Better than naive web scraping - Investment in data preprocessing pays off 3. **Parallelism Strategy Evolves with Scale** - <70B: TP + DP sufficient - 70-175B: Add PP - 175B+: 3D or 4D parallelism required - MoE: Add EP dimension 4. **Hardware Matters** - H100 vs A100: 2× speedup from better hardware - NVLink topology affects TP efficiency - InfiniBand essential for multi-node 5. **Monitoring is Essential** - Track MFU to catch performance issues - Monitor loss for training health - Watch memory usage to avoid OOM - Log everything for debugging ## References - Meta LLaMA 3 technical report - NVIDIA Nemotron blog posts - Microsoft Megatron-Turing NLG paper - BigScience BLOOM documentation - DeepSeek-V3 technical report