自适应训练系统

Install CUDA toolkit (11.8+ or 12.x)

Compile transformer + MoE kernels

cd Src/Main_Scripts/core ./compile_transformer_ops.sh ./compile_cuda_moe.sh

Compile training kernels

cd ../training ./compile_kernels.sh

**Automatic kernel detection:**
Framework automatically detects and loads compiled kernels at runtime. Falls back to PyTorch if kernels unavailable. No code changes required to use CUDA acceleration.

**Automatic JIT rebuild for current hardware:**
- If kernel `.so` files are missing or compiled for the wrong SM target, runtime wrappers trigger a rebuild automatically.
- Target architecture resolution order: `CUDA_TARGET_SM`  `TORCH_CUDA_ARCH_LIST`  detected GPU compute capability  fallback `sm_75`.
- Set `CUDA_TARGET_SM` when you need deterministic builds across machines.

**Supported architectures:**
- **sm_75:** Turing (T4, RTX 2080)
- **sm_80:** Ampere (A100, RTX 3090)
- **sm_86:** Ampere (RTX 3060/3070/3080)
- **sm_89:** Ada Lovelace (RTX 4090)
- **sm_90:** Hopper (H100, H200)

**Performance monitoring:**

Local Installation

Requirements: - Python 3.8+ (3.10+ recommended) - PyTorch 2.0+ (2.2+ recommended) - CUDA 11.8+ (for GPU with acceleration) or CPU - CUDA Toolkit with nvcc (for compiling custom kernels) - RAM: 16GB minimum, 32GB+ recommended - Disk: 50GB+ for dependencies, datasets, checkpoints

Installation: ```bash git clone https://github.com/matn23/AdaptiveTrainingSystem cd AdaptiveTrainingSystem pip install -r requirements.txt

Compile CUDA kernels (optional but recommended for 3-5x speedup)

cd Src/Main_Scripts/core ./compile_transformer_ops.sh ./compile_cuda_moe.sh cd ../training ./compile_kernels.sh

Recompile kernels after update

cd Src/Main_Scripts/core ./compile_transformer_ops.sh ./compile_cuda_moe.sh cd ../training ./compile_kernels.sh

Compilation and Usage

Compilation requirements: ```bash

Demo

Optional: force target architectures (comma or semicolon separated)

Model Configuration Presets

Pre-configured architecture presets for training models from scratch, spanning 500K to 300B parameters. These are configuration templates, not pre-trained models.

Each preset specifies architecture dimensions, MoE/MoD parameters, hardware targets, and expected performance with CUDA acceleration for initializing and training new models.

[!IMPORTANT] Performance Disclaimer: All benchmarks and throughput estimates provided are either verified on an NVIDIA T4 GPU or calculated as theoretical projections. Results on other hardware or larger scales are untested and listed for architectural reference only. Performance will vary based on your specific environment.

Memory estimates: Include model weights, optimizer states (Adam: 8 bytes/param), gradients, and activation memory at batch_size=1, mixed precision training. Actual memory scales with batch size and sequence length.

<p align="center"> <img src="assets/mem_usage_graph.png" width="800"> </p> Preset debug_200m not included in this graph

Throughput estimates: With CUDA acceleration enabled at batch_size=1, sequence_length=2048, mixed precision with gradient checkpointing. CUDA speedup column shows combined acceleration from all custom kernels vs. pure PyTorch.

Configuration selection:

• Development/testing: debug for pipeline validation, debug_200m for architecture testing
• Research: b1 for prototyping on consumer hardware with CUDA acceleration
• Production fine-tuning: b7 for quality/efficiency balance with significant CUDA speedup
• Large-scale pre-training: b30+ for maximum model capacity
• Extreme scale: b100+ requires cluster infrastructure and distributed expertise

Important:

These presets define untrained model architectures. Training starts from random initialization following standard practices (Xavier/Kaiming initialization for weights, zero initialization for biases). The framework does not provide pre-trained checkpoints.

Customization:

All presets are starting points. Architecture dimensions can be modified: hidden_size must be divisible by num_heads. Intermediate_size typically 8/3 hidden_size rounded to nearest 256 for optimal CUDA performance. Max_position_embeddings determines context window. Num_experts and moe_top_k can be adjusted independently. MoD capacity_factor controls compute/quality tradeoff. CUDA kernels automatically adapt to configuration changes.

---

Precision Configuration Parameters

Training precision: Format used during forward pass, backward pass, and gradient computation Inference precision: Format used during validation and evaluation Master precision: Format for optimizer's master parameter copy (typically FP32 in mixed precision) CUDA kernel precision: Automatic selection based on training precision

Separate training/inference precision: Common pattern: Train in mixed_bf16 for speed with CUDA acceleration, evaluate in fp32 for precise metrics. Or train in mixed_fp16 with CUDA kernels, deploy in int8 for inference.

Loss scaling parameters (FP16 only): - init_scale: Initial loss scaling factor (default: 2^16) - scale_factor: Multiplier for scale adjustment (default: 2.0) - scale_window: Steps without overflow before increasing scale (default: 2000) - min_scale: Minimum scale factor (default: 1.0)

Dynamic loss scaling adjusts automatically: scale increases every scale_window steps without overflow, decreases on overflow detection (NaN/Inf gradients). CUDA kernels maintain numerical stability with loss scaling. Most users do not need to modify these parameters.

---

Set in config:

API Reference

Supported Frameworks & Integrations

Quantization & Inference: - AutoGPTQ: 4-bit quantization support for efficient inference and fine-tuning on consumer hardware. - Optimum Quanto: Dynamic quantization support (8-bit/4-bit) for flexible deployment. - OpenAI Triton: High-performance FP8 kernels for H100+ architectures.

Model Compatibility: - DeepSeek Config Adapter: Auto-convert training configurations to DeepSeek-compatible formats. - HuggingFace Interop: Seamless integration with transformers for dataset loading and tokenization.