Get Started
Home
Topics
Search
Library
Research questionHow can sequence parallelism reduce communication costs for long-context LLM training across batch sizes?Long-context LLM training places heavy demands on memory and inter-GPU communication. Uniform sequence-parallel communication can remain inefficient across different batch sizes.
AI
LLM Pretraining & Post-training
Machine Learning
Technology
Latest papersRecent research connected to this question, newest first.BASP: Communication-Efficient Batch-Aware Sequence Parallelism for LLM TrainingThe source evaluates batch-aware sequence parallelism on NVIDIA A100 clusters with Llama and Qwen models, reporting 1.17–1.31× end-to-end training speedups over standard sequence-parallel baselines while preserving accuracy and memory usage.research paper · Sep 2, 2026
Related questions
How should partitioning, placement, and scheduling be coordinated to reduce bubbles in heterogeneous LLM training?How can long-context LLM inference reduce KV-cache memory without losing attention-head-specific information?How can autoregressive LLM decoding generate multiple tokens in parallel at large batch sizes without sacrificing quality?How can batched LLM serving handle request-specific read-write adaptation state without corrupting streams or losing throughput?