1. Abstract
HPC-Ops is an open-source, production-grade LLM inference operator library from Tencent's Hunyuan AI Infra team, with the goal of bringing mainstream inference cards (especially NVIDIA Hopper/SM90, such as H20) closer to peak hardware utilization. The project focuses on polishing the SOTA kernel from zero-use CUDA + CuTe/CUTLASS, and provides a relatively clean API for easy integration into inference frameworks such as vLLM and SGLang, and is suitable for multi-precision inference scenarios such as BF16/FP8.
2. Core features
1. Ultimate performance for production: For the in-depth optimization of inference critical paths, the official maximum observation acceleration ratio of Attention / GroupGEMM / FusedMoE (up to 2.22x) is given.
2. Key operator coverage: including Attention (Prefill/Decode, including paged attention), Grouped GEMM, Fused MoE and other inference high-frequency operators.
3. Multi-precision and quantization support: Natively supports BF16 and FP8, and covers different quantization scaling methods (such as block-wise/per-tensor).
4. Easy integration and readability: Emphasize "clean abstraction + customizability", and at the same time make the kernel implementation into a modern CUDA learning sample to lower the threshold for secondary development.
3. Installation
1. Environmental requirements: NVIDIA SM90 architecture GPU; Python 3.8+； Compiler with support for C++17; CUDA Toolkit 12.8+。
- Source code installation (build wheel):
git clone https://github.com/Tencent/hpc-ops.gitcd hpc-opsmake wheelpython3 -m pip install dist/*.whl- 3. Dependency suggestions: Prepare development/test dependencies according to the
requirements-dev.txtof the repository to facilitate running use cases and aligning versions.
4. Typical use cases
1. MoE model inference acceleration: Use FusedMoE / GroupGEMM to reduce GEMM and fusion overhead after expert routing, and improve QPS/QPM.
2. Long context and high concurrency services: Attention (including paged attention) is more sensitive to the throughput and latency of prefill/decode, making it suitable for squeezing GPUs under high load.
3. Secondary integration of inference framework: Replace/extend key operators in vLLM, SGLang, etc. to achieve the engineering path of "local kernel replacement and overall benefit".
4. Kernel R&D and Teaching: The engineering implementation of CuTe/CUTLASS is used as a readable sample for team training and new operator iteration.
5. Ecology and competing products
- Ecological location: HPC-Ops is more like a "high-performance operator base" and is suitable for use in combination with inference frameworks (vLLM, SGLang) rather than a direct replacement for a complete inference engine.
- Competitors/benchmarking: Attention direction common comparison FlashAttention/FlashInfer/TensorRT-LLM; MoE and GEMM directions are commonly compared with TensorRT-LLM, DeepGEMM, etc. The difference between HPC-Ops is the more aggressive microarchitecture optimization and engineering abstraction of specific hardware and production workloads.
6. Limitations and precautions
1. Hardware threshold: SM90 (Hopper) GPU is currently clearly required; The availability and benefits of other architectures need to be verified by themselves.
2. Version coupling: It is sensitive to CUDA versions, compilers, and dependencies, and it is recommended to lock the environment strictly according to the repository instructions.
3. The "maximum observation acceleration ratio" is not equal to stable returns: different batches, sequence lengths, concurrency and operator fusion methods will lead to differences in returns, and you need to benchmark your own workload.
4. Integration cost: When replacing operators with existing inference stacks, we should pay attention to numerical consistency, accuracy strategy (BF16/FP8) and fallback path to avoid online uncontrollable risks.
7. Project address
https://github.com/Tencent/hpc-ops
8. Frequently asked questions
Q: What is HPC-Ops and what is it suitable for?
A: It is a high-performance LLM inference operator library suitable for replacing key operators in frameworks such as vLLM/SGLang to improve throughput and GPU utilization.
Q: What GPU and CUDA versions does HPC-Ops support?
A: NVIDIA SM90 (Hopper) architecture GPUs are officially required, and CUDA Toolkit requires 12.8 or above.
Q: What inference links does the core operator of HPC-Ops cover?
A: Cover inference critical path operators such as Attention (prefill/decode, including paged attention), Grouped GEMM, and Fused MoE.
Q: How to choose the FP8 quantization/scaling method for HPC-Ops?
A: First, run through the interface and test case provided by the repository, then select a scaling strategy such as block-wise or per-tensor based on accuracy targets and performance indicators, and do end-to-end regression.
Q: How do I evaluate the benefits of HPC-Ops on my own model/business?
A: Use the real request distribution (sequence length, batch, concurrency, MoE configuration) as an A/B benchmark, and measure prefill, decode, and end-to-end throughput/latency respectively.
