claudegoodies
Skill

tensorrt-llm

From NousResearch

Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency. Use for production deployment on NVIDIA GPUs (A100/H100), when you need 10-100x faster inference than PyTorch, or for serving models with quantization (FP8/INT4), in-flight batching, and multi-GPU scaling.

Documents TensorRT-LLM setup, inference, and serving for NVIDIA GPU deployment with quantization and multi-GPU parallelism.

Use it when

  • Deploying LLMs on A100/H100/GB200 GPUs for production
  • Serving quantized models (FP8/INT4/FP4) for throughput or memory savings
  • Scaling inference across multiple GPUs or nodes with tensor/pipeline parallelism
  • Using trtllm-serve to expose an OpenAI-compatible chat completions endpoint

Skip it if

  • No NVIDIA GPU available (AMD, CPU, Apple Silicon not supported)
  • Need simple Python-first setup without TensorRT compilation step (use vLLM)
  • Requires CUDA 13.0.0/TensorRT 10.13.2/Python 3.10-3.12, adds environment constraints

Facts

Repository
NousResearch/hermes-agent
Status
Actively maintained
Last commit
Source file
optional-skills/mlops/tensorrt-llm/SKILL.md

Source preview

The instructions Claude Code reads when this skill runs.

# TensorRT-LLM

NVIDIA's open-source library for optimizing LLM inference with state-of-the-art performance on NVIDIA GPUs.

## When to use TensorRT-LLM

**Use TensorRT-LLM when:**
- Deploying on NVIDIA GPUs (A100, H100, GB200)
- Need maximum throughput (24,000+ tokens/sec on Llama 3)
- Require low latency for real-time applications
- Working with quantized models (FP8, INT4, FP4)
- Scaling across multiple GPUs or nodes

**Use vLLM instead when:**
- Need simpler setup and Python-first API
- Want PagedAttention without TensorRT compilation
- Working with AMD GPUs or non-NVIDIA hardware

**Use llama.cpp instead when:**
- Deploying on CPU or Apple Silicon
- Need edge deployment without NVIDIA GPUs
- Want simpler GGUF quantization format

## Quick start

### Installation

```bash
# Docker (recommended)
docker pull nvidia/tensorrt_llm:latest

# pip install
pip install tensorrt_llm==1.2.0rc3

# Requires CUDA 13.0.0, TensorRT 10.13.2, Python 3.10-3.12
```

### Basic inference

```python
from tensorrt_llm import LLM, SamplingParams

# Initialize model
llm = LLM(model="meta-llama/Meta-Llama-3-8B")

# Configure sampling
sampling_params = SamplingParams(
    max_tokens=100,
    temperature=0.7,
    top_p=0.9
)

# Generate
prompts = ["Explain quantum computing"]
outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    print(output.text)
```

### Serving with trtllm-serv
View full source on GitHub →

Other skills