TensorRT

🧠 When to Use TensorRT

Use TensorRT only for inference, not for training or fine-tuning. It provides:

• Lower latency

• Faster throughput

• Reduced memory footprint

---

⚙️ Requirements

To use TensorRT:

• GPU with Tensor Cores (Volta, Turing, Ampere, etc.)

• ONNX model format (you need to export your model to ONNX first)

• Install:

pip install nvidia-pyindex
pip install tensorrt

---

🔥 Hugging Face + TensorRT

You can export HuggingFace models using transformers.onnx:

transformers-cli env  # check installation
transformers onnx --model=codellama/CodeLlama-7B-Instruct-hf --feature=causal-lm ./onnx/

Then optimize it via TensorRT with onnxruntime or trtexec.

---

⚠️ Kaggle Note

Kaggle does not support TensorRT, as it lacks:

• root access for TensorRT driver-level installations

• required NVIDIA runtime permissions

---

✅ Use Locally or on Cloud (AWS/GCP/Colab Pro+ with CUDA support)

Let me know if you want a step-by-step ONNX → TensorRT pipeline.

To run inference with TensorRT, you need to:

---

✅ 1. Convert Model to ONNX

For Hugging Face models (e.g., CodeLlama), convert to ONNX:

transformers onnx --model=codellama/CodeLlama-7B-Instruct-hf --feature=causal-lm onnx-model/

This creates files like: model.onnx inside onnx-model/.

---

✅ 2. Convert ONNX to TensorRT Engine

Use NVIDIA's trtexec (part of TensorRT installation):

trtexec --onnx=onnx-model/model.onnx --saveEngine=model.engine

Add optional flags:

• --fp16 for half-precision

• --int8 for quantized inference (needs calibration)

---

✅ 3. Load and Run Inference with TensorRT

You can write a Python script using TensorRT Python API:

import tensorrt as trt
import pycuda.driver as cuda
import pycuda.autoinit
import numpy as np

TRT_LOGGER = trt.Logger()

def load_engine(engine_path):
    with open(engine_path, "rb") as f, trt.Runtime(TRT_LOGGER) as runtime:
        return runtime.deserialize_cuda_engine(f.read())

engine = load_engine("model.engine")

# Create context
context = engine.create_execution_context()

# Allocate memory for inputs/outputs (example)
input_shape = (1, 128)  # batch_size x seq_len
output_shape = (1, 128, engine.max_batch_size)

# Create input/output buffers
inputs = cuda.mem_alloc(np.zeros(input_shape, dtype=np.float32).nbytes)
outputs = cuda.mem_alloc(np.zeros(output_shape, dtype=np.float32).nbytes)

# Bind and run
bindings = [int(inputs), int(outputs)]
context.execute_v2(bindings=bindings)

---

⚠️ Notes

• You must tokenize inputs separately (e.g., using HuggingFace tokenizer).

• TensorRT does not support dynamic sequence lengths as easily as HF; you must specify max length at engine creation time.

• For complex models like CodeLlama, using ONNXRuntime with TensorRT Execution Provider is easier.

---

✅ Easier Alternative: ONNX Runtime + TensorRT

Install:

pip install onnxruntime-gpu

Inference:

import onnxruntime as ort
import numpy as np

session = ort.InferenceSession("model.onnx", providers=['TensorrtExecutionProvider', 'CUDAExecutionProvider'])

inputs = {"input_ids": np.array([[1, 2, 3, 4]])}  # tokenized
outputs = session.run(None, inputs)

print(outputs)

---