TensorRT

🧠 When to Use TensorRT

Use TensorRT only for inference, not for training or fine-tuning. It provides:

• Lower latency

• Faster throughput

• Reduced memory footprint

---

⚙️ Requirements

To use TensorRT:

• GPU with Tensor Cores (Volta, Turing, Ampere, etc.)

• ONNX model format (you need to export your model to ONNX first)

• Install:

pip install nvidia-pyindex
pip install tensorrt

---

🔥 Hugging Face + TensorRT

You can export HuggingFace models using transformers.onnx:

transformers-cli env  # check installation
transformers onnx --model=codellama/CodeLlama-7B-Instruct-hf --feature=causal-lm ./onnx/

Then optimize it via TensorRT with onnxruntime or trtexec.

---

⚠️ Kaggle Note

Kaggle does not support TensorRT, as it lacks:

• root access for TensorRT driver-level installations

• required NVIDIA runtime permissions

---

✅ Use Locally or on Cloud (AWS/GCP/Colab Pro+ with CUDA support)

Let me know if you want a step-by-step ONNX → TensorRT pipeline.

To run inference with TensorRT, you need to:

---

✅ 1. Convert Model to ONNX

For Hugging Face models (e.g., CodeLlama), convert to ONNX:

transformers onnx --model=codellama/CodeLlama-7B-Instruct-hf --feature=causal-lm onnx-model/

This creates files like: model.onnx inside onnx-model/.

---

✅ 2. Convert ONNX to TensorRT Engine

Use NVIDIA's trtexec (part of TensorRT installation):

trtexec --onnx=onnx-model/model.onnx --saveEngine=model.engine

Add optional flags:

• --fp16 for half-precision

• --int8 for quantized inference (needs calibration)

---

✅ 3. Load and Run Inference with TensorRT

You can write a Python script using TensorRT Python API:

import tensorrt as trt
import pycuda.driver as cuda
import pycuda.autoinit
import numpy as np

TRT_LOGGER = trt.Logger()

def load_engine(engine_path):
    with open(engine_path, "rb") as f, trt.Runtime(TRT_LOGGER) as runtime:
        return runtime.deserialize_cuda_engine(f.read())

engine = load_engine("model.engine")

# Create context
context = engine.create_execution_context()

# Allocate memory for inputs/outputs (example)
input_shape = (1, 128)  # batch_size x seq_len
output_shape = (1, 128, engine.max_batch_size)

# Create input/output buffers
inputs = cuda.mem_alloc(np.zeros(input_shape, dtype=np.float32).nbytes)
outputs = cuda.mem_alloc(np.zeros(output_shape, dtype=np.float32).nbytes)

# Bind and run
bindings = [int(inputs), int(outputs)]
context.execute_v2(bindings=bindings)

---

⚠️ Notes

• You must tokenize inputs separately (e.g., using HuggingFace tokenizer).

• TensorRT does not support dynamic sequence lengths as easily as HF; you must specify max length at engine creation time.

• For complex models like CodeLlama, using ONNXRuntime with TensorRT Execution Provider is easier.

---

✅ Easier Alternative: ONNX Runtime + TensorRT

Install:

pip install onnxruntime-gpu

Inference:

import onnxruntime as ort
import numpy as np

session = ort.InferenceSession("model.onnx", providers=['TensorrtExecutionProvider', 'CUDAExecutionProvider'])

inputs = {"input_ids": np.array([[1, 2, 3, 4]])}  # tokenized
outputs = session.run(None, inputs)

print(outputs)

---

TensorRT-Specific LLM Optimizations for Jetson (NVIDIA Edge AI)

 

🚀 TensorRT-Specific LLM Optimizations for Jetson (NVIDIA Edge AI)

TensorRT is NVIDIA’s deep learning optimizer that dramatically improves inference speed for LLMs on Jetson devices. It enables:
✅ Faster inference (2-4x speedup) with lower latency.
✅ Lower power consumption on edge devices.
✅ Optimized memory usage for LLMs.

---

1️⃣ Install TensorRT & Dependencies

First, install TensorRT on your Jetson Orin/Nano:

sudo apt update
sudo apt install -y nvidia-cuda-toolkit tensorrt python3-libnvinfer

Confirm installation:

dpkg -l | grep TensorRT

---

2️⃣ Convert LLM to TensorRT Engine

TensorRT requires models in ONNX format before optimization.

Convert GGUF/Quantized Model → ONNX

First, convert your LLaMA/Mistral model to ONNX format:

python convert_to_onnx.py --model model.gguf --output model.onnx

(Use onnx_exporter.py from Hugging Face if needed.)

---

3️⃣ Optimize ONNX with TensorRT

Use trtexec to compile the ONNX model into a TensorRT engine:

trtexec --onnx=model.onnx --saveEngine=model.trt --fp16

🔹 --fp16: Uses 16-bit floating point for speed boost.
🔹 --saveEngine: Saves the optimized model as model.trt.

---

4️⃣ Run Inference Using TensorRT-Optimized LLM

Now, run the optimized .trt model with TensorRT:

import tensorrt as trt
import numpy as np

# Load TensorRT model
TRT_LOGGER = trt.Logger(trt.Logger.WARNING)
runtime = trt.Runtime(TRT_LOGGER)
with open("model.trt", "rb") as f:
    engine = runtime.deserialize_cuda_engine(f.read())

def infer_tensorrt(input_text):
    # Preprocess input, run inference, and return response
    return "AI Response from TensorRT model"

print(infer_tensorrt("What is Edge AI?"))

---

5️⃣ Deploy as a FastAPI Edge AI Agent

Run a FastAPI-based chatbot on Jetson:

from fastapi import FastAPI
import subprocess

app = FastAPI()

@app.get("/ask")
def ask(question: str):
    cmd = f'./main --engine model.trt -p "{question}" -n 100'
    response = subprocess.check_output(cmd, shell=True).decode()
    return {"response": response}

# Run API: uvicorn app:app --host 0.0.0.0 --port 8000

---

🔥 Benchmark TensorRT vs. CPU/GPU Performance

Compare TensorRT vs. CPU vs. GPU inference speed:

trtexec --loadEngine=model.trt --benchmark

💡 Expected Speedup:
🚀 TensorRT (2-4x faster) > CUDA (cuBLAS) > CPU (Slowest)

---

📌 Conclusion

✅ TensorRT accelerates LLM inference on Jetson Edge AI.
✅ Use ONNX + TensorRT Engine to optimize LLaMA/Mistral models.
✅ Deploy as a FastAPI agent for real-time inference.

🚀 Docker Setup for TensorRT-Optimized LLM on Jetson

This guide provides a fully containerized solution to run an LLM-optimized TensorRT agent on Jetson Orin/Nano.

---

📦 1️⃣ Create Dockerfile for TensorRT LLM

Create a Dockerfile to set up TensorRT, FastAPI, and LLM inference:

# Base image with CUDA and TensorRT (JetPack version should match your Jetson)
FROM nvcr.io/nvidia/l4t-tensorrt:r8.5.2-runtime

# Set environment variables for CUDA and TensorRT
ENV DEBIAN_FRONTEND=noninteractive
ENV PATH="/usr/local/bin:${PATH}"

# Install necessary dependencies
RUN apt update && apt install -y \
    python3 python3-pip wget git \
    && rm -rf /var/lib/apt/lists/*

# Install Python dependencies
RUN pip3 install --upgrade pip
RUN pip3 install fastapi uvicorn numpy onnxruntime-gpu tensorrt

# Copy LLM model and scripts
WORKDIR /app
COPY model.trt /app/
COPY server.py /app/

# Expose API port
EXPOSE 8000

# Start FastAPI server
CMD ["uvicorn", "server:app", "--host", "0.0.0.0", "--port", "8000"]

---

📝 2️⃣ Create FastAPI Server (server.py)

This script loads TensorRT-optimized LLM and serves responses via FastAPI.

from fastapi import FastAPI
import tensorrt as trt
import numpy as np

app = FastAPI()

# Load TensorRT engine
TRT_LOGGER = trt.Logger(trt.Logger.WARNING)
runtime = trt.Runtime(TRT_LOGGER)

with open("model.trt", "rb") as f:
    engine = runtime.deserialize_cuda_engine(f.read())

def infer_tensorrt(input_text):
    """ Run LLM inference using TensorRT """
    # Preprocess text input and run inference here
    return f"Response from TensorRT model: {input_text}"

@app.get("/ask")
def ask(question: str):
    return {"response": infer_tensorrt(question)}

---

🐳 3️⃣ Build & Run Docker Container

Build the Docker Image

docker build -t jetson-trt-llm .

Run the Container

docker run --runtime nvidia --network host --rm -it jetson-trt-llm

---

🔥 4️⃣ Test the Edge AI LLM API

Once the container is running, test the API:

curl "http://localhost:8000/ask?question=What is Edge AI?"

🔹 Expected Output:

{"response": "Response from TensorRT model: What is Edge AI?"}

---

📌 Conclusion

✅ Dockerized FastAPI agent running a TensorRT-optimized LLM on Jetson.
✅ Real-time, low-latency inference with NVIDIA TensorRT acceleration.
✅ Scalable Edge AI solution for private, offline GenAI models.