ML self-service pipeline that abstracts Kubernetes complexity

 

To successfully bridge the gap between machine learning engineering and cluster operations, you need to build a self-service pipeline that abstracts Kubernetes complexity. The goal is to let ML practitioners provision GPUs and scale workloads using simple configurations, while Operations maintains guardrails around costs and resources.

Here is the operational blueprint to build, scale, and operationalize your ML-focused Kubernetes platform.

1. Provision the ML Development Cluster

Setting up a dedicated ML development cluster requires integrating hardware acceleration into the Kubernetes control plane from day one.

• Select the Infrastructure: Use cloud-managed services (AWS EKS, GCP GKE, or Azure AKS) for stable control planes and automated node OS provisioning.
• Install GPU Drivers: Deploy the NVIDIA GPU Operator via Helm. This automatically manages the NVIDIA driver, container toolkit, and device plug-in across all GPU nodes.
• Configure Node Pools: Create distinct, labeled node groups.
• CPU Pool: For the cluster control plane, core services, and light tooling (e.g., ).
• GPU Pool: For training and inference (e.g., or ).
• Apply Taints and Labels: Prevent non-ML workloads from scheduling on expensive GPU hardware.
• Add taint:
• Add label: 

2. Implement Automated GPU Provisioning

ML workloads have highly variable resource demands. Standard scaling is too slow; you need just-in-time provisioning.

• Deploy Karpenter: Replace the standard Cluster Autoscaler with Karpenter. It evaluates pending pods and launches the optimal EC2/Compute instance type in seconds.
• Define NodePools: Configure Karpenter to recognize ML constraints. Set limits on maximum cloud spend and allowed instance types.
• Enable Multi-Instance GPU (MIG): For dev workloads, split large physical GPUs (like A100s) into smaller, isolated instances. This allows multiple practitioners to share a single card for light experimentation.
• Enforce Resource Quotas: Implement Kubernetes and per namespace. This ensures a single user cannot accidentally spin up 50 GPU instances and drain the budget.

3. Enable Dynamic Autoscaling for ML Workloads

ML workloads require scaling based on real-time metrics like GPU utilization or API request queues, rather than standard CPU/Memory metrics.

• Deploy KEDA (Kubernetes Event-driven Autoscaling): KEDA extends the Horizontal Pod Autoscaler (HPA) to scale workloads based on external metrics.
• Inference Autoscaling: Configure KEDA to monitor your ingress controller or a message broker (like RabbitMQ/Kafka). Scale the model serving pods based on concurrent request volume or queue length.
• Training Autoscaling: For distributed training, use the Training Operator (Kubeflow). It orchestrates PyTorch or TensorFlow jobs, dynamically scaling worker pods up for the job and tearing them down immediately upon completion.
• Scale-to-Zero: Configure KEDA to scale inference models down to 0 replicas during off-hours if no requests are received, eliminating idle GPU costs.

4. Create the Self-Service Interface

ML practitioners should not write raw Kubernetes YAML. Abstract the infrastructure into familiar interfaces.

• Standardize with Helm/Kustomize: Create internal charts for standard ML patterns (e.g., "Jupyter Notebook Dev Environment" or "Triton Model Server Deployment").
• Expose User Interfaces:
• Kubeflow / OpenDataHub: Provide a centralized dashboard where users can click a button to launch a Jupyter, VS Code, or RStudio instance backed by a requested number of GPUs.
• Internal CLI/Internal Developer Portal (Backstage): Allow practitioners to deploy a model to production by filling out a simple form specifying only the model registry path and required GPU memory.
• Simplify the Pod Spec: Users should only need to add a simple block and a toleration to their configuration:

5. Establish Production Guardrails and Observability

Moving from dev to GPU-backed production requires strict monitoring and cost-attribution pipelines.

• Deploy DCGM Exporter: The Data Center GPU Manager (DCGM) exporter feeds hardware-level metrics (GPU temperature, VRAM usage, SM utilization) into Prometheus.
• Set Up Cost Allocation: Deploy KubeCost. Tag namespaces by team, project, or individual practitioner to track exactly who is driving cloud spend.
• Configure Spot Instances: Train Karpenter to use Spot GPUs for fault-tolerant, interruptible training jobs, reducing compute costs by up to 70%. Use On-Demand GPUs strictly for production inference.
• Implement Progressive Delivery: Use a service mesh (like Istio) or advanced ingress (like Argo Rollouts) to perform Canary deployments for new models, ensuring traffic is split safely before shifting full volume to new GPU pods.

To align this setup perfectly with your current workflow:

To build this self-service machine learning platform on Azure, you will use Azure Kubernetes Service (AKS), the NVIDIA GPU Operator, Karpenter (now natively supported on AKS via the Node Auto-Provisioning provider), and KEDA.

Here is the turnkey deployment blueprint, complete with configurations that abstract infrastructure management away from your ML practitioners.

---

1. Configure the Infrastructure (Operations Setup)

Operations must first provision an AKS cluster configured for automated node generation and GPU drivers.

Install the NVIDIA GPU Operator via Helm

Run this command to handle automated driver installation, container toolkit setup, and health monitoring across any GPU instances AKS spins up:

helm repo add nvidia https://github.io
helm repo update
helm install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator \
  --create-namespace \
  --set driver.enabled=true

Define the Karpenter NodePool (nodepool.yaml)

This configuration enables automated, just-in-time GPU node provisioning on Azure. It allows your cluster to dynamically spin up Standard_NCs_v3 (NVIDIA V100) or Standard_NDs_v4 (NVIDIA A100) machines based solely on developer requests, utilizing lower-cost Spot instances with an automatic fallback to On-Demand compute.

apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: ml-gpu-pool
spec:
  template:
    metadata:
      labels:
        tier: machine-learning
        accelerator: nvidia-gpu
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
        - key: node.kubernetes.io/instance-type
          operator: In
          values: ["Standard_NC6s_v3", "Standard_NC12s_v3", "Standard_ND96asr_v4"]
      taints:
        - key: ://nvidia.com
          operator: Exists
          effect: NoSchedule
  limits:
    cpu: "1000"
    memory: 4000Gi
  disruption:
    consolidationPolicy: WhenUnderutilized
    expireAfter: 720h

---

2. Implement Dynamic Metrics Scaling (KEDA)

ML workloads are rarely constrained by traditional CPU or Memory limits; traffic concurrency and GPU memory are the true scale metrics. Use this KEDA ScaledObject to monitor your model endpoint and automatically scale your inference pods to zero during periods of inactivity.

Create the KEDA Scaler (keda-scaler.yaml)

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: ml-model-scaler
  namespace: ml-workloads
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: vllm-inference-service
  minReplicaCount: 0  # Enables scale-to-zero to save Azure costs
  maxReplicaCount: 5
  cooldownPeriod: 300 # Wait 5 minutes before spinning down nodes
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://cluster.local
        # Scales based on concurrent requests processing inside your serving framework
        metricName: vllm_num_requests_waiting
        threshold: '5'
        query: sum(vllm_num_requests_waiting)

---

3. Build the Self-Service Interface (Developer Templates)

ML engineers should only interacting with high-level configurations. Package the infrastructure complexity into a standardized template.

The Simplified Developer Manifest (deployment.yaml)

To deploy a model, an ML practitioner only needs to fill out this streamlined schema. Karpenter reads the resources.limits and tolerations blocks, calls Azure APIs, creates the appropriate virtual machine, hooks up the drivers, and schedules the job with zero operational friction.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-inference-service
  namespace: ml-workloads
spec:
  replicas: 1
  selector:
    matchLabels:
      app: llama-3-inference
  template:
    metadata:
      labels:
        app: llama-3-inference
    spec:
      tolerations:
        - key: "://nvidia.com"
          operator: "Exists"
          effect: "NoSchedule"
      containers:
        - name: engine
          image: vllm/vllm-openai:latest
          args: ["--model", "meta-llama/Meta-Llama-3-8B-Instruct"]
          env:
            - name: HUGGING_FACE_HUB_TOKEN
              valueFrom:
                secretKeyRef:
                  name: hf-token
                  key: token
          ports:
            - containerPort: 8000
          resources:
            requests:
              cpu: "4"
              memory: "16Gi"
              ://nvidia.com: "1" # Operations setup triggers Karpenter node provisioning here
            limits:
              cpu: "8"
              memory: "32Gi"
              ://nvidia.com: "1"

---

4. Establish Operational Guardrails

To prevent run-away expenditure on expensive GPU clusters, enforce hard walls at the namespace level.

ResourceQuota Guardrail (quota.yaml)

Apply this file to individual team namespaces to ensure a single experiment cannot accidentally exceed your organization's budget allocations.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-alpha-gpu-limit
  namespace: ml-workloads
spec:
  hard:
    ://nvidia.com: "4" # Limits the team to a maximum of 4 concurrent GPUs
    ://nvidia.com: "4"

---

To support Triton Inference Server on your self-service Azure platform, you need to structure your deployment to handle multi-model repositories, optimize model loading from Azure Blob Storage, and track Triton-specific performance metrics for autoscaling.

Here is the production-ready implementation blueprint for both Operations and ML practitioners.

---

1. Configure the Infrastructure (Operations Setup)

Operations must ensure that Triton pods can securely access your model registry in Azure Blob Storage without hardcoding static credentials.

Step A: Configure Managed Identity for Blob Storage

Use Azure Workload Identity to map a Kubernetes ServiceAccount directly to an Azure User-Assigned Managed Identity. This identity must have at least Storage Blob Data Reader permissions on your Triton model repository container.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: triton-storage-sa
  namespace: ml-workloads
  annotations:
    azure.workload.identity/client-id: "00000000-0000-0000-0000-000000000000" # Your Azure Managed Identity Client ID

---

2. Dynamic Metric Scaling (KEDA Configuration)

Triton exposes dense execution metrics. Instead of relying on crude request counts, we scale based on Triton’s Queue Time (nv_inference_queue_duration_us). If requests are sitting in the queue waiting for a GPU engine execution slot, KEDA will instantly trigger Karpenter to provision more hardware. [1]

The Triton KEDA Scaler (triton-scaler.yaml)

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: triton-autoscaler
  namespace: ml-workloads
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: triton-inference-server
  minReplicaCount: 1  # Keep 1 warm instance for production pipelines
  maxReplicaCount: 8
  cooldownPeriod: 600 # 10 minutes to minimize GPU node thrashing
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://cluster.local
        metricName: triton_queue_delay
        threshold: '20000' # Trigger scaling if avg queue delay exceeds 20 milliseconds
        query: |
          sum(rate(nv_inference_queue_duration_us[1m])) 
          / 
          sum(rate(nv_inference_request_duration_us[1m]))

---

3. The Self-Service Interface (Developer Templates)

ML practitioners use this standardized blueprint to deploy their models. Triton streams the model folder directory structured layout (config.pbtxt, model.onnx, or model.pt) straight out of your Azure Storage account directly into GPU VRAM. [2]

The Simplified Triton Deployment (triton-deployment.yaml)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: triton-inference-server
  namespace: ml-workloads
  labels:
    app: triton-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: triton-server
  template:
    metadata:
      labels:
        app: triton-server
        azure.workload.identity/use: "true" # Triggers the Azure Workload Identity sidecar injection
    spec:
      serviceAccountName: triton-storage-sa
      tolerations:
        - key: "://nvidia.com"
          operator: "Exists"
          effect: "NoSchedule"
      containers:
        - name: triton-container
          image: nvcr.io/nvidia/tritonserver:24.01-py3 # Updated stable Triton runtime
          args:
            - "tritonserver"
            # Triton reads natively from Azure Storage Blob via the asys:// protocol
            - "--model-repository=asys://your-storage-account.blob.core.windows.net/model-registry-container"
            - "--strict-model-config=true"
            - "--log-verbose=1"
          ports:
            - containerPort: 8000
              name: http
            - containerPort: 8001
              name: grpc
            - containerPort: 8002
              name: metrics # Scraped by Prometheus to inform KEDA
          resources:
            requests:
              cpu: "4"
              memory: "16Gi"
              ://nvidia.com: "1" # Prompts Karpenter to provision an Azure GPU VM instantly
            limits:
              cpu: "8"
              memory: "32Gi"
              ://nvidia.com: "1"
          readinessProbe:
            httpGet:
              path: /v2/health/ready
              port: 8000
            initialDelaySeconds: 30
            periodSeconds: 10

The Accompanying Triton Service Routing (triton-service.yaml)

apiVersion: v1
kind: Service
metadata:
  name: triton-inference-service
  namespace: ml-workloads
  labels:
    app: triton-server
spec:
  ports:
    - port: 8000
      name: http
      targetPort: 8000
    - port: 8001
      name: grpc
      targetPort: 8001
    - port: 8002
      name: metrics
      targetPort: 8002
  selector:
    app: triton-server

---

4. Production Optimization Guardrail: Sharing GPUs (Optional)

If your practitioners run multiple small Triton models (e.g., Computer Vision classification or lightweight tabular nets), assigning an entire Azure GPU to a single container is cost-prohibitive.

You can configure Triton to split a single GPU across multiple execution model instances directly via its config.pbtxt or leverage MIG (Multi-Instance GPU) if using Standard_ND96asr_v4 (A100) nodes.

For software-level instance sharing inside Triton, direct your developers to add this snippet directly to their config.pbtxt inside their Azure Storage bucket:

# config.pbtxt snippet to maximize GPU utilization
instance_group [
  {
    count: 2  # Runs 2 concurrent instances of this model on a single assigned GPU
    kind: KIND_GPU
  }
]

To embed this setup into your Operations pipeline, we need to convert these requirements into structural infrastructure-as-code.

This Terraform root module provisions the core building blocks on Azure: it activates Azure Kubernetes Service (AKS) node auto-provisioning (Karpenter engine), configures the Azure Workload Identity infrastructure, sets up the Azure Storage Account model repository, and deploys the necessary Helm charts for Triton, KEDA, and the GPU operators.

1. Directory Architecture

Organize your Terraform repository layout to keep infrastructure declaration clean and separated from application space. [1, 2, 3]

├── main.tf                 # Core cluster, identity, and storage components
├── providers.tf            # Azure, Helm, and Kubernetes provider definitions
├── variables.tf            # Input configuration variables
└── outputs.tf              # Storage links, client IDs, and cluster endpoints

2. Infrastructure Declaration (main.tf)

# 1. Resource Group
resource "azurerm_resource_group" "ml_ops" {
  name     = var.resource_group_name
  location = var.location
}

# 2. Azure Kubernetes Service (AKS) with Node Auto-Provisioning (Karpenter)
resource "azurerm_kubernetes_cluster" "aks" {
  name                = "aks-ml-platform"
  location            = azurerm_resource_group.ml_ops.location
  resource_group_name = azurerm_resource_group.ml_ops.name
  dns_prefix          = "mlops-k8s"

  default_node_pool {
    name       = "systempool"
    node_count = 3
    vm_size    = "Standard_D4s_v5" # Cost-effective pool for core cluster overhead
  }

  identity {
    type = "SystemAssigned"
  }

  # Native Azure integration for Workload Identity & Open ID Connect
  oidc_issuer_enabled       = true
  workload_identity_enabled = true

  # Activates the native Azure Karpenter implementation (Node Auto-Provisioning)
  node_os_channel_upgrade = "NodeImage"
  
  # Ensure the cluster can run standard K8s scheduling optimizations
  sku_tier = "Standard"
}

# 3. Model Storage Infrastructure
resource "azurerm_storage_account" "model_registry" {
  name                     = var.storage_account_name
  resource_group_name      = azurerm_resource_group.ml_ops.name
  location                 = azurerm_resource_group.ml_ops.location
  account_tier             = "Standard"
  account_replication_type = "LRS"
}

resource "azurerm_storage_container" "models" {
  name                  = "triton-model-repository"
  storage_account_id    = azurerm_storage_account.model_registry.id
  container_access_type = "private"
}

# 4. Identity Mapping (Azure Workload Identity Setup)
resource "azurerm_user_assigned_identity" "triton_identity" {
  name                = "uai-triton-storage-reader"
  resource_group_name = azurerm_resource_group.ml_ops.name
  location            = azurerm_resource_group.ml_ops.location
}

# Give the Managed Identity read access to the storage bucket
resource "azurerm_role_assignment" "storage_reader" {
  scope                = azurerm_storage_account.model_registry.id
  role_definition_name = "Storage Blob Data Reader"
  principal_id         = azurerm_user_assigned_identity.triton_identity.principal_id
}

# Establish federated identity link between Azure and the cluster ServiceAccount
resource "azurerm_federated_identity_credential" "triton_fed_link" {
  name                = "fic-triton-sa-link"
  resource_group_name = azurerm_resource_group.ml_ops.name
  audience            = ["api://AzureADTokenExchange"]
  issuer              = azurerm_kubernetes_cluster.aks.oidc_issuer_url
  parent_id           = azurerm_user_assigned_identity.triton_identity.id
  subject             = "system:serviceaccount:ml-workloads:triton-storage-sa"
}

# 5. Helm Integrations
resource "helm_release" "gpu_operator" {
  name             = "gpu-operator"
  repository       = "https://nvidia.com"
  chart            = "gpu-operator"
  namespace        = "gpu-operator"
  create_namespace = true
  depends_on       = [azurerm_kubernetes_cluster.aks]

  set {
    name  = "driver.enabled"
    value = "true"
  }
}

resource "helm_release" "keda" {
  name             = "keda"
  repository       = "https://github.io"
  chart            = "keda"
  namespace        = "keda"
  create_namespace = true
  depends_on       = [azurerm_kubernetes_cluster.aks]
}

3. Providers System Config (providers.tf)

terraform {
  required_version = ">= 1.5.0"
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 4.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.23"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.11"
    }
  }
}

provider "azurerm" {
  features {}
}

provider "kubernetes" {
  host                   = azurerm_kubernetes_cluster.aks.kube_config[0].host
  client_certificate     = base64decode(azurerm_kubernetes_cluster.aks.kube_config[0].client_certificate)
  client_key             = base64decode(azurerm_kubernetes_cluster.aks.kube_config[0].client_key)
  cluster_ca_certificate = base64decode(azurerm_kubernetes_cluster.aks.kube_config[0].cluster_ca_certificate)
}

provider "helm" {
  kubernetes {
    host                   = azurerm_kubernetes_cluster.aks.kube_config[0].host
    client_certificate     = base64decode(azurerm_kubernetes_cluster.aks.kube_config[0].client_certificate)
    client_key             = base64decode(azurerm_kubernetes_cluster.aks.kube_config[0].client_key)
    cluster_ca_certificate = base64decode(azurerm_kubernetes_cluster.aks.kube_config[0].cluster_ca_certificate)
  }
}

4. Input Controls (variables.tf)

variable "resource_group_name" {
  type        = string
  default     = "rg-mlplatform-prod"
  description = "Target infrastructure resource group"
}

variable "location" {
  type        = string
  default     = "eastus2"
  description = "Target Azure data center location (Ensure GPU SKU availability here)"
}

variable "storage_account_name" {
  type        = string
  default     = "stmlmodelregistryprod"
  description = "Unique global naming string for the Blob Storage account"
}

5. Deployment Lifecycle Output (outputs.tf)

These output parameters expose the critical connection targets needed by ML Engineers to deploy their applications.

output "aks_cluster_name" {
  value = azurerm_kubernetes_cluster.aks.name
}

output "triton_workload_identity_client_id" {
  value       = azurerm_user_assigned_identity.triton_identity.client_id
  description = "Value to paste inside the developer's ServiceAccount annotation block"
}

output "storage_blob_endpoint" {
  value       = azurerm_storage_account.model_registry.primary_blob_endpoint
  description = "Value to paste inside Triton configuration arguments string"
}

Next Action Step

Run the compilation process to instantiate your cluster control plane:

terraform init
terraform apply -auto-approve

Once the terraform setup is complete, you can pass the output values to your practitioners. They can plug triton_workload_identity_client_id and storage_blob_endpoint directly into the templates we built previously to begin serving models.

To protect your budget and catch model degradation before it impacts users, you need alerts that target two specific areas: infrastructure failures (e.g., thermal throttling, stuck GPU drivers) and application inefficiencies (e.g., models wasting VRAM or requests piling up in Triton’s queue).

Here are the production-ready PrometheusAlertmanager rules, configured as a standard Kubernetes PrometheusRule custom resource definition (CRD).

1. Prometheus Alerts Configuration (ml-gpu-alerts.yaml)

apiVersion: ://coreos.com
kind: PrometheusRule
metadata:
  name: Triton-gpu-alerts
  namespace: monitoring
  labels:
    role: alert-rules
spec:
  groups:
    - name: ML.Platform.Hardware.Alerts
      rules:
        # 1. Stuck GPU Driver / Broken Hardware Detection
        - alert: GpuDriverOrHardwareFailure
          expr: dcgm_fi_driver_version == 0 or count(dcgm_fi_gpu_temp) == 0
          for: 2m
          labels:
            severity: critical
            tier: ml-platform
          annotations:
            summary: "GPU Driver or hardware failure detected on node {{ $labels.kubernetes_node }}"
            description: "The NVIDIA driver is reporting a crash or the card has dropped off the PCIe bus. Karpenter should replace this node."

        # 2. Thermal Throttling Guard
        - alert: GpuThermalThrottlingActive
          expr: dcgm_fi_therm_violation == 1
          for: 5m
          labels:
            severity: warning
            tier: ml-platform
          annotations:
            summary: "GPU Thermal Throttling active on {{ $labels.kubernetes_node }}"
            description: "The GPU clock speeds are being throttled due to high heat. Check the physical host health or cluster scheduling density."

        # 3. Running Out of Video RAM (VRAM)
        - alert: GpuVramCriticalExhaustion
          expr: (dcgm_fi_fb_used / (dcgm_fi_fb_used + dcgm_fi_fb_free)) * 100 > 92
          for: 3m
          labels:
            severity: critical
            tier: ml-platform
          annotations:
            summary: "GPU VRAM critical exhaustion on node {{ $labels.kubernetes_node }}"
            description: "VRAM utilization is at {{ $value | printf \"%.2f\" }}%. Triton is dangerously close to an Out-Of-Memory (OOM) crash."

    - name: ML.Platform.Application.Performance
      rules:
        # 4. Triton Model Inference Latency Breach
        - alert: TritonInferenceLatencyTooHigh
          expr: |
            (sum(rate(nv_inference_compute_input_duration_us[5m])) 
            + sum(rate(nv_inference_compute_infer_duration_us[5m])) 
            + sum(rate(nv_inference_compute_output_duration_us[5m]))) 
            / sum(rate(nv_inference_request_duration_us[5m])) > 150000
          for: 3m
          labels:
            severity: warning
            tier: ml-platform
          annotations:
            summary: "Triton model inference execution time exceeded 150ms"
            description: "Model execution pipelines are slowing down. Actual calculation overhead is running at {{ $value | humanizeDuration }}."

        # 5. Massive Request Queue (Autoscaler Bottleneck or Lock)
        - alert: TritonQueueSaturationBreach
          expr: |
            sum(rate(nv_inference_queue_duration_us[5m])) 
            / sum(rate(nv_inference_request_duration_us[5m])) > 50000
          for: 4m
          labels:
            severity: critical
            tier: ml-platform
          annotations:
            summary: "Triton request queue delay exceeded 50ms"
            description: "Requests are waiting too long in the memory queue. KEDA and Karpenter may have hit their max replica limits (Cluster Saturation)."

        # 6. Underutilized GPU Hardware (Wasted Budget Guard)
        - alert: GpuSeverelyUnderutilized
          expr: avg_over_time(dcgm_fi_gpu_util[1h]) < 10 and count(dcgm_fi_gpu_util) > 0
          for: 1h
          labels:
            severity: warning
            tier: ml-platform
          annotations:
            summary: "GPU Node {{ $labels.kubernetes_node }} is heavily underutilized"
            description: "Average GPU computing SM utilization over the past hour is only {{ $value | printf \"%.2f\" }}%. A practitioner may have forgotten to release an interactive Jupyter environment."

2. Connect Your Alert Routing (alertmanager-config.yaml)

To ensure these rules land in the right hands, route the hardware alerts straight to your Operations on-call rota (like PagerDuty) and the underutilization/latency alerts directly to the ML Engineering team's Slack/Teams channel.

apiVersion: ://coreos.com
kind: AlertmanagerConfig
metadata:
  name: ml-routing-rules
  namespace: monitoring
spec:
  route:
    groupBy: ['alertname', 'kubernetes_node', 'namespace']
    groupWait: 30s
    groupInterval: 5m
    repeatInterval: 4h
    receiver: 'Default Empty'
    routes:
      # Send hardware failures straight to Operations PagerDuty
      - matchers:
          - name: severity
            value: critical
          - name: tier
            value: ml-platform
        receiver: 'Ops-PagerDuty'
      # Send performance metrics and budget leaks to ML Team channels
      - matchers:
          - name: severity
            value: warning
          - name: tier
            value: ml-platform
        receiver: 'ML-Engineers-Slack'
  receivers:
    - name: 'Ops-PagerDuty'
      pagerdutyConfigs:
        - routingKey:
            name: pagerduty-keys
            key: ops-key
    - name: 'ML-Engineers-Slack'
      slackConfigs:
        - channel: '#mlops-alerts'
          apiURL:
            name: slack-webhook
            key: webhook-url
          text: "Notification: {{ .CommonAnnotations.summary }}\nDetails: {{ .CommonAnnotations.description }}"

How to apply this setup

Ensure your cluster's Prometheus installation is scraping the Triton pods on port 8002 (via a PodMonitor or standard annotation parsing), then run:

kubectl apply -f ml-gpu-alerts.yaml
kubectl apply -f alertmanager-config.yaml

To visually monitor your cluster, you can use pre-built community templates or a unified configuration file. This setup combines metrics from the NVIDIA DCGM Exporter and Triton Inference Server into a single glass view.

Option A: Use Official Dashboard IDs (Quickest Setup)

Instead of copying a long configuration file, you can import these two official IDs into your Grafana instance (Dashboards ➔ New ➔ Import):

1. NVIDIA DCGM Hardware Metrics: Use Grafana Dashboard ID: 12239. This tracks physical card health, VRAM limits, and temperatures.
2. NVIDIA Triton Performance Metrics: Use Grafana Dashboard ID: 12832. This tracks inference execution latency, queue times, and throughput counters.

---

Option B: The Unified Production JSON Manifest

To bundle your infrastructure, deploy this custom Kubernetes ConfigMap JSON schema. It creates a unified "ML Platform & Triton Model Insights" dashboard, pre-configured with panels for the Prometheus alerts we configured earlier.

Save this file as ml-grafana-dashboard.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: ml-platform-dashboard-export
  namespace: monitoring
  labels:
    # Triggers your Grafana sidecar provider to automatically load the dashboard
    grafana_dashboard: "1"
data:
  ml-platform-insights.json: |-
    {
      "annotations": { "list": [] },
      "editable": true,
      "fiscalYearStartMonth": 0,
      "graphTooltip": 1,
      "id": null,
      "links": [],
      "liveNow": false,
      "panels": [
        {
          "title": "Active GPU Compute Hardware Utilization",
          "type": "timeseries",
          "gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 },
          "targets": [
            {
              "datasource": { "type": "prometheus", "uid": "prometheus" },
              "editorMode": "code",
              "expr": "sum(dcgm_fi_gpu_util) by (kubernetes_node)",
              "legendFormat": "Node: {{kubernetes_node}}",
              "range": true
            }
          ],
          "fieldConfig": {
            "defaults": {
              "custom": { "drawStyle": "line", "lineInterpolation": "smooth" },
              "unit": "percent"
            }
          }
        },
        {
          "title": "VRAM Exhaustion Tracker",
          "type": "timeseries",
          "gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 },
          "targets": [
            {
              "datasource": { "type": "prometheus", "uid": "prometheus" },
              "editorMode": "code",
              "expr": "(dcgm_fi_fb_used / (dcgm_fi_fb_used + dcgm_fi_fb_free)) * 100",
              "legendFormat": "GPU Slot: {{gpu}} on {{pod}}",
              "range": true
            }
          ],
          "fieldConfig": {
            "defaults": {
              "unit": "percent",
              "thresholds": {
                "mode": "absolute",
                "steps": [
                  { "color": "green", "value": null },
                  { "color": "orange", "value": 80 },
                  { "color": "red", "value": 92 }
                ]
              }
            }
          }
        },
        {
          "title": "Triton Model Inference Latency Breakdown",
          "type": "timeseries",
          "gridPos": { "h": 8, "w": 12, "x": 0, "y": 8 },
          "targets": [
            {
              "datasource": { "type": "prometheus", "uid": "prometheus" },
              "editorMode": "code",
              "expr": "(sum(rate(nv_inference_compute_input_duration_us[5m])) by (model) + sum(rate(nv_inference_compute_infer_duration_us[5m])) by (model) + sum(rate(nv_inference_compute_output_duration_us[5m])) by (model)) / sum(rate(nv_inference_request_duration_us[5m])) by (model) / 1000",
              "legendFormat": "Model: {{model}} (v{{version}})",
              "range": true
            }
          ],
          "fieldConfig": {
            "defaults": {
              "custom": { "drawStyle": "line" },
              "unit": "ms"
            }
          }
        },
        {
          "title": "Triton Memory Queue Delays (Saturation Hook)",
          "type": "timeseries",
          "gridPos": { "h": 8, "w": 12, "x": 12, "y": 8 },
          "targets": [
            {
              "datasource": { "type": "prometheus", "uid": "prometheus" },
              "editorMode": "code",
              "expr": "sum(rate(nv_inference_queue_duration_us[5m])) by (model) / sum(rate(nv_inference_request_duration_us[5m])) by (model) / 1000",
              "legendFormat": "Queue Wait: {{model}}",
              "range": true
            }
          ],
          "fieldConfig": {
            "defaults": {
              "custom": { "drawStyle": "bars" },
              "unit": "ms"
            }
          }
        }
      ],
      "refresh": "10s",
      "schemaVersion": 38,
      "style": "dark",
      "tags": ["mlops", "triton", "gpu", "azure"],
      "time": { "from": "now-1h", "to": "now" },
      "timepicker": {},
      "timezone": "browser",
      "title": "ML Platform & Triton Model Insights",
      "version": 1
    }

Apply to Your Environment

Deploy this configuration mapping directly to your cluster backend:

kubectl apply -f ml-grafana-dashboard.yaml

If your platform uses Grafana’s standard dashboard-provider sidecar setup, this console will automatically populate under your shared directory tree.

Causal Modeling

 

                                                             generated by meta ai

Yes. In fact, causal modelling is one of the most advanced topics in quantitative finance and is becoming increasingly important because traditional ML models (LSTM, XGBoost, Transformers) often learn correlations, whereas causal models aim to discover why prices move.

Since I am pursuing an MSc in Financial Engineering at WQU, I'll teach it at that level, covering theory, mathematics, and Python implementations.

Learning Roadmap

We'll build from scratch.

1. Correlation vs Causation

2. Structural Causal Models (SCM)

3. Directed Acyclic Graphs (DAGs)

4. Causal Discovery Algorithms

5. Do-Calculus (Pearl)

6. Counterfactual Prediction

7. Causal Forecasting

8. Applying causal models to stock prediction

9. Building an end-to-end project

---

Step 1 — Correlation vs Causation

Suppose we have

Oil Price	Airline Stock
↑	↓
↓	↑

They are negatively correlated.

But does oil directly affect airlines?

Yes.

Higher fuel cost
→ higher operating expenses
→ lower profit
→ lower stock price

This is a causal relationship.

Now another example.

Ice Cream Sales	Shark Attacks
↑	↑

Highly correlated.

Does ice cream cause sharks?

No.

Common cause:

Summer
→ more people swim
→ more shark attacks

Summer
→ more ice cream sales

This is a confounder.

Traditional ML usually cannot distinguish these.

---

Step 2 — Financial Example

Imagine predicting Apple stock.

Variables

Interest Rate

Inflation

USD Index

Nasdaq Index

Apple Earnings

Apple Stock

The causal graph looks like

Interest Rate
        \
         \
          \
Inflation ---> USD
     \
      \
       \
     Nasdaq ----\
                  \
                   \
              Apple Earnings
                     \
                      \
                   Apple Stock

Notice

Interest rate does NOT directly cause Apple stock.

It changes

Interest Rate
↓

Bond Yield

↓

Investor Preference

↓

Nasdaq

↓

Apple

That matters.

---

Step 3 — Structural Causal Model (SCM)

Every node has an equation.

Example

Inflation = Noise

InterestRate = 0.8 × Inflation + Noise

USD = 0.5 × InterestRate + Noise

Nasdaq = -0.7 × InterestRate + Noise

Apple =

0.6 × Nasdaq

+0.4 × Earnings

+Noise

Unlike regression,

each equation represents a mechanism.

---

Python Example

Generate synthetic causal data.

import numpy as np
import pandas as pd

np.random.seed(42)

N = 1000

inflation = np.random.normal(2,0.5,N)

interest = 0.8*inflation + np.random.normal(0,0.2,N)

usd = 0.5*interest + np.random.normal(0,0.3,N)

nasdaq = -0.7*interest + np.random.normal(0,0.5,N)

earnings = np.random.normal(5,1,N)

apple = (
    0.6*nasdaq
    +0.4*earnings
    +np.random.normal(0,0.5,N)
)

df = pd.DataFrame({
    "Inflation":inflation,
    "Interest":interest,
    "USD":usd,
    "Nasdaq":nasdaq,
    "Earnings":earnings,
    "Apple":apple
})

print(df.head())

Notice that we explicitly generated the causal relationships.

---

Step 4 — Draw the DAG

Using NetworkX

import networkx as nx
import matplotlib.pyplot as plt

G = nx.DiGraph()

G.add_edges_from([
    ("Inflation","Interest"),
    ("Interest","USD"),
    ("Interest","Nasdaq"),
    ("Nasdaq","Apple"),
    ("Earnings","Apple")
])

nx.draw(G,
        with_labels=True,
        node_size=3000,
        arrows=True)

plt.show()

Produces

Inflation

↓

Interest

↙      ↘

USD   Nasdaq

          ↓

      Apple
↑

Earnings

---

Step 5 — Why This Is Better Than Regression

Suppose Fed raises rates.

Regression says

Past data:

Rate ↑

Apple ↓

Therefore predict Apple ↓

But suppose

Apple reports record earnings.

Causal model says

Interest ↑

↓

Nasdaq ↓

↓

Apple ↓

BUT

Earnings ↑↑

↓

Apple ↑

The model understands competing causes.

This is much closer to how human analysts reason.

---

Step 6 — Counterfactual Prediction

Question:

"What would Apple have done if the Fed had NOT increased rates?"

Regression cannot answer.

Causal model can.

interest = 5.5

# Intervention

interest = 2.0

# Recompute downstream variables

usd = 0.5*interest

nasdaq = -0.7*interest

apple = 0.6*nasdaq + 0.4*earnings

This is called an intervention:

do(Interest=2%)

instead of merely observing Interest = 2%.

---

Step 7 — Real Stock Prediction

For NIFTY prediction, a richer causal graph might be:

Fed Rate
     ↓
US Dollar
     ↓
Crude Oil
     ↓
Indian Inflation
     ↓
RBI Rate
     ↓
Bank Nifty
     ↓
NIFTY

GDP

↓

Corporate Earnings

↓

NIFTY

VIX

↓

NIFTY

FII Flow

↓

NIFTY

A causal model uses these relationships rather than feeding all variables into a black-box predictor.

---

Step 8 — Libraries

Popular Python libraries include:

• DoWhy — causal inference and effect estimation

• CausalNex — Bayesian networks and causal graphs

• pgmpy — Bayesian networks (already familiar from your coursework)

• econml — heterogeneous treatment effects

• lingam — causal discovery for linear non-Gaussian models

• causal-learn — PC, GES, FCI, NOTEARS and other discovery algorithms

• networkx — graph visualization

Example with DoWhy:

from dowhy import CausalModel

model = CausalModel(
    data=df,
    treatment="Interest",
    outcome="Apple",
    common_causes=["Inflation"]
)

identified_estimand = model.identify_effect()

estimate = model.estimate_effect(
    identified_estimand,
    method_name="backdoor.linear_regression"
)

print(estimate.value)

---

Final Project (MSc-Level)

A strong capstone could combine:

1. Data collection

   • Yahoo Finance

   • FRED (interest rates)

   • RBI

   • FII/DII flows

   • VIX

   • Crude oil

   • USD/INR

   • CPI

2. Causal discovery

   • PC Algorithm

   • NOTEARS

   • LiNGAM

3. Bayesian Network

   • Build the DAG

   • Learn conditional probability tables or continuous conditional distributions

4. Causal forecasting

   • Estimate intervention effects

   • Perform counterfactual analysis

   • Forecast index or stock prices under different macroeconomic scenarios

5. Explainability

   • Show which causal paths contributed most to the prediction

   • Compare against LSTM, XGBoost, and Transformer models

This kind of project aligns well with your financial engineering background because it integrates probabilistic graphical models, macroeconomic reasoning, and interpretable AI rather than relying solely on black-box prediction.