actual images from my colab

Building an AI-Powered Pothole Detection Dash Cam with Raspberry Pi Zero

Turn your car into a smart road condition monitor with computer vision and edge AI

---

Introduction

Potholes are more than just a nuisance—they cause billions of dollars in vehicle damage annually and pose serious safety risks to drivers. What if your daily commute could help map and document road conditions automatically? In this project, I'll show you how to build an intelligent dash cam using a Raspberry Pi Zero that detects potholes in real-time using computer vision.

This isn't just a hobbyist project. The system we're building could contribute to civic infrastructure monitoring, help municipalities prioritize road repairs, or simply alert you to hazards ahead during your drive.

What We're Building

An autonomous pothole detection system that:

• Runs entirely on a Raspberry Pi Zero (no cloud required)
• Detects potholes from dash cam footage in real-time
• Saves images of detected potholes with timestamps
• Can optionally log GPS coordinates for mapping
• Costs under $50 in hardware

Why Raspberry Pi Zero?

The Raspberry Pi Zero W is the perfect platform for this project:

• Compact: Fits anywhere on your dashboard
• Affordable: Around $15-20
• Low Power: Can run off a USB power bank
• WiFi Built-in: Easy to access and configure
• GPIO Support: Can interface with GPS modules and other sensors

While it's not the fastest computer, modern AI model optimization techniques make real-time inference possible even on this tiny device.

Hardware Requirements

Here's everything you'll need:

Essential Components

• Raspberry Pi Zero W or WH ($15-20)
• Raspberry Pi Camera Module or USB webcam ($10-25)
• MicroSD Card (16GB minimum, 32GB recommended) ($8-15)
• 5V 2.5A Power Supply or car USB adapter ($8-12)
• Dash Cam Mount (optional but recommended) ($5-15)

Optional Enhancements

• GPS Module (for location tracking) ($15-30)
• Real-Time Clock (RTC) Module (for accurate timestamps) ($5-10)
• Larger Power Bank (for extended operation) ($20-40)

Total Cost: $45-80 (basic setup)

The AI Model: YOLOv8 Nano

For pothole detection, we're using YOLOv8 Nano, the smallest and fastest variant of the popular YOLO (You Only Look Once) object detection model. Here's why:

• Lightweight: Only ~6MB model size
• Fast: Can achieve 1-3 FPS on Raspberry Pi Zero
• Accurate: Trained on thousands of pothole images
• Edge-Ready: Designed for deployment on resource-constrained devices

The model was trained on the Potholes Detection Dataset from Kaggle, which contains diverse road conditions, lighting scenarios, and pothole types.

Software Architecture

The system follows a simple but effective architecture:

Camera → Frame Capture → YOLOv8 Inference → Detection Logic → Save/Alert

Key Software Components:

1. MotionEyeOS: Provides camera streaming and management
2. Python 3: Main programming language
3. Ultralytics YOLOv8: AI inference engine
4. OpenCV: Image processing and camera interface

Training the Model

I trained the YOLOv8 Nano model on Kaggle's free GPU environment (Tesla T4). The training process involved:

• Dataset: 600+ labeled pothole images (train/validation/test split)
• Image Size: 416x416 pixels (optimized for speed)
• Training Time: ~30 minutes on Tesla T4
• Performance: 85%+ mAP@50 (mean Average Precision)

The model was specifically tuned for dash cam scenarios:

• Various lighting conditions (day/night)
• Different road surfaces (asphalt, concrete)
• Multiple pothole sizes and shapes
• Various viewing angles

Setting Up Your Raspberry Pi Zero

Now let's get to the practical part. After training the model, here's how to deploy it on your Raspberry Pi Zero.

Step 1: Install MotionEyeOS

MotionEyeOS is a lightweight Linux distribution designed for video surveillance. It's perfect for our dash cam application.

1. Download MotionEyeOS for Raspberry Pi Zero from the official GitHub releases

2. Flash to MicroSD Card:

   • Use Raspberry Pi Imager or Etcher
   • Select the MotionEyeOS image
   • Choose your SD card
   • Flash!

3. Enable SSH (for remote access):

   • Mount the SD card on your computer
   • Create an empty file named ssh in the boot partition

4. Boot Your Pi:

   • Insert the SD card into your Raspberry Pi Zero
   • Connect the camera module
   • Power it up
   • Wait 2-3 minutes for first boot

5. Access Web Interface:

   • Find your Pi's IP address (check your router or use a network scanner)
   • Open browser: http://<pi-ip>:8765
   • Default username: admin (no password)

Step 2: Install Dependencies

SSH into your Raspberry Pi:

ssh root@<your-pi-ip>
# Default password: blank (just press Enter)

Make the filesystem writable and update:

mount -o remount,rw /
mount -o remount,rw /boot

apt-get update
apt-get upgrade -y

Install Python and required libraries:

# Install system dependencies
apt-get install -y python3-pip python3-opencv libopencv-dev \
                   libatlas-base-dev libjpeg-dev libpng-dev

# Upgrade pip
pip3 install --upgrade pip

# Install Ultralytics and dependencies
pip3 install ultralytics opencv-python-headless pillow numpy --break-system-packages

Note: The --break-system-packages flag is required on newer Python versions.

Step 3: Transfer the Model

From your local machine, copy the trained model to your Pi:

scp best.pt root@<your-pi-ip>:/root/pothole_model.pt

Or use a USB drive:

1. Copy best.pt to a USB drive
2. Insert into Raspberry Pi
3. Mount and copy: cp /media/usb/best.pt /root/pothole_model.pt

Step 4: Deploy the Detection Script

Create the pothole detection script on your Pi. Below is the complete implementation:

#!/usr/bin/env python3
"""
Pothole Detection Script for Raspberry Pi Zero with MotionEyeOS
This script captures frames from the camera and runs pothole detection.
"""

import cv2
import numpy as np
from ultralytics import YOLO
import time
import argparse
from datetime import datetime
import os

class PotholeDetector:
    def __init__(self, model_path, camera_source=0, conf_threshold=0.25):
        """
        Initialize pothole detector
        
        Args:
            model_path: Path to YOLO model (.pt file)
            camera_source: Camera device index or RTSP stream URL
            conf_threshold: Confidence threshold for detections
        """
        print("Loading model...")
        self.model = YOLO(model_path)
        self.conf_threshold = conf_threshold
        self.camera_source = camera_source
        self.cap = None
        
        # Create output directory for detections
        self.output_dir = "/root/pothole_detections"
        os.makedirs(self.output_dir, exist_ok=True)
        
        print("Model loaded successfully!")
    
    def initialize_camera(self):
        """Initialize camera capture"""
        print(f"Initializing camera: {self.camera_source}")
        self.cap = cv2.VideoCapture(self.camera_source)
        
        if not self.cap.isOpened():
            raise RuntimeError("Failed to open camera")
        
        # Set camera properties for optimal performance on RPi Zero
        self.cap.set(cv2.CAP_PROP_FRAME_WIDTH, 416)
        self.cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 416)
        self.cap.set(cv2.CAP_PROP_FPS, 10)
        
        print("Camera initialized!")
    
    def detect_potholes(self, frame):
        """
        Run pothole detection on a frame
        
        Args:
            frame: Input image frame
            
        Returns:
            annotated_frame: Frame with bounding boxes
            detections: List of detection dictionaries
        """
        results = self.model.predict(
            source=frame,
            imgsz=416,
            conf=self.conf_threshold,
            verbose=False
        )
        
        annotated_frame = results[0].plot()
        
        # Extract detection information
        detections = []
        boxes = results[0].boxes
        
        for box in boxes:
            x1, y1, x2, y2 = box.xyxy[0].cpu().numpy()
            conf = float(box.conf[0])
            cls = int(box.cls[0])
            
            detections.append({
                'bbox': [int(x1), int(y1), int(x2), int(y2)],
                'confidence': conf,
                'class': cls
            })
        
        return annotated_frame, detections
    
    def save_detection(self, frame, detections):
        """Save frame when pothole is detected"""
        if len(detections) > 0:
            timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
            filename = f"pothole_{timestamp}.jpg"
            filepath = os.path.join(self.output_dir, filename)
            cv2.imwrite(filepath, frame)
            print(f"Pothole detected! Saved to {filepath}")
    
    def run(self, save_detections=True, display=False):
        """
        Main detection loop
        
        Args:
            save_detections: Whether to save images with detections
            display: Whether to display video (requires X server)
        """
        self.initialize_camera()
        
        fps_counter = []
        frame_count = 0
        
        print("Starting pothole detection...")
        print("Press Ctrl+C to stop")
        
        try:
            while True:
                ret, frame = self.cap.read()
                if not ret:
                    print("Failed to grab frame")
                    break
                
                start_time = time.time()
                
                # Run detection
                annotated_frame, detections = self.detect_potholes(frame)
                
                # Calculate FPS
                fps = 1 / (time.time() - start_time)
                fps_counter.append(fps)
                
                # Display FPS on frame
                cv2.putText(annotated_frame, f"FPS: {fps:.1f}", 
                           (10, 30), cv2.FONT_HERSHEY_SIMPLEX, 
                           0.7, (0, 255, 0), 2)
                
                # Save detection if enabled
                if save_detections:
                    self.save_detection(annotated_frame, detections)
                
                # Display frame if enabled
                if display:
                    cv2.imshow('Pothole Detection', annotated_frame)
                    if cv2.waitKey(1) & 0xFF == ord('q'):
                        break
                
                frame_count += 1
                
                # Print statistics every 30 frames
                if frame_count % 30 == 0:
                    avg_fps = np.mean(fps_counter[-30:])
                    print(f"Frame {frame_count} | Avg FPS: {avg_fps:.2f} | Detections: {len(detections)}")
        
        except KeyboardInterrupt:
            print("\nStopping detection...")
        
        finally:
            self.cleanup()
    
    def cleanup(self):
        """Release resources"""
        if self.cap is not None:
            self.cap.release()
        cv2.destroyAllWindows()
        print("Cleanup complete")


if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Pothole Detection for Raspberry Pi")
    parser.add_argument("--model", type=str, default="/root/pothole_model.pt",
                       help="Path to YOLO model")
    parser.add_argument("--camera", type=str, default="0",
                       help="Camera source (0 for USB cam, or RTSP URL for MotionEye stream)")
    parser.add_argument("--conf", type=float, default=0.25,
                       help="Confidence threshold")
    parser.add_argument("--no-save", action="store_true",
                       help="Don't save detected potholes")
    parser.add_argument("--display", action="store_true",
                       help="Display video feed (requires X server)")
    
    args = parser.parse_args()
    
    # Convert camera argument
    camera_source = int(args.camera) if args.camera.isdigit() else args.camera
    
    # Initialize and run detector
    detector = PotholeDetector(
        model_path=args.model,
        camera_source=camera_source,
        conf_threshold=args.conf
    )
    
    detector.run(
        save_detections=not args.no_save,
        display=args.display
    )

Save this script as /root/pothole_detection_rpi.py and make it executable:

chmod +x /root/pothole_detection_rpi.py

Step 5: Test the System

Run a test detection:

cd /root
python3 pothole_detection_rpi.py --model pothole_model.pt --camera 0

You should see output like:

Loading model...
Model loaded successfully!
Initializing camera: 0
Camera initialized!
Starting pothole detection...
Press Ctrl+C to stop
Frame 30 | Avg FPS: 2.1 | Detections: 0
Frame 60 | Avg FPS: 2.3 | Detections: 1
Pothole detected! Saved to /root/pothole_detections/pothole_20260131_143022.jpg

Step 6: Auto-Start on Boot (Optional)

To make the detector run automatically when your Pi boots, create a systemd service:

nano /etc/systemd/system/pothole-detector.service

Add this content:

[Unit]
Description=Pothole Detection Service
After=network.target

[Service]
Type=simple
User=root
WorkingDirectory=/root
ExecStart=/usr/bin/python3 /root/pothole_detection_rpi.py --model /root/pothole_model.pt --camera 0
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target

Enable and start the service:

systemctl daemon-reload
systemctl enable pothole-detector.service
systemctl start pothole-detector.service

# Check status
systemctl status pothole-detector.service

# View logs
journalctl -u pothole-detector.service -f

Performance Expectations

Based on testing, here's what you can expect:

Platform	FPS	Latency	Power Usage
Raspberry Pi Zero	1-3 FPS	~500ms	~1.5W
Raspberry Pi 3	5-10 FPS	~150ms	~4W
Raspberry Pi 4	10-15 FPS	~80ms	~6W
With Coral USB Accelerator	20-30 FPS	~40ms	+2W

Note: 1-3 FPS is sufficient for dash cam applications since your car typically covers only a few meters per second at city speeds.

Configuration Tips

Adjusting Confidence Threshold

The confidence threshold determines how certain the model must be before reporting a detection:

# More sensitive (more false positives)
python3 pothole_detection_rpi.py --conf 0.15

# Less sensitive (fewer false positives)
python3 pothole_detection_rpi.py --conf 0.40

Start with 0.25 and adjust based on your results.

Using MotionEye RTSP Stream

If you want to use MotionEye's camera stream:

python3 pothole_detection_rpi.py --camera "rtsp://localhost:8554/stream"

Reducing Storage Usage

Detected images can accumulate quickly. Set up automatic cleanup:

# Add to crontab
crontab -e

# Delete detections older than 7 days, run daily at 3 AM
0 3 * * * find /root/pothole_detections -name "*.jpg" -mtime +7 -delete

Adding GPS Logging (Optional Enhancement)

To create a pothole map, add GPS coordinates to your detections:

# Install GPS library
pip3 install gpsd-py3 --break-system-packages

# Connect GPS module to Pi GPIO
# Then modify the script to log coordinates

Example GPS integration snippet:

import gpsd

# Connect to GPS daemon
gpsd.connect()

# In save_detection method:
packet = gpsd.get_current()
if packet.mode >= 2:  # 2D or 3D fix
    lat = packet.lat
    lon = packet.lon
    # Save coordinates with image

Troubleshooting Common Issues

Camera Not Detected

# List video devices
ls -l /dev/video*

# Test camera
v4l2-ctl --list-devices

# If using CSI camera, enable it
raspi-config
# Interface Options → Camera → Enable

Low FPS / Sluggish Performance

1. Reduce image size:

   • Change imgsz=416 to imgsz=320 in the script
   • Lower resolution = faster processing

2. Increase confidence threshold:

   • Use --conf 0.35 to reduce processing overhead

3. Skip frames:

   • Process every 2nd or 3rd frame instead of all frames

4. Consider upgrading:

   • Raspberry Pi 3/4 offers 3-5x better performance
   • Coral USB Accelerator adds hardware AI acceleration

Out of Memory Errors

# Increase swap size
dphys-swapfile swapoff
nano /etc/dphys-swapfile
# Set: CONF_SWAPSIZE=1024
dphys-swapfile setup
dphys-swapfile swapon
reboot

Model Loading Errors

# Verify model file exists and has correct size
ls -lh /root/pothole_model.pt

# Should be ~6MB for YOLOv8 Nano
# If corrupted, re-transfer from source

Real-World Usage Tips

Mounting in Your Vehicle

1. Position: Mount behind rearview mirror for unobstructed view
2. Angle: Point slightly downward (15-30°) to capture road ahead
3. Power: Use cigarette lighter USB adapter (2.5A minimum)
4. Stability: Use adhesive mount or suction cup for vibration dampening

Best Practices

• Clean the lens regularly: Dust and rain spots reduce accuracy
• Test in various conditions: Day, night, rain, shadows
• Review detections weekly: Adjust confidence threshold as needed
• Backup regularly: Copy detection images to cloud storage
• Share with authorities: Report severe potholes with GPS coordinates

Data Collection for Community Mapping

If you want to contribute to community pothole mapping:

1. Enable GPS logging (see optional enhancement above)
2. Export detections with coordinates
3. Upload to platforms like:
   • OpenStreetMap (with appropriate tags)
   • SeeClickFix (report infrastructure issues)
   • Local municipality reporting portals

Cost-Benefit Analysis

Total Investment: $45-80 Annual Pothole Damage Average: $300-500 per driver (AAA estimate) ROI: If this system helps you avoid just 1-2 serious pothole hits, it pays for itself!

Additional Benefits:

• Contributes to civic infrastructure data
• Educational project for learning AI/computer vision
• Can be adapted for other detection tasks
• Reusable hardware for future projects

Limitations and Future Improvements

Current Limitations

1. Speed: 1-3 FPS on Pi Zero means some potholes might be missed at highway speeds
2. Night Performance: Accuracy drops in low-light conditions
3. Weather: Heavy rain or snow affects detection reliability
4. False Positives: Shadows, road markings, or debris may be misidentified

Future Enhancements

1. Multi-Class Detection: Detect cracks, manholes, and other road hazards
2. Speed Integration: Adjust detection sensitivity based on vehicle speed
3. Cloud Sync: Automatic upload of detections with GPS coordinates
4. Real-Time Alerts: Audio/visual warning when approaching detected pothole
5. Model Improvement: Continuous training with your collected data
6. Edge TPU Integration: 10x faster inference with Google Coral

Conclusion

Building an AI-powered pothole detection dash cam is a practical application of edge computing and computer vision. While the Raspberry Pi Zero has limitations, it proves that sophisticated AI can run on affordable, compact hardware.

This project demonstrates:

• Edge AI is accessible: You don't need expensive hardware or cloud services
• Computer vision has practical applications: Beyond facial recognition and self-driving cars
• DIY can make a difference: Individual data collection can contribute to community infrastructure

Whether you're a hobbyist exploring AI, a civic tech enthusiast, or someone tired of pothole damage, this project offers a hands-on way to engage with cutting-edge technology while potentially making your daily commute safer.

Resources and Links

• Dataset: Kaggle Potholes Detection Dataset
• YOLOv8 Documentation: Ultralytics Docs
• MotionEyeOS: GitHub Repository
• Raspberry Pi Forums: raspberrypi.org/forums

Acknowledgments

• Ultralytics for the amazing YOLOv8 framework
• Kaggle for free GPU training resources and datasets
• Raspberry Pi Foundation for affordable computing hardware
• MotionEye Project for excellent camera software

---

Links

https://github.com/dhirajpatra/jupyter_notebooks/blob/main/DataScienceProjects/yolo/Pothole_Detection_from_Dash_Cam.ipynb

https://github.com/dhirajpatra/jupyter_notebooks/blob/main/DataScienceProjects/kaggle/pothole-detection-from-dash-cam.ipynb

Have you built this project or have improvements to suggest? Share your experience in the comments below!

Last updated: January 2026

Self-contained Raspberry Pi surveillance System Without Continue Internet

                                                                gemini generated

A self-contained Raspberry Pi surveillance system that:

1. Runs autonomously (like EyeOS) — camera always active and streaming.

2. Keeps searching for a known Wi-Fi hotspot (your phone).

3. Starts streaming automatically when the phone hotspot is available.

4. Lets you view the camera feed on your phone (via browser or app).

Here’s a detailed, production-ready setup:

---

⚙️ Step 1: Setup Raspberry Pi Camera & Software

1. Enable the camera

sudo raspi-config

• Go to Interface Options → Camera → Enable

• Reboot:

sudo reboot

2. Install dependencies

sudo apt update
sudo apt install python3-picamera2 python3-flask git -y

---

📸 Step 2: Create a Local Flask Streaming App

Create file /home/pi/camera_stream.py:

from flask import Flask, Response
from picamera2 import Picamera2
import time

app = Flask(__name__)
piCam = Picamera2()
piCam.configure(piCam.create_preview_configuration(main={"size": (640, 480)}))
piCam.start()

def gen_frames():
    while True:
        frame = piCam.capture_array()
        yield (b'--frame\r\n'
               b'Content-Type: image/jpeg\r\n\r\n' + 
               piCam.encode_jpeg(frame) + b'\r\n')

@app.route('/')
def index():
    return Response(gen_frames(), mimetype='multipart/x-mixed-replace; boundary=frame')

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8080)

Test it:

python3 /home/pi/camera_stream.py

Open in your phone browser (later):
http://:8080

---

🔁 Step 3: Auto-Connect to Your Phone’s Hotspot

1. Turn on your phone hotspot (note SSID & password).

2. Edit Wi-Fi config:

sudo nano /etc/wpa_supplicant/wpa_supplicant.conf

Add your phone hotspot details:

network={
    ssid="YourHotspotName"
    psk="YourHotspotPassword"
    priority=10
}

3. Add a Wi-Fi reconnect loop service to always search for it:

Create file /usr/local/bin/wifi_reconnect.sh:

#!/bin/bash
while true
do
    if ! iwgetid -r > /dev/null; then
        echo "No Wi-Fi. Trying to reconnect..."
        sudo wpa_cli -i wlan0 reconfigure
    fi
    sleep 15
done

Make it executable:

sudo chmod +x /usr/local/bin/wifi_reconnect.sh

---

🔄 Step 4: Autostart on Boot

Create systemd service /etc/systemd/system/camera.service:

[Unit]
Description=Camera Stream Service
After=network.target

[Service]
ExecStart=/usr/bin/python3 /home/pi/camera_stream.py
Restart=always
User=pi

[Install]
WantedBy=multi-user.target

And Wi-Fi watcher service /etc/systemd/system/wifi-reconnect.service:

[Unit]
Description=Wi-Fi Auto Reconnect

[Service]
ExecStart=/usr/local/bin/wifi_reconnect.sh
Restart=always
User=root

[Install]
WantedBy=multi-user.target

Enable both:

sudo systemctl enable camera.service
sudo systemctl enable wifi-reconnect.service
sudo systemctl start camera.service
sudo systemctl start wifi-reconnect.service

---

📱 Step 5: Access from Phone

When you enable your phone hotspot:

• Pi connects automatically (within ~15s).

• Visit http://192.168.43.x:8080 (Pi IP shown in hotspot device list).

• You’ll see live camera feed.

---

💡 Optional Enhancements

• Use ngrok or localtunnel to access remotely later (if internet is available).

• Add motion detection with motion or opencv-python.

• Use systemd watchdog or cron to reboot if Wi-Fi drops too long.

---

Here’s how to extend your setup to add motion detection + offline snapshot saving, all still working without internet and auto-starting when your phone hotspot appears.

---

⚙️ Step 6: Add Motion Detection with Snapshots

We’ll modify the Flask camera script to:

• Continuously capture frames.

• Compare each frame to detect movement.

• Save snapshots locally when motion occurs.

• Keep the live stream accessible at http://<pi_ip>:8080.

---

📄 Replace your /home/pi/camera_stream.py with this:

from flask import Flask, Response
from picamera2 import Picamera2
import cv2, numpy as np, time, os

app = Flask(__name__)
piCam = Picamera2()
piCam.configure(piCam.create_preview_configuration(main={"size": (640, 480)}))
piCam.start()

last_frame = None
save_path = "/home/pi/motion_captures"
os.makedirs(save_path, exist_ok=True)

def gen_frames():
    global last_frame
    while True:
        frame = piCam.capture_array()
        gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
        gray = cv2.GaussianBlur(gray, (21, 21), 0)

        if last_frame is None:
            last_frame = gray
            continue

        delta = cv2.absdiff(last_frame, gray)
        thresh = cv2.threshold(delta, 25, 255, cv2.THRESH_BINARY)[1]
        thresh = cv2.dilate(thresh, None, iterations=2)
        cnts, _ = cv2.findContours(thresh, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)

        for c in cnts:
            if cv2.contourArea(c) < 1500:  # ignore small movements
                continue
            timestamp = time.strftime("%Y%m%d-%H%M%S")
            filename = f"{save_path}/motion_{timestamp}.jpg"
            cv2.imwrite(filename, frame)
            print(f"Motion detected, snapshot saved: {filename}")
            time.sleep(1)  # avoid saving too many per second
            break

        last_frame = gray

        # Stream JPEG frame
        _, jpeg = cv2.imencode('.jpg', frame)
        yield (b'--frame\r\n'
               b'Content-Type: image/jpeg\r\n\r\n' + jpeg.tobytes() + b'\r\n')

@app.route('/')
def index():
    return Response(gen_frames(), mimetype='multipart/x-mixed-replace; boundary=frame')

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8080)

---

🗂️ Step 7: Verify Motion Capture Folder

All motion-triggered snapshots will be stored in:

/home/pi/motion_captures/

You can access them later using:

ls /home/pi/motion_captures/

Or even download them through SFTP from your phone if you use an app like CX File Explorer or Termius.

---

🌀 Step 8: Auto-Restart Service (if not already)

Your existing systemd service (camera.service) already keeps this running on boot and restart.
Just restart it once to load the new version:

sudo systemctl restart camera.service

---

✅ Final Behavior

1. Raspberry Pi boots up and keeps scanning for your phone’s hotspot.

2. Once your hotspot is on, it connects automatically.

3. The Flask camera service starts streaming on http://192.168.43.x:8080.

4. Any motion triggers snapshots saved under /home/pi/motion_captures/.

5. Works completely offline.

---

Satellite Image Browser Database with Chrome DevTools

 

This demo application DevTools and Chrome browser database example focused on handling large satellite imagery data. This will demonstrate advanced browser storage techniques and DevTools debugging for big data scenarios.I've created an advanced Satellite Image Browser Database that demonstrates sophisticated DevTools integration and large data handling in the browser. Here's what makes this example particularly interesting:

🔥 Key Features for DevTools & Large Data:

1. Advanced IndexedDB Implementation

• Stores high-resolution satellite images (up to 25MB each)
• Custom indexing on region, satellite type, and timestamp
• Real-time storage quota monitoring
• Compression ratio calculations

2. DevTools Integration Points

• Application Tab: Inspect IndexedDB structure and stored images
• Network Tab: Monitor image generation performance
• Memory Tab: Track heap usage with large image datasets
• Console: Detailed performance logging and debugging hooks
• Performance Tab: Analyze rendering bottlenecks with large grids

3. Large Data Handling Techniques

• Progressive image loading to prevent UI blocking
• Canvas-based image generation with realistic terrain patterns
• Memory usage warnings when storage exceeds 100MB
• Efficient blob storage with JPEG compression
• Performance monitoring with load time analytics

4. Advanced Browser Features

• Quality Settings: Preview (500KB) to Ultra (25MB) per image
• Realistic Metadata: GPS coordinates, cloud cover, processing times
• Region-Specific Rendering: Amazon, Sahara, Himalayas, Arctic, Great Barrier Reef
• Multiple Satellite Sources: Landsat, Sentinel-2, MODIS, WorldView-3

🛠️ DevTools Debugging Workflow:

1. Generate Ultra-quality images → Watch Network tab for performance
2. Open Application → IndexedDB → Inspect image blob storage
3. Check Memory tab → Monitor heap growth with large datasets
4. Use Console hooks: window.satelliteDB.stats() for live data
5. Performance tab → Analyze rendering bottlenecks in image grid

💡 Perfect for Chrome Developer Content:

This example showcases cutting-edge browser capabilities that would be excellent for the Chrome for Developers team:

• Modern storage APIs (IndexedDB with large blobs)
• Performance monitoring integration
• Memory management best practices
• Canvas API for realistic data generation
• Progressive enhancement techniques

🛰️ Real Satellite Image Sources Added:

1. Image Sources by Region:

• Amazon: Real rainforest and forest canopy aerial photography
• Sahara: Actual desert dunes and sand pattern imagery
• Himalayas: Genuine mountain peaks and snow-covered ranges
• Arctic: Real ice formations and polar landscapes
• Great Barrier Reef: Actual coral reef and ocean aerial shots

2. Enhanced Features:

• Cross-origin image loading with fallback handling
• Satellite overlay effects (scan lines, coordinates, timestamps)
• Source tracking - shows "Real Satellite" vs "Generated" in metadata
• Fallback system - generates patterns if images fail to load
• Smart caching with timestamp parameters to avoid browser cache issues

3. Professional Satellite Effects:

• Green scan lines overlay (like real satellite imagery)
• GPS coordinates display in corner
• ISO timestamp watermarking
• Proper image resizing and compression

4. Robust Error Handling:

• If any online image fails to load, it automatically falls back to generated patterns
• Console logging shows which images are fetched vs generated
• CORS-friendly image loading with proper headers

🔥 DevTools Integration Benefits:

1. Network Tab: See real HTTP requests to image services
2. Application Tab: Inspect actual satellite image blobs in IndexedDB
3. Console: Track which images loaded successfully vs fallbacks
4. Performance: Monitor real network latency and image processing times

Now when you click "Generate Images", you'll get actual satellite and aerial photography that looks professional and realistic! The app will automatically handle any loading issues and provide fallbacks, making it perfect for demonstrating real-world browser database scenarios with large image datasets.

The images now have that authentic satellite look with professional overlays and metadata tracking! 🌍✨

The app generates realistic satellite imagery with proper metadata, making it feel like a real-world geospatial application while demonstrating advanced browser database techniques that developers can learn from!

Code here in Github

Technical Challenges to keep Character Consistency Across Image and Video Generations

                                                Google Veo

Character/image consistency across video generations is a major challenge in current AI video models like Veo 3. Let me help you understand the technical approaches and architectures that could address this problem.

Core Technical Challenges

The inconsistency issue stems from several factors:

• Latent space drift: Each generation samples from slightly different regions of the learned latent space
• Temporal coherence: Models struggle to maintain identity across time steps
• Reference conditioning: Insufficient mechanisms to anchor generation to specific visual features

Promising Technical Approaches

1. Identity-Conditioned Diffusion Models

Architecture Components:

• Identity Encoder: Extract robust identity embeddings from reference images
• Cross-attention mechanisms: Inject identity features at multiple scales
• Temporal consistency layers: Ensure coherent identity propagation across frames

# Conceptual architecture
class IdentityConditionedVideoDiffusion:
    def __init__(self):
        self.identity_encoder = IdentityEncoder()  # ResNet/Vision Transformer
        self.temporal_unet = TemporalUNet3D()
        self.cross_attention = CrossAttentionLayers()
    
    def forward(self, reference_image, text_prompt, noise):
        identity_features = self.identity_encoder(reference_image)
        # Inject identity at multiple resolution levels
        return self.temporal_unet(noise, text_prompt, identity_features)

Key Innovation: Use contrastive learning to learn identity-preserving embeddings that remain consistent across different poses, lighting, and contexts.

2. Multi-Reference Fusion Networks

Approach: Combine multiple reference images to create a robust identity representation

• Attention-based fusion: Weight different reference views based on relevance
• 3D-aware identity modeling: Build 3D representations from 2D references
• Pose-disentangled features: Separate identity from pose/expression

3. ControlNet-Inspired Identity Control

Architecture:

• Identity ControlNet: Additional network branch that conditions on reference images
• Feature alignment: Align generated features with reference features at multiple scales
• Adaptive conditioning strength: Dynamically adjust identity influence

4. Advanced Temporal Modeling

Transformer-Based Approaches:

class TemporalIdentityTransformer:
    def __init__(self):
        self.spatial_attention = MultiHeadAttention()
        self.temporal_attention = TemporalAttention()
        self.identity_memory = IdentityMemoryBank()
    
    def forward(self, frames, reference_identity):
        # Maintain identity memory across frames
        identity_context = self.identity_memory.retrieve(reference_identity)
        return self.process_with_identity_context(frames, identity_context)

5. GAN-Based Identity Preservation

StyleGAN-Inspired Approach:

• Identity-aware latent codes: Map reference images to consistent latent codes
• Disentangled generation: Separate identity, pose, lighting, and background
• Temporal GAN: Extend StyleGAN with temporal consistency losses

Practical Implementation Strategy

Phase 1: Identity Encoding

1. Train robust identity encoder using:

   • Contrastive learning (SimCLR, CLIP-style)
   • Face recognition datasets (VGGFace2, MS-Celeb-1M)
   • Multi-view consistency losses

2. Identity Feature Extraction:

def extract_identity_features(reference_image):
    # Multi-scale feature extraction
    features = []
    for scale in [64, 128, 256, 512]:
        resized = resize(reference_image, scale)
        feat = identity_encoder(resized)
        features.append(feat)
    return concatenate(features)

Phase 2: Conditioning Mechanism

Implement identity injection at multiple levels:

• Coarse level: Global identity embeddings
• Fine level: Local feature matching and alignment
• Temporal level: Consistency across frame sequences

Phase 3: Training Strategy

# Multi-objective loss function
total_loss = (
    reconstruction_loss +
    identity_consistency_loss +
    temporal_coherence_loss +
    perceptual_loss +
    adversarial_loss
)

Novel Architecture Proposal

Hierarchical Identity-Preserved Video Diffusion (HIP-VD)

Core Innovation: Multi-level identity conditioning with temporal memory

1. Identity Pyramid Network: Extract identity at multiple resolutions
2. Temporal Identity Memory: Maintain consistent identity representations across frames
3. Adaptive Identity Injection: Dynamically control identity strength based on scene context

Technical Details:

• Use Vision Transformers for identity encoding
• Implement memory-augmented attention for temporal consistency
• Apply progressive training from single frames to full videos

Research Directions

1. Self-Supervised Identity Learning: Learn identity representations without explicit labels
2. Few-Shot Identity Adaptation: Quickly adapt to new identities with minimal examples
3. Cross-Modal Identity Transfer: Maintain identity across different modalities (sketch→photo→video)

Implementation Roadmap

1. Start with existing diffusion models (Stable Video Diffusion)
2. Add identity conditioning layers using ControlNet architecture
3. Implement temporal consistency mechanisms
4. Train on curated datasets with identity annotations
5. Fine-tune for specific use cases (characters, objects, scenes)

This approach combines the best of diffusion models, GANs, and transformer architectures to create a robust identity-consistent video generation system. Would you like me to elaborate on any specific component or help you implement a particular part of this architecture?

Let me walk you through the major deep learning methods and technologies that laid the foundation before transformers revolutionized the field.

1. Perceptrons and Multi-Layer Perceptrons (MLPs)

Timeline: 1950s-1980s

Core Concepts:

• Single Perceptron: Linear classifier with step activation
• Multi-Layer Perceptron: Multiple layers with non-linear activations
• Backpropagation Algorithm (1986): Key breakthrough for training deep networks

# Basic MLP structure
class MLP:
    def __init__(self):
        self.layers = [
            LinearLayer(input_dim, hidden_dim),
            ActivationLayer('sigmoid'),
            LinearLayer(hidden_dim, output_dim)
        ]

Limitations: Vanishing gradient problem, limited representational power

2. Convolutional Neural Networks (CNNs)

Timeline: 1980s-2010s (mainstream adoption)

Key Architectures Evolution:

LeNet (1998)

• First successful CNN for digit recognition
• Convolution → Pooling → Fully Connected pattern

AlexNet (2012) - ImageNet breakthrough

# AlexNet-style architecture
class AlexNet:
    def __init__(self):
        self.conv_layers = [
            Conv2D(96, kernel_size=11, stride=4),  # Large kernels
            MaxPool2D(3, stride=2),
            Conv2D(256, kernel_size=5, padding=2),
            Conv2D(384, kernel_size=3, padding=1),
            # ... more layers
        ]
        self.classifier = [
            Linear(9216, 4096),
            Dropout(0.5),  # Key innovation
            Linear(4096, 1000)
        ]

VGGNet (2014)

• Deeper networks with smaller 3x3 kernels
• Showed importance of depth

ResNet (2015)

• Skip connections solved vanishing gradient problem
• Enabled very deep networks (152+ layers)

class ResidualBlock:
    def forward(self, x):
        identity = x
        out = self.conv1(x)
        out = self.conv2(out)
        out += identity  # Skip connection
        return self.relu(out)

DenseNet, EfficientNet, etc.

• Various architectural improvements

3. Recurrent Neural Networks (RNNs)

Timeline: 1980s-2010s

Vanilla RNN

class VanillaRNN:
    def forward(self, x_t, h_prev):
        h_t = tanh(W_hh @ h_prev + W_xh @ x_t + b)
        return h_t

Problems: Vanishing gradients, short-term memory

Long Short-Term Memory (LSTM) - 1997

Breakthrough: Solved vanishing gradient problem for sequences

class LSTMCell:
    def forward(self, x_t, h_prev, c_prev):
        # Forget gate
        f_t = sigmoid(W_f @ [h_prev, x_t] + b_f)
        # Input gate
        i_t = sigmoid(W_i @ [h_prev, x_t] + b_i)
        # Output gate
        o_t = sigmoid(W_o @ [h_prev, x_t] + b_o)
        # Cell state update
        c_t = f_t * c_prev + i_t * tanh(W_c @ [h_prev, x_t] + b_c)
        h_t = o_t * tanh(c_t)
        return h_t, c_t

Gated Recurrent Unit (GRU) - 2014

• Simplified version of LSTM
• Fewer parameters, similar performance

Bidirectional RNNs

• Process sequences in both directions
• Better context understanding

4. Autoencoders and Dimensionality Reduction

Timeline: 2000s-2010s

Basic Autoencoder

class Autoencoder:
    def __init__(self):
        self.encoder = Sequential([
            Linear(784, 400),
            ReLU(),
            Linear(400, 64)  # Bottleneck
        ])
        self.decoder = Sequential([
            Linear(64, 400),
            ReLU(),
            Linear(400, 784)
        ])

Variational Autoencoders (VAE) - 2013

• Probabilistic approach to representation learning
• Reparameterization trick for backpropagation through stochastic nodes

class VAE:
    def encode(self, x):
        mu = self.encoder_mu(x)
        logvar = self.encoder_logvar(x)
        return mu, logvar
    
    def reparameterize(self, mu, logvar):
        std = torch.exp(0.5 * logvar)
        eps = torch.randn_like(std)
        return mu + eps * std  # Reparameterization trick

Denoising Autoencoders

• Learn robust representations by reconstructing from corrupted inputs

5. Generative Adversarial Networks (GANs) - 2014

Breakthrough: Game-theoretic approach to generative modeling

class GAN:
    def __init__(self):
        self.generator = Generator()
        self.discriminator = Discriminator()
    
    def train_step(self, real_data):
        # Train Discriminator
        fake_data = self.generator(noise)
        d_loss = -log(D(real)) - log(1 - D(fake))
        
        # Train Generator
        g_loss = -log(D(G(noise)))

Major GAN Variants:

• DCGAN (2015): CNN-based architecture
• StyleGAN (2018): Style-based generation
• CycleGAN (2017): Unpaired image-to-image translation
• Progressive GAN: Gradual resolution increase

6. Deep Belief Networks (DBNs)

Timeline: 2000s

Structure: Stack of Restricted Boltzmann Machines (RBMs)

• Layer-wise pretraining: Train each RBM separately
• Fine-tuning: Backpropagation on entire network

class RBM:
    def __init__(self, visible_units, hidden_units):
        self.W = torch.randn(visible_units, hidden_units)
        self.contrastive_divergence_training()

7. Attention Mechanisms (Pre-Transformer)

Timeline: 2014-2017

Bahdanau Attention (2014)

class BahdanauAttention:
    def forward(self, decoder_hidden, encoder_outputs):
        # Compute attention scores
        scores = self.attention_net(decoder_hidden, encoder_outputs)
        weights = softmax(scores)
        context = sum(weights * encoder_outputs)
        return context

Luong Attention (2015)

• Different scoring functions (dot, general, concat)

Self-Attention (2016)

• Attention within the same sequence
• Predecessor to transformer self-attention

8. Reinforcement Learning Integration

Deep Q-Networks (DQN) - 2013

class DQN:
    def __init__(self):
        self.q_network = CNN()  # For Atari games
        self.target_network = CNN()
        self.replay_buffer = ReplayBuffer()

Policy Gradient Methods

• REINFORCE: Basic policy gradient
• Actor-Critic: Combines value and policy learning
• PPO, A3C: Advanced policy optimization

9. Optimization and Training Techniques

Activation Functions Evolution:

• Sigmoid/Tanh → ReLU → LeakyReLU → ELU → Swish/GELU

Normalization Techniques:

# Batch Normalization (2015)
class BatchNorm:
    def forward(self, x):
        mean = x.mean(dim=0)
        var = x.var(dim=0)
        return (x - mean) / sqrt(var + eps)

# Layer Normalization (2016) - Important for RNNs
class LayerNorm:
    def forward(self, x):
        mean = x.mean(dim=-1, keepdim=True)
        var = x.var(dim=-1, keepdim=True)
        return (x - mean) / sqrt(var + eps)

Advanced Optimizers:

• SGD → Momentum → AdaGrad → Adam → AdamW

10. Regularization Techniques

# Dropout (2012)
class Dropout:
    def forward(self, x, training=True):
        if training:
            mask = torch.bernoulli(torch.full_like(x, 1-self.p))
            return x * mask / (1 - self.p)
        return x

# Weight Decay
optimizer = Adam(params, lr=0.001, weight_decay=1e-4)

Timeline Summary

1950s: Perceptron
1980s: Backpropagation, CNNs (LeNet)
1990s: LSTM, SVMs
2000s: Deep Belief Networks, RBMs
2006: Deep Learning Renaissance (Hinton et al.)
2012: AlexNet (CNN breakthrough)
2013: VAE, DQN
2014: GAN, Attention (Bahdanau)
2015: ResNet, Batch Norm
2016: Layer Norm, Self-Attention concepts
2017: Attention is All You Need (Transformer) 🚀

Key Limitations That Led to Transformers

1. RNNs: Sequential processing, vanishing gradients
2. CNNs: Limited receptive fields, not suitable for sequences
3. Attention + RNN: Still sequential bottleneck
4. Memory: Limited long-range dependencies

Transformers solved these by:

• Pure attention mechanisms (no recurrence)
• Parallel processing
• Unlimited context (in theory)
• Better gradient flow

Each of these pre-transformer technologies contributed crucial insights that eventually culminated in the transformer architecture. 

Google Cloud VertexAI AutoML Vision Identifying Damaged Cars

Vertex AI brings together the Google Cloud services for building ML under one, unified UI and API. In Vertex AI, you can now easily train and compare models using AutoML or custom code training and all your models are stored in one central model repository. These models can now be deployed to the same endpoints on Vertex AI.

AutoML Vision helps anyone with limited Machine Learning (ML) expertise train high quality image classification models. In this hands-on lab, you will learn how to produce a custom ML model that automatically recognizes damaged car parts. Since the time it takes to train the model is above the time limit of the lab, you will interact and request predictions from a hosted model in a different project trained on the same dataset. You will then tweak the values of the data for the prediction request and examine how it changes the resulting prediction from the model.

Screen Shots from Google Cloud 

Objectives

In this lab, you learn how to:

• Upload a labeled dataset to Cloud Storage using a CSV file and connect it to Vertex AI as a Managed Dataset.
• Inspect uploaded images to ensure there are no errors in your dataset.
• Kick off an AutoML Vision model training job.
• Request predictions from a hosted model trained on the same dataset.

Note: Should you have trouble viewing images contained in the lab below, click on the images themselves to enlarge.

Setup and requirements

For each lab, you get a new Google Cloud project and set of resources for a fixed time at no cost.

1. Sign in to Qwiklabs using an incognito window.

2. Note the lab's access time (for example, 1:15:00), and make sure you can finish within that time.
   There is no pause feature. You can restart if needed, but you have to start at the beginning.

3. When ready, click Start lab.

4. Note your lab credentials (Username and Password). You will use them to sign in to the Google Cloud Console.

5. Click Open Google Console.

6. Click Use another account and copy/paste credentials for this lab into the prompts.
   If you use other credentials, you'll receive errors or incur charges.

7. Accept the terms and skip the recovery resource page.

Note: Do not click End Lab unless you have finished the lab or want to restart it. This clears your work and removes the project.

Activate Cloud Shell

Cloud Shell is a virtual machine that contains development tools. It offers a persistent 5-GB home directory and runs on Google Cloud. Cloud Shell provides command-line access to your Google Cloud resources. gcloud is the command-line tool for Google Cloud. It comes pre-installed on Cloud Shell and supports tab completion.

1. Click the Activate Cloud Shell button () at the top right of the console.

2. Click Continue.
   It takes a few moments to provision and connect to the environment. When you are connected, you are also authenticated, and the project is set to your PROJECT_ID.

Sample commands

• List the active account name:

gcloud auth list

Copied!

(Output)

Credentialed accounts:
 - <myaccount>@<mydomain>.com (active)

(Example output)

Credentialed accounts:
 - google1623327_student@qwiklabs.net

• List the project ID:

gcloud config list project

Copied!

(Output)

[core]
project = <project_ID>

(Example output)

[core]
project = qwiklabs-gcp-44776a13dea667a6

Note: Full documentation of gcloud is available in the gcloud CLI overview guide.

Task 1. Upload training images to Cloud Storage

In this task you will upload the training images you want to use to Cloud Storage. This will make it easier to import the data into Vertex AI later.

To train a model to classify images of damaged car parts, you need to provide the machine with labeled training data. The model will use the data to develop an understanding of each image, differentiating between car parts and those with damages on them.

Note: For the purposes of this lab, you won’t need to label images because a labeled dataset (i.e. image plus label) in a CSV file has been provided. The next section outlines the steps to use the CSV file.

In this example, your model will learn to classify five different damaged car parts: bumper, engine compartment, hood, lateral, and windshield.

Create a Cloud Storage bucket

1. To start, open a new Cloud Shell window and execute the following commands to set some environment variables:

export PROJECT_ID=$DEVSHELL_PROJECT_ID
export BUCKET=$PROJECT_ID

Copied!

2. Next, to create a Cloud Storage bucket, execute the following command:

   gsutil mb -p $PROJECT_ID \
    -c standard    \
    -l REGION \
    gs://${BUCKET}

   Copied!

Upload car images to your Storage Bucket

The training images are publicly available in a Cloud Storage bucket. Again, copy and paste the script template below into Cloud Shell to copy the images into your own bucket.

1. To copy images into your Cloud Storage bucket, execute the following command:

gsutil -m cp -r gs://car_damage_lab_images/* gs://${BUCKET}

Copied!

2. In the navigation pane, click Cloud Storage > Buckets.

3. Click the Refresh button at the top of the Cloud Storage browser.

4. Click on your bucket name. You should see five folders of photos for each of the five different damaged car parts to be classified:

5. Optionally, you can click one of the folders and check out the images inside.

Great! Your car images are now organized and ready for training.

Click Check my progress to verify the objective.

Upload car images to your Storage Bucket

Task 2. Create a dataset

In this task you create a new dataset and connect your dataset to your training images to allow Vertex AI to access them.

Normally, you would create a CSV file where each row contains a URL to a training image and the associated label for that image. In this case, the CSV file has been created for you; you just need to update it with your bucket name and upload the CSV file to your Cloud Storage bucket.

Update the CSV file

Copy and paste the script templates below into Cloud Shell and press Enter to update, and upload the CSV file.

1. To create a copy of the file, execute the following command:

gsutil cp gs://car_damage_lab_metadata/data.csv .

Copied!

2. To update the CSV with the path to your storage, execute the following command:

sed -i -e "s/car_damage_lab_images/${BUCKET}/g" ./data.csv

Copied!

3. Verify your bucket name was inserted into the CSV properly:

cat ./data.csv

Copied!

4. To upload the CSV file to your Cloud Storage bucket, execute the following command:

gsutil cp ./data.csv gs://${BUCKET}

Copied!

5. Once the command completes, click the Refresh button at the top of the Cloud Storage bucket and open your bucket.

6. Confirm that the data.csv file is listed in your bucket.

Create a managed dataset

1. In the Google Cloud Console, on the Navigation menu () click Vertex AI > Dashboard.

2. Click Enable all recommended API.

3. From the Vertex AI navigation menu on the left, click Datasets.

4. At the top of the console, click + Create.

5. For Dataset name, type damaged_car_parts.

6. Select Image classification (Single label). (Note: in your own projects, you may want to check the "Multi-label Classification" box if you're doing multi-class classification).

7. Select region as REGION

8. Click Create.

Connect your dataset to your training images

In this section, you will choose the location of your training images that you uploaded in the previous step.

1. In the Select an import method section, click Select import files from Cloud Storage.

2. In the Select import files from Cloud Storage section, click Browse.

3. Follow the prompts to navigate to your storage bucket and click your data.csv file. Click Select.

4. Once you've properly selected your file, a green checkbox appears to the left of the file path. Click Continue to proceed.

Note: It will take around 9 to 12 minutes for your images to import and be aligned with their categories. You’ll need to wait for this step to complete before checking your progress.

5. Once the import has completed, prepare for the next section by clicking the Browse tab. (Hint: You may need to refresh the page to confirm.)

Click Check my progress to verify the objective.

Create a dataset

Task 3. Inspect images

In this task, you examine the images to ensure there are no errors in your dataset.

Check image labels

1. If your browser page has refreshed, click Datasets , select your image name, and then click Browse.

2. Under Filter labels, click any one of the labels to view the specific training images.

Note: If you were building a production model, you'd want at least 100 images per label to ensure high accuracy. This is just a demo so only 20 images of each type were used so the model could train quickly.

3. If an image is labeled incorrectly, you can click on it to select the correct label or delete the image from your training set:

4. Next, click on the Analyze tab to view the number of images per label. The Label Stats window appears on the right side of your browser.

Note: If you need help labeling your dataset, Vertex AI Labeling Services lets you work with human labelers to generate highly accurate labels.

Task 4. Train your model

You're ready to start training your model! Vertex AI handles this for you automatically, without requiring you to write any of the model code.

1. From the right-hand side, click Train New Model.

2. From the Training method window, leave the default configurations and select AutoML as the training method. Click Continue.

3. From the Model details window, enter a name for your model, use: damaged_car_parts_model. Click Continue.

4. From the Training options window, click Continue.

5. From the Explainability window, click continue and for Compute and pricing window, set your budget to 8 maximum node hours.

6. Click Start Training.

Note: Model training can take longer than the allotted time to complete the lab. The model does not need to finish training to continue to the next section.

Click Check my progress to verify the objective.

Train your model

Task 5. Request a prediction from a hosted model

For the purposes of this lab, a model trained on the exact same dataset is hosted in a different project so that you can request predictions from it while your local model finishes training, as it is likely that the local model training will exceed the limit of this lab.

A proxy to the pre-trained model is set up for you so you don't need to run through any extra steps to get it working within your lab environment.

To request predictions from the model, you will send predictions to an endpoint inside of your project that will forward the request to the hosted model and return back the output. Sending a prediction to the AutoML Proxy is very similar to the way that you would interact with your model you just created, so you can use this as practice.

Get the name of AutoML proxy endpoint

1. In the Google Cloud Console, on the Navigation menu (≡) click Cloud Run.

2. Click automl-proxy.

3. Copy the URL to the endpoint. It should look something like: https://automl-proxy-xfpm6c62ta-uc.a.run.app.

You will use this endpoint for the prediction request in the next section.

Create a prediction request

1. Open a new Cloud Shell window.

2. On the Cloud Shell toolbar, click Open Editor.

3. Click File > New File.

4. Click on the link, copy the file content into the new file you just created:

5. Save the file and name it payload.json.

For reference, the content you supplied is a Base64 string from the following image.

6. Next, set the following environment variables. Copy in your AutoML Proxy URL you retrieved in earlier.

AUTOML_PROXY=<automl-proxy url>
INPUT_DATA_FILE=payload.json

Copied!

7. Perform a API request to the AutoML Proxy endpoint to request the prediction from the hosted model:

curl -X POST -H "Content-Type: application/json" $AUTOML_PROXY/v1 -d "@${INPUT_DATA_FILE}"

Copied!

If you ran a successful prediction, your output should resemble the following:

{"predictions":[{"confidences":[0.951557755],"displayNames":["bumper"],"ids":["1960986684719890432"]}],"deployedModelId":"4271461936421404672","model":"projects/1030115194620/locations/us-central1/models/2143634257791156224","modelDisplayName":"damaged_car_parts_vertex","modelVersionId":"1"}

For this model, the prediction results are pretty self-explanatory. The displayNames field should correctly predict a bumper with a high confidence threshold. Now, you can change the Base64 encoded image value in the JSON file you created.

Click Check my progress to verify the objective.

Create the prediction request

8. Right-click on each image below, then select Save image As….

9. Follow the prompts to save each image with a unique name. (Hint: Assign a simple name like 'Image1' and 'Image2' to assist with uploading).

 

10. Open the Base64 Image Encoder follow the instructions to upload and encode an image to a Base64 string.

11. Replace the Base64 encoded string value in the content field in your JSON payload file, and run the prediction again. Repeat for the other image(s).

How did your model do? Did it predict all three images correctly? You should see the the following outputs, respectively:

{"predictions":[{"ids":["5419751198540431360"],"confidences":[0.985487759],"displayNames":["engine_compartment"]}],"deployedModelId":"4271461936421404672","model":"projects/1030115194620/locations/us-central1/models/2143634257791156224","modelDisplayName":"damaged_car_parts_vertex","modelVersionId":"1"}

{"predictions":[{"displayNames":["hood"],"ids":["3113908189326737408"],"confidences":[0.962432086]}],"deployedModelId":"4271461936421404672","model":"projects/1030115194620/locations/us-central1/models/2143634257791156224","modelDisplayName":"damaged_car_parts_vertex","modelVersionId":"1"}

Task 6. Review

In this lab, you learned how to train your own custom machine learning model and generate predictions on hosted model via an API request. You uploaded training images to Cloud Storage and used a CSV file for Vertex AI to find these images. You inspected the labeled images for any discrepancies before finally evaluating a trained model. Now you've got what it takes to train a model on your own image dataset!

Next steps / learn more

• Watch the intro video to Cloud AutoML.
• Learn more about how AutoML Vision works by listening to the GoogleCloudPlatform Podcast episode.
• Read the Cloud AutoML announcement blog post.
• Learn how to perform each step with the API.

Google QuikLabs courtsy.