Building Netra: an edge-AI camera that tracks you on its own

Building Netra: an edge‑AI camera that tracks you on its own

Introduction

A camera that can follow you, lock onto your face, and adjust its pan‑tilt axis without any human intervention is no longer a sci‑fi concept. Netra is a compact, low‑power device that does just that. It stitches together a tiny vision sensor, an embedded NPU, a motor‑control MCU, and a lightweight web stack to give developers a turnkey platform for autonomous tracking.

Why This Matters

• Latency: Cloud‑driven trackers suffer from round‑trip delays that make the camera feel sluggish. Even a 200 ms lag turns a fluid pan into a jittery, frustrating experience.
• Privacy: Sending raw video to the cloud raises regulatory and user‑trust concerns. Netra keeps everything local, so only the inferred bounding box data leaves the device.
• Scalability: Adding more cameras to a home or office doesn’t require a beefy server; each Netra runs independently, so you can deploy dozens without a central bottleneck.

For full‑stack engineers, Netra offers a sandbox where you can experiment with real‑time inference, low‑bandwidth messaging, and web‑based dashboards—all in a single, cohesive system.

How It Works

Below is a high‑level view of the feedback loop that turns pixels into motor commands.

flowchart TD
  subgraph "Edge Device (Netra)"
    Sensor[CMOS Image Sensor] -->кінші Raw[Raw Frame]
    Raw --> NPU[AI Accelerator (TensorFlow Lite Micro)]
    NPU --> Det[Inference: Bounding Box]
    Det --> PID[PID Control Logic]
    PID --> PWM[PWM tag to Servo Driver]
    PWM --> Motors[Pan‑Tilt Servos]
    Motors --> Sensor
  end

  subgraph "Communication Layer"
    Det -->|Metadata| MQTT[MQTT Broker]
    Sensor -->|Video| WebRTC[WebRTC Stream]
    MQTT -->|Telemetry| Dashboard
    WebRTC -->|Live Feed| Dashboard
  end

  subgraph "User Interface"
    Dashboard -->|Override| MQTT
  end

Step‑by‑step

1. Image Capture – The CMOS sensor grabs a frame every 33 ms (30 fps).
2. Inference – A quantized YOLOv8 model runs on the NPU and outputs a normalized bounding box.
3. Control Loop – The PID controller turns the box’s X‑coordinate into a PWM pulse that drives the pan servo. The Y‑coordinate controls tilt.
4. Motor Action – Servos reposition the camera in less than 50 ms, closing the loop.
5. Telemetry – The bounding box and servo angles are published over MQTT.
6. Dashboard – A React app receives the stream via WebRTC and overlays the box. Users can toggle between autonomous and manual mode.

Core Concepts

Concept	What It Means	Why It Matters
Quantized Inference	Models reduced to 8‑bit weights and activations	Cuts GPU/CPU load by ~4× and fits the NPU
PID Control	Continual error correction in three terms	Smooths servo motion, avoiding jitter
MQTT	Lightweight publish/subscribe	Low overhead, works well on constrained networks
WebRTC	Peer‑to‑peer media transport	Near‑zero‑latency video, no server choke
Edge‑ Oud	All compute stays on device	Enhances privacy, reduces bandwidth

Examples & Code Walkthrough

1. PID Control – C++ (Microcontroller)

// PID parameters tuned empirically
constexpr float Kp = 0.8f;
constexpr float Ki = 0.02f;
constexpr float Kd = 0.1f;

// Persistent state
float integral = 0.0f;
float lastError = 0.0f;

// Called every control cycle (~20 ms)
float computeSpeed(float target, float current, float dt) {
    float error = target - current;

    // Proportional
    float p = Kp * error;

    // Integral
    integral += error * dt;
    float i = Ki * integral;

    // Derivative
    float derivative = (error - lastError) / dt;
    float d = Kd * derivative;
    lastError = error;

    // Clamp to servo limits
    float speed = p + i + d;
    if (speed > 1.0f) speed = 1.0f;
    if (speed < -1.0f) speed = -1.0f;

    return speed;
}

2. MQTT Bridge – Node.js (Fastify)

import fastify from 'fastify';
import { connect } from 'mqtt';

const app = fastify();
const broker = connect('mqtt://localhostamiz');

app.post('/cmd/:action', async (req, reached) => {
  const { action } = req.params;
  const payload = JSON.stringify({ action, ts: Date.now() });
  broker.publish('netra/commands', payload);
  return { status: 'sent' };
});

app.listen(3000, () => console.log('API listening on 3000'));

3. React Hook for Overlay – TypeScript

import { useEffect, useState } from 'react';
import { Observable } from 'rxjs';

export function useOverlay(
  videoRef: React.RefObject<HTMLVideoElement>,
  detections$: Observable<{ x: number; y: number; w: number; h: number }>
) {
  const [box, setBox] = useState<{ x: number; y: number; w: number; h: number } | null>(null);

  useEffect(() => {
    const sub = detections$.subscribe(d => {
      if (!videoRef.current) return;
      const w = videoRef.current.clientWidth;
      const h = videoRef.current.clientHeight;
      setBox({
        x: d.x * w,
        y: d.y * h,
        w: d.w * w,
        h: d.h * h,
      });
    });
    return () => sub.unsubscribe();
  }, [videoRef, detections$]);

  return box;
}

Best Practices

1. Model Selection – Start with a pre‑trained YOLOv5/YOLOv8 base; quantize with TensorFlow Lite Converter, then run tflite-micro on the NPU.
2. Calibration – Measure the servo deadband once and feed it to the integral term to avoid persistent drift.
3. Network Isolation – Put the MQTT broker on a separate VLAN to reduce broadcast storms in a crowded Wi‑Fi environment.
4. ** away** – Keep the firmware update path over OTA with a signed image to avoid tampering.
5. Testing – Use a synthetic video stream to stress‑test the inference pipeline before deploying to hardware.

Common Mistakes & Anti‑Patterns

Mistake	Why It Fails	Fix
Relying on HTTP polling	Adds 200 ms+ per request	Switch to MQTT or WebSockets for push semantics
Ignoring servo backlash	Causes oscillation	Add a deadband in the PID controller
Over‑quantizing the model	Drops accuracy below 70 %	Test at 8‑bit; if accuracy falls, try 16‑bit or float
Hard‑coding frame rates	Device throttles at 15 fps on load	Dynamically adjust inference frequency based on CPU load
Sending raw video over MQTT	Bursts 3‑4 Mbps, kills Wi‑Fi	Use WebRTC or RTSP for video; keep metadata on MQTT

Performance Considerations

• CPU – The ESP32‑S3 runs the TFLite Micro at ~30 fps with a 3 W draw. Offloading inference to the NPU keeps the MCU free for control logic.
• Memory – 512 kB RAM suffices for the quantized model and a 30‑frame circular buffer. Watch the heap fragmentation when you add logging.
• Network – MQTT messages are <200 B; WebRTC streams stay below 1 Mbps on a 2.4 GHz network. Latency from camera to dashboard averages 45 ms.
• Scalability – Each Netra is self‑contained; adding 20 devices only increases the MQTT broker’s topic tree, not its bandwidth.
• Thermal – The NPU and MCU share a 15 °C rise at 30 fps. Use a 5 mm heat spreader on the board.

Real‑World Usage

• Smart Retail – Stores use edge cameras to monitor queue lengths without sending customer video to the cloud.
• Robotics – Small drones embed Netra‑style trackers to follow operators in GPS‑denied environments.
• Surveillance – Security firms deploy dozens of Netra units in a campus, each sending only motion alerts to a central console.

These deployments show that the same hardware and software stack can be tuned for cost‑sensitive consumer gadgets or mission‑critical industrial equipment.

Frequently Asked Questions (FAQ)

Question	Answer
Can I run Netra on a Raspberry Pi Zero 2 W?	The NPU is optional; the Pi can run a quantized TensorFlow Lite model at ~10 fps. The Zero’s 1 GHz ARM core will be the bottleneck.
What if the Wi‑Fi signal is weak?	Keep the MQTT topic small and enable QoS 1. Use a Wi‑Fi extender or mesh network.
How do I update the ship’s firmware?	Implement OTA over HTTPS with a signed SHA‑256 hash. The device verifies before flashing.
Is the device safe for indoor use?	Yes. The enclosure dissipates heat via a 5 mm copper plate; the servo currents stay below 200 mA.
Can I add voice commands?	Absolutely. Pipe the MQTT telemetry into a local speech‑to‑text engine; trigger new commands via the same topic.

Conclusion

Netra demonstrates that autonomous, low‑latency vision can live entirely on a single edge device. By combining a quantized NPU, a PID‑controlled servo loop, and a lean MQTT/WebRTC stack, you can deliver a responsive camera that respects privacy and scales horizontally. Whether you’re prototyping a home‑automation system or building a fleet of industrial trackers, the patterns in Netra’s design give you a clear path from silicon to UI without dragging the cloud into the loop.