espdl – Model Inference

[中文]

The espdl module runs ESP-DL .espdl models on captured images. It provides task-specific wrappers for object detection (PedestrianDetect, ESPDet, YOLO11), pose estimation (YOLO11nPose), image classification (ImageNetCls), and single-object tracking (NanoTrack), plus Model for exposing raw ESP-DL output tensors to Python.

Button-Confirmed Single-Object Tracking

espdl.NanoTrack runs NanoTrack V2 full-INT8 models. Copy the three .espdl files from models/nanotrack_v2/ to /sdcard/nanotrack_v2/. Local quantization experiments live under ignored build/experiments/nanotrack-v2 and are not included in a clean checkout. Use the ESP32-P4 exports for P4/S31; the mixed INT16 experiment exports are not accepted by this wrapper. S3 requires target-specific quantization and separate validation.

example/03-Machine-Learning/00-ESP-DL/nanotrack.py displays a centered 64x64 selection box. Place the target inside it and press BOOT (GPIO35 on ESP32-P4X-EYE and P4 Function EV Board) to start tracking; press again to return to selection. Missed matches keep the original template and continue local searching. The camera image is displayed at native size, center-cropped on smaller LCDs. Change BUTTON_PIN and BOX_SIZE for other hardware or target sizes. Inference precedes drawing so overlays do not enter model inputs.

The example uses a configurable MATCH_THRESHOLD = 0.95. The displayed match value is a model score, not a target-identity probability.

from machine import Pin
import display
import espdl
import sensor
import time


MODEL_DIR = "/sdcard/nanotrack_v2"
# Active-low BOOT button on ESP32-P4X-EYE and P4 Function EV Board.
# Change this pin for other boards.
BUTTON_PIN = 35
BOX_SIZE = 64
DEBOUNCE_MS = 30
# A match score is not a calibrated target-identity probability.
MATCH_THRESHOLD = 0.95


lcd = None
tracker = None
try:
    button = Pin(BUTTON_PIN, Pin.IN, Pin.PULL_UP)
    sensor.reset()
    sensor.set_pixformat(sensor.RGB565)
    sensor.set_framesize(sensor.QVGA)
    sensor.skip_frames(time=1000)
    lcd = display.Display()
    tracker = espdl.NanoTrack(
        MODEL_DIR + "/template_deploy.espdl",
        MODEL_DIR + "/search_deploy.espdl",
        MODEL_DIR + "/head_deploy.espdl",
    )
    frame = sensor.snapshot()
    width, height = frame.width(), frame.height()
    canvas = frame.copy()
    # Center at native size; smaller displays show the center crop.
    visible_w = min(width, lcd.width())
    visible_h = min(height, lcd.height())
    view = ((width - visible_w) // 2, (height - visible_h) // 2,
            visible_w, visible_h)
    offset_x = (lcd.width() - visible_w) // 2
    offset_y = (lcd.height() - visible_h) // 2
    size = min(BOX_SIZE, visible_w, visible_h)
    roi = ((width - size) // 2, (height - size) // 2, size, size)
    tracking = False
    last_button = stable_button = button.value()
    changed_at = time.ticks_ms()
    # A button held at startup must be released before it can confirm.
    armed = stable_button == 1

    while True:
        now = time.ticks_ms()
        level = button.value()
        pressed = False
        if level != last_button:
            last_button = level
            changed_at = now
        if level != stable_button and time.ticks_diff(now, changed_at) >= DEBOUNCE_MS:
            stable_button = level
            if level:
                armed = True
            elif armed:
                armed = False
                pressed = True

        if pressed:
            if tracking:
                tracking = False
            else:
                canvas.draw_image(frame, 0, 0)
                canvas.draw_string(view[0] + 2, view[1] + 2, "Initializing...",
                                   color=(255, 255, 0))
                lcd.write(canvas, x=offset_x, y=offset_y, roi=view, fit=False)
                # Use the last displayed raw frame, without any annotations.
                tracker.init(frame, roi)
                tracking = True

        frame = sensor.snapshot()
        begin = time.ticks_ms()
        result = tracker.track(frame, threshold=MATCH_THRESHOLD) if tracking else None
        canvas.draw_image(frame, 0, 0)
        if result is not None:
            x, y, w, h, score = result
            message = "Tracking - BOOT: select"
            canvas.draw_rectangle(x, y, w, h, color=(0, 255, 0), thickness=2)
            canvas.draw_string(view[0] + 2, view[1] + visible_h - 14,
                               "match %.2f  %d ms" % (score, time.ticks_diff(time.ticks_ms(), begin)),
                               color=(0, 255, 0))
        elif tracking:
            # Preserve the template and retry near the last accepted position.
            message = "Lost - searching..."
        else:
            message = "Aim box, press BOOT"
            canvas.draw_rectangle(roi, color=(255, 255, 0), thickness=2)
        canvas.draw_string(view[0] + 2, view[1] + 2, message, color=(255, 255, 255))
        lcd.write(canvas, x=offset_x, y=offset_y, roi=view, fit=False)
        time.sleep_ms(0 if tracking else 10)
finally:
    try:
        if tracker is not None:
            tracker.deinit()
    finally:
        if lcd is not None:
            lcd.deinit()

init(image, roi) caches target features, then frees the template model. Search and head models remain loaded; selecting a new target reloads the template model temporarily. Use unannotated RGB565 or grayscale images of the same dimensions throughout a track. The returned tuple is (x, y, w, h, score); below-threshold results return None without updating the previous search state. Scores do not guarantee target visibility. This is local single-object tracking without automatic global reacquisition or training of network weights.

Raw Output Tensors

import sensor, espdl

model = espdl.Model("/sdcard/custom.espdl", mean=(0, 0, 0), std=(255, 255, 255), letterbox=True)
try:
    print("inputs:", model.inputs())
    print("outputs:", model.outputs())
    img = sensor.snapshot()
    outputs = model.predict(img)
    for name, tensor in outputs.items():
        _, shape, dtype, exponent, raw = tensor
        print(name, shape, dtype, exponent, len(raw))
finally:
    model.deinit()

Use Model when the built-in task wrappers do not match a model’s output layout. ESP-DL still performs image preprocessing and inference; predict() returns the raw output tensors so Python code can run the model-specific decode. inputs() and outputs() return TensorInfo tuples (name, shape, dtype, exponent). predict() returns RawTensor tuples (name, shape, dtype, exponent, bytes) keyed by output tensor name. Decode the raw bytes according to dtype, apply the ESP-DL power-of-two exponent scale, and run logic such as sigmoid, box decode, NMS, softmax, or top-k selection. When letterbox=True, remove the model-input padding before mapping coordinates back to the source image.

Object Detection

import sensor, espdl

det = espdl.ESPDet("/sdcard/espdet_pico_224_224_face.espdl", score=0.5, nms=0.7)
try:
    img = sensor.snapshot()
    for x, y, w, h, score, category in det.detect(img):
        img.draw_rectangle(x, y, w, h, color=(255, 0, 0), thickness=2)
        img.draw_string(x, max(0, y - 12), "%.2f:%d" % (score, category))
finally:
    det.deinit()

score removes low-confidence candidates, while nms controls suppression of overlapping boxes. Detection coordinates are mapped back to the input image and can be used directly for drawing. Load a model once and reuse it across frames; deinit() releases model weights and intermediate buffers.

PicoDet

PicoDet is the detector family used by the bundled pedestrian model. Its anchor-point output tensors are decoded by ESP-DL’s Pico postprocessor; this output layout is distinct from ESPDet and YOLO11.

PedestrianDetect

PedestrianDetect is the Python API for the bundled 224x224 PicoDet pedestrian model. It uses the model’s preprocessing defaults and returns the standard detection tuples:

det = espdl.PedestrianDetect(
    "/sdcard/pedestrian_detect/pedestrian_detect_pico.espdl",
    score=0.7,
    nms=0.5,
)

Image Classification

import espdl, image

img = image.Image("/sdcard/cat.jpg").to_rgb565(copy=True)
classifier = espdl.ImageNetCls(
    "/sdcard/imagenet_cls_mobilenetv2_s8_v1.espdl",
    topk=5,
    score=0.0,
)
try:
    for label, score in classifier.classify(img):
        print(label, "%.4f" % score)
finally:
    classifier.deinit()

Classification returns up to topk (label, score) pairs ordered by the model output. Convert file images to RGB565 when necessary so the input format is accepted by the wrapper.

Pose Estimation

pose = espdl.YOLO11nPose("/sdcard/espdet_yolo11n_pose_160_160_coco.espdl", score=0.35, topk=5)
try:
    img = sensor.snapshot()
    for x, y, w, h, score, category, keypoints in pose.detect(img):
        img.draw_rectangle(x, y, w, h, color=(255, 0, 0))
        for px, py in keypoints:
            if px > 0 and py > 0:
                img.draw_circle(px, py, 2, color=(0, 255, 0), fill=True)
finally:
    pose.deinit()

Each pose result contains 17 COCO keypoints. A missing or low-confidence point is returned as (0, 0) and should be skipped before drawing or computing joint geometry.

Adjust Thresholds at Runtime

det.set_thresholds(score=0.65, nms=0.6)

Thresholds can be changed without reloading the model, which is useful when adapting an application to lighting, distance, or scene-density changes.

Result tuples

TensorInfo: (name, shape, dtype, exponent)
RawTensor: (name, shape, dtype, exponent, bytes)
Detection: (x, y, w, h, score, category)
Pose: (x, y, w, h, score, category, keypoints) with 17 COCO keypoints
Classification: (label, score)
Tracking: (x, y, w, h, score) or None below the match threshold

Functions

espdl.load_model(path, *, profile=...)

Preload an ESP-DL model file.

Parameters:

path – model path, for example “/sdcard/model.espdl” or “/flash/model.espdl”.
profile – True enables ESP-DL profiling output when supported.

Classes

class espdl.NanoTrack(template, search, head)

NanoTrack V2 single-object tracker using full-INT8 NHWC ESP-DL models.

Load search and head models. The template model is loaded only during init().

Parameters:

template – 127x127 template backbone .espdl path.
search – 255x255 search backbone .espdl path.
head – two-input NanoTrack V2 head .espdl path.

init(image, roi)

Cache target features from an unannotated RGB565/grayscale image, then release the template model.

Parameters:: roi – (x, y, w, h), inside the image, at least 10x10 pixels.

track(image, *, threshold=...)

Track in a same-size image; return (x, y, w, h, score), or None below threshold.

Parameters:: threshold – raw foreground score in [0, 1]; rejected frames do not update the search state.

deinit(): Release models and cached features. Repeated calls are safe.

class espdl.Model(path, *, mean=..., std=..., rgb_swap=..., letterbox=..., pad=...)

Generic ESP-DL model runner for examples that implement model-specific decoding in Python.

Create a raw model runner from an .espdl model.

Parameters:

path – model path.
mean – optional RGB mean values for image preprocessing.
std – optional RGB standard deviation values for image preprocessing.
rgb_swap – True converts RGB input to BGR before model inference.
letterbox – True keeps aspect ratio and pads the model input.
pad – optional RGB padding values used when letterbox is enabled.

deinit(): Release model resources.

inputs(): Return model input metadata keyed by tensor name.

outputs(): Return model output metadata keyed by tensor name.

predict(image)

Run model inference on an image and return raw output tensors keyed by tensor name.

Parameters:: image – RGB565 or grayscale image.

class espdl.ESPDet(path, *, score=..., nms=..., mean=..., std=...)

ESP-DL object detection wrapper for ESPDet models.

Create a detector from an .espdl model.

Parameters:

path – model path.
score – optional confidence threshold.
nms – optional non-maximum suppression threshold.
mean – optional RGB mean values for preprocessing.
std – optional RGB standard deviation values for preprocessing.

deinit(): Release model resources.

detect(image)

Run object detection on an image.

Parameters:: image – RGB565 or grayscale image.

set_thresholds(*, score=..., nms=...)

Update detector thresholds.

Parameters:

score – new confidence threshold, or None to keep current value.
nms – new NMS threshold, or None to keep current value.

class espdl.PedestrianDetect(path, *, score=..., nms=..., topk=...)

ESP-DL PicoDet pedestrian detection wrapper.

Create a pedestrian detector from the bundled ESP-DL model.

Parameters:

path – model path.
score – optional confidence threshold; defaults to the upstream model’s 0.7.
nms – optional non-maximum suppression threshold; defaults to the upstream model’s 0.5.
topk – maximum number of detections returned per frame.

deinit(): Release model resources.

detect(image): Run pedestrian detection on an RGB565 or grayscale image.

set_thresholds(*, score=..., nms=...)

Update detector thresholds.

Parameters:

score – new confidence threshold, or None to keep current value.
nms – new NMS threshold, or None to keep current value.

class espdl.YOLO11(path, *, score=..., nms=..., topk=..., mean=..., std=...)

ESP-DL YOLO11 object detection wrapper.

Create a YOLO11 detector from an .espdl model.

Parameters:

path – model path.
score – optional confidence threshold.
nms – optional non-maximum suppression threshold.
topk – maximum number of detections returned per frame.
mean – optional RGB mean values for preprocessing.
std – optional RGB standard deviation values for preprocessing.

deinit(): Release model resources.

detect(image)

Run object detection on an image.

Parameters:: image – RGB565 or grayscale image.

set_thresholds(*, score=..., nms=...)

Update detector thresholds.

Parameters:

score – new confidence threshold, or None to keep current value.
nms – new NMS threshold, or None to keep current value.

class espdl.YOLO11nPose(path, *, score=..., nms=..., topk=..., mean=..., std=...)

ESP-DL YOLO11n COCO pose wrapper.

Create a YOLO11n pose detector from an .espdl model.

Parameters:

path – model path.
score – optional confidence threshold.
nms – optional non-maximum suppression threshold.
topk – maximum number of pose results returned per frame.
mean – optional RGB mean values for preprocessing.
std – optional RGB standard deviation values for preprocessing.

deinit(): Release model resources.

detect(image)

Run COCO pose detection on an image. Returns 17 COCO keypoints for each person.

Parameters:: image – RGB565 or grayscale image.

set_thresholds(*, score=..., nms=...)

Update detector thresholds.

Parameters:

score – new confidence threshold, or None to keep current value.
nms – new NMS threshold, or None to keep current value.

class espdl.ImageNetCls(path, *, topk=..., score=..., mean=..., std=..., softmax=...)

ESP-DL ImageNet classification wrapper.

Create a classifier from an .espdl model.

Parameters:

path – model path.
topk – maximum number of classes returned.
score – optional minimum score threshold.
mean – optional RGB mean values for preprocessing.
std – optional RGB standard deviation values for preprocessing.
softmax – True applies softmax to output scores.

deinit(): Release model resources.

classify(image)

Run image classification on an image.

Parameters:: image – RGB565 or grayscale image.

set_thresholds(*, topk=..., score=...)

Update classifier thresholds.

Parameters:

topk – new maximum result count, or None to keep current value.
score – new minimum score, or None to keep current value.