espdl – 模型推理

[English]

espdl 模块在采集到的图像上运行 ESP-DL 的 .espdl 模型，并提供常见任务封装：目标检测（PedestrianDetect、ESPDet、YOLO11）、姿态估计（YOLO11nPose）、图像分类（ImageNetCls）和单目标追踪（NanoTrack），同时可通过 Model 将 ESP-DL 原始输出 tensor 暴露给 Python。

按键确认单目标追踪

espdl.NanoTrack 运行 NanoTrack V2 全 INT8 模型。将 models/nanotrack_v2/ 中的三个 .espdl 文件复制到设备的 /sdcard/nanotrack_v2/ 即可，本地量化实验保存在被忽略的 build/experiments/nanotrack-v2，不包含在全新检出的仓库中。P4/S31 使用 ESP32-P4 导出物；本封装不接受混合 INT16 实验模型。S3 需要面向 S3 重新量化并单独验证。

example/03-Machine-Learning/00-ESP-DL/nanotrack.py 默认显示中心 64x64 选框。将目标放入框内，按 BOOT 键（ESP32-P4X-EYE 和 P4 Function EV Board 均为 GPIO35）开始追踪；再次按键返回选框状态。丢失目标后保留原模板并继续局部搜索。相机图像按原尺寸显示，小屏幕只裁切中心区域，不缩放。可通过 BUTTON_PIN 和 BOX_SIZE 适配其他硬件或目标尺寸。推理在绘制标注前完成，避免图形进入模型输入。

示例使用可配置的 MATCH_THRESHOLD = 0.95，画面上的 match 是模型匹配分数，不是目标身份正确的概率。

from machine import Pin
import display
import espdl
import sensor
import time


MODEL_DIR = "/sdcard/nanotrack_v2"
# Active-low BOOT button on ESP32-P4X-EYE and P4 Function EV Board.
# Change this pin for other boards.
BUTTON_PIN = 35
BOX_SIZE = 64
DEBOUNCE_MS = 30
# A match score is not a calibrated target-identity probability.
MATCH_THRESHOLD = 0.95


lcd = None
tracker = None
try:
    button = Pin(BUTTON_PIN, Pin.IN, Pin.PULL_UP)
    sensor.reset()
    sensor.set_pixformat(sensor.RGB565)
    sensor.set_framesize(sensor.QVGA)
    sensor.skip_frames(time=1000)
    lcd = display.Display()
    tracker = espdl.NanoTrack(
        MODEL_DIR + "/template_deploy.espdl",
        MODEL_DIR + "/search_deploy.espdl",
        MODEL_DIR + "/head_deploy.espdl",
    )
    frame = sensor.snapshot()
    width, height = frame.width(), frame.height()
    canvas = frame.copy()
    # Center at native size; smaller displays show the center crop.
    visible_w = min(width, lcd.width())
    visible_h = min(height, lcd.height())
    view = ((width - visible_w) // 2, (height - visible_h) // 2,
            visible_w, visible_h)
    offset_x = (lcd.width() - visible_w) // 2
    offset_y = (lcd.height() - visible_h) // 2
    size = min(BOX_SIZE, visible_w, visible_h)
    roi = ((width - size) // 2, (height - size) // 2, size, size)
    tracking = False
    last_button = stable_button = button.value()
    changed_at = time.ticks_ms()
    # A button held at startup must be released before it can confirm.
    armed = stable_button == 1

    while True:
        now = time.ticks_ms()
        level = button.value()
        pressed = False
        if level != last_button:
            last_button = level
            changed_at = now
        if level != stable_button and time.ticks_diff(now, changed_at) >= DEBOUNCE_MS:
            stable_button = level
            if level:
                armed = True
            elif armed:
                armed = False
                pressed = True

        if pressed:
            if tracking:
                tracking = False
            else:
                canvas.draw_image(frame, 0, 0)
                canvas.draw_string(view[0] + 2, view[1] + 2, "Initializing...",
                                   color=(255, 255, 0))
                lcd.write(canvas, x=offset_x, y=offset_y, roi=view, fit=False)
                # Use the last displayed raw frame, without any annotations.
                tracker.init(frame, roi)
                tracking = True

        frame = sensor.snapshot()
        begin = time.ticks_ms()
        result = tracker.track(frame, threshold=MATCH_THRESHOLD) if tracking else None
        canvas.draw_image(frame, 0, 0)
        if result is not None:
            x, y, w, h, score = result
            message = "Tracking - BOOT: select"
            canvas.draw_rectangle(x, y, w, h, color=(0, 255, 0), thickness=2)
            canvas.draw_string(view[0] + 2, view[1] + visible_h - 14,
                               "match %.2f  %d ms" % (score, time.ticks_diff(time.ticks_ms(), begin)),
                               color=(0, 255, 0))
        elif tracking:
            # Preserve the template and retry near the last accepted position.
            message = "Lost - searching..."
        else:
            message = "Aim box, press BOOT"
            canvas.draw_rectangle(roi, color=(255, 255, 0), thickness=2)
        canvas.draw_string(view[0] + 2, view[1] + 2, message, color=(255, 255, 255))
        lcd.write(canvas, x=offset_x, y=offset_y, roi=view, fit=False)
        time.sleep_ms(0 if tracking else 10)
finally:
    try:
        if tracker is not None:
            tracker.deinit()
    finally:
        if lcd is not None:
            lcd.deinit()

init(image, roi) 缓存目标特征，然后释放模板模型；搜索模型和 head 常驻，重新框选时临时重载模板模型。同一次追踪使用尺寸不变、未绘制标注的 RGB565 或灰度图像。结果为 (x, y, w, h, score)；低于阈值时返回 None，不更新上一搜索状态。分数不能保证目标仍然可见。这是局部单目标追踪，不包含全局自动找回或网络权重训练。

原始输出 tensor

import sensor, espdl

model = espdl.Model("/sdcard/custom.espdl", mean=(0, 0, 0), std=(255, 255, 255), letterbox=True)
try:
    print("inputs:", model.inputs())
    print("outputs:", model.outputs())
    img = sensor.snapshot()
    outputs = model.predict(img)
    for name, tensor in outputs.items():
        _, shape, dtype, exponent, raw = tensor
        print(name, shape, dtype, exponent, len(raw))
finally:
    model.deinit()

当内置任务封装不匹配模型输出布局时，可使用 Model。ESP-DL 仍负责图像预处理和推理；predict() 返回原始输出 tensor，Python 代码再执行模型相关解码。inputs() 和 outputs() 返回 TensorInfo 元组 (name, shape, dtype, exponent)。predict() 返回按输出 tensor 名称索引的 RawTensor 元组 (name, shape, dtype, exponent, bytes)。应根据 dtype 解包原始字节，应用 ESP-DL 的二次幂 exponent 量化比例，再执行 sigmoid、框解码、NMS、softmax 或 top-k 选择等逻辑。若启用 letterbox=True，将坐标映射回源图像前需要先去除模型输入中的 padding。

目标检测

import sensor, espdl

det = espdl.ESPDet("/sdcard/espdet_pico_224_224_face.espdl", score=0.5, nms=0.7)
try:
    img = sensor.snapshot()
    for x, y, w, h, score, category in det.detect(img):
        img.draw_rectangle(x, y, w, h, color=(255, 0, 0), thickness=2)
        img.draw_string(x, max(0, y - 12), "%.2f:%d" % (score, category))
finally:
    det.deinit()

score 用于过滤低置信度候选框，nms 用于控制重叠检测框的抑制。检测坐标会映射回输入图像，可直接用于绘图。模型应加载一次并在多帧之间复用；deinit() 会释放模型权重和中间缓冲区。

PicoDet

PicoDet 是仓库自带行人模型采用的检测器家族。它的 anchor-point 输出 tensor 由 ESP-DL 的 Pico 后处理器解码，输出布局与 ESPDet、YOLO11 不同。

PedestrianDetect

PedestrianDetect 是仓库自带 224x224 PicoDet 行人模型的 Python API。它使用该模型的预处理默认值，并返回标准检测元组：

det = espdl.PedestrianDetect(
    "/sdcard/pedestrian_detect/pedestrian_detect_pico.espdl",
    score=0.7,
    nms=0.5,
)

图像分类

import espdl, image

img = image.Image("/sdcard/cat.jpg").to_rgb565(copy=True)
classifier = espdl.ImageNetCls(
    "/sdcard/imagenet_cls_mobilenetv2_s8_v1.espdl",
    topk=5,
    score=0.0,
)
try:
    for label, score in classifier.classify(img):
        print(label, "%.4f" % score)
finally:
    classifier.deinit()

分类结果按照模型输出顺序返回最多 topk 个 (label, score) 元组。必要时应将文件图像转换为 RGB565，确保输入格式可被封装层接受。

姿态估计

pose = espdl.YOLO11nPose("/sdcard/espdet_yolo11n_pose_160_160_coco.espdl", score=0.35, topk=5)
try:
    img = sensor.snapshot()
    for x, y, w, h, score, category, keypoints in pose.detect(img):
        img.draw_rectangle(x, y, w, h, color=(255, 0, 0))
        for px, py in keypoints:
            if px > 0 and py > 0:
                img.draw_circle(px, py, 2, color=(0, 255, 0), fill=True)
finally:
    pose.deinit()

每个姿态结果包含 17 个 COCO 关键点。缺失或低置信度关键点会返回为 (0, 0)，绘图或计算关节几何关系前应将其跳过。

运行时调整阈值

det.set_thresholds(score=0.65, nms=0.6)

无需重新加载模型即可修改阈值，适合应用根据光照、距离或场景目标密度的变化进行动态调整。

结果元组

TensorInfo：(name, shape, dtype, exponent)
RawTensor：(name, shape, dtype, exponent, bytes)
检测：(x, y, w, h, score, category)
姿态：(x, y, w, h, score, category, keypoints)，含 17 个 COCO 关键点
分类：(label, score)
追踪：(x, y, w, h, score)，低于匹配阈值时为 None

参见

AI 推理介绍 ESP-DL 推理流程、.espdl 格式、量化以及前后处理。部署新模型请参见引入新的模型。

可运行示例：example/03-Machine-Learning/00-ESP-DL（行人检测、ESP-DL 原始输出解码、ESPDet、YOLO11、姿态、ImageNet 分类、BOOT 按键确认的 NanoTrack 追踪）。

Functions

espdl.load_model(path, *, profile=...)

Preload an ESP-DL model file.

参数:

path – model path, for example “/sdcard/model.espdl” or “/flash/model.espdl”.
profile – True enables ESP-DL profiling output when supported.

Classes

class espdl.NanoTrack(template, search, head)

NanoTrack V2 single-object tracker using full-INT8 NHWC ESP-DL models.

Load search and head models. The template model is loaded only during init().

参数:

template – 127x127 template backbone .espdl path.
search – 255x255 search backbone .espdl path.
head – two-input NanoTrack V2 head .espdl path.

init(image, roi)

Cache target features from an unannotated RGB565/grayscale image, then release the template model.

参数:: roi – (x, y, w, h), inside the image, at least 10x10 pixels.

track(image, *, threshold=...)

Track in a same-size image; return (x, y, w, h, score), or None below threshold.

参数:: threshold – raw foreground score in [0, 1]; rejected frames do not update the search state.

deinit(): Release models and cached features. Repeated calls are safe.

class espdl.Model(path, *, mean=..., std=..., rgb_swap=..., letterbox=..., pad=...)

Generic ESP-DL model runner for examples that implement model-specific decoding in Python.

Create a raw model runner from an .espdl model.

参数:

path – model path.
mean – optional RGB mean values for image preprocessing.
std – optional RGB standard deviation values for image preprocessing.
rgb_swap – True converts RGB input to BGR before model inference.
letterbox – True keeps aspect ratio and pads the model input.
pad – optional RGB padding values used when letterbox is enabled.

deinit(): Release model resources.

inputs(): Return model input metadata keyed by tensor name.

outputs(): Return model output metadata keyed by tensor name.

predict(image)

Run model inference on an image and return raw output tensors keyed by tensor name.

参数:: image – RGB565 or grayscale image.

class espdl.ESPDet(path, *, score=..., nms=..., mean=..., std=...)

ESP-DL object detection wrapper for ESPDet models.

Create a detector from an .espdl model.

参数:

path – model path.
score – optional confidence threshold.
nms – optional non-maximum suppression threshold.
mean – optional RGB mean values for preprocessing.
std – optional RGB standard deviation values for preprocessing.

deinit(): Release model resources.

detect(image)

Run object detection on an image.

参数:: image – RGB565 or grayscale image.

set_thresholds(*, score=..., nms=...)

Update detector thresholds.

参数:

score – new confidence threshold, or None to keep current value.
nms – new NMS threshold, or None to keep current value.

class espdl.PedestrianDetect(path, *, score=..., nms=..., topk=...)

ESP-DL PicoDet pedestrian detection wrapper.

Create a pedestrian detector from the bundled ESP-DL model.

参数:

path – model path.
score – optional confidence threshold; defaults to the upstream model’s 0.7.
nms – optional non-maximum suppression threshold; defaults to the upstream model’s 0.5.
topk – maximum number of detections returned per frame.

deinit(): Release model resources.

detect(image): Run pedestrian detection on an RGB565 or grayscale image.

set_thresholds(*, score=..., nms=...)

Update detector thresholds.

参数:

score – new confidence threshold, or None to keep current value.
nms – new NMS threshold, or None to keep current value.

class espdl.YOLO11(path, *, score=..., nms=..., topk=..., mean=..., std=...)

ESP-DL YOLO11 object detection wrapper.

Create a YOLO11 detector from an .espdl model.

参数:

path – model path.
score – optional confidence threshold.
nms – optional non-maximum suppression threshold.
topk – maximum number of detections returned per frame.
mean – optional RGB mean values for preprocessing.
std – optional RGB standard deviation values for preprocessing.

deinit(): Release model resources.

detect(image)

Run object detection on an image.

参数:: image – RGB565 or grayscale image.

set_thresholds(*, score=..., nms=...)

Update detector thresholds.

参数:

score – new confidence threshold, or None to keep current value.
nms – new NMS threshold, or None to keep current value.

class espdl.YOLO11nPose(path, *, score=..., nms=..., topk=..., mean=..., std=...)

ESP-DL YOLO11n COCO pose wrapper.

Create a YOLO11n pose detector from an .espdl model.

参数:

path – model path.
score – optional confidence threshold.
nms – optional non-maximum suppression threshold.
topk – maximum number of pose results returned per frame.
mean – optional RGB mean values for preprocessing.
std – optional RGB standard deviation values for preprocessing.

deinit(): Release model resources.

detect(image)

Run COCO pose detection on an image. Returns 17 COCO keypoints for each person.

参数:: image – RGB565 or grayscale image.

set_thresholds(*, score=..., nms=...)

Update detector thresholds.

参数:

score – new confidence threshold, or None to keep current value.
nms – new NMS threshold, or None to keep current value.

class espdl.ImageNetCls(path, *, topk=..., score=..., mean=..., std=..., softmax=...)

ESP-DL ImageNet classification wrapper.

Create a classifier from an .espdl model.

参数:

path – model path.
topk – maximum number of classes returned.
score – optional minimum score threshold.
mean – optional RGB mean values for preprocessing.
std – optional RGB standard deviation values for preprocessing.
softmax – True applies softmax to output scores.

deinit(): Release model resources.

classify(image)

Run image classification on an image.

参数:: image – RGB565 or grayscale image.

set_thresholds(*, topk=..., score=...)

Update classifier thresholds.

参数:

topk – new maximum result count, or None to keep current value.
score – new minimum score, or None to keep current value.