USB Stream 组件说明

usb_stream 是基于 ESP32-S2/ESP32-S3 的 USB UVC + UAC 主机驱动程序，支持从 USB 设备读取/写入/控制多媒体流。例如最多同时支持 1 路摄像头 + 1 路麦克风 + 1 路播放器数据流。

特性：

支持通过 UVC Stream 接口获取视频流，支持批量和同步两种传输模式
支持通过 UAC Stream 接口获取麦克风数据流，发送播放器数据流
支持通过 UAC Control 接口控制麦克风音量、静音等特性
支持自动解析设备配置描述符
支持对数据流暂停和恢复

USB Stream 用户指南

开发板
1. 可以使用任何带有 USB 接口的 ESP32-S2/ESP32-S3 开发板，注意该 USB 接口需要能够向外供电
USB UVC 功能
1. 摄像头必须兼容 USB1.1 全速（Fullspeed）模式
2. 摄像头需要自带 MJPEG 压缩
3. 用户可通过 uvc_streaming_config() 函数手动指定相机接口、传输模式和图像帧参数
4. 如使用同步传输摄像头，需要支持设置接口 MPS（Max Packet Size）为 512
5. 同步传输模式，图像数据流 USB 传输总带宽应小于 4 Mbps （500 KB/s）
6. 批量传输模式，图像数据流 USB 传输总带宽应小于 8.8 Mbps （1100 KB/s）
7. 其它特殊相机要求，请参考示例程序 README.md
USB UAC 功能
1. 音频功能必须兼容 UAC 1.0 协议
2. 用户需要通过 uac_streaming_config() 函数手动指定 spk/mic 采样率，位宽参数
USB UVC + UAC 功能
1. UVC 和 UAC 功能可以单独启用，例如仅配置 UAC 来驱动一个 USB 耳机，或者仅配置 UVC 来驱动一个 USB 摄像头
2. 如需同时启用 UVC + UAC，该驱动程序目前仅支持具有摄像头和音频接口的复合设备（Composite Device），不支持同时连接两个单独的设备

USB Stream API 参考

用户可通过 uvc_config_t 配置摄像头分辨率、帧率参数。通过 uac_config_t 配置音频采样率、位宽等参数。参数说明如下:

uvc_config_t uvc_config = {
    .frame_width = 320, // mjpeg width pixel, for example 320
    .frame_height = 240, // mjpeg height pixel, for example 240
    .frame_interval = FPS2INTERVAL(15), // frame interval (100µs units), such as 15fps
    .xfer_buffer_size = 32 * 1024, // single frame image size, need to be determined according to actual testing, 320 * 240 generally less than 35KB
    .xfer_buffer_a = pointer_buffer_a, // the internal transfer buffer
    .xfer_buffer_b = pointer_buffer_b, // the internal transfer buffer
    .frame_buffer_size = 32 * 1024, // single frame image size, need to determine according to actual test
    .frame_buffer = pointer_frame_buffer, // the image frame buffer
    .frame_cb = &camera_frame_cb, //camera callback, can block in here
    .frame_cb_arg = NULL, // camera callback args
}

uac_config_t uac_config = {
    .mic_bit_resolution = 16, //microphone resolution, bits
    .mic_samples_frequence = 16000, //microphone frequency, Hz
    .spk_bit_resolution = 16, //speaker resolution, bits
    .spk_samples_frequence = 16000, //speaker frequency, Hz
    .spk_buf_size = 16000, //size of speaker send buffer, should be a multiple of spk_ep_mps
    .mic_buf_size = 0, //mic receive buffer size, 0 if not use, else should be a multiple of mic_min_bytes
    .mic_cb = &mic_frame_cb, //mic callback, can not block in here
    .mic_cb_arg = NULL, //mic callback args
};

使用 uvc_streaming_config() 配置 UVC 驱动，如果设备同时支持音频，可使用 uac_streaming_config() 配置 UAC 驱动
使用 usb_streaming_start() 打开数据流，之后驱动将响应设备连接和协议协商
之后，主机将根据用户参数配置，匹配已连接设备的描述符，如果设备无法满足配置要求，驱动将以警告提示
如果设备满足用户配置要求，主机将持续接收 IN 数据流（UVC 和 UAC 麦克风），并在新帧准备就绪后调用用户的回调
1. 接收到新的 MJPEG 图像后，将触发 UVC 回调函数。用户可在回调函数中阻塞，因为它在独立的任务上下文中工作
2. 接收到 mic_min_bytes 字节数据后，将触发 mic 回调。但是这里的回调一定不能以任何方式阻塞，否则会影响下一帧的接收。如果需要对 mic 进行阻塞操作，可以通过 uac_mic_streaming_read 轮询方式代替回调方式
发送扬声器数据时，用户可通过 uac_spk_streaming_write() 将数据写入内部 ringbuffer，主机将在 USB 空闲时从中获取数据并发送 OUT 数据
使用 usb_streaming_control() 控制流挂起/恢复。如果 UAC 支持特性单元，可通过其分别控制麦克风和播放器的音量和静音
使用 usb_streaming_stop() 停止 USB 流，USB 资源将被完全释放。

Bug 报告

ESP32-S2 ECO0 芯片 SPI 屏幕和 USB 同时启用，可能导致屏幕抖动

在最早版本的 ESP32-S2 ECO0 芯片上, USB 可能污染 SPI 数据 (ESP32-S2 新版本>=ECO1 和 ESP32-S3 均不存在该 Bug)
软件解决方案

spi_ll.h 添加以下接口

//components/hal/esp32s2/include/hal/spi_ll.h
static inline uint32_t spi_ll_tx_get_fifo_cnt(spi_dev_t *hw)
{
    return hw->dma_out_status.out_fifo_cnt;
}

如下在 spi_new_trans 中添加额外检查

// The function is called to send a new transaction, in ISR or in the task.
// Setup the transaction-specified registers and linked-list used by the DMA (or FIFO if DMA is not used)
static void SPI_MASTER_ISR_ATTR spi_new_trans(spi_device_t *dev, spi_trans_priv_t *trans_buf)
{
    //...................
    spi_hal_setup_trans(hal, hal_dev, &hal_trans);
    spi_hal_prepare_data(hal, hal_dev, &hal_trans);

    //Call pre-transmission callback, if any
    if (dev->cfg.pre_cb) dev->cfg.pre_cb(trans);
#if 1
    //USB Bug workaround
    //while (!((spi_ll_tx_get_fifo_cnt(SPI_LL_GET_HW(host->id)) == 12) || (spi_ll_tx_get_fifo_cnt(SPI_LL_GET_HW(host->id)) == trans->length / 8))) {
    while (trans->length && spi_ll_tx_get_fifo_cnt(SPI_LL_GET_HW(host->id)) == 0) {
        __asm__ __volatile__("nop");
        __asm__ __volatile__("nop");
        __asm__ __volatile__("nop");
    }
#endif
    //Kick off transfer
    spi_hal_user_start(hal);
}

Examples

API Reference

Header File

components/usb/usb_stream/include/usb_stream.h

Functions

esp_err_t uvc_streaming_config(const uvc_config_t *config)

Config UVC streaming with user defined parameters.For normal use, user only need to specify no-optional parameters, and set optional parameters to 0 (the driver will find the correct value from the device descriptors). For quick start mode, user should specify all parameters manually to skip get and process descriptors steps.

参数: config – parameters defined in uvc_config_t
返回: esp_err_t ESP_ERR_INVALID_STATE USB streaming is running, user need to stop streaming first ESP_ERR_INVALID_ARG Invalid argument ESP_OK Success

esp_err_t uac_streaming_config(const uac_config_t *config)

Config UAC streaming with user defined parameters.For normal use, user only need to specify no-optional parameters, and set optional parameters to 0 (the driver will find the correct value from the device descriptors). For quick start mode, user should specify all parameters manually to skip get and process descriptors steps.

参数: config – parameters defined in uvc_config_t
返回: esp_err_t ESP_ERR_INVALID_STATE USB streaming is running, user need to stop streaming first ESP_ERR_INVALID_ARG Invalid argument ESP_OK Success

esp_err_t usb_streaming_start(void)

Start usb streaming with pre-configs, usb driver will create internal tasks to handle usb data from stream pipe, and run user’s callback after new frame ready.

返回: ESP_ERR_INVALID_STATE streaming not configured, or streaming running already ESP_FAIL start failed ESP_OK start succeed

esp_err_t usb_streaming_stop(void)

Stop current usb streaming, internal tasks will be delete, related resource will be free.

返回: ESP_ERR_INVALID_STATE streaming not started ESP_ERR_TIMEOUT stop wait timeout ESP_OK stop succeed

esp_err_t usb_streaming_connect_wait(size_t timeout_ms)

Wait for USB device connection.

参数: timeout_ms – timeout in ms
返回: esp_err_t ESP_ERR_INVALID_STATE: usb streaming not started ESP_ERR_TIMEOUT: timeout ESP_OK: device connected

esp_err_t usb_streaming_state_register(state_callback_t cb, void *user_ptr)

This function registers a callback for USB streaming, please note that only one callback can be registered, the later registered callback will overwrite the previous one.

参数

cb – A pointer to a function that will be called when the USB streaming state changes.
user_ptr – user_ptr is a void pointer.

返回

esp_err_t

ESP_OK Success
ESP_ERR_INVALID_STATE USB streaming is running, callback need register before start

esp_err_t usb_streaming_control(usb_stream_t stream, stream_ctrl_t ctrl_type, void *ctrl_value)

Control USB streaming with specific stream and control type.

参数

stream – stream type defined in usb_stream_t
ctrl_type – control type defined in stream_ctrl_t
ctrl_value – control value

返回

ESP_ERR_INVALID_ARG invalid arg ESP_ERR_INVALID_STATE driver not configured or not started ESP_ERR_NOT_SUPPORTED current device not support this control type ESP_FAIL control failed ESP_OK succeed

esp_err_t uac_spk_streaming_write(void *data, size_t data_bytes, size_t timeout_ms)

Write data to the speaker buffer, will be send out when USB device is ready.

参数

data – The data to be written.
data_bytes – The size of the data to be written.
timeout_ms – The timeout value for writing data to the buffer.

返回

ESP_ERR_INVALID_STATE spk stream not config ESP_ERR_NOT_FOUND spk interface not found ESP_ERR_TIMEOUT spk ringbuf full ESP_OK succeed

esp_err_t uac_mic_streaming_read(void *buf, size_t buf_size, size_t *data_bytes, size_t timeout_ms)

Read data from internal mic buffer, the actual size will be returned.

参数

buf – pointer to the buffer to store the received data
buf_size – The size of the data buffer.
data_bytes – The actual size read from buffer
timeout_ms – The timeout value for the read operation.

返回

ESP_ERR_INVALID_ARG parameter error ESP_ERR_INVALID_STATE mic stream not config ESP_ERR_NOT_FOUND mic interface not found ESP_TIMEOUT timeout ESP_OK succeed

esp_err_t uac_frame_size_list_get(usb_stream_t stream, uac_frame_size_t *frame_list, size_t *list_size, size_t *cur_index)

Get the audio frame size list of current stream, the list contains audio channel number, bit resolution and samples frequency. IF list_size equals 1 and the samples_frequence equals 0, which means the frequency can be set to any value between samples_frequence_min and samples_frequence_max.

参数

stream – the stream type
frame_list – the output frame list, NULL to only get the list size
list_size – frame list size
cur_index – current frame index

返回

esp_err_t

ESP_ERR_INVALID_ARG Parameter error
ESP_ERR_INVALID_STATE USB device not active
ESP_OK Success

esp_err_t uac_frame_size_reset(usb_stream_t stream, uint8_t ch_num, uint16_t bit_resolution, uint32_t samples_frequence)

Reset audio channel number, bit resolution and samples frequency, please reset when the streaming in suspend state. The new configs will be effective after streaming resume.

参数

stream – stream type
ch_num – audio channel numbers
bit_resolution – audio bit resolution
samples_frequence – audio samples frequency

返回

esp_err_t

ESP_ERR_INVALID_ARG Parameter error
ESP_ERR_INVALID_STATE USB device not active
ESP_ERR_NOT_FOUND frequency not found
ESP_OK Success
ESP_FAIL Reset failed

esp_err_t uvc_frame_size_list_get(uvc_frame_size_t *frame_list, size_t *list_size, size_t *cur_index)

Get the frame size list of current connected camera.

参数

frame_list – the frame size list, can be NULL if only want to get list size
list_size – the list size
cur_index – current frame index

返回

esp_err_t ESP_ERR_INVALID_ARG parameter error ESP_ERR_INVALID_STATE uvc stream not config or not active ESP_OK succeed

esp_err_t uvc_frame_size_reset(uint16_t frame_width, uint16_t frame_height, uint32_t frame_interval)

Reset the expected frame size and frame interval, please reset when uvc streaming in suspend state.The new configs will be effective after streaming resume.

Note: frame_width and frame_height can be set to 0 at the same time, which means no change on frame size.

参数

frame_width – frame width, FRAME_RESOLUTION_ANY means any width
frame_height – frame height, FRAME_RESOLUTION_ANY means any height
frame_interval – frame interval, 0 means no change

返回

esp_err_t

Structures

struct uvc_config

UVC configurations, for params with (optional) label, users do not need to specify manually, unless there is a problem with descriptors, or users want to skip the get and process descriptors steps.

Public Members

uint16_t frame_width: Picture width, set FRAME_RESOLUTION_ANY for any resolution

uint16_t frame_height: Picture height, set FRAME_RESOLUTION_ANY for any resolution

uint32_t frame_interval: Frame interval in 100-ns units, 666666 ~ 15 Fps

uint32_t xfer_buffer_size: Transfer buffer size, using double buffer here, must larger than one frame size

uint8_t *xfer_buffer_a: Buffer a for usb payload

uint8_t *xfer_buffer_b: Buffer b for usb payload

uint32_t frame_buffer_size: Frame buffer size, must larger than one frame size

uint8_t *frame_buffer: Buffer for one frame

uvc_frame_callback_t frame_cb: callback function to handle incoming frame

void *frame_cb_arg: callback function arg

uvc_format_t format: (optional) UVC stream format, default using MJPEG Optional configs, Users need to specify parameters manually when they want to skip the get and process descriptors steps (used to speed up startup)

uvc_xfer_t xfer_type: (optional) UVC stream transfer type, UVC_XFER_ISOC or UVC_XFER_BULK

uint8_t format_index: (optional) Format index

uint8_t frame_index: (optional) Frame index, to choose resolution

uint16_t interface: (optional) UVC stream interface number

uint16_t interface_alt: (optional) UVC stream alternate interface, to choose MPS (Max Packet Size), bulk fix to 0

uint8_t ep_addr: (optional) endpoint address of selected alternate interface

uint32_t ep_mps: (optional) MPS of selected interface_alt

uint32_t flags: (optional) flags to control the driver behavers

struct mic_frame_t

mic frame type

Public Members

void *data: mic data

uint32_t data_bytes: mic data size

uint16_t bit_resolution: bit resolution in buffer

uint32_t samples_frequence: mic sample frequency

struct uvc_frame_size_t

uvc frame type

Public Members

uint16_t width: frame width

uint16_t height: frame height

uint32_t interval: frame interval

uint32_t interval_min: frame min interval

uint32_t interval_max: frame max interval

uint32_t interval_step: frame interval step

struct uac_frame_size_t

uac frame type

Public Members

uint8_t ch_num: channel numbers

uint16_t bit_resolution: bit resolution in buffer

uint32_t samples_frequence: sample frequency

uint32_t samples_frequence_min: min sample frequency

uint32_t samples_frequence_max: max sample frequency

struct uac_config_t

UAC configurations, for params with (optional) label, users do not need to specify manually, unless there is a problem with descriptor parse, or a problem with the device descriptor.

Public Members

uint8_t spk_ch_num: speaker channel numbers, UAC_CH_ANY for any channel number

uint8_t mic_ch_num: microphone channel numbers, UAC_CH_ANY for any channel number

uint16_t mic_bit_resolution: microphone resolution(bits), UAC_BITS_ANY for any bit resolution

uint32_t mic_samples_frequence: microphone frequency(Hz), UAC_FREQUENCY_ANY for any frequency

uint16_t spk_bit_resolution: speaker resolution(bits), UAC_BITS_ANY for any

uint32_t spk_samples_frequence: speaker frequency(Hz), UAC_FREQUENCY_ANY for any frequency

uint32_t spk_buf_size: size of speaker send buffer, should be a multiple of spk_ep_mps

uint32_t mic_buf_size: mic receive buffer size, 0 if not use

mic_callback_t mic_cb: mic callback, can not block in here!, NULL if not use

void *mic_cb_arg: mic callback args, NULL if not use Optional configs, Users need to specify parameters manually when they want to skip the get and process descriptors steps (used to speed up startup)

uint16_t mic_interface: (optional) microphone stream interface number, set 0 if not use

uint8_t mic_ep_addr: (optional) microphone interface endpoint address

uint32_t mic_ep_mps: (optional) microphone interface endpoint mps

uint16_t spk_interface: (optional) speaker stream interface number, set 0 if not use

uint8_t spk_ep_addr: (optional) speaker interface endpoint address

uint32_t spk_ep_mps: (optional) speaker interface endpoint mps

uint16_t ac_interface: (optional) audio control interface number, set 0 if not use

uint8_t mic_fu_id: (optional) microphone feature unit id, set 0 if not use

uint8_t spk_fu_id: (optional) speaker feature unit id, set 0 if not use

uint32_t flags: (optional) flags to control the driver behavers

Macros

FRAME_RESOLUTION_ANY: any uvc frame resolution

UAC_FREQUENCY_ANY: any uac sample frequency

UAC_BITS_ANY: any uac bit resolution

UAC_CH_ANY: any uac channel number

FPS2INTERVAL(fps): convert fps to uvc frame interval

FRAME_INTERVAL_FPS_5: 5 fps

FRAME_INTERVAL_FPS_10: 10 fps

FRAME_INTERVAL_FPS_15: 15 fps

FRAME_INTERVAL_FPS_20: 20 fps

FRAME_INTERVAL_FPS_30: 25 fps

FLAG_UVC_SUSPEND_AFTER_START: suspend uvc after usb_streaming_start

FLAG_UAC_SPK_SUSPEND_AFTER_START: suspend uac speaker after usb_streaming_start

FLAG_UAC_MIC_SUSPEND_AFTER_START: suspend uac microphone after usb_streaming_start

Type Definitions

typedef struct uvc_config uvc_config_t: UVC configurations, for params with (optional) label, users do not need to specify manually, unless there is a problem with descriptors, or users want to skip the get and process descriptors steps.

typedef void (*mic_callback_t)(mic_frame_t *frame, void *user_ptr): user callback function to handle incoming mic frames

typedef void (*state_callback_t)(usb_stream_state_t state, void *user_ptr): user callback function to handle usb device connection status

Enumerations

enum uvc_xfer_t

UVC stream usb transfer type, most camera using isochronous mode, bulk mode can also be support for higher bandwidth.

Values:

enumerator UVC_XFER_ISOC: Isochronous Transfer Mode

enumerator UVC_XFER_BULK: Bulk Transfer Mode

enumerator UVC_XFER_UNKNOWN: Unknown Mode

enum usb_stream_t

UVC stream format type, default using MJPEG format,.

Stream id, used for control

Values:

enumerator STREAM_UVC: usb video stream

enumerator STREAM_UAC_SPK: usb audio speaker stream

enumerator STREAM_UAC_MIC: usb audio microphone stream

enumerator STREAM_MAX: max stream id

enum usb_stream_state_t

USB device connection status.

Values:

enumerator STREAM_CONNECTED

enumerator STREAM_DISCONNECTED

enum stream_ctrl_t

Stream control type, which also depends on if device support.

Values:

enumerator CTRL_NONE: None

enumerator CTRL_SUSPEND: streaming suspend control. ctrl_data NULL

enumerator CTRL_RESUME: streaming resume control. ctrl_data NULL

enumerator CTRL_UAC_MUTE: mute control. ctrl_data (false/true)

enumerator CTRL_UAC_VOLUME: volume control. ctrl_data (0~100)

enumerator CTRL_MAX: max type value

提供有关此文档的反馈