USB Stream Component

[中文]

usb_stream is an USB UVC + UAC host driver for ESP32-S2/ESP32-S3, which supports read/write/control multimedia streaming from usb device. For example, at most one UVC + one Microphone + one Speaker streaming can be supported at the same time.

Features:

Support video stream through UVC Stream interface, support both isochronous and bulk mode
Support microphone stream and speaker stream through the UAC Stream interface
Support volume, mute and other features control through the UAC Control interface
Supports automatic parse of device configuration descriptors
Support stream separately suspend and resume

USB Stream User Guide

Development board
1. Any ESP32-S2 / ESP32-S3 board with USB Host port can be used. Note that the port must be able to output voltage.
USB UVC function
1. Camera must be compatible with USB1.1 full-speed mode
2. Camera must support MJPEG output
3. Users can manually specify the camera interface, transmission mode, and image frame params through uvc_streaming_config()
4. For isochronous mode camera, interface max packet size should no more than 512 bytes
5. For isochronous mode camera, frame stream bandwidth should be less than 4 Mbps (500 KB/s)
6. For bulk mode camera, frame stream bandwidth should be less than 8.8 Mbps (1100 KB/s)
7. Please refer example readme for special camera requirements
USB UAC function
1. Audio function must be compatible with UAC1.0 protocol
2. Users need to manually specify the spk/mic sampling rate, bit width params through uac_streaming_config() function
USB UVC + UAC
1. The UVC and UAC functions can be enabled separately. For example, only the UAC be configured to drive a usb headset, or only the UVC be configured to drive a USB camera
2. If you need to enable UVC and UAC at the same time, note that the driver currently only supports Composite devices with both camera and audio interfaces, rather than two separate devices.

USB Stream Config Reference

Using uvc_config_t to configure camera resolution and frame rate parameters. Using uac_config_t to configure the audio sampling rate, bit width and other parameters. The parameters are described as follows:

uvc_config_t uvc_config = {
    .frame_width = 320, // mjpeg width pixel, for example 320
    .frame_height = 240, // mjpeg height pixel, for example 240
    .frame_interval = FPS2INTERVAL(15), // frame interval (100µs units), such as 15fps
    .xfer_buffer_size = 32 * 1024, // single frame image size, need to be determined according to actual testing, 320 * 240 generally less than 35KB
    .xfer_buffer_a = pointer_buffer_a, // the internal transfer buffer
    .xfer_buffer_b = pointer_buffer_b, // the internal transfer buffer
    .frame_buffer_size = 32 * 1024, // single frame image size, need to determine according to actual test
    .frame_buffer = pointer_frame_buffer, // the image frame buffer
    .frame_cb = &camera_frame_cb, //camera callback, can block in here
    .frame_cb_arg = NULL, // camera callback args
}

uac_config_t uac_config = {
    .mic_bit_resolution = 16, //microphone resolution, bits
    .mic_samples_frequence = 16000, //microphone frequency, Hz
    .spk_bit_resolution = 16, //speaker resolution, bits
    .spk_samples_frequence = 16000, //speaker frequency, Hz
    .spk_buf_size = 16000, //size of speaker send buffer, should be a multiple of spk_ep_mps
    .mic_buf_size = 0, //mic receive buffer size, 0 if not use, else should be a multiple of mic_min_bytes
    .mic_cb = &mic_frame_cb, //mic callback, can not block in here
    .mic_cb_arg = NULL, //mic callback args
};

Use the uvc_streaming_config() to config the UVC driver, or use the uac_streaming_config() to config the UAC driver
Use the usb_streaming_start() to turn on the stream, then the driver will handle USB connection and negotiation.
The host will matches the descriptors of the connected devices according to the user parameters. If the device fails to meet the configuration requirements, the driver prompt warning message
If the device meets user configuration requirements, the Host will continue to receive the IN stream (UVC and UAC mic), and will call the user’s callbacks when new frames ready.
1. The camera callback will be triggered after a new MJPEG image ready. The callback can block during processing, because which works in an independent task context.
2. The mic callback will be triggered after mic_min_bytes() bytes data received. But the callback here must not block in any way, otherwise it will affect the reception of the next frame. If the block operations for mic is necessary, you can use the polling mode instead of the callback mode through uac_mic_streaming_read() api.
User can send speaker OUT stream using uac_spk_streaming_write() through a ringbuffer, the Host will fetch the data when USB is free to send.
Use the usb_streaming_control() to control the stream suspend/resume, uac volume/mute control can also be support if the USB device has such feature unit;
Use the usb_streaming_stop() to stop the usb stream, USB resource will be completely released.

Bug report

ESP32-S2 ECO0 Chip SPI screen jitter when work with usb camera

In earlier versions of the ESP32-S2 chip, USB transfers can cause SPI data contamination (esp32s2>=ECO1 and esp32s3 do not have this bug)
Software workaround

spi_ll.h add below function

//components/hal/esp32s2/include/hal/spi_ll.h
static inline uint32_t spi_ll_tx_get_fifo_cnt(spi_dev_t *hw)
{
    return hw->dma_out_status.out_fifo_cnt;
}

modify spi_new_trans implement as below

// The function is called to send a new transaction, in ISR or in the task.
// Setup the transaction-specified registers and linked-list used by the DMA (or FIFO if DMA is not used)
static void SPI_MASTER_ISR_ATTR spi_new_trans(spi_device_t *dev, spi_trans_priv_t *trans_buf)
{
    //...................
    spi_hal_setup_trans(hal, hal_dev, &hal_trans);
    spi_hal_prepare_data(hal, hal_dev, &hal_trans);

    //Call pre-transmission callback, if any
    if (dev->cfg.pre_cb) dev->cfg.pre_cb(trans);
#if 1
    //USB Bug workaround
    //while (!((spi_ll_tx_get_fifo_cnt(SPI_LL_GET_HW(host->id)) == 12) || (spi_ll_tx_get_fifo_cnt(SPI_LL_GET_HW(host->id)) == trans->length / 8))) {
    while (trans->length && spi_ll_tx_get_fifo_cnt(SPI_LL_GET_HW(host->id)) == 0) {
        __asm__ __volatile__("nop");
        __asm__ __volatile__("nop");
        __asm__ __volatile__("nop");
    }
#endif
    //Kick off transfer
    spi_hal_user_start(hal);
}

Examples

API Reference

Header File

components/usb/usb_stream/include/usb_stream.h

Functions

esp_err_t uvc_streaming_config(const uvc_config_t *config)

Config UVC streaming with user defined parameters.For normal use, user only need to specify no-optional parameters, and set optional parameters to 0 (the driver will find the correct value from the device descriptors). For quick start mode, user should specify all parameters manually to skip get and process descriptors steps.

Parameters: config – parameters defined in uvc_config_t
Returns: esp_err_t ESP_ERR_INVALID_STATE USB streaming is running, user need to stop streaming first ESP_ERR_INVALID_ARG Invalid argument ESP_OK Success

esp_err_t uac_streaming_config(const uac_config_t *config)

Config UAC streaming with user defined parameters.For normal use, user only need to specify no-optional parameters, and set optional parameters to 0 (the driver will find the correct value from the device descriptors). For quick start mode, user should specify all parameters manually to skip get and process descriptors steps.

Parameters: config – parameters defined in uvc_config_t
Returns: esp_err_t ESP_ERR_INVALID_STATE USB streaming is running, user need to stop streaming first ESP_ERR_INVALID_ARG Invalid argument ESP_OK Success

esp_err_t usb_streaming_start(void)

Start usb streaming with pre-configs, usb driver will create internal tasks to handle usb data from stream pipe, and run user’s callback after new frame ready.

Returns: ESP_ERR_INVALID_STATE streaming not configured, or streaming running already ESP_FAIL start failed ESP_OK start succeed

esp_err_t usb_streaming_stop(void)

Stop current usb streaming, internal tasks will be delete, related resource will be free.

Returns: ESP_ERR_INVALID_STATE streaming not started ESP_ERR_TIMEOUT stop wait timeout ESP_OK stop succeed

esp_err_t usb_streaming_connect_wait(size_t timeout_ms)

Wait for USB device connection.

Parameters: timeout_ms – timeout in ms
Returns: esp_err_t ESP_ERR_INVALID_STATE: usb streaming not started ESP_ERR_TIMEOUT: timeout ESP_OK: device connected

esp_err_t usb_streaming_state_register(state_callback_t cb, void *user_ptr)

This function registers a callback for USB streaming, please note that only one callback can be registered, the later registered callback will overwrite the previous one.

Parameters

cb – A pointer to a function that will be called when the USB streaming state changes.
user_ptr – user_ptr is a void pointer.

Returns

esp_err_t

ESP_OK Success
ESP_ERR_INVALID_STATE USB streaming is running, callback need register before start

esp_err_t usb_streaming_control(usb_stream_t stream, stream_ctrl_t ctrl_type, void *ctrl_value)

Control USB streaming with specific stream and control type.

Parameters

stream – stream type defined in usb_stream_t
ctrl_type – control type defined in stream_ctrl_t
ctrl_value – control value

Returns

ESP_ERR_INVALID_ARG invalid arg ESP_ERR_INVALID_STATE driver not configured or not started ESP_ERR_NOT_SUPPORTED current device not support this control type ESP_FAIL control failed ESP_OK succeed

esp_err_t uac_spk_streaming_write(void *data, size_t data_bytes, size_t timeout_ms)

Write data to the speaker buffer, will be send out when USB device is ready.

Parameters

data – The data to be written.
data_bytes – The size of the data to be written.
timeout_ms – The timeout value for writing data to the buffer.

Returns

ESP_ERR_INVALID_STATE spk stream not config ESP_ERR_NOT_FOUND spk interface not found ESP_ERR_TIMEOUT spk ringbuf full ESP_OK succeed

esp_err_t uac_mic_streaming_read(void *buf, size_t buf_size, size_t *data_bytes, size_t timeout_ms)

Read data from internal mic buffer, the actual size will be returned.

Parameters

buf – pointer to the buffer to store the received data
buf_size – The size of the data buffer.
data_bytes – The actual size read from buffer
timeout_ms – The timeout value for the read operation.

Returns

ESP_ERR_INVALID_ARG parameter error ESP_ERR_INVALID_STATE mic stream not config ESP_ERR_NOT_FOUND mic interface not found ESP_TIMEOUT timeout ESP_OK succeed

esp_err_t uac_frame_size_list_get(usb_stream_t stream, uac_frame_size_t *frame_list, size_t *list_size, size_t *cur_index)

Get the audio frame size list of current stream, the list contains audio channel number, bit resolution and samples frequency. IF list_size equals 1 and the samples_frequence equals 0, which means the frequency can be set to any value between samples_frequence_min and samples_frequence_max.

Parameters

stream – the stream type
frame_list – the output frame list, NULL to only get the list size
list_size – frame list size
cur_index – current frame index

Returns

esp_err_t

ESP_ERR_INVALID_ARG Parameter error
ESP_ERR_INVALID_STATE USB device not active
ESP_OK Success

esp_err_t uac_frame_size_reset(usb_stream_t stream, uint8_t ch_num, uint16_t bit_resolution, uint32_t samples_frequence)

Reset audio channel number, bit resolution and samples frequency, please reset when the streaming in suspend state. The new configs will be effective after streaming resume.

Parameters

stream – stream type
ch_num – audio channel numbers
bit_resolution – audio bit resolution
samples_frequence – audio samples frequency

Returns

esp_err_t

ESP_ERR_INVALID_ARG Parameter error
ESP_ERR_INVALID_STATE USB device not active
ESP_ERR_NOT_FOUND frequency not found
ESP_OK Success
ESP_FAIL Reset failed

esp_err_t uvc_frame_size_list_get(uvc_frame_size_t *frame_list, size_t *list_size, size_t *cur_index)

Get the frame size list of current connected camera.

Parameters

frame_list – the frame size list, can be NULL if only want to get list size
list_size – the list size
cur_index – current frame index

Returns

esp_err_t ESP_ERR_INVALID_ARG parameter error ESP_ERR_INVALID_STATE uvc stream not config or not active ESP_OK succeed

esp_err_t uvc_frame_size_reset(uint16_t frame_width, uint16_t frame_height, uint32_t frame_interval)

Reset the expected frame size and frame interval, please reset when uvc streaming in suspend state.The new configs will be effective after streaming resume.

Note: frame_width and frame_height can be set to 0 at the same time, which means no change on frame size.

Parameters

frame_width – frame width, FRAME_RESOLUTION_ANY means any width
frame_height – frame height, FRAME_RESOLUTION_ANY means any height
frame_interval – frame interval, 0 means no change

Returns

esp_err_t

Structures

struct uvc_config

UVC configurations, for params with (optional) label, users do not need to specify manually, unless there is a problem with descriptors, or users want to skip the get and process descriptors steps.

Public Members

uint16_t frame_width: Picture width, set FRAME_RESOLUTION_ANY for any resolution

uint16_t frame_height: Picture height, set FRAME_RESOLUTION_ANY for any resolution

uint32_t frame_interval: Frame interval in 100-ns units, 666666 ~ 15 Fps

uint32_t xfer_buffer_size: Transfer buffer size, using double buffer here, must larger than one frame size

uint8_t *xfer_buffer_a: Buffer a for usb payload

uint8_t *xfer_buffer_b: Buffer b for usb payload

uint32_t frame_buffer_size: Frame buffer size, must larger than one frame size

uint8_t *frame_buffer: Buffer for one frame

uvc_frame_callback_t frame_cb: callback function to handle incoming frame

void *frame_cb_arg: callback function arg

uvc_format_t format: (optional) UVC stream format, default using MJPEG Optional configs, Users need to specify parameters manually when they want to skip the get and process descriptors steps (used to speed up startup)

uvc_xfer_t xfer_type: (optional) UVC stream transfer type, UVC_XFER_ISOC or UVC_XFER_BULK

uint8_t format_index: (optional) Format index

uint8_t frame_index: (optional) Frame index, to choose resolution

uint16_t interface: (optional) UVC stream interface number

uint16_t interface_alt: (optional) UVC stream alternate interface, to choose MPS (Max Packet Size), bulk fix to 0

uint8_t ep_addr: (optional) endpoint address of selected alternate interface

uint32_t ep_mps: (optional) MPS of selected interface_alt

uint32_t flags: (optional) flags to control the driver behavers

struct mic_frame_t

mic frame type

Public Members

void *data: mic data

uint32_t data_bytes: mic data size

uint16_t bit_resolution: bit resolution in buffer

uint32_t samples_frequence: mic sample frequency

struct uvc_frame_size_t

uvc frame type

Public Members

uint16_t width: frame width

uint16_t height: frame height

uint32_t interval: frame interval

uint32_t interval_min: frame min interval

uint32_t interval_max: frame max interval

uint32_t interval_step: frame interval step

struct uac_frame_size_t

uac frame type

Public Members

uint8_t ch_num: channel numbers

uint16_t bit_resolution: bit resolution in buffer

uint32_t samples_frequence: sample frequency

uint32_t samples_frequence_min: min sample frequency

uint32_t samples_frequence_max: max sample frequency

struct uac_config_t

UAC configurations, for params with (optional) label, users do not need to specify manually, unless there is a problem with descriptor parse, or a problem with the device descriptor.

Public Members

uint8_t spk_ch_num: speaker channel numbers, UAC_CH_ANY for any channel number

uint8_t mic_ch_num: microphone channel numbers, UAC_CH_ANY for any channel number

uint16_t mic_bit_resolution: microphone resolution(bits), UAC_BITS_ANY for any bit resolution

uint32_t mic_samples_frequence: microphone frequency(Hz), UAC_FREQUENCY_ANY for any frequency

uint16_t spk_bit_resolution: speaker resolution(bits), UAC_BITS_ANY for any

uint32_t spk_samples_frequence: speaker frequency(Hz), UAC_FREQUENCY_ANY for any frequency

uint32_t spk_buf_size: size of speaker send buffer, should be a multiple of spk_ep_mps

uint32_t mic_buf_size: mic receive buffer size, 0 if not use

mic_callback_t mic_cb: mic callback, can not block in here!, NULL if not use

void *mic_cb_arg: mic callback args, NULL if not use Optional configs, Users need to specify parameters manually when they want to skip the get and process descriptors steps (used to speed up startup)

uint16_t mic_interface: (optional) microphone stream interface number, set 0 if not use

uint8_t mic_ep_addr: (optional) microphone interface endpoint address

uint32_t mic_ep_mps: (optional) microphone interface endpoint mps

uint16_t spk_interface: (optional) speaker stream interface number, set 0 if not use

uint8_t spk_ep_addr: (optional) speaker interface endpoint address

uint32_t spk_ep_mps: (optional) speaker interface endpoint mps

uint16_t ac_interface: (optional) audio control interface number, set 0 if not use

uint8_t mic_fu_id: (optional) microphone feature unit id, set 0 if not use

uint8_t spk_fu_id: (optional) speaker feature unit id, set 0 if not use

uint32_t flags: (optional) flags to control the driver behavers

Macros

FRAME_RESOLUTION_ANY: any uvc frame resolution

UAC_FREQUENCY_ANY: any uac sample frequency

UAC_BITS_ANY: any uac bit resolution

UAC_CH_ANY: any uac channel number

FPS2INTERVAL(fps): convert fps to uvc frame interval

FRAME_INTERVAL_FPS_5: 5 fps

FRAME_INTERVAL_FPS_10: 10 fps

FRAME_INTERVAL_FPS_15: 15 fps

FRAME_INTERVAL_FPS_20: 20 fps

FRAME_INTERVAL_FPS_30: 25 fps

FLAG_UVC_SUSPEND_AFTER_START: suspend uvc after usb_streaming_start

FLAG_UAC_SPK_SUSPEND_AFTER_START: suspend uac speaker after usb_streaming_start

FLAG_UAC_MIC_SUSPEND_AFTER_START: suspend uac microphone after usb_streaming_start

Type Definitions

typedef struct uvc_config uvc_config_t: UVC configurations, for params with (optional) label, users do not need to specify manually, unless there is a problem with descriptors, or users want to skip the get and process descriptors steps.

typedef void (*mic_callback_t)(mic_frame_t *frame, void *user_ptr): user callback function to handle incoming mic frames

typedef void (*state_callback_t)(usb_stream_state_t state, void *user_ptr): user callback function to handle usb device connection status

Enumerations

enum uvc_xfer_t

UVC stream usb transfer type, most camera using isochronous mode, bulk mode can also be support for higher bandwidth.

Values:

enumerator UVC_XFER_ISOC: Isochronous Transfer Mode

enumerator UVC_XFER_BULK: Bulk Transfer Mode

enumerator UVC_XFER_UNKNOWN: Unknown Mode

enum usb_stream_t

UVC stream format type, default using MJPEG format,.

Stream id, used for control

Values:

enumerator STREAM_UVC: usb video stream

enumerator STREAM_UAC_SPK: usb audio speaker stream

enumerator STREAM_UAC_MIC: usb audio microphone stream

enumerator STREAM_MAX: max stream id

enum usb_stream_state_t

USB device connection status.

Values:

enumerator STREAM_CONNECTED

enumerator STREAM_DISCONNECTED

enum stream_ctrl_t

Stream control type, which also depends on if device support.

Values:

enumerator CTRL_NONE: None

enumerator CTRL_SUSPEND: streaming suspend control. ctrl_data NULL

enumerator CTRL_RESUME: streaming resume control. ctrl_data NULL

enumerator CTRL_UAC_MUTE: mute control. ctrl_data (false/true)

enumerator CTRL_UAC_VOLUME: volume control. ctrl_data (0~100)

enumerator CTRL_MAX: max type value

Provide feedback about this document