USB Stream Component

[中文]

usb_stream is an USB UVC + UAC host driver for ESP32-S2/ESP32-S3, which supports read/write/control multimedia streaming from usb device. For example, at most one UVC + one Microphone + one Speaker streaming can be supported at the same time.

Features:

  1. Support video stream through UVC Stream interface, support both isochronous and bulk mode

  2. Support microphone stream and speaker stream through the UAC Stream interface

  3. Support volume, mute and other features control through the UAC Control interface

  4. Supports automatic parse of device configuration descriptors

  5. Support stream separately suspend and resume

USB Stream User Guide

  • Development board

    1. Any ESP32-S2 / ESP32-S3 board with USB Host port can be used. Note that the port must be able to output voltage.

  • USB UVC function

    1. Camera must be compatible with USB1.1 full-speed mode

    2. Camera must support MJPEG output

    3. Users can manually specify the camera interface, transmission mode, and image frame params through uvc_streaming_config()

    4. For isochronous mode camera, interface max packet size should no more than 512 bytes

    5. For isochronous mode camera, frame stream bandwidth should be less than 4 Mbps (500 KB/s)

    6. For bulk mode camera, frame stream bandwidth should be less than 8.8 Mbps (1100 KB/s)

    7. Please refer example readme for special camera requirements

  • USB UAC function

    1. Audio function must be compatible with UAC1.0 protocol

    2. Users need to manually specify the spk/mic sampling rate, bit width params through uac_streaming_config() function

  • USB UVC + UAC

    1. The UVC and UAC functions can be enabled separately. For example, only the UAC be configured to drive a usb headset, or only the UVC be configured to drive a USB camera

    2. If you need to enable UVC and UAC at the same time, note that the driver currently only supports Composite devices with both camera and audio interfaces, rather than two separate devices.

USB Stream Config Reference

  1. Using uvc_config_t to configure camera resolution and frame rate parameters. Using uac_config_t to configure the audio sampling rate, bit width and other parameters. The parameters are described as follows:

uvc_config_t uvc_config = {
    .frame_width = 320, // mjpeg width pixel, for example 320
    .frame_height = 240, // mjpeg height pixel, for example 240
    .frame_interval = FPS2INTERVAL(15), // frame interval (100µs units), such as 15fps
    .xfer_buffer_size = 32 * 1024, // single frame image size, need to be determined according to actual testing, 320 * 240 generally less than 35KB
    .xfer_buffer_a = pointer_buffer_a, // the internal transfer buffer
    .xfer_buffer_b = pointer_buffer_b, // the internal transfer buffer
    .frame_buffer_size = 32 * 1024, // single frame image size, need to determine according to actual test
    .frame_buffer = pointer_frame_buffer, // the image frame buffer
    .frame_cb = &camera_frame_cb, //camera callback, can block in here
    .frame_cb_arg = NULL, // camera callback args
}

uac_config_t uac_config = {
    .mic_bit_resolution = 16, //microphone resolution, bits
    .mic_samples_frequence = 16000, //microphone frequency, Hz
    .spk_bit_resolution = 16, //speaker resolution, bits
    .spk_samples_frequence = 16000, //speaker frequency, Hz
    .spk_buf_size = 16000, //size of speaker send buffer, should be a multiple of spk_ep_mps
    .mic_buf_size = 0, //mic receive buffer size, 0 if not use, else should be a multiple of mic_min_bytes
    .mic_cb = &mic_frame_cb, //mic callback, can not block in here
    .mic_cb_arg = NULL, //mic callback args
};
  1. Use the uvc_streaming_config() to config the UVC driver, or use the uac_streaming_config() to config the UAC driver

  2. Use the usb_streaming_start() to turn on the stream, then the driver will handle USB connection and negotiation.

  3. The host will matches the descriptors of the connected devices according to the user parameters. If the device fails to meet the configuration requirements, the driver prompt warning message

  4. If the device meets user configuration requirements, the Host will continue to receive the IN stream (UVC and UAC mic), and will call the user’s callbacks when new frames ready.

    1. The camera callback will be triggered after a new MJPEG image ready. The callback can block during processing, because which works in an independent task context.

    2. The mic callback will be triggered after mic_min_bytes() bytes data received. But the callback here must not block in any way, otherwise it will affect the reception of the next frame. If the block operations for mic is necessary, you can use the polling mode instead of the callback mode through uac_mic_streaming_read() api.

  5. User can send speaker OUT stream using uac_spk_streaming_write() through a ringbuffer, the Host will fetch the data when USB is free to send.

  6. Use the usb_streaming_control() to control the stream suspend/resume, uac volume/mute control can also be support if the USB device has such feature unit;

  7. Use the usb_streaming_stop() to stop the usb stream, USB resource will be completely released.

Bug report

ESP32-S2 ECO0 Chip SPI screen jitter when work with usb camera

  1. In earlier versions of the ESP32-S2 chip, USB transfers can cause SPI data contamination (esp32s2>=ECO1 and esp32s3 do not have this bug)

  2. Software workaround

  • spi_ll.h add below function

//components/hal/esp32s2/include/hal/spi_ll.h
static inline uint32_t spi_ll_tx_get_fifo_cnt(spi_dev_t *hw)
{
    return hw->dma_out_status.out_fifo_cnt;
}
  • modify spi_new_trans implement as below

// The function is called to send a new transaction, in ISR or in the task.
// Setup the transaction-specified registers and linked-list used by the DMA (or FIFO if DMA is not used)
static void SPI_MASTER_ISR_ATTR spi_new_trans(spi_device_t *dev, spi_trans_priv_t *trans_buf)
{
    //...................
    spi_hal_setup_trans(hal, hal_dev, &hal_trans);
    spi_hal_prepare_data(hal, hal_dev, &hal_trans);

    //Call pre-transmission callback, if any
    if (dev->cfg.pre_cb) dev->cfg.pre_cb(trans);
#if 1
    //USB Bug workaround
    //while (!((spi_ll_tx_get_fifo_cnt(SPI_LL_GET_HW(host->id)) == 12) || (spi_ll_tx_get_fifo_cnt(SPI_LL_GET_HW(host->id)) == trans->length / 8))) {
    while (trans->length && spi_ll_tx_get_fifo_cnt(SPI_LL_GET_HW(host->id)) == 0) {
        __asm__ __volatile__("nop");
        __asm__ __volatile__("nop");
        __asm__ __volatile__("nop");
    }
#endif
    //Kick off transfer
    spi_hal_user_start(hal);
}

Examples

  1. usb/host/usb_camera_mic_spk

  2. usb/host/usb_camera_lcd_display

  3. usb/host/usb_audio_player

API Reference

Header File

Functions

esp_err_t uvc_streaming_config(const uvc_config_t *config)

Config UVC streaming with user defined parameters.For normal use, user only need to specify no-optional parameters, and set optional parameters to 0 (the driver will find the correct value from the device descriptors). For quick start mode, user should specify all parameters manually to skip get and process descriptors steps.

Parameters

config – parameters defined in uvc_config_t

Returns

esp_err_t ESP_ERR_INVALID_STATE USB streaming is running, user need to stop streaming first ESP_ERR_INVALID_ARG Invalid argument ESP_OK Success

esp_err_t uac_streaming_config(const uac_config_t *config)

Config UAC streaming with user defined parameters.For normal use, user only need to specify no-optional parameters, and set optional parameters to 0 (the driver will find the correct value from the device descriptors). For quick start mode, user should specify all parameters manually to skip get and process descriptors steps.

Parameters

config – parameters defined in uvc_config_t

Returns

esp_err_t ESP_ERR_INVALID_STATE USB streaming is running, user need to stop streaming first ESP_ERR_INVALID_ARG Invalid argument ESP_OK Success

esp_err_t usb_streaming_start(void)

Start usb streaming with pre-configs, usb driver will create internal tasks to handle usb data from stream pipe, and run user’s callback after new frame ready.

Returns

ESP_ERR_INVALID_STATE streaming not configured, or streaming running already ESP_FAIL start failed ESP_OK start succeed

esp_err_t usb_streaming_stop(void)

Stop current usb streaming, internal tasks will be delete, related resource will be free.

Returns

ESP_ERR_INVALID_STATE streaming not started ESP_ERR_TIMEOUT stop wait timeout ESP_OK stop succeed

esp_err_t usb_streaming_connect_wait(size_t timeout_ms)

Wait for USB device connection.

Parameters

timeout_ms – timeout in ms

Returns

esp_err_t ESP_ERR_INVALID_STATE: usb streaming not started ESP_ERR_TIMEOUT: timeout ESP_OK: device connected

esp_err_t usb_streaming_state_register(state_callback_t cb, void *user_ptr)

This function registers a callback for USB streaming, please note that only one callback can be registered, the later registered callback will overwrite the previous one.

Parameters
  • cb – A pointer to a function that will be called when the USB streaming state changes.

  • user_ptr – user_ptr is a void pointer.

Returns

esp_err_t

  • ESP_OK Success

  • ESP_ERR_INVALID_STATE USB streaming is running, callback need register before start

esp_err_t usb_streaming_control(usb_stream_t stream, stream_ctrl_t ctrl_type, void *ctrl_value)

Control USB streaming with specific stream and control type.

Parameters
  • stream – stream type defined in usb_stream_t

  • ctrl_type – control type defined in stream_ctrl_t

  • ctrl_value – control value

Returns

ESP_ERR_INVALID_ARG invalid arg ESP_ERR_INVALID_STATE driver not configured or not started ESP_ERR_NOT_SUPPORTED current device not support this control type ESP_FAIL control failed ESP_OK succeed

esp_err_t uac_spk_streaming_write(void *data, size_t data_bytes, size_t timeout_ms)

Write data to the speaker buffer, will be send out when USB device is ready.

Parameters
  • data – The data to be written.

  • data_bytes – The size of the data to be written.

  • timeout_ms – The timeout value for writing data to the buffer.

Returns

ESP_ERR_INVALID_STATE spk stream not config ESP_ERR_NOT_FOUND spk interface not found ESP_ERR_TIMEOUT spk ringbuf full ESP_OK succeed

esp_err_t uac_mic_streaming_read(void *buf, size_t buf_size, size_t *data_bytes, size_t timeout_ms)

Read data from internal mic buffer, the actual size will be returned.

Parameters
  • buf – pointer to the buffer to store the received data

  • buf_size – The size of the data buffer.

  • data_bytes – The actual size read from buffer

  • timeout_ms – The timeout value for the read operation.

Returns

ESP_ERR_INVALID_ARG parameter error ESP_ERR_INVALID_STATE mic stream not config ESP_ERR_NOT_FOUND mic interface not found ESP_TIMEOUT timeout ESP_OK succeed

esp_err_t uac_frame_size_list_get(usb_stream_t stream, uac_frame_size_t *frame_list, size_t *list_size, size_t *cur_index)

Get the audio frame size list of current stream, the list contains audio channel number, bit resolution and samples frequency. IF list_size equals 1 and the samples_frequence equals 0, which means the frequency can be set to any value between samples_frequence_min and samples_frequence_max.

Parameters
  • stream – the stream type

  • frame_list – the output frame list, NULL to only get the list size

  • list_size – frame list size

  • cur_index – current frame index

Returns

esp_err_t

  • ESP_ERR_INVALID_ARG Parameter error

  • ESP_ERR_INVALID_STATE USB device not active

  • ESP_OK Success

esp_err_t uac_frame_size_reset(usb_stream_t stream, uint8_t ch_num, uint16_t bit_resolution, uint32_t samples_frequence)

Reset audio channel number, bit resolution and samples frequency, please reset when the streaming in suspend state. The new configs will be effective after streaming resume.

Parameters
  • stream – stream type

  • ch_num – audio channel numbers

  • bit_resolution – audio bit resolution

  • samples_frequence – audio samples frequency

Returns

esp_err_t

  • ESP_ERR_INVALID_ARG Parameter error

  • ESP_ERR_INVALID_STATE USB device not active

  • ESP_ERR_NOT_FOUND frequency not found

  • ESP_OK Success

  • ESP_FAIL Reset failed

esp_err_t uvc_frame_size_list_get(uvc_frame_size_t *frame_list, size_t *list_size, size_t *cur_index)

Get the frame size list of current connected camera.

Parameters
  • frame_list – the frame size list, can be NULL if only want to get list size

  • list_size – the list size

  • cur_index – current frame index

Returns

esp_err_t ESP_ERR_INVALID_ARG parameter error ESP_ERR_INVALID_STATE uvc stream not config or not active ESP_OK succeed

esp_err_t uvc_frame_size_reset(uint16_t frame_width, uint16_t frame_height, uint32_t frame_interval)

Reset the expected frame size and frame interval, please reset when uvc streaming in suspend state.The new configs will be effective after streaming resume.

Note: frame_width and frame_height can be set to 0 at the same time, which means no change on frame size.

Parameters
  • frame_width – frame width, FRAME_RESOLUTION_ANY means any width

  • frame_height – frame height, FRAME_RESOLUTION_ANY means any height

  • frame_interval – frame interval, 0 means no change

Returns

esp_err_t

Structures

struct uvc_config

UVC configurations, for params with (optional) label, users do not need to specify manually, unless there is a problem with descriptors, or users want to skip the get and process descriptors steps.

Public Members

uint16_t frame_width

Picture width, set FRAME_RESOLUTION_ANY for any resolution

uint16_t frame_height

Picture height, set FRAME_RESOLUTION_ANY for any resolution

uint32_t frame_interval

Frame interval in 100-ns units, 666666 ~ 15 Fps

uint32_t xfer_buffer_size

Transfer buffer size, using double buffer here, must larger than one frame size

uint8_t *xfer_buffer_a

Buffer a for usb payload

uint8_t *xfer_buffer_b

Buffer b for usb payload

uint32_t frame_buffer_size

Frame buffer size, must larger than one frame size

uint8_t *frame_buffer

Buffer for one frame

uvc_frame_callback_t frame_cb

callback function to handle incoming frame

void *frame_cb_arg

callback function arg

uvc_format_t format

(optional) UVC stream format, default using MJPEG Optional configs, Users need to specify parameters manually when they want to skip the get and process descriptors steps (used to speed up startup)

uvc_xfer_t xfer_type

(optional) UVC stream transfer type, UVC_XFER_ISOC or UVC_XFER_BULK

uint8_t format_index

(optional) Format index

uint8_t frame_index

(optional) Frame index, to choose resolution

uint16_t interface

(optional) UVC stream interface number

uint16_t interface_alt

(optional) UVC stream alternate interface, to choose MPS (Max Packet Size), bulk fix to 0

uint8_t ep_addr

(optional) endpoint address of selected alternate interface

uint32_t ep_mps

(optional) MPS of selected interface_alt

uint32_t flags

(optional) flags to control the driver behavers

struct mic_frame_t

mic frame type

Public Members

void *data

mic data

uint32_t data_bytes

mic data size

uint16_t bit_resolution

bit resolution in buffer

uint32_t samples_frequence

mic sample frequency

struct uvc_frame_size_t

uvc frame type

Public Members

uint16_t width

frame width

uint16_t height

frame height

uint32_t interval

frame interval

uint32_t interval_min

frame min interval

uint32_t interval_max

frame max interval

uint32_t interval_step

frame interval step

struct uac_frame_size_t

uac frame type

Public Members

uint8_t ch_num

channel numbers

uint16_t bit_resolution

bit resolution in buffer

uint32_t samples_frequence

sample frequency

uint32_t samples_frequence_min

min sample frequency

uint32_t samples_frequence_max

max sample frequency

struct uac_config_t

UAC configurations, for params with (optional) label, users do not need to specify manually, unless there is a problem with descriptor parse, or a problem with the device descriptor.

Public Members

uint8_t spk_ch_num

speaker channel numbers, UAC_CH_ANY for any channel number

uint8_t mic_ch_num

microphone channel numbers, UAC_CH_ANY for any channel number

uint16_t mic_bit_resolution

microphone resolution(bits), UAC_BITS_ANY for any bit resolution

uint32_t mic_samples_frequence

microphone frequency(Hz), UAC_FREQUENCY_ANY for any frequency

uint16_t spk_bit_resolution

speaker resolution(bits), UAC_BITS_ANY for any

uint32_t spk_samples_frequence

speaker frequency(Hz), UAC_FREQUENCY_ANY for any frequency

uint32_t spk_buf_size

size of speaker send buffer, should be a multiple of spk_ep_mps

uint32_t mic_buf_size

mic receive buffer size, 0 if not use

mic_callback_t mic_cb

mic callback, can not block in here!, NULL if not use

void *mic_cb_arg

mic callback args, NULL if not use Optional configs, Users need to specify parameters manually when they want to skip the get and process descriptors steps (used to speed up startup)

uint16_t mic_interface

(optional) microphone stream interface number, set 0 if not use

uint8_t mic_ep_addr

(optional) microphone interface endpoint address

uint32_t mic_ep_mps

(optional) microphone interface endpoint mps

uint16_t spk_interface

(optional) speaker stream interface number, set 0 if not use

uint8_t spk_ep_addr

(optional) speaker interface endpoint address

uint32_t spk_ep_mps

(optional) speaker interface endpoint mps

uint16_t ac_interface

(optional) audio control interface number, set 0 if not use

uint8_t mic_fu_id

(optional) microphone feature unit id, set 0 if not use

uint8_t spk_fu_id

(optional) speaker feature unit id, set 0 if not use

uint32_t flags

(optional) flags to control the driver behavers

Macros

FRAME_RESOLUTION_ANY

any uvc frame resolution

UAC_FREQUENCY_ANY

any uac sample frequency

UAC_BITS_ANY

any uac bit resolution

UAC_CH_ANY

any uac channel number

FPS2INTERVAL(fps)

convert fps to uvc frame interval

FRAME_INTERVAL_FPS_5

5 fps

FRAME_INTERVAL_FPS_10

10 fps

FRAME_INTERVAL_FPS_15

15 fps

FRAME_INTERVAL_FPS_20

20 fps

FRAME_INTERVAL_FPS_30

25 fps

FLAG_UVC_SUSPEND_AFTER_START

suspend uvc after usb_streaming_start

FLAG_UAC_SPK_SUSPEND_AFTER_START

suspend uac speaker after usb_streaming_start

FLAG_UAC_MIC_SUSPEND_AFTER_START

suspend uac microphone after usb_streaming_start

Type Definitions

typedef struct uvc_config uvc_config_t

UVC configurations, for params with (optional) label, users do not need to specify manually, unless there is a problem with descriptors, or users want to skip the get and process descriptors steps.

typedef void (*mic_callback_t)(mic_frame_t *frame, void *user_ptr)

user callback function to handle incoming mic frames

typedef void (*state_callback_t)(usb_stream_state_t state, void *user_ptr)

user callback function to handle usb device connection status

Enumerations

enum uvc_xfer_t

UVC stream usb transfer type, most camera using isochronous mode, bulk mode can also be support for higher bandwidth.

Values:

enumerator UVC_XFER_ISOC

Isochronous Transfer Mode

enumerator UVC_XFER_BULK

Bulk Transfer Mode

enumerator UVC_XFER_UNKNOWN

Unknown Mode

enum usb_stream_t

UVC stream format type, default using MJPEG format,.

Stream id, used for control

Values:

enumerator STREAM_UVC

usb video stream

enumerator STREAM_UAC_SPK

usb audio speaker stream

enumerator STREAM_UAC_MIC

usb audio microphone stream

enumerator STREAM_MAX

max stream id

enum usb_stream_state_t

USB device connection status.

Values:

enumerator STREAM_CONNECTED
enumerator STREAM_DISCONNECTED
enum stream_ctrl_t

Stream control type, which also depends on if device support.

Values:

enumerator CTRL_NONE

None

enumerator CTRL_SUSPEND

streaming suspend control. ctrl_data NULL

enumerator CTRL_RESUME

streaming resume control. ctrl_data NULL

enumerator CTRL_UAC_MUTE

mute control. ctrl_data (false/true)

enumerator CTRL_UAC_VOLUME

volume control. ctrl_data (0~100)

enumerator CTRL_MAX

max type value