ESP Media Service

[中文]

Introduction

ESP Media Service defines a common media interface for audio and video services in ESP-ADF, based on ESP Service. An application creates a service, configures the media stream, and then links a source stream to a sink stream. Once linked and started, media frames flow through the read and write interfaces, so the application does not need to forward frames by hand. Audio and video capture and playback are documented in ESP-GMF General Multimedia Framework.

Feature List

  • A unified audio/video service model with three roles: source, sink, and source-sink (ESP_MEDIA_ROLE_SRC / SINK / SRC_SINK)

  • Multi-media endpoints based on a stream ID (esp_media_stream_id_t); a single service can expose multiple streams at the same time

  • Request negotiation at link time: a sink can express requirements to the source through esp_media_service_request_t (for example, whether a global cache is needed)

  • The provider read interface (esp_media_provider_t) is decoupled from the track manager write interface

  • A built-in default in-memory track manager (esp_media_track_mngr_t) that supports either per-track independent caching or global arrival-order caching

  • Service lifecycle is uniformly managed through ESP Service, with media-related operations layered on top of the base class as an independent vtable (esp_media_service_ops_t)

Technical Deep Dive

Service Roles and Linking

ESP Media Service declares whether it is a source, a sink, or both, through get_role in esp_media_service_ops_t; esp_media_service_link() verifies that the roles of the selected source and sink are compatible when linking, and then passes the source’s provider to the sink.

        flowchart TD
    Create["Create service"] --> Config["Configure service"]
    Config --> Link["Link source/sink streams"]
    Link --> Start["Start service"]
    Start --> Flow["Media frames flow"]
    Flow --> Stop["Stop service"]
    Stop --> Unlink["Unlink and destroy"]
    
esp_media_stream_id_t stream = ESP_MEDIA_DEFAULT_STREAM;

esp_media_service_link(src_service, stream, sink_service, stream);

esp_service_start(sink_service);
esp_service_start(src_service);

Once the link is established, media data flows from the provider exported by the source service to the sink; removing the link requires calling esp_media_service_unlink(), which clears the provider currently set on the sink stream.

Provider and Track Manager

The delivery of media data relies on two mutually decoupled interfaces: at link time, the sink obtains a read-only esp_media_provider_t from the source service, and acquires and releases frames through esp_media_provider_acquire_frame() / esp_media_provider_release_frame(); the source service, in turn, holds an esp_media_track_mngr_t, produces frames through its write API, and exposes the provider handle it exports to downstream consumers.

/* Source service side: create the track manager, register a track, and export the provider */
esp_media_track_mngr_cfg_t cfg = { .max_track_num = 2 };
esp_media_track_mngr_create(&cfg, &svc->mngr);
esp_media_track_mngr_add_track(svc->mngr, &audio_track);
esp_media_track_mngr_get_provider(svc->mngr, &svc->provider);

/* Sink side: acquire and release a frame */
esp_media_frame_t frame = {0};
if (esp_media_provider_acquire_frame(&sink->provider, &frame, timeout_ms) == ESP_OK) {
    process_frame(&frame);
    esp_media_provider_release_frame(&sink->provider, &frame);
}

Warning

A frame acquired through esp_media_provider_acquire_frame() must be released by calling esp_media_provider_release_frame(); after it is released, frame.data must no longer be accessed.

Track Manager Cache Modes

The default esp_media_track_mngr_t supports two payload ownership modes: ESP_MEDIA_TRACK_CACHE_INTERNAL, in which the manager copies and holds the frame data itself, and ESP_MEDIA_TRACK_CACHE_USER, which caches only the frame metadata while the payload remains owned by the user and is returned through the frame_release callback after it has been consumed. A global cache is also supported, letting multiple tracks share a single queue ordered by arrival time, which is suitable for scenarios such as RTMP where audio and video are interleaved; enabling the global cache must be configured before tracks are added.

Stop and Abort Ordering

The media interface is designed to tolerate the stop sequence: when the source service stops, it should call esp_media_track_write_abort(), which notifies downstream consumers through the ESP_MEDIA_PROVIDER_EVENT_TRACKS_ABORT event; when the sink stops, it should first set a local stop flag, then call esp_media_provider_abort() to wake up any blocked read, wait for the task to exit and release any frames it has acquired, and only then unlink. If a track manager is shared by multiple services through linking, it must be unlinked before reset or destroy is performed on it; as long as any task still holds an acquired frame or is blocked on the queue, reset or destroy must not be performed on the track manager.

Application Examples

Complete source/sink examples are in the examples directory of the esp_media_service component repository. See ESP Service for the service base class.

FAQ

Q1: Can a service implement only one of ``get_provider`` or ``set_provider``?

Not necessarily. A service with the role ESP_MEDIA_ROLE_SRC_SINK can implement both at the same time, acting as a source for downstream consumers as well as a sink for upstream producers; esp_media_service_link() only checks role compatibility for the selected pair of source/sink services.

API Reference

Header File

Functions

esp_err_t esp_media_service_init(esp_media_service_t *service, const esp_media_service_config_t *config)

Initialize a media service base object. 

    Derived services call this after allocating or embedding
    esp_media_service_t, before exposing the base esp_service_t
Parameters:

  • service – [inout] Media service object to initialize

  • config – [in] Media service configuration

Returns:

  • ESP_OK On success

  • ESP_ERR_INVALID_ARG service or config is NULL

  • Others Error returned by esp_service_init()

esp_err_t esp_media_service_deinit(esp_service_t *service)

Deinitialize a media service base object.

Parameters:

service – [in] Base service pointer returned by ESP_SERVICE_BASE()

Returns:

  • ESP_OK On success

  • ESP_ERR_INVALID_ARG service is NULL

  • Others Error returned by esp_service_deinit()

esp_err_t esp_media_service_get_role(esp_service_t *service, esp_media_role_t *out_role)

Query the media role of a service.

Parameters:

  • service – [in] Base media service handle

  • out_role – [out] Output media role

Returns:

  • ESP_OK On success

  • ESP_ERR_INVALID_ARG service or out_role is NULL

  • ESP_ERR_NOT_SUPPORTED Operation is not implemented by the service

  • Others Error returned by the service implementation

esp_err_t esp_media_service_get_provider(esp_service_t *service, esp_media_stream_id_t stream, esp_media_provider_t *out_provider)

Get a provider from a source stream.

Parameters:

  • service – [in] Base media service handle

  • stream – [in] Source stream ID

  • out_provider – [out] Output provider

Returns:

  • ESP_OK On success

  • ESP_ERR_INVALID_ARG Service or provider is NULL

  • ESP_ERR_NOT_SUPPORTED Operation is not implemented by the service

  • Others Error returned by the service implementation

esp_err_t esp_media_service_set_provider(esp_service_t *service, esp_media_stream_id_t stream, const esp_media_provider_t *provider)

Set a provider on a sink stream. 

    Pass NULL provider to disconnect the sink stream
Parameters:

  • service – [in] Base media service handle

  • stream – [in] Sink stream ID

  • provider – [in] Provider handle to consume, or NULL to clear

Returns:

  • ESP_OK On success

  • ESP_ERR_INVALID_ARG service is NULL, or provider has no ops

  • ESP_ERR_NOT_SUPPORTED Operation is not implemented by the service

  • Others Error returned by the service implementation

esp_err_t esp_media_service_get_request(esp_service_t *service, esp_media_stream_id_t stream, esp_media_service_request_t *out_request)

Get link request hints for a media stream.

Parameters:

  • service – [in] Base media service handle

  • stream – [in] Stream ID

  • out_request – [out] Output request hints

Returns:

  • ESP_OK On success

  • ESP_ERR_INVALID_ARG service or out_request is NULL

  • ESP_ERR_NOT_SUPPORTED Operation is not implemented by the service

  • Others Error returned by the service implementation

esp_err_t esp_media_service_set_request(esp_service_t *service, esp_media_stream_id_t stream, const esp_media_service_request_t *request)

Set link request hints for a media stream.

Parameters:

  • service – [in] Base media service handle

  • stream – [in] Stream ID

  • request – [in] Request hints to apply

Returns:

  • ESP_OK On success

  • ESP_ERR_INVALID_ARG service or request is NULL

  • ESP_ERR_NOT_SUPPORTED Operation is not implemented by the service

  • Others Error returned by the service implementation

esp_err_t esp_media_service_link(esp_service_t *src_service, esp_media_stream_id_t src_stream, esp_service_t *sink_service, esp_media_stream_id_t sink_stream)

Link a source stream to a sink stream. 

    The sink request hints are applied to the source when supported, then
    the source provider is passed to the sink
Parameters:

  • src_service – [in] Source media service handle

  • src_stream – [in] Source stream ID

  • sink_service – [in] Sink media service handle

  • sink_stream – [in] Sink stream ID

Returns:

  • ESP_OK On success

  • ESP_ERR_INVALID_ARG src_service or sink_service is NULL

  • ESP_ERR_NOT_SUPPORTED Role check failed or required operation is missing

  • Others Error returned by source or sink implementation

esp_err_t esp_media_service_unlink(esp_service_t *src_service, esp_media_stream_id_t src_stream, esp_service_t *sink_service, esp_media_stream_id_t sink_stream)

Unlink a source stream from a sink stream. 

    Clears the provider currently set on the sink stream
Parameters:

  • src_service – [in] Source media service handle

  • src_stream – [in] Source stream ID

  • sink_service – [in] Sink media service handle

  • sink_stream – [in] Sink stream ID

Returns:

  • ESP_OK On success

  • ESP_ERR_INVALID_ARG src_service or sink_service is NULL

  • ESP_ERR_NOT_SUPPORTED Role check failed or sink operation is missing

  • Others Error returned by source or sink implementation

Structures

struct esp_media_service_request_t

Media-service link request/capability hints. 

    The structure is intentionally extensible so future services can add
    link-time requests without changing the media op shape

Public Members

bool need_global_cache

Sink requests source frames in one arrival-order cache

struct esp_media_service_ops_t

Media-service virtual operations.

Public Members

esp_err_t (*get_role)(esp_service_t *service, esp_media_role_t *out_role)

Query source/sink role

esp_err_t (*get_provider)(esp_service_t *service, esp_media_stream_id_t stream, esp_media_provider_t *out_provider)

Get provider exported by a source stream

esp_err_t (*set_provider)(esp_service_t *service, esp_media_stream_id_t stream, const esp_media_provider_t *provider)

Set provider consumed by a sink stream

esp_err_t (*get_request)(esp_service_t *service, esp_media_stream_id_t stream, esp_media_service_request_t *request)

Get sink link requests

esp_err_t (*set_request)(esp_service_t *service, esp_media_stream_id_t stream, const esp_media_service_request_t *request)

Apply sink requests to a source

struct esp_media_service_config_t

Media service configuration.

Public Members

const char *name

Service instance name

void *user_data

User data passed to esp_service

const esp_service_ops_t *service_ops

Optional esp_service lifecycle ops

const esp_media_service_ops_t *media_ops

Optional media ops

struct esp_media_service

Base media service structure Derived services embed this as the first member.

Public Members

esp_service_t base

Base service, must stay first

const esp_media_service_ops_t *media_ops

Media virtual operations

Macros

ESP_MEDIA_SERVICE_CONFIG_DEFAULT()

SPDX-FileCopyrightText: 2026 Espressif Systems (Shanghai) CO., LTD SPDX-License-Identifier: LicenseRef-Espressif-Modified-MIT

See LICENSE file for details. Default media service configuration

ESP_MEDIA_DEFAULT_STREAM

Default media stream ID

Type Definitions

typedef uint16_t esp_media_stream_id_t

Instance-aware media stream address.

typedef struct esp_media_service esp_media_service_t

Base media service structure Derived services embed this as the first member.

Enumerations

enum esp_media_role_t

Media service data direction.

Values:

enumerator ESP_MEDIA_ROLE_NONE
enumerator ESP_MEDIA_ROLE_SRC

Service produces media frames

enumerator ESP_MEDIA_ROLE_SINK

Service consumes media frames

enumerator ESP_MEDIA_ROLE_SRC_SINK

Service produces and consumes frames

Header File

Functions

esp_err_t esp_media_provider_get_track_num(const esp_media_provider_t *provider, uint16_t *out_num)

Get the number of tracks exposed by a provider.

SPDX-FileCopyrightText: 2026 Espressif Systems (Shanghai) CO., LTD SPDX-License-Identifier: LicenseRef-Espressif-Modified-MIT

See LICENSE file for details.

Parameters:

  • provider – [in] Provider handle

  • out_num – [out] Output track count

Returns:

  • ESP_OK On success

  • ESP_ERR_INVALID_ARG provider or out_num is NULL

  • ESP_ERR_NOT_SUPPORTED Operation is not implemented by the provider

  • Others Error returned by the provider implementation

esp_err_t esp_media_provider_get_track_info(const esp_media_provider_t *provider, uint16_t index, esp_media_track_info_t *out_info)

Get track metadata by provider track index.

Parameters:

  • provider – [in] Provider handle

  • index – [in] Track index in the provider

  • out_info – [out] Output track metadata

Returns:

  • ESP_OK On success

  • ESP_ERR_INVALID_ARG provider or out_info is NULL

  • ESP_ERR_NOT_SUPPORTED Operation is not implemented by the provider

  • Others Error returned by the provider implementation

esp_err_t esp_media_provider_set_event_cb(const esp_media_provider_t *provider, esp_media_provider_event_cb_t cb, void *event_ctx)

Register a provider event callback.

Parameters:

  • provider – [in] Provider handle

  • cb – [in] Event callback, or NULL to clear it

  • event_ctx – [in] User context passed to cb

Returns:

  • ESP_OK On success

  • ESP_ERR_INVALID_ARG provider is NULL

  • ESP_ERR_NOT_SUPPORTED Operation is not implemented by the provider

  • Others Error returned by the provider implementation

esp_err_t esp_media_provider_abort(const esp_media_provider_t *provider)

Abort provider-side blocking read operations.

Parameters:

provider – [in] Provider handle

Returns:

  • ESP_OK On success

  • ESP_ERR_INVALID_ARG provider is NULL

  • ESP_ERR_NOT_SUPPORTED Operation is not implemented by the provider

  • Others Error returned by the provider implementation

esp_err_t esp_media_provider_acquire_frame(const esp_media_provider_t *provider, esp_media_frame_t *out_frame, uint32_t timeout_ms)

Acquire the next frame from a provider. 

    The returned frame must be released with esp_media_provider_release_frame().
    Set fields such as type or track_id in out_frame before the call to
    select a specific track when the provider supports it
Parameters:

  • provider – [in] Provider handle

  • out_frame – [inout] Input track selector, output acquired frame

  • timeout_ms – [in] Timeout in milliseconds; 0 means no wait, UINT32_MAX means wait forever

Returns:

  • ESP_OK On success

  • ESP_ERR_INVALID_ARG provider or out_frame is NULL

  • ESP_ERR_NOT_SUPPORTED Operation is not implemented by the provider

  • Others Error returned by the provider implementation

esp_err_t esp_media_provider_read_frame(const esp_media_provider_t *provider, esp_media_frame_t *out_frame, uint32_t timeout_ms)

Read the next frame into caller-provided storage. 

    Set out_frame->data and out_frame->size before calling. On success,
    out_frame->size is updated to the actual payload size
Parameters:

  • provider – [in] Provider handle

  • out_frame – [inout] Input buffer descriptor, output frame descriptor

  • timeout_ms – [in] Timeout in milliseconds; 0 means no wait, UINT32_MAX means wait forever

Returns:

  • ESP_OK On success

  • ESP_ERR_INVALID_ARG provider or out_frame is NULL

  • ESP_ERR_INVALID_SIZE Provided buffer is too small

  • ESP_ERR_NOT_SUPPORTED Operation is not implemented by the provider

  • Others Error returned by the provider implementation

esp_err_t esp_media_provider_release_frame(const esp_media_provider_t *provider, esp_media_frame_t *frame)

Release a frame acquired from a provider.

Parameters:

  • provider – [in] Provider handle

  • frame – [in] Frame returned by esp_media_provider_acquire_frame()

Returns:

  • ESP_OK On success

  • ESP_ERR_INVALID_ARG provider or frame is NULL

  • ESP_ERR_NOT_SUPPORTED Operation is not implemented by the provider

  • Others Error returned by the provider implementation

Header File

Functions

esp_err_t esp_media_track_mngr_create(const esp_media_track_mngr_cfg_t *cfg, esp_media_track_mngr_t **out_mngr)

Create a media track manager.

Parameters:

  • cfg – [in] Provider configuration

  • out_mngr – [out] Output track manager handle

Returns:

  • ESP_OK On success

  • ESP_ERR_INVALID_ARG cfg is NULL, max_track_num is 0, or out_provider is NULL

  • ESP_ERR_NO_MEM Allocation failed

esp_err_t esp_media_track_mngr_destroy(esp_media_track_mngr_t *mngr)

Destroy a media track manager. 

    Wakes and destroys internal queues, releases pending user-owned
    frames through their release callbacks, and frees the track manager
Parameters:

mngr – [in] Track manager handle returned by esp_media_track_mngr_create()

Returns:

  • ESP_OK On success

  • ESP_ERR_INVALID_ARG Track manager is NULL

esp_err_t esp_media_track_mngr_reset(esp_media_track_mngr_t *mngr)

Reset tracks and queued data. 

    Removes all tracks and clears abort state. Global-cache mode is kept.
    User must re-add tracks after reset
Parameters:

mngr – [in] Track manager handle

Returns:

  • ESP_OK On success

  • ESP_ERR_INVALID_ARG Track manager is NULL

esp_err_t esp_media_track_mngr_set_global_cache(esp_media_track_mngr_t *mngr, bool enable, size_t cache_size)

Configure global cache mode before tracks are added.

Parameters:

  • mngr – [in] Track manager handle

  • enable – [in] true to use one arrival-order cache shared by all tracks

  • cache_size – [in] Shared queue byte size, 0 uses default

Returns:

  • ESP_OK On success

  • ESP_ERR_INVALID_ARG Track manager is NULL

  • ESP_ERR_INVALID_STATE Tracks have already been added

  • ESP_ERR_NO_MEM Allocation failed

esp_err_t esp_media_track_mngr_add_track(esp_media_track_mngr_t *mngr, const esp_media_track_mngr_track_cfg_t *cfg)

Add a track to a track manager.

Parameters:

  • mngr – [in] Track manager handle

  • cfg – [in] Track configuration

Returns:

  • ESP_OK On success

  • ESP_ERR_INVALID_ARG Track manager or cfg is NULL

  • ESP_ERR_NO_MEM Track limit reached or queue allocation failed

esp_err_t esp_media_track_mngr_update_track(esp_media_track_mngr_t *mngr, uint16_t index, const esp_media_track_info_t *info)

Queue a track metadata update. 

    The update is committed when provider acquire/read reaches the queued
    zero-size frame with ESP_MEDIA_FRAME_FLAG_TRACK_CHANGED. The provider
    event callback is invoked synchronously at that point
Parameters:

  • mngr – [in] Track manager handle

  • index – [in] Track index to update

  • info – [in] New track metadata

Returns:

  • ESP_OK On success

  • ESP_ERR_INVALID_ARG Track manager or info is NULL

  • ESP_ERR_NOT_FOUND index is out of range

  • ESP_ERR_NO_MEM Track queue is unavailable

  • ESP_ERR_TIMEOUT Failed to queue update frame

  • ESP_FAIL Failed to commit update frame

esp_err_t esp_media_track_mngr_remove_track(esp_media_track_mngr_t *mngr, uint16_t index)

Queue a track removal notification. 

    The removal is committed when provider acquire/read reaches the queued
    zero-size frame with ESP_MEDIA_FRAME_FLAG_TRACK_REMOVED. The provider
    event callback is invoked synchronously at that point
Parameters:

  • mngr – [in] Track manager handle

  • index – [in] Track index to remove

Returns:

  • ESP_OK On success

  • ESP_ERR_INVALID_ARG Track manager is NULL

  • ESP_ERR_NOT_FOUND index is out of range

  • ESP_ERR_NO_MEM Track queue is unavailable

  • ESP_ERR_TIMEOUT Failed to queue removal frame

  • ESP_FAIL Failed to commit removal frame

esp_err_t esp_media_track_mngr_get_provider(esp_media_track_mngr_t *mngr, esp_media_provider_t *provider)

Get the media provider handle exported by a track manager.

Parameters:

  • mngr – [in] Track manager handle

  • provider – [out] Output media provider

Returns:

  • ESP_OK On success

  • ESP_ERR_INVALID_ARG Track manager or provider is NULL

Structures

struct esp_media_track_mngr_cfg_t

Media track manager configuration.

Note

The global cache used or not can be reconfigured by API esp_media_track_mngr_set_global_cache

Public Members

uint16_t max_track_num

Maximum number of tracks

bool use_global_cache

Use one arrival-order cache shared by all tracks

size_t cache_size

Shared queue byte size, 0 uses default

struct esp_media_track_mngr_cache_cfg_t

Media provider track buffer configuration.

Public Members

esp_media_track_cache_type_t cache_type

Track cache type

size_t cache_size

Queue byte size for track manager-owned payload mode

uint16_t addr_align

Track manager-owned frame data alignment, 0 uses pointer size

uint16_t size_align

Cached frame size alignment, 0 no special request

uint16_t queue_num

Metadata queue depth

esp_media_frame_release_cb_t frame_release

Release callback for user-owned frames

void *release_ctx

Context for frame_release

struct esp_media_track_mngr_track_cfg_t

Media track manager track configuration.

Public Members

esp_media_track_info_t info

Track metadata

esp_media_track_mngr_cache_cfg_t cache_cfg

Track buffer/cache settings

Type Definitions

typedef struct esp_media_track_mngr esp_media_track_mngr_t

Definition of media track manager.

SPDX-FileCopyrightText: 2026 Espressif Systems (Shanghai) CO., LTD SPDX-License-Identifier: LicenseRef-Espressif-Modified-MIT

See LICENSE file for details.

typedef void (*esp_media_frame_release_cb_t)(const esp_media_frame_t *frame, void *ctx)

Release callback for user-owned frames after consumed.

Param frame:

[in] Frame whose payload can be released by the owner

Param ctx:

[in] User context from esp_media_track_mngr_cache_cfg_t

Enumerations

enum esp_media_track_cache_type_t

Media track cache type.

Values:

enumerator ESP_MEDIA_TRACK_CACHE_INTERNAL

Track manager owns queued frame payload data

enumerator ESP_MEDIA_TRACK_CACHE_USER

Track manager queues frame metadata only, user owns payload data

Provide feedback about this document