# ESP Media Service

[中文]

## Introduction

ESP Media Service defines a common media interface for audio and video services in ESP-ADF, based on ESP Service. An application creates a service, configures the media stream, and then links a source stream to a sink stream. Once linked and started, media frames flow through the read and write interfaces, so the application does not need to forward frames by hand. Audio and video capture and playback are documented in ESP-GMF General Multimedia Framework.

## Feature List

- A unified audio/video service model with three roles: source, sink, and source-sink (`ESP_MEDIA_ROLE_SRC` / `SINK` / `SRC_SINK`)

- Multi-media endpoints based on a stream ID (`esp_media_stream_id_t`); a single service can expose multiple streams at the same time

- Request negotiation at link time: a sink can express requirements to the source through `esp_media_service_request_t` (for example, whether a global cache is needed)

- The provider read interface (`esp_media_provider_t`) is decoupled from the track manager write interface

- A built-in default in-memory track manager (`esp_media_track_mngr_t`) that supports either per-track independent caching or global arrival-order caching

- Service lifecycle is uniformly managed through ESP Service, with media-related operations layered on top of the base class as an independent vtable (`esp_media_service_ops_t`)

## Technical Deep Dive

### Service Roles and Linking

ESP Media Service declares whether it is a source, a sink, or both, through `get_role` in `esp_media_service_ops_t`; `esp_media_service_link()` verifies that the roles of the selected source and sink are compatible when linking, and then passes the source’s provider to the sink.

            flowchart TD
        Create["Create service"] --> Config["Configure service"]
        Config --> Link["Link source/sink streams"]
        Link --> Start["Start service"]
        Start --> Flow["Media frames flow"]
        Flow --> Stop["Stop service"]
        Stop --> Unlink["Unlink and destroy"]

    esp_media_stream_id_t stream = ESP_MEDIA_DEFAULT_STREAM;

    esp_media_service_link(src_service, stream, sink_service, stream);

    esp_service_start(sink_service);
    esp_service_start(src_service);

Once the link is established, media data flows from the provider exported by the source service to the sink; removing the link requires calling `esp_media_service_unlink()`, which clears the provider currently set on the sink stream.

### Provider and Track Manager

The delivery of media data relies on two mutually decoupled interfaces: at link time, the sink obtains a read-only `esp_media_provider_t` from the source service, and acquires and releases frames through `esp_media_provider_acquire_frame()` / `esp_media_provider_release_frame()`; the source service, in turn, holds an `esp_media_track_mngr_t`, produces frames through its write API, and exposes the provider handle it exports to downstream consumers.

    /* Source service side: create the track manager, register a track, and export the provider */
    esp_media_track_mngr_cfg_t cfg = { .max_track_num = 2 };
    esp_media_track_mngr_create(&cfg, &svc->mngr);
    esp_media_track_mngr_add_track(svc->mngr, &audio_track);
    esp_media_track_mngr_get_provider(svc->mngr, &svc->provider);

    /* Sink side: acquire and release a frame */
    esp_media_frame_t frame = {0};
    if (esp_media_provider_acquire_frame(&sink->provider, &frame, timeout_ms) == ESP_OK) {
        process_frame(&frame);
        esp_media_provider_release_frame(&sink->provider, &frame);
    }

Warning

A frame acquired through `esp_media_provider_acquire_frame()` must be released by calling `esp_media_provider_release_frame()`; after it is released, `frame.data` must no longer be accessed.

### Track Manager Cache Modes

The default `esp_media_track_mngr_t` supports two payload ownership modes: `ESP_MEDIA_TRACK_CACHE_INTERNAL`, in which the manager copies and holds the frame data itself, and `ESP_MEDIA_TRACK_CACHE_USER`, which caches only the frame metadata while the payload remains owned by the user and is returned through the `frame_release` callback after it has been consumed. A global cache is also supported, letting multiple tracks share a single queue ordered by arrival time, which is suitable for scenarios such as RTMP where audio and video are interleaved; enabling the global cache must be configured before tracks are added.

### Stop and Abort Ordering

The media interface is designed to tolerate the stop sequence: when the source service stops, it should call `esp_media_track_write_abort()`, which notifies downstream consumers through the `ESP_MEDIA_PROVIDER_EVENT_TRACKS_ABORT` event; when the sink stops, it should first set a local stop flag, then call `esp_media_provider_abort()` to wake up any blocked read, wait for the task to exit and release any frames it has acquired, and only then unlink. If a track manager is shared by multiple services through linking, it must be unlinked before reset or destroy is performed on it; as long as any task still holds an acquired frame or is blocked on the queue, reset or destroy must not be performed on the track manager.

## Application Examples

Complete source/sink examples are in the examples directory of the esp_media_service component repository. See ESP Service for the service base class.

## FAQ

Q1: Can a service implement only one of ``get_provider`` or ``set_provider``?

Not necessarily. A service with the role `ESP_MEDIA_ROLE_SRC_SINK` can implement both at the same time, acting as a source for downstream consumers as well as a sink for upstream producers; `esp_media_service_link()` only checks role compatibility for the selected pair of source/sink services.

## API Reference

### Header File

- components/esp_media_service/include/esp_media_service.h

### Functions

esp_err_t esp_media_service_init(esp_media_service_t *service, const esp_media_service_config_t *config)

  Initialize a media service base object.

          Derived services call this after allocating or embedding
          esp_media_service_t, before exposing the base esp_service_t

  Parameters:

    - service – [inout] Media service object to initialize

    - config – [in] Media service configuration

  Returns:

    - ESP_OK On success

    - ESP_ERR_INVALID_ARG service or config is NULL

    - Others Error returned by esp_service_init()

esp_err_t esp_media_service_deinit(esp_service_t *service)

  Deinitialize a media service base object.

  Parameters:

    service – [in] Base service pointer returned by ESP_SERVICE_BASE()

  Returns:

    - ESP_OK On success

    - ESP_ERR_INVALID_ARG service is NULL

    - Others Error returned by esp_service_deinit()

esp_err_t esp_media_service_get_role(esp_service_t *service, esp_media_role_t *out_role)

  Query the media role of a service.

  Parameters:

    - service – [in] Base media service handle

    - out_role – [out] Output media role

  Returns:

    - ESP_OK On success

    - ESP_ERR_INVALID_ARG service or out_role is NULL

    - ESP_ERR_NOT_SUPPORTED Operation is not implemented by the service

    - Others Error returned by the service implementation

esp_err_t esp_media_service_get_provider(esp_service_t *service, esp_media_stream_id_t stream, esp_media_provider_t *out_provider)

  Get a provider from a source stream.

  Parameters:

    - service – [in] Base media service handle

    - stream – [in] Source stream ID

    - out_provider – [out] Output provider

  Returns:

    - ESP_OK On success

    - ESP_ERR_INVALID_ARG Service or provider is NULL

    - ESP_ERR_NOT_SUPPORTED Operation is not implemented by the service

    - Others Error returned by the service implementation

esp_err_t esp_media_service_set_provider(esp_service_t *service, esp_media_stream_id_t stream, const esp_media_provider_t *provider)

  Set a provider on a sink stream.

          Pass NULL provider to disconnect the sink stream

  Parameters:

    - service – [in] Base media service handle

    - stream – [in] Sink stream ID

    - provider – [in] Provider handle to consume, or NULL to clear

  Returns:

    - ESP_OK On success

    - ESP_ERR_INVALID_ARG service is NULL, or provider has no ops

    - ESP_ERR_NOT_SUPPORTED Operation is not implemented by the service

    - Others Error returned by the service implementation

esp_err_t esp_media_service_get_request(esp_service_t *service, esp_media_stream_id_t stream, esp_media_service_request_t *out_request)

  Get link request hints for a media stream.

  Parameters:

    - service – [in] Base media service handle

    - stream – [in] Stream ID

    - out_request – [out] Output request hints

  Returns:

    - ESP_OK On success

    - ESP_ERR_INVALID_ARG service or out_request is NULL

    - ESP_ERR_NOT_SUPPORTED Operation is not implemented by the service

    - Others Error returned by the service implementation

esp_err_t esp_media_service_set_request(esp_service_t *service, esp_media_stream_id_t stream, const esp_media_service_request_t *request)

  Set link request hints for a media stream.

  Parameters:

    - service – [in] Base media service handle

    - stream – [in] Stream ID

    - request – [in] Request hints to apply

  Returns:

    - ESP_OK On success

    - ESP_ERR_INVALID_ARG service or request is NULL

    - ESP_ERR_NOT_SUPPORTED Operation is not implemented by the service

    - Others Error returned by the service implementation

esp_err_t esp_media_service_link(esp_service_t *src_service, esp_media_stream_id_t src_stream, esp_service_t *sink_service, esp_media_stream_id_t sink_stream)

  Link a source stream to a sink stream.

          The sink request hints are applied to the source when supported, then
          the source provider is passed to the sink

  Parameters:

    - src_service – [in] Source media service handle

    - src_stream – [in] Source stream ID

    - sink_service – [in] Sink media service handle

    - sink_stream – [in] Sink stream ID

  Returns:

    - ESP_OK On success

    - ESP_ERR_INVALID_ARG src_service or sink_service is NULL

    - ESP_ERR_NOT_SUPPORTED Role check failed or required operation is missing

    - Others Error returned by source or sink implementation

esp_err_t esp_media_service_unlink(esp_service_t *src_service, esp_media_stream_id_t src_stream, esp_service_t *sink_service, esp_media_stream_id_t sink_stream)

  Unlink a source stream from a sink stream.

          Clears the provider currently set on the sink stream

  Parameters:

    - src_service – [in] Source media service handle

    - src_stream – [in] Source stream ID

    - sink_service – [in] Sink media service handle

    - sink_stream – [in] Sink stream ID

  Returns:

    - ESP_OK On success

    - ESP_ERR_INVALID_ARG src_service or sink_service is NULL

    - ESP_ERR_NOT_SUPPORTED Role check failed or sink operation is missing

    - Others Error returned by source or sink implementation

### Structures

struct esp_media_service_request_t

  Media-service link request/capability hints.

          The structure is intentionally extensible so future services can add
          link-time requests without changing the media op shape

  Public Members

  bool need_global_cache

    Sink requests source frames in one arrival-order cache

struct esp_media_service_ops_t

  Media-service virtual operations.

  Public Members

  esp_err_t (*get_role)(esp_service_t *service, esp_media_role_t *out_role)

    Query source/sink role

  esp_err_t (*get_provider)(esp_service_t *service, esp_media_stream_id_t stream, esp_media_provider_t *out_provider)

    Get provider exported by a source stream

  esp_err_t (*set_provider)(esp_service_t *service, esp_media_stream_id_t stream, const esp_media_provider_t *provider)

    Set provider consumed by a sink stream

  esp_err_t (*get_request)(esp_service_t *service, esp_media_stream_id_t stream, esp_media_service_request_t *request)

    Get sink link requests

  esp_err_t (*set_request)(esp_service_t *service, esp_media_stream_id_t stream, const esp_media_service_request_t *request)

    Apply sink requests to a source

struct esp_media_service_config_t

  Media service configuration.

  Public Members

  const char *name

    Service instance name

  void *user_data

    User data passed to esp_service

  const esp_service_ops_t *service_ops

    Optional esp_service lifecycle ops

  const esp_media_service_ops_t *media_ops

    Optional media ops

struct esp_media_service

  Base media service structure Derived services embed this as the first member.

  Public Members

  esp_service_t base

    Base service, must stay first

  const esp_media_service_ops_t *media_ops

    Media virtual operations

### Macros

ESP_MEDIA_SERVICE_CONFIG_DEFAULT()

  SPDX-FileCopyrightText: 2026 Espressif Systems (Shanghai) CO., LTD SPDX-License-Identifier: LicenseRef-Espressif-Modified-MIT

  See LICENSE file for details. Default media service configuration

ESP_MEDIA_DEFAULT_STREAM

  Default media stream ID

### Type Definitions

typedef uint16_t esp_media_stream_id_t

  Instance-aware media stream address.

typedef struct esp_media_service esp_media_service_t

  Base media service structure Derived services embed this as the first member.

### Enumerations

enum esp_media_role_t

  Media service data direction.

  Values:

  enumerator ESP_MEDIA_ROLE_NONE
  enumerator ESP_MEDIA_ROLE_SRC

    Service produces media frames

  enumerator ESP_MEDIA_ROLE_SINK

    Service consumes media frames

  enumerator ESP_MEDIA_ROLE_SRC_SINK

    Service produces and consumes frames

### Header File

- components/esp_media_service/include/esp_media_provider.h

### Functions

esp_err_t esp_media_provider_get_track_num(const esp_media_provider_t *provider, uint16_t *out_num)

  Get the number of tracks exposed by a provider.

  SPDX-FileCopyrightText: 2026 Espressif Systems (Shanghai) CO., LTD SPDX-License-Identifier: LicenseRef-Espressif-Modified-MIT

  See LICENSE file for details.

  Parameters:

    - provider – [in] Provider handle

    - out_num – [out] Output track count

  Returns:

    - ESP_OK On success

    - ESP_ERR_INVALID_ARG provider or out_num is NULL

    - ESP_ERR_NOT_SUPPORTED Operation is not implemented by the provider

    - Others Error returned by the provider implementation

esp_err_t esp_media_provider_get_track_info(const esp_media_provider_t *provider, uint16_t index, esp_media_track_info_t *out_info)

  Get track metadata by provider track index.

  Parameters:

    - provider – [in] Provider handle

    - index – [in] Track index in the provider

    - out_info – [out] Output track metadata

  Returns:

    - ESP_OK On success

    - ESP_ERR_INVALID_ARG provider or out_info is NULL

    - ESP_ERR_NOT_SUPPORTED Operation is not implemented by the provider

    - Others Error returned by the provider implementation

esp_err_t esp_media_provider_set_event_cb(const esp_media_provider_t *provider, esp_media_provider_event_cb_t cb, void *event_ctx)

  Register a provider event callback.

  Parameters:

    - provider – [in] Provider handle

    - cb – [in] Event callback, or NULL to clear it

    - event_ctx – [in] User context passed to cb

  Returns:

    - ESP_OK On success

    - ESP_ERR_INVALID_ARG provider is NULL

    - ESP_ERR_NOT_SUPPORTED Operation is not implemented by the provider

    - Others Error returned by the provider implementation

esp_err_t esp_media_provider_abort(const esp_media_provider_t *provider)

  Abort provider-side blocking read operations.

  Parameters:

    provider – [in] Provider handle

  Returns:

    - ESP_OK On success

    - ESP_ERR_INVALID_ARG provider is NULL

    - ESP_ERR_NOT_SUPPORTED Operation is not implemented by the provider

    - Others Error returned by the provider implementation

esp_err_t esp_media_provider_acquire_frame(const esp_media_provider_t *provider, esp_media_frame_t *out_frame, uint32_t timeout_ms)

  Acquire the next frame from a provider.

          The returned frame must be released with esp_media_provider_release_frame().
          Set fields such as type or track_id in out_frame before the call to
          select a specific track when the provider supports it

  Parameters:

    - provider – [in] Provider handle

    - out_frame – [inout] Input track selector, output acquired frame

    - timeout_ms – [in] Timeout in milliseconds; 0 means no wait, UINT32_MAX means wait forever

  Returns:

    - ESP_OK On success

    - ESP_ERR_INVALID_ARG provider or out_frame is NULL

    - ESP_ERR_NOT_SUPPORTED Operation is not implemented by the provider

    - Others Error returned by the provider implementation

esp_err_t esp_media_provider_read_frame(const esp_media_provider_t *provider, esp_media_frame_t *out_frame, uint32_t timeout_ms)

  Read the next frame into caller-provided storage.

          Set out_frame->data and out_frame->size before calling. On success,
          out_frame->size is updated to the actual payload size

  Parameters:

    - provider – [in] Provider handle

    - out_frame – [inout] Input buffer descriptor, output frame descriptor

    - timeout_ms – [in] Timeout in milliseconds; 0 means no wait, UINT32_MAX means wait forever

  Returns:

    - ESP_OK On success

    - ESP_ERR_INVALID_ARG provider or out_frame is NULL

    - ESP_ERR_INVALID_SIZE Provided buffer is too small

    - ESP_ERR_NOT_SUPPORTED Operation is not implemented by the provider

    - Others Error returned by the provider implementation

esp_err_t esp_media_provider_release_frame(const esp_media_provider_t *provider, esp_media_frame_t *frame)

  Release a frame acquired from a provider.

  Parameters:

    - provider – [in] Provider handle

    - frame – [in] Frame returned by esp_media_provider_acquire_frame()

  Returns:

    - ESP_OK On success

    - ESP_ERR_INVALID_ARG provider or frame is NULL

    - ESP_ERR_NOT_SUPPORTED Operation is not implemented by the provider

    - Others Error returned by the provider implementation

### Header File

- components/esp_media_service/include/esp_media_track_mngr.h

### Functions

esp_err_t esp_media_track_mngr_create(const esp_media_track_mngr_cfg_t *cfg, esp_media_track_mngr_t **out_mngr)

  Create a media track manager.

  Parameters:

    - cfg – [in] Provider configuration

    - out_mngr – [out] Output track manager handle

  Returns:

    - ESP_OK On success

    - ESP_ERR_INVALID_ARG cfg is NULL, max_track_num is 0, or out_provider is NULL

    - ESP_ERR_NO_MEM Allocation failed

esp_err_t esp_media_track_mngr_destroy(esp_media_track_mngr_t *mngr)

  Destroy a media track manager.

          Wakes and destroys internal queues, releases pending user-owned
          frames through their release callbacks, and frees the track manager

  Parameters:

    mngr – [in] Track manager handle returned by esp_media_track_mngr_create()

  Returns:

    - ESP_OK On success

    - ESP_ERR_INVALID_ARG Track manager is NULL

esp_err_t esp_media_track_mngr_reset(esp_media_track_mngr_t *mngr)

  Reset tracks and queued data.

          Removes all tracks and clears abort state. Global-cache mode is kept.
          User must re-add tracks after reset

  Parameters:

    mngr – [in] Track manager handle

  Returns:

    - ESP_OK On success

    - ESP_ERR_INVALID_ARG Track manager is NULL

esp_err_t esp_media_track_mngr_set_global_cache(esp_media_track_mngr_t *mngr, bool enable, size_t cache_size)

  Configure global cache mode before tracks are added.

  Parameters:

    - mngr – [in] Track manager handle

    - enable – [in] true to use one arrival-order cache shared by all tracks

    - cache_size – [in] Shared queue byte size, 0 uses default

  Returns:

    - ESP_OK On success

    - ESP_ERR_INVALID_ARG Track manager is NULL

    - ESP_ERR_INVALID_STATE Tracks have already been added

    - ESP_ERR_NO_MEM Allocation failed

esp_err_t esp_media_track_mngr_add_track(esp_media_track_mngr_t *mngr, const esp_media_track_mngr_track_cfg_t *cfg)

  Add a track to a track manager.

  Parameters:

    - mngr – [in] Track manager handle

    - cfg – [in] Track configuration

  Returns:

    - ESP_OK On success

    - ESP_ERR_INVALID_ARG Track manager or cfg is NULL

    - ESP_ERR_NO_MEM Track limit reached or queue allocation failed

esp_err_t esp_media_track_mngr_update_track(esp_media_track_mngr_t *mngr, uint16_t index, const esp_media_track_info_t *info)

  Queue a track metadata update.

          The update is committed when provider acquire/read reaches the queued
          zero-size frame with ESP_MEDIA_FRAME_FLAG_TRACK_CHANGED. The provider
          event callback is invoked synchronously at that point

  Parameters:

    - mngr – [in] Track manager handle

    - index – [in] Track index to update

    - info – [in] New track metadata

  Returns:

    - ESP_OK On success

    - ESP_ERR_INVALID_ARG Track manager or info is NULL

    - ESP_ERR_NOT_FOUND index is out of range

    - ESP_ERR_NO_MEM Track queue is unavailable

    - ESP_ERR_TIMEOUT Failed to queue update frame

    - ESP_FAIL Failed to commit update frame

esp_err_t esp_media_track_mngr_remove_track(esp_media_track_mngr_t *mngr, uint16_t index)

  Queue a track removal notification.

          The removal is committed when provider acquire/read reaches the queued
          zero-size frame with ESP_MEDIA_FRAME_FLAG_TRACK_REMOVED. The provider
          event callback is invoked synchronously at that point

  Parameters:

    - mngr – [in] Track manager handle

    - index – [in] Track index to remove

  Returns:

    - ESP_OK On success

    - ESP_ERR_INVALID_ARG Track manager is NULL

    - ESP_ERR_NOT_FOUND index is out of range

    - ESP_ERR_NO_MEM Track queue is unavailable

    - ESP_ERR_TIMEOUT Failed to queue removal frame

    - ESP_FAIL Failed to commit removal frame

esp_err_t esp_media_track_mngr_get_provider(esp_media_track_mngr_t *mngr, esp_media_provider_t *provider)

  Get the media provider handle exported by a track manager.

  Parameters:

    - mngr – [in] Track manager handle

    - provider – [out] Output media provider

  Returns:

    - ESP_OK On success

    - ESP_ERR_INVALID_ARG Track manager or provider is NULL

### Structures

struct esp_media_track_mngr_cfg_t

  Media track manager configuration.

  Note

  The global cache used or not can be reconfigured by API `esp_media_track_mngr_set_global_cache`

  Public Members

  uint16_t max_track_num

    Maximum number of tracks

  bool use_global_cache

    Use one arrival-order cache shared by all tracks

  size_t cache_size

    Shared queue byte size, 0 uses default

struct esp_media_track_mngr_cache_cfg_t

  Media provider track buffer configuration.

  Public Members

  esp_media_track_cache_type_t cache_type

    Track cache type

  size_t cache_size

    Queue byte size for track manager-owned payload mode

  uint16_t addr_align

    Track manager-owned frame data alignment, 0 uses pointer size

  uint16_t size_align

    Cached frame size alignment, 0 no special request

  uint16_t queue_num

    Metadata queue depth

  esp_media_frame_release_cb_t frame_release

    Release callback for user-owned frames

  void *release_ctx

    Context for frame_release

struct esp_media_track_mngr_track_cfg_t

  Media track manager track configuration.

  Public Members

  esp_media_track_info_t info

    Track metadata

  esp_media_track_mngr_cache_cfg_t cache_cfg

    Track buffer/cache settings

### Type Definitions

typedef struct esp_media_track_mngr esp_media_track_mngr_t

  Definition of media track manager.

  SPDX-FileCopyrightText: 2026 Espressif Systems (Shanghai) CO., LTD SPDX-License-Identifier: LicenseRef-Espressif-Modified-MIT

  See LICENSE file for details.

typedef void (*esp_media_frame_release_cb_t)(const esp_media_frame_t *frame, void *ctx)

  Release callback for user-owned frames after consumed.

  Param frame:

    [in] Frame whose payload can be released by the owner

  Param ctx:

    [in] User context from esp_media_track_mngr_cache_cfg_t

### Enumerations

enum esp_media_track_cache_type_t

  Media track cache type.

  Values:

  enumerator ESP_MEDIA_TRACK_CACHE_INTERNAL

    Track manager owns queued frame payload data

  enumerator ESP_MEDIA_TRACK_CACHE_USER

    Track manager queues frame metadata only, user owns payload data

Provide feedback about this document