ESP-Player Component

[中文]

Note

This document is automatically translated using AI. Please excuse any detailed errors. The official English version is still in progress.

Introduction to ESP Player

ESP Player is an embedded multimedia playback component launched by Espressif, which completes the entire playback chain of demuxing → decoding → rendering within a single instance, targeting resource-constrained IoT and multimedia products.

ESP Player serves as a coordination layer between the application and the underlying GMF multimedia pipeline, responsible for state machines, command distribution, and audio-video synchronization. The application only needs to set the playback source and control the playback through a simple API, which can support various scenarios such as local files, HTTP(S) streams, HLS, and feeding of external frames without containers.

Feature Highlights

Capability

Description

Multiple Input Sources

Local files, HTTP/HTTPS streams, HLS (on-demand/live, automatically recognizes .m3u8), and container-less external frame feeding (Bluetooth A2DP raw frames, microphone PCM, custom encoding frames; can also feed audio and video simultaneously)

Rich Format Support

Supports common encapsulation containers and mainstream audio and video encoding formats, covering music playback, voice prompts, audio/video monitoring playback, etc.

Audio-Video Synchronization

Supports system clock, audio as the main clock, video as the main clock, and no synchronization (freerun) modes, meeting different synchronization accuracy requirements

Complete Playback Control

Play, pause, resume, stop, Seek (millisecond level, preset next start position), speed playback

Audio-Video Track Selection

Can separately switch audio/video paths, multi-track containers support track enumeration and switching (such as multi-language audio tracks)

ID3 Metadata

Supports parsing raw MP3 ID3 information (such as song title, cover)

Status query

Can query the current playback status, not solely dependent on event callbacks

Network Buffering

Based on queue water level startup pre-buffering and runtime re-buffering, improving the smoothness of network streaming media playback

Event Notification

Supports synchronous callback and asynchronous event queue modes, covering playback status, buffering, errors, track information, etc.

Flexible Trimming and Tuning

Can trim functions according to format and hardware in menuconfig to reduce memory; supports multiple instances and individual configuration of task/buffer; supports registration of custom decoders

Supported Media Formats

Category

Supported Content

Input Source

Local files (file:///), HTTP/HTTPS streams, HLS (.m3u8), external frame mode (fill:///, block:///)

Encapsulation Containers

WAV, MP4, M4A, TS, OGG, AVI, FLV, CAF; and raw ES stream files (.mp3, .aac, .flac, .amr, without container headers)

Audio Decoding

AAC, MP3, Vorbis, Opus, FLAC, AMR-NB/WB, G.711 A-law/μ-law, ALAC, ADPCM, SBC, LC3

Video Decoding

H.264, MJPEG

Input Source and URL

ESP Player specifies the playback source through a unified URL: the scheme determines the input channel, and the file extension (.codec) determines the decoder. URLs are divided into two major categories: standard URI (local/network) and external frame mode (without containers).

Standard URI (Local and Network)

Type

URL Example

Description

Local File

file:///sdcard/music/test.mp3

Standard triple-slash format; you can also write the bare path /sdcard/music/test.mp3 directly, which will be standardized to file:///

Local Headless PCM

file:///sdcard/test.pcm?sr=48000&ch=2&bits=16

Headless PCM without a container must specify the sampling rate, number of channels, and bit width through ?query

HTTP(S) Stream

https://host/audio/test.mp4

Network on-demand; supports specifying the port, such as http://192.168.1.10:8080/stream.aac

HTTP(S) Basic Authentication

https://user:pass@host/audio/test.mp4

Optional user:pass@ basic authentication

HLS

https://host/live/playlist.m3u8

Paths containing .m3u8 are automatically recognized as HLS, supporting on-demand and live streaming

Network Headless PCM

http://host/audio/test.pcm?sr=48000&ch=2&bits=16

Network headless PCM also needs to carry ?query

External Frame Mode (Containerless Frame Feeding)

The application directly sends raw frames into decoding, suitable for Bluetooth A2DP raw frames, microphone PCM, custom encoding frames, and feeding of audio and video dual channels without containers. The URL is in the form of fill:///name.codec[?params] or block:///name.codec[?params].

  • fill: A complete copy of the frame is made each time it is called, which is simple to use;

  • block: Zero-copy, the caller must ensure that the buffer is valid before decoding is completed;

  • name has no actual semantics, the extension .codec determines the decoder;

  • When feeding audio and video simultaneously, each frame needs to be marked as either audio or video.

After running, send frames through esp_player_submit_frame(). Common examples:

Scenario

URL Example

Description

Raw PCM

fill:///test.pcm?sr=16000&ch=1&bits=16

Full frame copy

Raw PCM (Zero-copy)

block:///test.pcm?sr=16000&ch=1&bits=16

Save a memory copy

AAC (Standard ADTS)

fill:///test.aac

With ADTS header

AAC (No ADTS header, such as BT A2DP)

fill:///test.aac?no_adts=1

Bluetooth A2DP and other headerless AAC

HE-AAC Raw Frame

fill:///test.aac?no_adts=1&aac_plus=1

HE-AAC (AAC+)

OPUS Raw Frame

fill:///test.opus?sr=16000&ch=2&frame_dms=20

Specify sampling rate/channel/frame duration

Common query parameters

Parameter

Meaning

sr

Sampling rate (Hz)

ch

Number of channels

bits

Bit width (bit)

no_adts

AAC without ADTS header (such as Bluetooth A2DP raw frame)

aac_plus

HE-AAC (AAC+)

frame_dms

OPUS/LC3 frame duration

plc

SBC/LC3 packet loss compensation

For the complete parameter list, see esp_player_set_url(). Network, HLS, and other inputs also need to enable the corresponding IO in the menuconfig’s ESP Player → Input IO sources. If the input type is not enabled, an error will be returned when setting the URL.

Typical Application Scenarios

Application Scenario

Description

Audio playback products

Music players, story machines, audiobooks, etc., support local and network audio sources, speed and Seek

Voice prompts and notification sounds

Device boot sound, operation prompt sound, alarm prompt, etc., can be played simultaneously through multi-instance mixing

Screened audio and video playback

Play local or network MP4 (H.264 + AAC) on devices with LCD + audio, synchronized audio and video output

Network streaming media and live broadcast

Play on-demand or live content through HTTP(S)/HLS

Bluetooth audio and custom frames

Play Bluetooth A2DP raw frames, microphone PCM, or custom encoding frames (no container required; audio and video can also be fed in simultaneously) through external frame mode.

Multi-channel audio mixing

Multiple player instances decode in parallel and mix output to the same DAC

Advanced Business Usage

Seamless Source Switching

For scenarios such as switching songs in a playlist or changing channels, there is no need to reinitialize the player: simply call esp_player_stop() → esp_player_set_url() → esp_player_run() in order, and the ESP Player will automatically dismantle the old pipeline and rebuild it according to the new source. After the source is set, you can also Seek before playing, which is used for resuming playback or bookmarks.

Multi-instance Mixing

You can create multiple player instances to play in parallel and mix output to the same DAC. Suitable for scenarios such as simultaneous playback of background music and prompt sounds/voice broadcasts, overlay of multiple audio sources, etc.

External Frame Mode

When there is no standard container, the application can directly feed the raw frames to the player for decoding and playback: specified by fill:/// (full frame copy) or block:/// (zero copy), and then call esp_player_submit_frame() to submit frames. Both single-channel or audio and video can be fed in simultaneously. Typical scenarios include Bluetooth A2DP raw frames, microphone PCM, custom encoded frames, and synchronized audio and video playback without a container.

Cooperation with Peripheral Components

ESP Player is only responsible for the playback core. In actual solutions, it usually cooperates with other GMF components, each performing its own duties:

Component

Responsibility

esp_board_manager

Initialize LCD, audio DAC, SD card and other board-level resources

esp_player

Unpacking, decoding, audio and video synchronization and playback control (core)

esp_audio_render

Output the decoded audio PCM to the speaker/DAC, support multi-channel mixing

esp_video_render

Display the decoded video frame to the screen through the LCD backend

Data flow: Data source → esp_player(Demux → Decode → Sync)→ esp_audio_render / esp_video_render → Speaker / LCD.

Performance Data

The following data comes from the component README, playing local SD card files. The memory is divided into built-in SRAM and expanded PSRAM (MALLOC_CAP_SPIRAM).

Test Environment

Chip

CPU

SPI RAM

Flash

ESP32-S3

240 MHz

80 MHz

QIO

ESP32-S31

320 MHz

200 MHz

QIO

ESP32-P4

400 MHz

250 MHz

QIO

Audio Playback Performance

Chip

Format

SRAM (KB)

PSRAM (KB)

Total (KB)

CPU (%)

Start-up delay (ms)

Resume delay (ms)

ESP32-S3

MP3

8.8

91.8

100.6

5

32

27

ESP32-S3

AAC

10.1

106.4

116.5

3

19

6

ESP32-S31

MP3

7.2

91.8

99.0

4

16

6

ESP32-S31

AAC

8.6

106.4

115.0

5

16

6

ESP32-P4

MP3

7.2

91.8

99.0

3

15

3

ESP32-P4

AAC

10.1

106.4

116.5

3

11

2

  • Start-up delay: Time from esp_player_run() to the first frame of audio output

  • Resume delay: Time from esp_player_stop() + esp_player_run()(without changing URL) to the first frame of audio output

Compared with ADF v2.x esp_audio (ESP32-S3, same format and test conditions; the value in parentheses is the reduction relative to esp_audio):

Format

Component

Built-in SRAM (KB)

PSRAM (KB)

Total (KB)

CPU (%)

Start broadcasting (ms)

Replay (ms)

MP3

esp_audio

19.0

109.1

128.1

8

74

74

MP3

esp_player

8.8（53.7%）

91.8（15.9%）

100.6（21.5%）

5（37.5%）

32（56.8%）

27（63.5%）

AAC

esp_audio

35.7

182.9

218.6

5

80

80

AAC

esp_player

10.1（71.7%）

106.4（41.8%）

116.5（46.7%）

3（40.0%）

19（76.3%）

6（92.5%）

Video Playback Performance

The video playback of the ESP Player is an end-to-end process (IO + Demuxing + Decoding + Rendering). The table below provides the maximum end-to-end performance given by the component README:

Chip

Format

Maximum performance

ESP32-S3

H.264

320×240@18 fps

ESP32-S31

H.264

320×240@55 fps；640×480@13 fps

ESP32-P4

H.264

320×240@68 fps；640×480@18 fps

ESP32-S3

MJPEG

320×240@28 fps

ESP32-S31

MJPEG

320×240@>120 fps；640×480@66 fps；1280×720@30 fps

ESP32-P4

MJPEG

320×240@>120 fps；640×480@91 fps；1280×720@32 fps

  • The above table shows H.264 as software decoding; MJPEG is software decoding on ESP32-S3, and hardware decoding on ESP32-S31 / ESP32-P4.

  • Video playback requires high memory and computational power, it is recommended to use ESP32-S31, ESP32-P4 with PSRAM; for higher resolution MJPEG, hardware JPEG decoding of these two models is recommended.

  • For individual decoding frame rates of each decoder under different chips and resolutions, please refer to the corresponding decoding component documents (such as esp_h264, esp_new_jpeg, esp_video_codec).

Memory Optimization Suggestions

  • Only enable the actual used encapsulation containers, decoders and on-board codecs in menuconfig, and close all unused ones;

  • For pure local audio scenarios, you can close the video path and network buffer related configurations;

  • Enabling PSRAM (SPIRAM, FREERTOS_TASK_CREATE_ALLOW_EXT_MEM) allows larger stacks and buffers to be placed externally, reducing the usage of built-in SRAM; by default, the player task stack is already placed in PSRAM;

  • When there are multiple instances and the resources of each are significantly different, each handle can be individually configured for task and buffer.

How to Use ESP Player

ESP Player is hosted on GitHub. You can add ESP Player to your project by entering the following command in your project: idf.py add-dependency “espressif/esp_player”.

The basic usage process is: initialize the player and configure the audio/video rendering handle → set the playback source (AV mask, URL, sync mode) → start playback, and handle playback end, error and other states through event callbacks.

The examples folder of the component provides runnable example projects, it is recommended to refer to and debug before use:

For more API descriptions, URL formats and advanced usage, please refer to README.

FAQ

What are the capabilities of ESP Player?

ESP Player is only responsible for the playback link (demux → decode → render), and does not include the network transport protocol stack. It can directly play local files, HTTP(S)/HLS streams, or receive raw frames sent by the upstream for decoding and playback through the external frame mode. Therefore, when it comes to real-time two-way communication, protocol signaling interaction or push-pull streaming(such as SIP, RTSP, RTMP, WebRTC), the corresponding transport/communication component is responsible for receiving and sending media data, ESP Player and it belong to different links, and can be used as needed.

How to choose between ESP Player, Audio Simple Player and bare GMF?

All three can “play/process media”, but the encapsulation level and positioning are different: Audio Simple Player (esp_audio_simple_player) is a lightweight pure audio player, which decodes/transforms and hands over PCM to the application through callbacks; ESP Player (esp_player) is an end-to-end complete audio and video player; bare GMF is directly based on esp-gmf’s pipeline/element/task arrangement, with the highest flexibility but needs to implement various capabilities by itself.

Capability

Audio Simple Player

ESP Player

Bare GMF (self-built)

Pure audio local / HTTP

Supported

Supported

Supported (self-built)

Audio and video (MP4, etc.)

Not supported

Supported

Supported (self-built)

HLS

Not supported

Supported

Need to integrate yourself

Seek

Not supported

Supported (millisecond level)

Self-implemented

Pause / Continue

Supported

Supported

Self-implemented

Audio and video synchronization

— (Audio only)

Supported (4 modes)

Self-implemented

Network buffering / re-buffering

Weak

Supported

Self-implemented

Multi-track / track switching

Not supported

Supported

Self-implemented

Fill/block frame feeding

Not supported

Supported

Can be self-built

Output method

PCM callback

Direct render output (speaker/screen)

Any

Custom processing elements

Supported (register element)

Supported (register custom decoder element)

Supported (any element)

Multiple pipelines / mixing

Not supported (single chain)

Single instance (audio and video integrated)

Supported

Development volume

Low

Low

High (need to arrange yourself)

In short: Choose Audio Simple Player if you only need audio playback without any additional capabilities; for a ready-to-use comprehensive audio and video player, choose ESP Player; consider bare GMF self-build only if you need non-standard, highly customized media processing chains. If you also need multi-channel mixing, playlists, or direct connections with collection/SIP/RTSP services, see ESP-Player-Service Component.

Which chips are supported?

Supports the full range of Espressif chips. Pure audio playback has lower resource requirements and can run on a wider range of chips; video playback requires more memory and computing power, it is recommended to use chips with PSRAM such as ESP32-S3, ESP32-S31, ESP32-P4, etc. For higher resolution MJPEG video, it is recommended to use ESP32-P4/S31 that supports hardware JPEG decoding.

How to reduce memory usage?

Refer to the previous section “Memory Optimization Suggestions”: In menuconfig, trim the container/decoder/codec according to the actual format and hardware, turn off unnecessary video paths and network buffers, and turn on PSRAM (the default task stack is already in PSRAM).