ESP-Player Component

Note

This document is automatically translated using AI. Please excuse any detailed errors. The official English version is still in progress.

Introduction to ESP Player

ESP Player is an embedded multimedia playback component launched by Espressif, which completes the entire playback chain of demuxing → decoding → rendering within a single instance, targeting resource-constrained IoT and multimedia products.

ESP Player serves as a coordination layer between the application and the underlying GMF multimedia pipeline, responsible for state machines, command distribution, and audio-video synchronization. The application only needs to set the playback source and control the playback through a simple API, which can support various scenarios such as local files, HTTP(S) streams, HLS, and feeding of external frames without containers.

Feature Highlights

Capability	Description
Multiple Input Sources	Local files, HTTP/HTTPS streams, HLS (on-demand/live, automatically recognizes `.m3u8`), and feeding of external frames without containers (Bluetooth A2DP raw frames, microphone PCM, custom encoded frames)
Rich Format Support	Supports common encapsulation containers and mainstream audio and video encoding formats, covering music playback, voice prompts, audio/video monitoring playback, etc.
Audio-Video Synchronization	Supports system clock, audio as the main clock, video as the main clock, and no synchronization (freerun) modes, meeting different synchronization accuracy requirements
Complete Playback Control	Play, pause, resume, stop, Seek (millisecond level), speed playback
Audio-Video Track Selection	Can separately switch audio/video paths, multi-track containers support track enumeration and switching (such as multi-language audio tracks)
Network Buffering	Based on queue water level startup pre-buffering and runtime re-buffering, improving the smoothness of network streaming media playback
Event Notification	Supports synchronous callback and asynchronous event queue modes, covering playback status, buffering, errors, track information, etc.
Flexible Trimming and Tuning	Can trim functions according to format and hardware in menuconfig to reduce memory; supports multiple instances and individual configuration of task/buffer; supports registration of custom decoders

Supported Media Formats

Category	Supported Content
Input Source	Local files (`file:///`), HTTP/HTTPS streams, HLS (`.m3u8`), external frame mode (`fill:///`, `block:///`)
Encapsulation Containers	WAV, MP4, M4A, TS, OGG, AVI, FLV, CAF; and raw ES stream files (`.mp3`, `.aac`, `.flac`, `.amr`, without container headers)
Audio Decoding	AAC, MP3, Vorbis, Opus, FLAC, AMR-NB/WB, G.711 A-law/μ-law, ALAC, ADPCM, SBC, LC3
Video Decoding	H.264, MJPEG

Input Source and URL

ESP Player specifies the playback source through a unified URL: the scheme determines the input channel, and the file extension (.codec) determines the decoder. URLs are divided into two major categories: standard URI (local/network) and external frame mode (without containers).

Standard URI (Local and Network)

Type	URL Example	Description
Local File	`file:///sdcard/music/test.mp3`	Standard triple-slash format; you can also write the bare path `/sdcard/music/test.mp3` directly, which will be standardized to `file:///`
Local Headless PCM	`file:///sdcard/test.pcm?sr=48000&ch=2&bits=16`	Headless PCM without a container must specify the sampling rate, number of channels, and bit width through `?query`
HTTP(S) Stream	`https://host/audio/test.mp4`	Network on-demand; supports specifying the port, such as `http://192.168.1.10:8080/stream.aac`
HTTP(S) Basic Authentication	`https://user:pass@host/audio/test.mp4`	Optional `user:pass@` basic authentication
HLS	`https://host/live/playlist.m3u8`	Paths containing `.m3u8` are automatically recognized as HLS, supporting on-demand and live streaming
Network Headless PCM	`http://host/audio/test.pcm?sr=48000&ch=2&bits=16`	Network headless PCM also needs to carry `?query`

External Frame Mode (Containerless Frame Feeding)

Applications directly send raw frames into decoding, suitable for scenarios such as Bluetooth A2DP raw frames, microphone PCM, custom encoding frames, etc. without standard containers. The URL is like fill:///name.codec[?params] or block:///name.codec[?params]:

fill: A complete copy of the frame is made each time it is called, which is simple to use;
block: Zero-copy, the caller must ensure that the buffer is valid before decoding is completed;
name has no actual semantics, the extension .codec determines the decoder;
External frame mode does not support audio and video synchronization (ESP_PLAYER_MASK_AV).

After running, send frames through esp_player_submit_frame(). Common examples:

Scenario	URL Example	Description
Raw PCM	`fill:///test.pcm?sr=16000&ch=1&bits=16`	Full frame copy
Raw PCM (Zero-copy)	`block:///test.pcm?sr=16000&ch=1&bits=16`	Save a memory copy
AAC (Standard ADTS)	`fill:///test.aac`	With ADTS header
AAC (No ADTS header, such as BT A2DP)	`fill:///test.aac?no_adts=1`	Bluetooth A2DP and other headerless AAC
HE-AAC Raw Frame	`fill:///test.aac?no_adts=1&aac_plus=1`	HE-AAC (AAC+)
OPUS Raw Frame	`fill:///test.opus?sr=16000&ch=2&frame_dms=20`	Specify sampling rate/channel/frame duration

Common query parameters

Parameter	Meaning
`sr`	Sampling rate (Hz)
`ch`	Number of channels
`bits`	Bit width (bit)
`no_adts`	AAC without ADTS header (such as Bluetooth A2DP raw frame)
`aac_plus`	HE-AAC (AAC+)
`frame_dms`	OPUS/LC3 frame duration
`plc`	SBC/LC3 packet loss compensation

For the complete parameter list, see esp_player_set_url(). Network, HLS, and other inputs also need to enable the corresponding IO in the menuconfig’s ESP Player → Input IO sources. If the input type is not enabled, an error will be returned when setting the URL.

Typical Application Scenarios

Application Scenario	Description
Audio playback products	Music players, story machines, audiobooks, etc., support local and network audio sources, speed and Seek
Voice prompts and notification sounds	Device boot sound, operation prompt sound, alarm prompt, etc., can be played simultaneously through multi-instance mixing
Screened audio and video playback	Play local or network MP4 (H.264 + AAC) on devices with LCD + audio, synchronized audio and video output
Network streaming media and live broadcast	Play on-demand or live content through HTTP(S)/HLS
Bluetooth audio and custom frames	Play Bluetooth A2DP raw frames, microphone PCM or custom encoded frames (no container required) through external frame mode
Multi-channel audio mixing	Multiple player instances decode in parallel and mix output to the same DAC

Advanced Business Usage

Seamless Source Switching

For scenarios such as switching songs in the playlist, changing channels, etc., there is no need to reinitialize the player: just call esp_player_stop() → esp_player_set_url() → esp_player_run() in order, and ESP Player will automatically dismantle the old pipeline and rebuild according to the new source.

Multi-instance Mixing

You can create multiple player instances to play in parallel and mix output to the same DAC. Suitable for scenarios such as simultaneous playback of background music and prompt sounds/voice broadcasts, overlay of multiple audio sources, etc.

External Frame Mode

When there is no standard container, the application can directly feed the raw frame to the player for decoding and playback: specify through fill:/// (whole frame copy) or block:/// (zero copy), and call esp_player_submit_frame() to send the frame after running. Typical scenarios are the playback of Bluetooth A2DP raw frames, microphone PCM, and custom encoded frames.

Cooperation with Peripheral Components

ESP Player is only responsible for the playback core. In actual solutions, it usually cooperates with other GMF components, each performing its own duties:

Component	Responsibility
`esp_board_manager`	Initialize LCD, audio DAC, SD card and other board-level resources
`esp_player`	Unpacking, decoding, audio and video synchronization and playback control (core)
`esp_audio_render`	Output the decoded audio PCM to the speaker/DAC, support multi-channel mixing
`esp_video_render`	Display the decoded video frame to the screen through the LCD backend

Data flow: Data source → esp_player(Demux → Decode → Sync)→ esp_audio_render / esp_video_render → Speaker / LCD.

Performance Data

Audio Playback Performance

Test environment:

Chip	CPU	SPI RAM	Flash
ESP32-P4	360 MHz	200 MHz	QIO

Comparison of playing local SD card files, MP3 and AAC:

Format	Memory (KB)	CPU (%)	Start-up delay (ms)	Resume delay (ms)
MP3	95	3.1	188	124
AAC	105	3.6	185	126

Start-up delay: Time from esp_player_run() to the first frame of audio output
Resume delay: Time from esp_player_stop() + esp_player_run()(without changing URL) to the first frame of audio output

Video Playback Performance

The video playback of ESP Player is an end-to-end process (IO + Demux + Decode + Render), and the overall frame rate is the result of the combined effects of each link. Take ESP32-P4 playing H.264 as an example:

Resolution	End-to-end frame rate (fps)
320×240	About 40
640×480	About 15

For the individual decoding frame rate of each decoder under different chips and resolutions, please refer to the corresponding decoding component documents (such as esp_h264, esp_new_jpeg, esp_video_codec).

Main Links Affecting Frame Rate

The end-to-end frame rate is usually restricted by the following links (arranged according to the common impact degree):

Link	Description
Video decoding	Usually the main bottleneck. The higher the resolution of software decoding (such as H.264, soft decoding MJPEG), the greater the overhead; the hardware JPEG decoding of ESP32-P4/S31 can significantly reduce CPU usage and improve frame rate
Display / Rendering	The decoding output is mostly YUV, and color space conversion (such as YUV→RGB565) and full-frame transfer to the screen are often required before delivery, which is limited by the LCD interface bandwidth (SPI/8080/RGB/MIPI-DSI) and refresh rate, especially at high resolutions
Input / IO	Local playback is affected by the reading speed of SD card/Flash; network playback is affected by bandwidth, delay and jitter
Memory bandwidth	When the frame buffer is large, it is often placed in PSRAM, and its read and write bandwidth and Cache hit will directly affect the decoding and rendering speed
Task scheduling and synchronization	Task priority, dual-core allocation, buffer water level and audio and video synchronization strategy will all affect actual throughput

Memory Optimization Suggestions

Only enable the actual used encapsulation containers, decoders and on-board codecs in menuconfig, and close all unused ones;
For pure local audio scenarios, you can close the video path and network buffer related configurations;
Enabling PSRAM (SPIRAM, FREERTOS_TASK_CREATE_ALLOW_EXT_MEM) allows larger stacks and buffers to be placed externally, reducing internal RAM usage;
When there are multiple instances and the resources of each are significantly different, each handle can be individually configured for task and buffer.

How to Use ESP Player

ESP Player is hosted on GitHub. You can add ESP Player to your project by entering the following command in your project: idf.py add-dependency “espressif/esp_player”.

The basic usage process is: initialize the player and configure the audio/video rendering handle → set the playback source (AV mask, URL, sync mode) → start playback, and handle playback end, error and other states through event callbacks.

The examples folder of the component provides runnable example projects, it is recommended to refer to and debug before use:

audio_player : Audio playback (including multi-channel mixing)
video_player : Screen audio and video playback

For more API descriptions, URL formats and advanced usage, please refer to README.

FAQ

What are the capabilities of ESP Player?

ESP Player is only responsible for the playback link (demux → decode → render), and does not include the network transport protocol stack. It can directly play local files, HTTP(S)/HLS streams, or receive raw frames sent by the upstream for decoding and playback through the external frame mode. Therefore, when it comes to real-time two-way communication, protocol signaling interaction or push-pull streaming(such as SIP, RTSP, RTMP, WebRTC), the corresponding transport/communication component is responsible for receiving and sending media data, ESP Player and it belong to different links, and can be used as needed.

How to choose between ESP Player, Audio Simple Player and bare GMF?

All three can “play/process media”, but the encapsulation level and positioning are different: Audio Simple Player (esp_audio_simple_player) is a lightweight pure audio player, which decodes/transforms and hands over PCM to the application through callbacks; ESP Player (esp_player) is an end-to-end complete audio and video player; bare GMF is directly based on esp-gmf’s pipeline/element/task arrangement, with the highest flexibility but needs to implement various capabilities by itself.

Capability	Audio Simple Player	ESP Player	Bare GMF (self-built)
Pure audio local / HTTP	Supported	Supported	Supported (self-built)
Audio and video (MP4, etc.)	Not supported	Supported	Supported (self-built)
HLS	Not supported	Supported	Need to integrate yourself
Seek	Not supported	Supported (millisecond level)	Self-implemented
Pause / Continue	Supported	Supported	Self-implemented
Audio and video synchronization	— (Audio only)	Supported (4 modes)	Self-implemented
Network buffering / re-buffering	Weak	Supported	Self-implemented
Multi-track / track switching	Not supported	Supported	Self-implemented
Fill/block frame feeding	Not supported	Supported	Can be self-built
Output method	PCM callback	Direct render output (speaker/screen)	Any
Custom processing elements	Supported (register element)	Supported (register custom decoder element)	Supported (any element)
Multiple pipelines / mixing	Not supported (single chain)	Single instance (audio and video integrated)	Supported
Development volume	Low	Low	High (need to arrange yourself)

In short: Choose Audio Simple Player for audio playback only, with no additional capabilities required; choose ESP Player for a complete audio and video playback out of the box; consider bare GMF self-built only for non-standard, highly customized media processing chains.

Which chips are supported?

Supports the full range of Espressif chips. Pure audio playback has lower resource requirements and can run on a wider range of chips; video playback requires more memory and computing power, it is recommended to use chips with PSRAM such as ESP32-S3, ESP32-S31, ESP32-P4, etc. For higher resolution MJPEG video, it is recommended to use ESP32-P4/S31 that supports hardware JPEG decoding.

How to reduce memory usage?

Refer to the above “Memory Optimization Suggestions”: Trim the container/decoder/codec according to the actual format and hardware in menuconfig, turn off unnecessary video paths and network buffering, and enable PSRAM.