# GMF-Video

[中文]

gmf_video is the video processing component of ESP-GMF, providing 9 elements covering codec, pixel accelerator, software image effects, and frame rate / overlay tools. All elements inherit from `esp_gmf_video_element_t`, exposing unified lifecycle functions, acquire-release data interfaces, and runtime method calls. The underlying algorithms are implemented by ESP32-P4’s hardware PPA / 2D-DMA, the `esp_video_codec` library, and the `esp_image_effects` software image effects library. This document covers the purpose, configuration, and runtime control of each element by category. For the element base class and runtime method mechanism, see GMF Elements; for the data path, see Data Flow.

## Feature List

- vid_dec: video decoding, supporting H.264 and MJPEG; output pixel format can be specified (e.g., YUV420P, RGB565LE)

- vid_enc: video encoding, supporting H.264 and MJPEG; bitrate, GOP and QP range(H.264 only) can be adjusted at runtime

- vid_ppa: ESP32-P4 pixel processing accelerator composite element, merging color conversion, scaling, cropping, and rotation into a single hardware pass

- vid_fps_cvt: frame rate conversion, dropping frames by PTS to reduce input frame rate to a specified output frame rate

- vid_overlay: overlay mixer, blending additional content (watermarks, UI, timestamps) onto the original video via alpha blending or transparent-color (colorkey) compositing

- vid_color_cvt: software color conversion, covering common format conversions such as YUYV, RGB565, RGB888, and YUV420P

- vid_crop: software cropping, extracting a specified rectangular region from the original frame

- vid_scale: software scaling, resampling video frames to a target resolution

- vid_rotate: software rotation, supporting arbitrary angles (in degrees)

- Common: bypass optimization, alignment auto-query, `need_recfg`-triggered runtime reconfiguration, PTS pass-through and frame drop logic

## Technical Details

### Element Hierarchy and Common Mechanisms

Each specific element is constructed by “wrapping an underlying video algorithm / hardware handle”: the derived struct uses `esp_gmf_video_element_t` as its first field, binding the codec / PPA / imgfx handle and this element’s configuration; `open` creates the underlying handle and reports the output `esp_gmf_info_video_t` (width, height, frame rate, pixel format FourCC) to the pipeline; `process` fetches a frame from the input port at each scheduling interval, performs the conversion, and writes to the output port; `close` releases hardware resources.

            classDiagram
        direction TD

        class esp_gmf_video_element_t
        class Codec {
            vid_dec
            vid_enc
        }
        class HwAccel {
            vid_ppa
        }
        class SwEffects {
            vid_color_cvt
            vid_crop
            vid_scale
            vid_rotate
        }
        class Utility {
            vid_fps_cvt
            vid_overlay
        }

        esp_gmf_video_element_t <|-- Codec
        esp_gmf_video_element_t <|-- HwAccel
        esp_gmf_video_element_t <|-- SwEffects
        esp_gmf_video_element_t <|-- Utility

Four common mechanisms:

- Bypass optimization: vid_dec enters zero-copy pass-through when source and destination formats are identical; software effect elements also set bypass when source and destination parameters are identical, directly passing the input payload to the output port.

- need_recfg reconfiguration: All setters (`set_dst_format` / `set_dst_resolution` / `set_angle`, etc.) set the `need_recfg` flag after a change; at the next `process` boundary, the old handle is destroyed and a new one is rebuilt, allowing resolution or pixel format changes at runtime without the application rebuilding the element.

- Buffer alignment auto-query: Underlying hardware (PPA, H.264 encoder) typically has alignment requirements for input/output buffers; the element queries alignment parameters from the hardware driver at `open` and writes them to the port attr so the framework allocates payloads with the correct alignment, preventing hardware from rejecting processing.

- Format info reporting: After the upstream element resolves the width/height / pixel format of the current frame, it reports via the `REPORT_INFO` event; the pipeline delivers the info to downstream elements, which open or trigger reconfiguration accordingly. For details see the REPORT_INFO description in GMF Pipeline and Task Scheduling.

### Video Codec

vid_dec and vid_enc are the start or end elements of a video pipeline; a typical assembly is as follows.

            flowchart LR
        subgraph rec ["Recording Pipeline"]
            direction LR
            CapIn(("camera/raw")) --> CC1["vid_color_cvt"]
            CC1 --> Scale["vid_scale"]
            Scale --> Enc["vid_enc"]
            Enc --> FileOut(("file/http"))
        end
        subgraph play ["Playback Pipeline"]
            direction LR
            FileIn(("file/http")) --> Dec["vid_dec"]
            Dec --> PPA["vid_ppa"]
            PPA --> Disp(("display"))
        end

vid_dec: Decodes compressed streams to raw pixels. `esp_gmf_video_dec_init()` accepts `esp_gmf_video_dec_cfg_t`; the `codec_cc` field uses FourCC to specify the decoder implementation; when zero, the framework automatically selects based on available hardware / software decoders.

Use `esp_gmf_video_dec_get_dst_formats()` to query the list of available output pixel formats for a given source codec (H.264 / MJPEG, etc.), then use `esp_gmf_video_dec_set_dst_format()` to fix the output format before starting the element. If the source and destination formats are identical (rare but useful, e.g., a remux-only pipeline), vid_dec automatically enters bypass mode and forwards payloads directly.

    esp_gmf_video_dec_cfg_t cfg = { .codec_cc = 0 };  /* Auto select */
    esp_gmf_element_handle_t dec = NULL;
    esp_gmf_video_dec_init(&cfg, &dec);

    const uint32_t *fmts = NULL;
    uint8_t fmt_num = 0;
    esp_gmf_video_dec_get_dst_formats(dec, ESP_FOURCC_H264, &fmts, &fmt_num);
    esp_gmf_video_dec_set_dst_format(dec, fmts[0]);

vid_enc: Encodes raw pixels to a compressed stream. `esp_gmf_video_enc_init()` accepts `esp_gmf_video_enc_cfg_t`; `codec_cc` specifies the encoder implementation.

The normal workflow is for the element to obtain source info (`esp_gmf_info_video_t`) from upstream after startup, then auto-select the specific implementation based on the target codec specified by `set_dst_codec` and open. To query encoder capabilities before startup (e.g., query supported source pixel formats), first call `esp_gmf_video_enc_preset()` with the source info and target codec, then call `esp_gmf_video_enc_get_src_formats()` and `esp_gmf_video_enc_get_out_size()`.

Three parameters can be adjusted at runtime (H.264 encoder additionally supports the last two):

    esp_gmf_video_enc_set_bitrate(enc, 2 * 1000 * 1000);  /* 2 Mbps */
    esp_gmf_video_enc_set_gop(enc, 30);                    /* I-frame every 30 frames */
    esp_gmf_video_enc_set_qp(enc, 20, 40);                 /* QP range */

Output buffer size is estimated by `esp_gmf_video_enc_get_out_size()`, using a 10:1 compression ratio upper bound for MJPEG and 2:1 for H.264; the framework allocates output payloads at this size.

### Hardware-Accelerated PPA

vid_ppa is supported on ESP32-P4/ESP32S31 only, merging color conversion, scaling, cropping, and rotation into a single hardware pass. `esp_gmf_video_ppa_init()` requires no configuration; four setters configure the target format, target resolution, crop region, and rotation angle (only 0 / 90 / 180 / 270 degrees are supported). These setters can only be called before the element starts; `open` submits the combined parameters to hardware in one shot.

PPA internally selects the hardware path based on the current request: pure pixel format conversion can be completed by 2D-DMA with higher throughput; scaling, rotation, or format combinations not supported by 2D-DMA fall back to the main PPA path. Path selection is handled internally; the element only exposes a unified interface.

    esp_gmf_element_handle_t ppa = NULL;
    esp_gmf_video_ppa_init(NULL, &ppa);

    esp_gmf_video_resolution_t dst_res = { .width = 480, .height = 320 };
    esp_gmf_video_ppa_set_dst_resolution(ppa, &dst_res);
    esp_gmf_video_ppa_set_dst_format(ppa, ESP_FOURCC_RGB16);
    esp_gmf_video_ppa_set_rotation(ppa, 90);

    esp_gmf_video_rgn_t crop = { .x = 0, .y = 0, .width = 800, .height = 600 };
    esp_gmf_video_ppa_set_cropped_rgn(ppa, &crop);

On non-ESP32-P4 chips, vid_ppa is unavailable; use the four software effect elements in series (vid_color_cvt → vid_crop → vid_scale → vid_rotate) for equivalent functionality, at the cost of increased CPU usage.

### Software Image Effects

The four software effect elements wrap the corresponding algorithms in the `esp_image_effects` library, exposing only their respective setters. All effects follow the same path: `acquire_in` to get the input frame, generate output frame size based on configuration, `acquire_out` to get the output frame, call `esp_imgfx_*_process` to process, then release both.

vid_color_cvt: Pixel format conversion, e.g., RGB565 → YUV420P, YUYV → RGB888. The color space standard (BT601 / BT709) is selected via the `esp_imgfx_color_convert_cfg_t.color_space_std` field at initialization. `esp_gmf_video_color_convert_dst_format()` switches the target format at runtime, setting `need_recfg`.

vid_crop: Crop element. Initialized with `DEFAULT_ESP_GMF_CROP_CONFIG` for a default 320×240 input and 160×120 crop region. Use `esp_gmf_video_crop_rgn()` at runtime to switch the crop rectangle (`esp_gmf_video_rgn_t` provides x / y / width / height).

vid_scale: Scale element. The `filter_type` field determines the algorithm (e.g., `ESP_IMGFX_SCALE_FILTER_TYPE_DOWN_RESAMPLE`). Use `esp_gmf_video_scale_dst_resolution()` at runtime to change the target resolution.

vid_rotate: Rotate element. Unlike vid_ppa which only supports four angles, software rotation supports arbitrary angles (in degrees), suitable for tilted display or image correction scenarios. `esp_gmf_video_rotate_set_rotation()` switches the angle at runtime.

    esp_imgfx_scale_cfg_t cfg = DEFAULT_ESP_GMF_SCALE_CONFIG();
    cfg.in_res       = (esp_imgfx_res_t){ .width = 640, .height = 480 };
    cfg.in_pixel_fmt = ESP_IMGFX_PIXEL_FMT_RGB565_LE;
    cfg.scale_res    = (esp_imgfx_res_t){ .width = 320, .height = 240 };

    esp_gmf_element_handle_t scale = NULL;
    esp_gmf_video_scale_init(&cfg, &scale);

### Frame Rate and Overlay

vid_fps_cvt: Reduces input frame rate to a specified output frame rate by dropping frames. The element maintains a start PTS and accumulated output frame count, computing `expected_pts` at `1 / fps` steps. For each arriving frame: the frame is passed to downstream and the count is incremented only if its PTS is >= `expected_pts`; otherwise it is dropped via `release_in`.

Since pure frame dropping cannot upsample, the value set by `set_fps` must be lower than the upstream frame rate; otherwise it has no effect.

    esp_gmf_element_handle_t fps = NULL;
    esp_gmf_video_fps_cvt_init(NULL, &fps);
    esp_gmf_video_fps_cvt_set_fps(fps, 15);  /* Reduce from 30 fps to 15 fps */

vid_overlay: Overlays additional content onto the main video. The element has multiple inputs: the main video enters through the default in port, and the overlay layer enters through the extra port registered by the user via `esp_gmf_video_overlay_set_overlay_port()`. Compositing supports alpha blending and transparent-color (colorkey) modes. Alpha blending formula:

    Output = Original × (255 − alpha) + Overlay × alpha

`set_rgn` specifies the position and format of the overlay layer on the main frame (`esp_gmf_overlay_rgn_info_t` contains `format_id`, `dst_rgn`, and optional `has_trans_color` / `trans_color` for colorkey compositing—pixels near the specified RGB value are treated as transparent); `set_alpha` changes transparency at runtime (0 fully transparent, 255 fully opaque); `esp_gmf_video_overlay_enable()`(false) temporarily disables the overlay, the element no longer acquires the overlay port, and the main frame passes through unchanged.

    esp_gmf_element_handle_t ovl = NULL;
    esp_gmf_video_overlay_init(NULL, &ovl);

    esp_gmf_video_overlay_set_overlay_port(ovl, overlay_port);
    esp_gmf_overlay_rgn_info_t rgn = {
        .format_id = ESP_FOURCC_RGB16,
        .dst_rgn   = { .x = 16, .y = 16, .width = 128, .height = 32 },
    };
    esp_gmf_video_overlay_set_rgn(ovl, &rgn);
    esp_gmf_video_overlay_set_alpha(ovl, 200);

## Performance

Video processing throughput is primarily determined by the underlying hardware / algorithm; the overhead of element scheduling and acquire-release is relatively small. By bottleneck category:

Category | Main Bottleneck | Optimization Direction
--- | --- | ---
vid_dec / vid_enc | Throughput of hardware codec or software library | ESP32-P4 supports H.264 hardware encoding; call `preset` early to let the framework pre-allocate buffers
vid_ppa | PPA / 2D-DMA clock and buffer bandwidth | Merge adjacent pixel operations into the same vid_ppa instance to reduce intermediate buffers
Software effects | CPU and memory bandwidth | Prefer vid_ppa on ESP32-P4; control resolution to within SoC processing capacity for other SoCs
vid_fps_cvt | Only PTS comparison and frame dropping; very low CPU overhead | Suitable for placement before encoding or display to reduce downstream load
vid_overlay | CPU cycles for alpha blending or colorkey compositing | Smaller overlay region and fewer bytes per pixel format means faster processing

## Application Examples

- `elements/test_apps/main/elements/gmf_video_el_test.c`: Typical video element test cases, including `gen_pattern_color_bar` for synthesizing test frames and format reporting validation at the open stage

- `elements/test_apps/main/elements/gmf_image_effects_test.c`: Image effects pipeline test, covering chaining of software effect elements

The upper-level application components `esp_video_render` and `esp_capture` also connect video pipelines through this component and can serve as advanced references.

## SoC Compatibility

Support matrix of elements on Espressif SoCs:

element | ESP32 | ESP32-S2 | ESP32-S3 | ESP32-P4
--- | --- | --- | --- | ---
vid_ppa | Not supported | Not supported | Not supported | Supported
vid_fps_cvt | Supported | Supported | Supported | Supported
vid_overlay | Supported | Supported | Supported | Supported
vid_dec | MJPEG | MJPEG | SW H.264 / MJPEG | SW H.264 / HW MJPEG
vid_enc | MJPEG | MJPEG | SW H.264 / MJPEG | HW H.264 / HW MJPEG
vid_color_cvt / vid_crop / vid_scale / vid_rotate | Supported | Supported | Supported | Supported

ESP32 and ESP32-S2 do not support H.264. On ESP32-S3, H.264 encode and decode are implemented in software. On ESP32-P4, H.264 hardware encoding and MJPEG hardware encode/decode are supported; H.264 decoding is implemented in software. For ESP32-P4 PPA acceleration, omit the software effect elements and use vid_ppa instead.

## FAQ

Q: How do I select the output format for vid_dec?

First use `esp_gmf_video_dec_get_dst_formats()` to query the list of output formats supported by the current source codec, then select a format that meets the input requirements of the downstream element (e.g., vid_ppa needs RGB565, display needs RGB888), and call `esp_gmf_video_dec_set_dst_format()` to fix the output format before starting the element. Output frame size queries require the target format to be set first, or the corresponding query interface to be called after the element starts.

Q: Cannot query output frame size before vid_enc starts?

Normally the encoder can only determine the output size after obtaining `esp_gmf_info_video_t` from upstream. To query before startup, first call `esp_gmf_video_enc_preset()` with the expected source info and target codec; then `esp_gmf_video_enc_get_out_size()` can return a value.

Q: Is vid_ppa supported on ESP32?

No. vid_ppa depends on ESP32-P4 hardware PPA / 2D-DMA. On other SoCs, use the software implementations vid_color_cvt / vid_scale / vid_rotate / vid_crop instead; resolution and frame rate should be constrained according to actual CPU capability.

Q: What happens if the frame rate set in vid_fps_cvt is higher than upstream?

The element does not insert frames. If `set_fps` is higher than the upstream frame rate, every frame’s PTS will be >= `expected_pts`, all frames pass through, and the output frame rate is limited by upstream. The configuration takes effect only when reducing the frame rate; it must not exceed the upstream rate.

Q: How are the two inputs of vid_overlay synchronized?

The main input is a regular GMF input port; the overlay input is the extra port registered by the user via `set_overlay_port`. The element acquires the main input first then the overlay input in each `process`; the application is responsible for ensuring the two streams are frame-aligned (e.g., two upstream elements in the same pipeline producing frames synchronously); otherwise overlay frames may lag or jitter.

Q: How to choose between software effect elements and vid_ppa on ESP32-P4?

When combining color conversion + scaling + rotation / cropping, prefer vid_ppa: a single hardware pass completes all conversions with low CPU usage and stable latency. Use the corresponding software elements when only a single operation is needed, or when the angle / pixel format required is not supported by PPA.

## API Reference

Header files for this component:

- `esp_gmf_video_dec.h` / `esp_gmf_video_enc.h`: Video codec elements

- `esp_gmf_video_ppa.h`: Pixel processing accelerator

- `esp_gmf_video_fps_cvt.h`: Frame rate conversion

- `esp_gmf_video_overlay.h`: Overlay mixer

- `esp_gmf_video_color_convert.h` / `esp_gmf_video_crop.h` / `esp_gmf_video_scale.h` / `esp_gmf_video_rotate.h`: Software image effects

- `esp_gmf_video_types.h`: Shared types for video resolution / region / overlay configuration

- `esp_gmf_video_param.h`: Video parameter helper interface

The interface for the element base class `esp_gmf_video_element_t` is located in GMF Elements.

### Header File

- elements/gmf_video/include/esp_gmf_video_dec.h

### Functions

esp_gmf_err_t esp_gmf_video_dec_init(esp_gmf_video_dec_cfg_t *cfg, esp_gmf_element_handle_t *handle)

  Initializes the GMF Video decoder with the provided configuration.

  Parameters:

    - cfg – [in] Pointer to the Video decoder configuration (optional)

    - handle – [out] Pointer to the Video decoder handle to be initialized

  Returns:

    - ESP_GMF_ERR_OK Success

    - ESP_GMF_ERR_INVALID_ARG Invalid configuration provided

    - ESP_GMF_ERR_MEMORY_LACK Failed to allocate memory

esp_gmf_err_t esp_gmf_video_dec_get_out_size(esp_gmf_element_handle_t handle, uint32_t *frame_size)

  Get Video decoder output frame size.

  Note

  This API should only called after element running or `esp_gmf_video_dec_set_dst_format` is called

  Parameters:

    - handle – [in] Video decoder element handle

    - frame_size – [out] Output frame size to store

  Returns:

    - ESP_GMF_ERR_OK Success

    - ESP_GMF_ERR_INVALID_ARG Invalid configuration provided

esp_gmf_err_t esp_gmf_video_dec_set_dst_format(esp_gmf_element_handle_t handle, uint32_t dst_fmt)

  Setting for video decoder destination format (e.g., YUV420P, RGB565LE)

  Note

  This API should only called before element running

  Parameters:

    - handle – [in] Video decoder element handle

    - dst_fmt – [in] Destination format (GMF FourCC representation of video codec)

  Returns:

    - ESP_GMF_ERR_OK Success

    - ESP_GMF_ERR_INVALID_ARG Invalid configuration provided

esp_gmf_err_t esp_gmf_video_dec_get_dst_formats(esp_gmf_element_handle_t handle, uint32_t src_codec, const uint32_t **dst_fmts, uint8_t *dst_fmts_num)

  Get supported output formats for a given video decoder type This function retrieves the list of output formats (e.g., YUV420P, RGB565LE) that the decoder can produce when decoding from the specified source codec.

  Parameters:

    - handle – [in] Video decoder element handle

    - src_codec – [in] Decode type (GMF FourCC representation of video codec)

    - dst_fmts – [out] Output video formats to store (GMF FourCC representation of video format)

    - dst_fmts_num – [out] Output video format number to store

  Returns:

    - ESP_GMF_ERR_OK Success

    - ESP_GMF_ERR_INVALID_ARG Invalid configuration provided

### Structures

struct esp_gmf_video_dec_cfg_t

  Video decoder configuration (optional)

  Public Members

  uint32_t codec_cc

    FourCC code to force a specific decoder implementation See `esp_video_codec_desc_t` in `esp_video_codec_types.h` for details If zero, the system will select a suitable decoder (HW preferred)

### Header File

- elements/gmf_video/include/esp_gmf_video_enc.h

### Functions

esp_gmf_err_t esp_gmf_video_enc_init(esp_gmf_video_enc_cfg_t *cfg, esp_gmf_element_handle_t *handle)

  Initializes the GMF video encoder with the provided configuration.

  Parameters:

    - cfg – [in] Pointer to the video encoder configuration

    - handle – [out] Pointer to the video encoder handle to be initialized

  Returns:

    - ESP_GMF_ERR_OK Success

    - ESP_GMF_ERR_INVALID_ARG Invalid configuration provided

    - ESP_GMF_ERR_MEMORY_LACK Failed to allocate memory

esp_gmf_err_t esp_gmf_video_enc_set_dst_codec(esp_gmf_element_handle_t handle, uint32_t dst_codec)

  Setting for video encoder destination codec (e.g., H.264, MJPEG)

  Note

  This API should only called before element running

  Parameters:

    - handle – [in] Video encoder element handle

    - dst_codec – [in] Video encode destination codec (use GMF FourCC presentation)

  Returns:

    - ESP_GMF_ERR_OK Success

    - ESP_GMF_ERR_INVALID_ARG Invalid configuration provided

esp_gmf_err_t esp_gmf_video_enc_preset(esp_gmf_element_handle_t handle, esp_gmf_info_video_t *src_info, uint32_t dst_codec)

  Do presetting for video encoder.

  Note

  This function pre-configures the video encoder with both source and destination information This allows querying encoder capabilities before runtime Normally, source information is reported from dependent element in the running state Call this function only before element running

  Parameters:

    - handle – [in] Video encoder element handle

    - src_info – [in] Video encode input video information

    - dst_codec – [in] Destination video codec (use GMF FourCC presentation)

  Returns:

    - ESP_GMF_ERR_OK Success

    - ESP_GMF_ERR_INVALID_ARG Invalid configuration provided

esp_gmf_err_t esp_gmf_video_enc_get_out_size(esp_gmf_element_handle_t handle, uint32_t *frame_size)

  Get video encoder output frame size.

  Note

  This API should only be called after `esp_gmf_video_enc_preset` set or after element running

  Parameters:

    - handle – [in] Video encoder element handle

    - frame_size – [out] Output frame size to store

  Returns:

    - ESP_GMF_ERR_OK Success

    - ESP_GMF_ERR_INVALID_ARG Invalid configuration provided

esp_gmf_err_t esp_gmf_video_enc_get_src_formats(esp_gmf_element_handle_t handle, uint32_t dst_codec, const uint32_t **src_fmts, uint8_t *src_fmts_num)

  Get supported source formats (e.g., YUV420P, RGB565LE etc) for a given video encoder type This function retrieves the list of supported input formats that can be encoded into the specified destination codec.

  Parameters:

    - handle – [in] Video encoder element handle

    - dst_codec – [in] Video encode destination codec (GMF FourCC representation of video codec)

    - src_fmts – [out] Source formats to stored (GMF FourCC representation of video format)

    - src_fmts_num – [out] Source format number to store

  Returns:

    - ESP_GMF_ERR_OK Success

    - ESP_GMF_ERR_INVALID_ARG Invalid configuration provided

esp_gmf_err_t esp_gmf_video_enc_set_bitrate(esp_gmf_element_handle_t handle, uint32_t bitrate)

  Setting for video encoder bitrate.

  Note

  Support setting both before and after element running

  Parameters:

    - handle – [in] Video encoder element handle

    - bitrate – [in] Output bitrate

  Returns:

    - ESP_GMF_ERR_OK Success

    - ESP_GMF_ERR_INVALID_ARG Invalid configuration provided

esp_gmf_err_t esp_gmf_video_enc_set_gop(esp_gmf_element_handle_t handle, uint32_t gop)

  Set the GOP (Group of Pictures) for video encoder.

  Note

  This setting applies specifically to the H.264 encoder The H.264 encoder generates an I-frame at the start of each GOP Can be configured both before and after the encoder element is running

  Parameters:

    - handle – [in] Handle to the video encoder element

    - gop – [in] GOP size for H.264 (in number of frames)

  Returns:

    - ESP_GMF_ERR_OK On success

    - ESP_GMF_ERR_INVALID_ARG Invalid configuration provided

esp_gmf_err_t esp_gmf_video_enc_set_qp(esp_gmf_element_handle_t handle, uint32_t min_qp, uint32_t max_qp)

  Set the QP (Quantization Parameter) range for video encoder.

  Note

  This setting applies specifically to the H.264 encoder Higher QP values generally result in lower image quality Can be configured both before and after the encoder element is running

  Parameters:

    - handle – [in] Handle to the video encoder element

    - min_qp – [in] Minimum QP value

    - max_qp – [in] Maximum QP value

  Returns:

    - ESP_GMF_ERR_OK On success

    - ESP_GMF_ERR_INVALID_ARG Invalid configuration provided

### Structures

struct esp_gmf_video_enc_cfg_t

  Video encoder configuration.

  Public Members

  uint32_t codec_cc

    FourCC code to force a specific encoder implementation See `esp_video_codec_desc_t` in `esp_video_codec_types.h` for details If zero, the system will select a suitable encoder (HW preferred)

### Header File

- elements/gmf_video/include/esp_gmf_video_ppa.h

### Functions

esp_gmf_err_t esp_gmf_video_ppa_init(void *config, esp_gmf_element_handle_t *handle)

  Video PPA (Pixel Processing Accelerator) is a compact module Currently it only support on ESP32P4 and including following video processing: 1) Color convert 2) Scale 3) Rotate 5) Crop These operations can be mixed so that can do them all together for better performance.

  Initializes the GMF Video PPA

  Parameters:

    - config – [in] No need to set

    - handle – [out] Video PPA handle to store

  Returns:

    - ESP_GMF_ERR_OK Success

    - ESP_GMF_ERR_INVALID_ARG Invalid configuration provided

    - ESP_GMF_ERR_MEMORY_LACK Failed to allocate memory

esp_gmf_err_t esp_gmf_video_ppa_set_dst_resolution(esp_gmf_element_handle_t handle, esp_gmf_video_resolution_t *res)

  Set video PPA destination resolution.

  Note

  This API should only called before element running

  Parameters:

    - handle – [in] Video PPA handle

    - res – [in] Video PPA output resolution

  Returns:

    - ESP_GMF_ERR_OK Success

    - ESP_GMF_ERR_INVALID_ARG Invalid configuration provided

    - ESP_GMF_ERR_MEMORY_LACK Failed to allocate memory

esp_gmf_err_t esp_gmf_video_ppa_set_dst_format(esp_gmf_element_handle_t handle, uint32_t format)

  Set video PPA destination format.

  Note

  This API should only called before element running

  Parameters:

    - handle – [in] Video PPA handle

    - format – [in] Video PPA destination format (GMF FourCC representation of video format)

  Returns:

    - ESP_GMF_ERR_OK Success

    - ESP_GMF_ERR_INVALID_ARG Invalid configuration provided

    - ESP_GMF_ERR_MEMORY_LACK Failed to allocate memory

esp_gmf_err_t esp_gmf_video_ppa_set_cropped_rgn(esp_gmf_element_handle_t handle, esp_gmf_video_rgn_t *rgn)

  Set video PPA cropped region.

  Note

  This API should only called before element running

  Parameters:

    - handle – [in] Video PPA handle

    - rgn – [in] Region to be kept in original video frame

  Returns:

    - ESP_GMF_ERR_OK Success

    - ESP_GMF_ERR_INVALID_ARG Invalid configuration provided

    - ESP_GMF_ERR_MEMORY_LACK Failed to allocate memory

esp_gmf_err_t esp_gmf_video_ppa_set_rotation(esp_gmf_element_handle_t handle, uint16_t degree)

  Set video PPA rotation degree.

  Note

  This API should only called before element running

  Parameters:

    - handle – [in] Video PPA handle

    - degree – [in] Currently only support (0, 90, 180, 270) unit(one degree)

  Returns:

    - ESP_GMF_ERR_OK Success

    - ESP_GMF_ERR_INVALID_ARG Invalid configuration provided

### Header File

- elements/gmf_video/include/esp_gmf_video_overlay.h

### Functions

esp_gmf_err_t esp_gmf_video_overlay_init(void *config, esp_gmf_element_handle_t *handle)

  Initializes the GMF overlay mixer with the provided configuration.

  Diagram: +————————-&#8212;+ | Original Plane | | (Background) | | +——————&#8212;+ | | | Overlay Plane | | | | (Foreground) | | | +——————&#8212;+ | +————————-&#8212;+

  Note

  This module mixes the overlay plane (foreground) into the original plane (background) using alpha blending The formula is: Output_Plane_Pixel = Original_Plane_Pixel * (255 - alpha) + Overlay_Plane_Pixel * alpha where `alpha` is the transparency level (0 = fully transparent, 255 = fully opaque)

  Parameters:

    - config – [in] No need to set

    - handle – [out] Pointer to the overlay mixer handle to be initialized

  Returns:

    - ESP_GMF_ERR_OK Success

    - ESP_GMF_ERR_INVALID_ARG Invalid configuration provided

    - ESP_GMF_ERR_MEMORY_LACK Failed to allocate memory

esp_gmf_err_t esp_gmf_video_overlay_set_rgn(esp_gmf_element_handle_t handle, esp_gmf_overlay_rgn_info_t *rgn_info)

  Set the output region for video overlay.

  Note

  This API should only called before element running

  Parameters:

    - handle – [in] Video overlay element handle

    - rgn_info – [out] Region information

  Returns:

    - ESP_GMF_ERR_OK Success

    - ESP_GMF_ERR_INVALID_ARG Invalid configuration provided

    - ESP_GMF_ERR_MEMORY_LACK Failed to allocate memory

esp_gmf_err_t esp_gmf_video_overlay_set_overlay_port(esp_gmf_element_handle_t handle, esp_gmf_port_handle_t port)

  Set the GMF port for video overlay input.

  Note

  The video overlay element will use this port to acquire overlay data, mix it into the original video plane, and then release the port The port is managed by the user, the overlay element only accesses it This API can be called before element running and allow reset

  Parameters:

    - handle – [in] Video overlay element handle

    - port – [in] GMF port for video overlay data

  Returns:

    - ESP_GMF_ERR_OK Success

    - ESP_GMF_ERR_INVALID_ARG Invalid configuration provided

    - ESP_GMF_ERR_MEMORY_LACK Failed to allocate memory

esp_gmf_err_t esp_gmf_video_overlay_set_alpha(esp_gmf_element_handle_t handle, uint8_t alpha)

  Set the alpha value of the overlay window.

  Note

  Allow to set both before and after element running

  Parameters:

    - handle – [in] Video overlay element handle

    - alpha – [in] Overlay window alpha setting: 0 means fully transparent, 255 means fully opaque

  Returns:

    - ESP_GMF_ERR_OK Success

    - ESP_GMF_ERR_INVALID_ARG Invalid configuration provided

esp_gmf_err_t esp_gmf_video_overlay_enable(esp_gmf_element_handle_t handle, bool enable)

  Enable overlay mixer or not.

  Note

  When overlay mixer is disabled will not acquire data from overlay port and do mixer anymore Original plane data leave unchanged Allow to set both before and after element running

  Parameters:

    - handle – [in] Video overlay mixer handle

    - enable – [in] Enable mixer or not

  Returns:

    - ESP_GMF_ERR_OK Success

    - ESP_GMF_ERR_INVALID_ARG Invalid configuration provided

### Header File

- elements/gmf_video/include/esp_gmf_video_fps_cvt.h

### Functions

esp_gmf_err_t esp_gmf_video_fps_cvt_init(void *config, esp_gmf_element_handle_t *handle)

  Initializes the GMF video frame rate convert.

  Parameters:

    - config – [in] No need to set

    - handle – [out] Pointer to the video frame rate convert handle to be initialized

  Returns:

    - ESP_GMF_ERR_OK Success

    - ESP_GMF_ERR_INVALID_ARG Invalid configuration provided

    - ESP_GMF_ERR_MEMORY_LACK Failed to allocate memory

esp_gmf_err_t esp_gmf_video_fps_cvt_set_fps(esp_gmf_element_handle_t handle, uint16_t fps)

  Set GMF video frame rate converter output frame rate.

  Note

  This API should only called before element running

  Parameters:

    - handle – [in] Video frame rate converter handle

    - fps – [out] Destination frame rate to set (unit frame per second)

  Returns:

    - ESP_GMF_ERR_OK Success

    - ESP_GMF_ERR_INVALID_ARG Invalid configuration provided

### Header File

- elements/gmf_video/include/esp_gmf_video_color_convert.h

### Functions

esp_gmf_err_t esp_gmf_video_color_convert_init(esp_imgfx_color_convert_cfg_t *config, esp_gmf_element_handle_t *handle)

  Initialize the video color convert element with the specified configuration.

  Parameters:

    - config – [in] Pointer to a color convert configuration

    - handle – [out] Pointer to a `esp_gmf_element_handle_t` where the initialized video color convert handle will be stored

  Returns:

    - ESP_GMF_ERR_OK Initialization successful

    - ESP_GMF_ERR_INVALID_ARG Invalid arguments

    - ESP_GMF_ERR_MEMORY_LACK No memory for video color convert element

esp_gmf_err_t esp_gmf_video_color_convert_dst_format(esp_gmf_element_handle_t handle, uint32_t format)

  Set video color convert destination format.

  Note

  This API have no special timing request, user can call it freely

  Parameters:

    - handle – [in] Video color convert handle

    - format – [in] Video color convert destination format (GMF FourCC representation of video format)

  Returns:

    - ESP_GMF_ERR_OK Successful

    - ESP_GMF_ERR_INVALID_ARG Invalid arguments

### Macros

DEFAULT_ESP_GMF_COLOR_CONVERT_CONFIG()

### Header File

- elements/gmf_video/include/esp_gmf_video_crop.h

### Functions

esp_gmf_err_t esp_gmf_video_crop_init(esp_imgfx_crop_cfg_t *config, esp_gmf_element_handle_t *handle)

  Initialize the video crop element with the specified configuration.

  Parameters:

    - config – [in] Pointer to a `esp_imgfx_crop_cfg_t` structure that contains the configuration settings for the video crop

    - handle – [out] Pointer to a `esp_gmf_element_handle_t` where the initialized video crop handle will be stored

  Returns:

    - ESP_GMF_ERR_OK Initialization successful

    - ESP_GMF_ERR_INVALID_ARG Invalid arguments

    - ESP_GMF_ERR_MEMORY_LACK No memory for video crop element

esp_gmf_err_t esp_gmf_video_crop_rgn(esp_gmf_element_handle_t handle, esp_gmf_video_rgn_t *rgn)

  Set video cropped region.

  Note

  This API have no special timing request, user can call it freely

  Parameters:

    - handle – [in] Video video crop handle

    - rgn – [in] Region to be kept in original video frame

  Returns:

    - ESP_GMF_ERR_OK Successful

    - ESP_GMF_ERR_INVALID_ARG Invalid arguments

### Macros

DEFAULT_ESP_GMF_CROP_CONFIG()

### Header File

- elements/gmf_video/include/esp_gmf_video_scale.h

### Functions

esp_gmf_err_t esp_gmf_video_scale_init(esp_imgfx_scale_cfg_t *config, esp_gmf_element_handle_t *handle)

  Initialize the video scale element with the specified configuration.

  Parameters:

    - config – [in] Pointer to a `esp_imgfx_scale_cfg_t` structure that contains the configuration settings for the video scale

    - handle – [out] Pointer to a `esp_gmf_element_handle_t` where the initialized video scale handle will be stored

  Returns:

    - ESP_GMF_ERR_OK Initialization successful

    - ESP_GMF_ERR_INVALID_ARG Invalid arguments

    - ESP_GMF_ERR_MEMORY_LACK No memory for video scale element

esp_gmf_err_t esp_gmf_video_scale_dst_resolution(esp_gmf_element_handle_t handle, esp_gmf_video_resolution_t *res)

  Set video scale destination resolution.

  Note

  This API have no special timing request, user can call it freely

  Parameters:

    - handle – [in] Video video scale handle

    - res – [in] Video video scale output resolution

  Returns:

    - ESP_GMF_ERR_OK Successful

    - ESP_GMF_ERR_INVALID_ARG Invalid arguments

### Macros

DEFAULT_ESP_GMF_SCALE_CONFIG()

### Header File

- elements/gmf_video/include/esp_gmf_video_rotate.h

### Functions

esp_gmf_err_t esp_gmf_video_rotate_init(esp_imgfx_rotate_cfg_t *config, esp_gmf_element_handle_t *handle)

  Initialize the video rotate element with the specified configuration.

  Parameters:

    - config – [in] Pointer to a `esp_imgfx_rotate_cfg_t` structure that contains the configuration settings for the video rotate

    - handle – [out] Pointer to a `esp_gmf_element_handle_t` where the initialized video rotate handle will be stored

  Returns:

    - ESP_GMF_ERR_OK Initialization successful

    - ESP_GMF_ERR_INVALID_ARG Invalid arguments

    - ESP_GMF_ERR_MEMORY_LACK No memory for video rotate element

esp_gmf_err_t esp_gmf_video_rotate_set_rotation(esp_gmf_element_handle_t handle, uint16_t degree)

  Set video rotate rotation degree.

  Note

  This API have no special timing request, user can call it freely

  Parameters:

    - handle – [in] Video video rotate handle

    - degree – [in] Any angle, unit(one degree)

  Returns:

    - ESP_GMF_ERR_OK Successful

    - ESP_GMF_ERR_INVALID_ARG Invalid arguments

### Macros

DEFAULT_ESP_GMF_ROTATE_CONFIG()

### Header File

- elements/gmf_video/include/esp_gmf_video_types.h

### Structures

struct esp_gmf_video_resolution_t

  Video resolution definition.

  Public Members

  uint16_t width

    Width of the video in pixels

  uint16_t height

    Height of the video in pixels

struct esp_gmf_video_rgn_t

  Video region definition Represents a rectangular region of the video frame.

  Public Members

  uint16_t x

    X-coordinate of the top-left corner

  uint16_t y

    Y-coordinate of the top-left corner

  uint16_t width

    Width of the region in pixels

  uint16_t height

    Height of the region in pixels

struct esp_gmf_overlay_rgn_info_t

  Specifies the format and destination region for a video overlay.

  Note

  This structure defines the parameters needed to position and format an overlay on a video frame

  Public Members

  uint32_t format_id

    FourCC of overlay region frame

  esp_gmf_video_rgn_t dst_rgn

    Region position to put the overlay

  uint8_t rgn_index

    Region index start from 0 (first region)

  bool has_trans_color

    Whether region has transparent color (if near it will become transparent)

  uint8_t trans_color[3]

    Transparent color RGB order

### Header File

- elements/gmf_video/include/esp_gmf_video_param.h

### Functions

esp_gmf_err_t esp_gmf_video_param_set_dst_format(esp_gmf_element_handle_t handle, uint32_t dst_fmt)

  This file provides general parameter setting functions for common video processing using GMF method.

          These functions simplify setting parameters for video processing elements,
          handling the underlying method calls. They do not enforce a specific order
          or argument count, but apply the basic settings where supported. Refer to
          the individual element APIs for details on supported settings and timing.

  Set destination format for video element (e.g., color converter, video decoder)

  Parameters:

    - handle – [in] Video element handle

    - dst_fmt – [in] Destination format

  Returns:

    - ESP_GMF_ERR_OK On success

    - ESP_GMF_ERR_NOT_FOUND Not found the method

    - ESP_GMF_ERR_MEMORY_LACK Failed to allocate memory

    - Others Failed to apply method

esp_gmf_err_t esp_gmf_video_param_set_fps(esp_gmf_element_handle_t handle, uint16_t fps)

  Set frame rate for video element (mainly for FPS converter)

  Parameters:

    - handle – [in] Video element handle

    - fps – [in] Destination frame rate

  Returns:

    - ESP_GMF_ERR_OK On success

    - ESP_GMF_ERR_NOT_FOUND Not found the method

    - ESP_GMF_ERR_MEMORY_LACK Failed to allocate memory

    - Others Failed to apply method

esp_gmf_err_t esp_gmf_video_param_set_dst_resolution(esp_gmf_element_handle_t handle, esp_gmf_video_resolution_t *res)

  Set destination resolution for video element (mainly for video scaler)

  Parameters:

    - handle – [in] Video element handle

    - res – [in] Destination resolution

  Returns:

    - ESP_GMF_ERR_OK On success

    - ESP_GMF_ERR_NOT_FOUND Not found the method

    - ESP_GMF_ERR_MEMORY_LACK Failed to allocate memory

    - Others Failed to apply method

esp_gmf_err_t esp_gmf_video_param_set_dst_codec(esp_gmf_element_handle_t handle, uint32_t dst_codec)

  Set destination codec for video element (e.g., video encoder)

  Parameters:

    - handle – [in] Video element handle

    - dst_codec – [in] Destination codec

  Returns:

    - ESP_GMF_ERR_OK On success

    - ESP_GMF_ERR_NOT_FOUND Not found the method

    - ESP_GMF_ERR_MEMORY_LACK Failed to allocate memory

    - Others Failed to apply method

esp_gmf_err_t esp_gmf_video_param_venc_preset(esp_gmf_element_handle_t handle, esp_gmf_info_video_t *vid_info, uint32_t dst_codec)

  Preset source video information and destination codec for video encoder.

  Note

  In special cases call this API then query some information from video encoder before element running

  Parameters:

    - handle – [in] Video element handle

    - vid_info – [in] Source video information

    - dst_codec – [in] Destination codec

  Returns:

    - ESP_GMF_ERR_OK On success

    - ESP_GMF_ERR_NOT_FOUND Not found the method

    - ESP_GMF_ERR_MEMORY_LACK Failed to allocate memory

    - Others Failed to apply method

esp_gmf_err_t esp_gmf_video_param_get_src_fmts_by_codec(esp_gmf_element_handle_t handle, uint32_t dst_codec, const uint32_t **src_fmts, uint8_t *src_fmts_num)

  Get supported source formats by codec for video element (mainly for video encoder)

  Parameters:

    - handle – [in] Video element handle

    - dst_codec – [in] Destination codec

    - src_fmts – [in] Pointer to store source formats array

    - src_fmts_num – [in] Pointer to store source formats number

  Returns:

    - ESP_GMF_ERR_OK On success

    - ESP_GMF_ERR_NOT_FOUND Not found the method

    - ESP_GMF_ERR_MEMORY_LACK Failed to allocate memory

    - Others Failed to apply method

esp_gmf_err_t esp_gmf_video_param_set_src_codec(esp_gmf_element_handle_t handle, uint32_t src_codec)

  Set source codec for video element (e.g., video decoder)

  Parameters:

    - handle – [in] Video element handle

    - src_codec – [in] Source codec

  Returns:

    - ESP_GMF_ERR_OK On success

    - ESP_GMF_ERR_NOT_FOUND Not found the method

    - ESP_GMF_ERR_MEMORY_LACK Failed to allocate memory

    - Others Failed to apply method

esp_gmf_err_t esp_gmf_video_param_set_rotate_angle(esp_gmf_element_handle_t handle, uint16_t degree)

  Set the rotation angle for the video element (mainly for video rotator)

  Parameters:

    - handle – [in] Video element handle

    - degree – [in] Rotation angle in degrees (e.g., 0, 90, 180, 270)

  Returns:

    - ESP_GMF_ERR_OK On success

    - ESP_GMF_ERR_NOT_FOUND Not found the method

    - ESP_GMF_ERR_MEMORY_LACK Failed to allocate memory

    - Others Failed to apply method

esp_gmf_err_t esp_gmf_video_param_set_cropped_region(esp_gmf_element_handle_t handle, esp_gmf_video_rgn_t *rgn)

  Set the cropped region for video element (mainly for video cropper)

  Parameters:

    - handle – [in] Video element handle

    - rgn – [in] Kept region after crop

  Returns:

    - ESP_GMF_ERR_OK On success

    - ESP_GMF_ERR_NOT_FOUND Not found the method

    - ESP_GMF_ERR_MEMORY_LACK Failed to allocate memory

    - Others Failed to apply method

esp_gmf_err_t esp_gmf_video_param_overlay_enable(esp_gmf_element_handle_t self, bool enable)

  Enable overlay for video element (mainly for video overlay)

  Parameters:

    - handle – [in] Video element handle

    - enable – [in] Whether enable video overlay or not

  Returns:

    - ESP_GMF_ERR_OK On success

    - ESP_GMF_ERR_NOT_FOUND Not found the method

    - ESP_GMF_ERR_MEMORY_LACK Failed to allocate memory

    - Others Failed to apply method

esp_gmf_err_t esp_gmf_video_param_set_overlay_port(esp_gmf_element_handle_t self, void *port)

  Set overlay port for video element (mainly for video overlay)

  Parameters:

    - handle – [in] Video element handle

    - port – [in] Handle to the input GMF port providing overlay data

  Returns:

    - ESP_GMF_ERR_OK On success

    - ESP_GMF_ERR_NOT_FOUND Not found the method

    - ESP_GMF_ERR_MEMORY_LACK Failed to allocate memory

    - Others Failed to apply method

esp_gmf_err_t esp_gmf_video_param_set_overlay_rgn(esp_gmf_element_handle_t self, esp_gmf_overlay_rgn_info_t *rgn)

  Set overlay region information for video element (mainly for video overlay)

  Parameters:

    - handle – [in] Video element handle

    - rgn – [in] Overlay region information

  Returns:

    - ESP_GMF_ERR_OK On success

    - ESP_GMF_ERR_NOT_FOUND Not found the method

    - ESP_GMF_ERR_MEMORY_LACK Failed to allocate memory

    - Others Failed to apply method

esp_gmf_err_t esp_gmf_video_param_get_dst_fmts_by_codec(esp_gmf_element_handle_t handle, uint32_t src_codec, const uint32_t **dst_fmts, uint8_t *dst_fmts_num)

  Get supported destination formats by codec for video element (mainly for video decoder)

  Parameters:

    - handle – [in] Video element handle

    - src_codec – [in] Source codec

    - src_fmts – [in] Pointer to store source formats array

    - src_fmts_num – [in] Pointer to store source formats number

  Returns:

    - ESP_GMF_ERR_OK On success

    - ESP_GMF_ERR_NOT_FOUND Not found the method

    - ESP_GMF_ERR_MEMORY_LACK Failed to allocate memory

    - Others Failed to apply method

Was this page helpful?

- Thank you! We received your feedback.
- If you have any comments, fill in Espressif Documentation Feedback Form.

- We value your feedback.
- Let us know how we can improve this page by filling in Espressif Documentation Feedback Form.