Image Signal Processor

Introduction

ESP32-P4 includes an image signal processor (ISP), which is a feature pipeline that consists of many image processing algorithms. ISP receives image data from the DVP camera or MIPI-CSI camera, or system memory, and writes the processed image data to the system memory through DMA. ISP shall work with other modules to read and write data, it can not work alone.

Terminology

MIPI-CSI: Camera serial interface, a high-speed serial interface for cameras compliant with MIPI specifications
DVP: Digital video parallel interface, generally composed of vsync, hsync, de, and data signals
RAW: Unprocessed data directly output from an image sensor, typically divided into R, Gr, Gb, and B four channels classified into RAW8, RAW10, RAW12, etc., based on bit width
RGB: Colored image format composed of red, green, and blue colors classified into RGB888, RGB565, etc., based on the bit width of each color
YUV: Colored image format composed of luminance and chrominance classified into YUV444, YUV422, YUV420, etc., based on the data arrangement
AF: Auto-focus

Functional Overview

The ISP driver offers following services:

Resource Allocation - covers how to allocate ISP resources with properly set of configurations. It also covers how to recycle the resources when they finished working.
Enable and disable ISP processor - covers how to enable and disable an ISP processor.
Get AF oneshot result - covers how to get AF oneshot result.
Start AF statistics continuously - covers how to start the continuous AF statistics.
Register callback - covers how to hook user specific code to ISP driver event callback function.
Thread Safety - lists which APIs are guaranteed to be thread safe by the driver.
Kconfig Options - lists the supported Kconfig options that can bring different effects to the driver.
IRAM SAFE - describes tips on how to make the ISP interrupt and control functions work better along with a disabled cache.

Resource Allocation

Install Image Signal Processor (ISP) Driver

ISP driver requires the configuration that specified by esp_isp_processor_cfg_t.

If the configurations in esp_isp_processor_cfg_t is specified, users can call esp_isp_new_processor() to allocate and initialize an ISP processor. This function will return an ISP processor handle if it runs correctly. You can take following code as reference.

esp_isp_processor_cfg_t isp_config = {
    .clk_src = ISP_CLK_SRC_DEFAULT,
    .clk_hz = 80 * 1000 * 1000,
    .input_data_source = ISP_INPUT_DATA_SOURCE_CSI,
    .input_data_color_type = ISP_COLOR_RAW8,
    .output_data_color_type = ISP_COLOR_RGB565,
};

isp_proc_handle_t isp_proc = NULL;
ESP_ERROR_CHECK(esp_isp_new_processor(&isp_config, &isp_proc));

You can use the created handle to do driver enable / disable the ISP driver and do other ISP module installation.

Install Image Signal Processor (ISP) Auto-Focus (AF) Driver

ISP auto-focus (AF) driver requires the configuration that specified by esp_isp_af_config_t.

If the configurations in esp_isp_af_config_t is specified, users can call esp_isp_new_af_controller() to allocate and initialize an ISP AF processor. This function will return an ISP AF processor handle if it runs correctly. You can take following code as reference.

esp_isp_af_config_t af_config = {
    .edge_thresh = 128,
};
isp_af_ctrlr_t af_ctrlr = NULL;
ESP_ERROR_CHECK(esp_isp_new_af_controller(isp_proc, &af_config, &af_ctrlr));

You can use the created handle to do driver enable / disable the ISP AF driver and ISP AF Env module installation.

Uninstall Image Signal Processor (ISP) Driver

If a previously installed ISP processor is no longer needed, it's recommended to recycle the resource by calling esp_isp_del_processor(), so that to release the underlying hardware.

UnInstall Image Signal Processor (ISP) Auto-Focus (AF) Driver

If a previously installed ISP AF processor is no longer needed, it's recommended to recycle the resource by calling esp_isp_del_af_controller(), so that to release the underlying hardware.

Enable and Disable Image Signal Processor (ISP)

Image Signal Processor (ISP)

Before doing ISP pipeline, you need to enable the ISP processor first, by calling esp_isp_enable(). This function:

Switches the driver state from init to enable.

Calling esp_isp_disable() does the opposite, that is, put the driver back to the init state.

Image Signal Processor (ISP) Auto-Focus (AF) Processor

Before doing ISP AF, you need to enable the ISP AF processor first, by calling esp_isp_af_controller_enable(). This function:

Switches the driver state from init to enable.

Calling esp_isp_af_controller_disable() does the opposite, that is, put the driver back to the init state.

Get Auto-Focus (AF) Oneshot Result

Calling esp_isp_af_controller_get_oneshot_statistics() to get oneshot AF statistics result. You can take following code as reference.

esp_isp_af_config_t af_config = {
    .edge_thresh = 128,
};
isp_af_ctrlr_t af_ctrlr = NULL;
ESP_ERROR_CHECK(esp_isp_new_af_controller(isp_proc, &af_config, &af_ctrlr));
ESP_ERROR_CHECK(esp_isp_af_controller_enable(af_ctrlr));
isp_af_result_t result = {};
/* Trigger the AF statistics and get its result for one time with timeout value 2000ms. */
ESP_ERROR_CHECK(esp_isp_af_controller_get_oneshot_statistics(af_ctrlr, 2000, &result));

Start Auto-Focus (AF) Statistics Continuously

Aside from the above oneshot API, the ISP AF driver also provides a way to start AF statistics continuously. Calling esp_isp_af_controller_start_continuous_statistics() to start the continuous statistics and esp_isp_af_controller_stop_continuous_statistics() to stop it.

Note that if you want to use the continuous statistics, you need to register the esp_isp_af_env_detector_evt_cbs_t::on_env_statistics_done or esp_isp_af_env_detector_evt_cbs_t::on_env_change callback to get the statistics result. See how to register in Register Event Callbacks

isp_af_ctrlr_t af_ctrlr = NULL;
esp_isp_af_config_t af_config = {
    .edge_thresh = 128,
};
isp_af_result_t stat_res = {};
/* Create the af controller */
ESP_ERROR_CHECK(esp_isp_new_af_controller(isp_proc, &af_config, &af_ctrlr));
/* Enabled the af controller */
ESP_ERROR_CHECK(esp_isp_af_controller_enable(af_ctrlr));
/* Start continuous AF statistics */
ESP_ERROR_CHECK(esp_isp_af_controller_start_continuous_statistics(af_ctrlr));
// You can do other stuffs here, the statistics result can be obtained in the callback
// ......
// vTaskDelay(pdMS_TO_TICKS(1000));
/* Stop continuous AF statistics */
ESP_ERROR_CHECK(esp_isp_af_controller_stop_continuous_statistics(af_ctrlr));
/* Disable the af controller */
ESP_ERROR_CHECK(esp_isp_af_controller_disable(af_ctrlr));
/* Delete the af controller and free the resources */
ESP_ERROR_CHECK(esp_isp_del_af_controller(af_ctrlr));

Set Auto-Focus (AF) Environment Detector

Calling esp_isp_af_controller_set_env_detector() to set an ISP AF environment detector. You can take following code as reference.

esp_isp_af_env_config_t env_config = {
    .interval = 10,
};
isp_af_ctrlr_t af_ctrlr = NULL;
ESP_ERROR_CHECK(esp_isp_new_af_controller(isp_proc, &af_config, &af_ctrlr));
ESP_ERROR_CHECK(esp_isp_af_controller_set_env_detector(af_ctrlr, &env_config));

Set Auto-Focus (AF) Environment Detector Threshold

Calling esp_isp_af_env_detector_set_threshold() to set the threshold of an ISP AF environment detector.

int definition_thresh = 0;
int luminance_thresh = 0;
ESP_ERROR_CHECK(esp_isp_af_env_detector_set_threshold(env_detector, definition_thresh, luminance_thresh));

Register Event Callbacks

Register Image Signal Processor (ISP) Auto-Focus (AF) Environment Detector Event Callbacks

After the ISP AF environment detector starts up, it can generate a specific event dynamically. If you have some functions that should be called when the event happens, please hook your function to the interrupt service routine by calling esp_isp_af_env_detector_register_event_callbacks(). All supported event callbacks are listed in esp_isp_af_env_detector_evt_cbs_t:

esp_isp_af_env_detector_evt_cbs_t::on_env_statistics_done sets a callback function for environment statistics done. As this function is called within the ISR context, you must ensure that the function does not attempt to block (e.g., by making sure that only FreeRTOS APIs with ISR suffix are called from within the function). The function prototype is declared in esp_isp_af_env_detector_callback_t.
esp_isp_af_env_detector_evt_cbs_t::on_env_change sets a callback function for environment change. As this function is called within the ISR context, you must ensure that the function does not attempt to block (e.g., by making sure that only FreeRTOS APIs with ISR suffix are called from within the function). The function prototype is declared in esp_isp_af_env_detector_callback_t.

You can save your own context to esp_isp_af_env_detector_register_event_callbacks() as well, via the parameter user_data. The user data will be directly passed to the callback function.

Thread Safety

The factory function esp_isp_new_processor(), esp_isp_del_processor(), esp_isp_new_af_controller(), esp_isp_del_af_controller(), esp_isp_new_af_env_detector(), and esp_isp_del_af_env_detector() are guaranteed to be thread safe by the driver, which means, user can call them from different RTOS tasks without protection by extra locks.

Kconfig Options

CONFIG_ISP_ISR_IRAM_SAFE controls whether the default ISR handler should be masked when the cache is disabled

IRAM Safe

By default, the ISP interrupt will be deferred when the cache is disabled because of writing or erasing the flash.

There is a Kconfig option CONFIG_ISP_ISR_IRAM_SAFE that:

Enables the interrupt being serviced even when the cache is disabled
Places all functions that used by the ISR into IRAM
Places driver object into DRAM (in case it is mapped to PSRAM by accident)

This allows the interrupt to run while the cache is disabled, but comes at the cost of increased IRAM consumption.

API Reference

Header File

components/esp_driver_isp/include/driver/isp.h
This header file can be included with:
#include "driver/isp.h"
This header file is a part of the API provided by the esp_driver_isp component. To declare that your component depends on esp_driver_isp, add the following to your CMakeLists.txt:
REQUIRES esp_driver_isp
or
PRIV_REQUIRES esp_driver_isp

Header File

components/esp_driver_isp/include/driver/isp_types.h
This header file can be included with:
#include "driver/isp_types.h"
This header file is a part of the API provided by the esp_driver_isp component. To declare that your component depends on esp_driver_isp, add the following to your CMakeLists.txt:
REQUIRES esp_driver_isp
or
PRIV_REQUIRES esp_driver_isp

Type Definitions

typedef struct isp_processor_t *isp_proc_handle_t: Type of ISP processor handle.

typedef struct isp_af_controller_t *isp_af_ctrlr_t: Type of ISP AF controller handle.

Header File

components/esp_driver_isp/include/driver/isp_af.h
This header file can be included with:
#include "driver/isp_af.h"
This header file is a part of the API provided by the esp_driver_isp component. To declare that your component depends on esp_driver_isp, add the following to your CMakeLists.txt:
REQUIRES esp_driver_isp
or
PRIV_REQUIRES esp_driver_isp

Functions

esp_err_t esp_isp_new_af_controller(isp_proc_handle_t isp_proc, const esp_isp_af_config_t *af_config, isp_af_ctrlr_t *ret_hdl)

New an ISP AF controller.

Parameters

isp_proc -- [in] ISP Processor handle
af_config -- [in] Pointer to AF config. Refer to esp_isp_af_config_t.
ret_hdl -- [out] AF controller handle

Returns

ESP_OK On success
ESP_ERR_INVALID_ARG If the combination of arguments is invalid
ESP_ERR_INVALID_STATE Invalid state
ESP_ERR_NOT_FOUND No free interrupt found with the specified flags
ESP_ERR_NO_MEM If out of memory

esp_err_t esp_isp_del_af_controller(isp_af_ctrlr_t af_ctrlr)

Delete an ISP AF controller.

Parameters

af_ctrlr -- [in] AF controller handle

Returns

ESP_OK On success
ESP_ERR_INVALID_ARG If the combination of arguments is invalid.
ESP_ERR_INVALID_STATE Driver state is invalid.

esp_err_t esp_isp_af_controller_enable(isp_af_ctrlr_t af_ctrlr)

Enable an ISP AF controller.

Parameters

af_ctrlr -- [in] AF controller handle

Returns

ESP_OK On success
ESP_ERR_INVALID_ARG If the combination of arguments is invalid.
ESP_ERR_INVALID_STATE Driver state is invalid.

esp_err_t esp_isp_af_controller_disable(isp_af_ctrlr_t af_ctrlr)

Disable an ISP AF controller.

Parameters

af_ctrlr -- [in] AF controller handle

Returns

ESP_OK On success
ESP_ERR_INVALID_ARG If the combination of arguments is invalid.
ESP_ERR_INVALID_STATE Driver state is invalid.

esp_err_t esp_isp_af_controller_get_oneshot_statistics(isp_af_ctrlr_t af_ctrlr, int timeout_ms, isp_af_result_t *out_res)

Trigger AF luminance and definition statistics for one time and get the result.

Note

This function is a synchronous and block function, it only returns when AF luminance and definition statistics is done or timeout. It's a simple method to get the result directly for one time.

Parameters

af_ctrlr -- [in] AF controller handle
timeout_ms -- [in] Timeout in millisecond
- timeout_ms < 0: Won't return until finished
- timeout_ms = 0: No timeout, trigger one time statistics and return immediately, in this case, the result won't be assigned in this function, but you can get the result in the callback esp_isp_af_env_detector_evt_cbs_t::on_env_statistics_done
- timeout_ms > 0: Wait for specified milliseconds, if not finished, then return timeout error
out_res -- [out] AF luminance and definition statistics result, can be NULL if timeout_ms = 0

Returns

ESP_OK On success
ESP_ERR_TIMEOUT If the waiting time exceeds the specified timeout.
ESP_ERR_INVALID_ARG If the combination of arguments is invalid.
ESP_ERR_INVALID_STATE Driver state is invalid.

esp_err_t esp_isp_af_controller_start_continuous_statistics(isp_af_ctrlr_t af_ctrlr)

Start AF continuous statistics of the luminance and definition in the windows.

Note

This function is an asynchronous and non-block function, it will start the continuous statistics and return immediately. You have to register the AF callback and get the result from the callback event data.

Parameters

af_ctrlr -- [in] AF controller handle

Returns

ESP_OK On success
ESP_ERR_INVALID_ARG Null pointer
ESP_ERR_INVALID_STATE Driver state is invalid.

esp_err_t esp_isp_af_controller_stop_continuous_statistics(isp_af_ctrlr_t af_ctrlr)

Stop AF continuous statistics of the luminance and definition in the windows.

Parameters

af_ctrlr -- [in] AF controller handle

Returns

ESP_OK On success
ESP_ERR_INVALID_ARG Null pointer
ESP_ERR_INVALID_STATE Driver state is invalid.

esp_err_t esp_isp_af_controller_set_env_detector(isp_af_ctrlr_t af_ctrlr, const esp_isp_af_env_config_t *env_config)

Set ISP AF environment detector.

Parameters

af_ctrlr -- [in] AF controller handle
env_config -- [in] AF Env detector configuration

Returns

ESP_OK On success
ESP_ERR_INVALID_ARG If the combination of arguments is invalid.
ESP_ERR_INVALID_STATE Driver state is invalid.

esp_err_t esp_isp_af_controller_set_env_detector_threshold(isp_af_ctrlr_t af_ctrlr, int definition_thresh, int luminance_thresh)

Set ISP AF environment detector detecting threshold.

Parameters

af_ctrlr -- [in] AF controller handle
definition_thresh -- [in] Threshold for definition
luminance_thresh -- [in] Threshold for luminance

Returns

ESP_OK On success
ESP_ERR_INVALID_ARG If the combination of arguments is invalid.
ESP_ERR_INVALID_STATE Driver state is invalid.

esp_err_t esp_isp_af_env_detector_register_event_callbacks(isp_af_ctrlr_t af_ctrlr, const esp_isp_af_env_detector_evt_cbs_t *cbs, void *user_data)

Register AF environment detector event callbacks.

Note

User can deregister a previously registered callback by calling this function and setting the to-be-deregistered callback member in the cbs structure to NULL.

Note

When CONFIG_ISP_ISR_IRAM_SAFE is enabled, the callback itself and functions called by it should be placed in IRAM. Involved variables (including user_data) should be in internal RAM as well.

Parameters

af_ctrlr -- [in] AF controller handle
cbs -- [in] Group of callback functions
user_data -- [in] User data, which will be delivered to the callback functions directly

Returns

ESP_OK: On success
ESP_ERR_INVALID_ARG: Invalid arguments
ESP_ERR_INVALID_STATE: Driver state is invalid, you shouldn't call this API at this moment

Structures

struct esp_isp_af_config_t

AF controller config.

Public Members

isp_af_window_t window[ISP_AF_WINDOW_NUM]: The sampling windows of AF.

int edge_thresh: Edge threshold, definition higher than this value will be counted as a valid pixel for calculating AF result.

int intr_priority: The interrupt priority, range 0~7, if set to 0, the driver will try to allocate an interrupt with a relative low priority (1,2,3) otherwise the larger the higher, 7 is NMI.

struct esp_isp_af_env_config_t

AF environment detector config.

Public Members

int interval: Interval between environment detection, in frames. i.e., AF controller will trigger the statistic periodically to detect the environment change.

struct esp_isp_af_env_detector_evt_data_t

Event data structure.

Public Members

isp_af_result_t af_result: The AF statistics result

struct esp_isp_af_env_detector_evt_cbs_t

Group of ISP AF Env detector callbacks.

Note

These callbacks are all running in an ISR environment.

Note

When CONFIG_ISP_ISR_IRAM_SAFE is enabled, the callback itself and functions called by it should be placed in IRAM. Involved variables should be in internal RAM as well.

Public Members

esp_isp_af_env_detector_callback_t on_env_statistics_done: Event callback, invoked when environment sample done.

esp_isp_af_env_detector_callback_t on_env_change: Event callback, invoked when environment change happens.

Type Definitions

typedef bool (*esp_isp_af_env_detector_callback_t)(isp_af_ctrlr_t af_ctrlr, const esp_isp_af_env_detector_evt_data_t *edata, void *user_data)

Prototype of ISP AF Env detector event callback.

Param handle: [in] ISP AF controller handle
Param edata: [in] ISP AF Env detector event data
Param user_data: [in] User registered context, registered when in esp_isp_af_env_detector_register_event_callbacks()
Return: Whether a high priority task is woken up by this function

Provide feedback about this document