Image Signal Processor

Introduction

ESP32-P4 includes an Image Signal Processor (ISP), which is a feature pipeline that consists of many image processing algorithms. ISP receives image data from the DVP camera or MIPI-CSI camera, or system memory, and writes the processed image data to the system memory through DMA. ISP shall work with other modules to read and write data, it can not work alone.

Terminology

  • MIPI-CSI: Camera serial interface, a high-speed serial interface for cameras compliant with MIPI specifications

  • DVP: Digital video parallel interface, generally composed of vsync, hsync, de, and data signals

  • RAW: Unprocessed data directly output from an image sensor, typically divided into R, Gr, Gb, and B four channels classified into RAW8, RAW10, RAW12, etc., based on bit width

  • RGB: Colored image format composed of red, green, and blue colors classified into RGB888, RGB565, etc., based on the bit width of each color

  • YUV: Colored image format composed of luminance and chrominance classified into YUV444, YUV422, YUV420, etc., based on the data arrangement

  • AF: Auto-focus

  • AWB: Auto-white balance

  • BF: Bayer noise filter

  • CCM: Color correction matrix

ISP Pipeline

ISP Pipeline

Functional Overview

The ISP driver offers following services:

Resource Allocation

Install ISP Driver

ISP driver requires the configuration that specified by esp_isp_processor_cfg_t.

If the configurations in esp_isp_processor_cfg_t is specified, users can call esp_isp_new_processor() to allocate and initialize an ISP processor. This function will return an ISP processor handle if it runs correctly. You can take following code as reference.

esp_isp_processor_cfg_t isp_config = {
    .clk_src = ISP_CLK_SRC_DEFAULT,
    ...
};

isp_proc_handle_t isp_proc = NULL;
ESP_ERROR_CHECK(esp_isp_new_processor(&isp_config, &isp_proc));

You can use the created handle to do driver enable / disable the ISP driver and do other ISP module installation.

Install ISP Auto-Focus (AF) Driver

ISP auto-focus (AF) driver requires the configuration that specified by esp_isp_af_config_t.

If the configurations in esp_isp_af_config_t is specified, users can call esp_isp_new_af_controller() to allocate and initialize an ISP AF processor. This function will return an ISP AF processor handle if it runs correctly. You can take following code as reference.

esp_isp_af_config_t af_config = {
    .edge_thresh = 128,
};
isp_af_ctlr_t af_ctrlr = NULL;
ESP_ERROR_CHECK(esp_isp_new_af_controller(isp_proc, &af_config, &af_ctrlr));

You can use the created handle to do driver enable / disable the ISP AF driver and ISP AF Env module installation.

Install ISP Auto-White-Balance (AWB) Driver

ISP auto-white-balance (AWB) driver requires the configuration specified by esp_isp_awb_config_t.

If an esp_isp_awb_config_t configuration is specified, you can call esp_isp_new_awb_controller() to allocate and initialize an ISP AWB processor. This function will return an ISP AWB processor handle on success. You can take following code as reference.

isp_awb_ctlr_t awb_ctlr = NULL;
uint32_t image_width = 800;
uint32_t image_height = 600;
/* The AWB configuration, please refer to the API comment for how to tune these parameters */
esp_isp_awb_config_t awb_config = {
    .sample_point = ISP_AWB_SAMPLE_POINT_AFTER_CCM,
    ...
};
ESP_ERROR_CHECK(esp_isp_new_awb_controller(isp_proc, &awb_config, &awb_ctlr));

The AWB handle created in this step is required by other AWB APIs and AWB scheme.

Uninstall ISP Driver

If a previously installed ISP processor is no longer needed, it's recommended to recycle the resource by calling esp_isp_del_processor(), so that to release the underlying hardware.

UnInstall ISP AF Driver

If a previously installed ISP AF processor is no longer needed, it's recommended to recycle the resource by calling esp_isp_del_af_controller(), so that to release the underlying hardware.

UnInstall ISP AWB Driver

If a previously installed ISP AWB processor is no longer needed, it's recommended to free the resource by calling esp_isp_del_awb_controller(), it will also release the underlying hardware.

Enable and Disable ISP

ISP

Before doing ISP pipeline, you need to enable the ISP processor first, by calling esp_isp_enable(). This function:

  • Switches the driver state from init to enable.

Calling esp_isp_disable() does the opposite, that is, put the driver back to the init state.

ISP AF Processor

Before doing ISP AF, you need to enable the ISP AF processor first, by calling esp_isp_af_controller_enable(). This function:

  • Switches the driver state from init to enable.

Calling esp_isp_af_controller_disable() does the opposite, that is, put the driver back to the init state.

AF One-shot and Continuous Statistics

Calling esp_isp_af_controller_get_oneshot_statistics() to get oneshot AF statistics result. You can take following code as reference.

Aside from the above oneshot API, the ISP AF driver also provides a way to start AF statistics continuously. Calling esp_isp_af_controller_start_continuous_statistics() to start the continuous statistics and esp_isp_af_controller_stop_continuous_statistics() to stop it.

Note that if you want to use the continuous statistics, you need to register the esp_isp_af_env_detector_evt_cbs_t::on_env_statistics_done or esp_isp_af_env_detector_evt_cbs_t::on_env_change callback to get the statistics result. See how to register in Register Event Callbacks

esp_isp_af_config_t af_config = {
    .edge_thresh = 128,
};
isp_af_ctlr_t af_ctrlr = NULL;
ESP_ERROR_CHECK(esp_isp_new_af_controller(isp_proc, &af_config, &af_ctrlr));
ESP_ERROR_CHECK(esp_isp_af_controller_enable(af_ctrlr));
isp_af_result_t result = {};
/* Trigger the AF statistics and get its result for one time with timeout value 2000ms. */
ESP_ERROR_CHECK(esp_isp_af_controller_get_oneshot_statistics(af_ctrlr, 2000, &result));

/* Start continuous AF statistics */
ESP_ERROR_CHECK(esp_isp_af_controller_start_continuous_statistics(af_ctrlr));
// You can do other stuffs here, the statistics result can be obtained in the callback
// ......
// vTaskDelay(pdMS_TO_TICKS(1000));
/* Stop continuous AF statistics */
ESP_ERROR_CHECK(esp_isp_af_controller_stop_continuous_statistics(af_ctrlr));

/* Disable the af controller */
ESP_ERROR_CHECK(esp_isp_af_controller_disable(af_ctrlr));
/* Delete the af controller and free the resources */
ESP_ERROR_CHECK(esp_isp_del_af_controller(af_ctrlr));

Set AF Environment Detector

Calling esp_isp_af_controller_set_env_detector() to set an ISP AF environment detector. You can take following code as reference.

esp_isp_af_env_config_t env_config = {
    .interval = 10,
};
isp_af_ctlr_t af_ctrlr = NULL;
ESP_ERROR_CHECK(esp_isp_new_af_controller(isp_proc, &af_config, &af_ctrlr));
ESP_ERROR_CHECK(esp_isp_af_controller_set_env_detector(af_ctrlr, &env_config));

Set AF Environment Detector Threshold

Calling esp_isp_af_env_detector_set_threshold() to set the threshold of an ISP AF environment detector.

int definition_thresh = 0;
int luminance_thresh = 0;
ESP_ERROR_CHECK(esp_isp_af_env_detector_set_threshold(env_detector, definition_thresh, luminance_thresh));

ISP AWB Processor

Before doing ISP AWB, you need to enable the ISP AWB processor first, by calling esp_isp_awb_controller_enable(). This function:

  • Switches the driver state from init to enable.

Calling esp_isp_awb_controller_disable() does the opposite, that is, put the driver back to the init state.

AWB One-shot and Continuous Statistics

Calling esp_isp_awb_controller_get_oneshot_result() to get oneshot AWB statistics result of white patches. You can take following code as reference.

Aside from the above oneshot API, the ISP AWB driver also provides a way to start AWB statistics continuously. Calling esp_isp_awb_controller_start_continuous_statistics() starts the continuous statistics and esp_isp_awb_controller_stop_continuous_statistics() stops it.

Note that if you want to use the continuous statistics, you need to register the esp_isp_awb_cbs_t::on_statistics_done callback to get the statistics result. See how to register it in Register Event Callbacks

bool example_isp_awb_on_statistics_done_cb(isp_awb_ctlr_t awb_ctlr, const esp_isp_awb_evt_data_t *edata, void *user_data);
// ...
isp_awb_ctlr_t awb_ctlr = NULL;
uint32_t image_width = 800;
uint32_t image_height = 600;
/* The AWB configuration, please refer to the API comment for how to tune these parameters */
esp_isp_awb_config_t awb_config = {
    .sample_point = ISP_AWB_SAMPLE_POINT_AFTER_CCM,
    ...
};
isp_awb_stat_result_t stat_res = {};
/* Create the awb controller */
ESP_ERROR_CHECK(esp_isp_new_awb_controller(isp_proc, &awb_config, &awb_ctlr));
/* Register AWB callback */
esp_isp_awb_cbs_t awb_cb = {
    .on_statistics_done = example_isp_awb_on_statistics_done_cb,
};
ESP_ERROR_CHECK(esp_isp_awb_register_event_callbacks(awb_ctlr, &awb_cb, NULL));
/* Enabled the awb controller */
ESP_ERROR_CHECK(esp_isp_awb_controller_enable(awb_ctlr));

/* Get oneshot AWB statistics result */
ESP_ERROR_CHECK(esp_isp_awb_controller_get_oneshot_statistics(awb_ctlr, -1, &stat_res));

/* Start continuous AWB statistics, note that continuous statistics requires `on_statistics_done` callback */
ESP_ERROR_CHECK(esp_isp_awb_controller_start_continuous_statistics(awb_ctlr));
// You can do other stuffs here, the statistics result can be obtained in the callback
// ......
// vTaskDelay(pdMS_TO_TICKS(1000));
/* Stop continuous AWB statistics */
ESP_ERROR_CHECK(esp_isp_awb_controller_stop_continuous_statistics(awb_ctlr));

/* Disable the awb controller */
ESP_ERROR_CHECK(esp_isp_awb_controller_disable(awb_ctlr));
/* Delete the awb controller and free the resources */
ESP_ERROR_CHECK(esp_isp_del_awb_controller(awb_ctlr));

ISP BF Processor

This pipeline is used for doing image input denoising under bayer mode.

Calling esp_isp_bf_configure() to configure BF function, you can take following code as reference.

esp_isp_bf_config_t bf_config = {
    .denoising_level = 5,
    ...
};
ESP_ERROR_CHECK(esp_isp_bf_configure(isp_proc, &bf_config));
ESP_ERROR_CHECK(esp_isp_bf_enable(isp_proc));

esp_isp_bf_config_t::bf_template is used for bayer denoise. You can set the esp_isp_bf_config_t::bf_template with a Gaussian filter template or an average filter template.

After calling esp_isp_bf_configure(), you need to enable the ISP BF processor, by calling esp_isp_bf_enable(). This function:

  • Switches the driver state from init to enable.

Calling esp_isp_bf_disable() does the opposite, that is, put the driver back to the init state. .. _isp-ccm-config:

Configure CCM

Color Correction Matrix can scale the color ratio of RGB888 pixels. It can be used for adjusting the image color via some algorithms, for example, used for white balance by inputting the AWB computed result, or used as a Filter with some filter algorithms.

To adjust the color correction matrix, here is the formula:

[ R' ]     [ RR  RG  RB  ]   [ R ]
[ G' ] =   [ GR  GG  GB  ] * [ G ]
[ B' ]     [ BR  BG  BB  ]   [ B ]

, and you can refer to the following code:

// ...
// Configure CCM
esp_isp_ccm_config_t ccm_cfg = {
    .matrix = {
        1.0, 0.0, 0.0,
        0.0, 1.0, 0.0,
        0.0, 0.0, 1.0
    },
    .saturation = false,
    ...
};
ESP_ERROR_CHECK(esp_isp_ccm_configure(isp_proc, &ccm_cfg));
// The configured CCM will be applied to the image once the CCM module is enabled
ESP_ERROR_CHECK(esp_isp_ccm_enable(isp_proc));
// CCM can also be configured after it is enabled
ccm_cfg.matrix[0][0] = 2.0;
ESP_ERROR_CHECK(esp_isp_ccm_configure(isp_proc, &ccm_cfg));
// Disable CCM if no longer needed
ESP_ERROR_CHECK(esp_isp_ccm_disable(isp_proc));

Register Event Callbacks

Register ISP AF Environment Detector Event Callbacks

After the ISP AF environment detector starts up, it can generate a specific event dynamically. If you have some functions that should be called when the event happens, please hook your function to the interrupt service routine by calling esp_isp_af_env_detector_register_event_callbacks(). All supported event callbacks are listed in esp_isp_af_env_detector_evt_cbs_t:

You can save your own context to esp_isp_af_env_detector_register_event_callbacks() as well, via the parameter user_data. The user data will be directly passed to the callback function.

Register ISP AWB Statistics Done Event Callbacks

After the ISP AWB controller finished statistics of white patches, it can generate a specific event dynamically. If you want to be informed when the statistics done event takes place, please hook your function to the interrupt service routine by calling esp_isp_awb_register_event_callbacks(). All supported event callbacks are listed in esp_isp_awb_cbs_t:

  • esp_isp_awb_cbs_t::on_statistics_done sets a callback function when finished statistics of the white patches. As this function is called within the ISR context, you must ensure that the function does not attempt to block (e.g., by making sure that only FreeRTOS APIs with ISR suffix are called from within the function). The function prototype is declared in esp_isp_awb_callback_t.

You can save your own context via the parameter user_data of esp_isp_awb_register_event_callbacks(). The user data will be directly passed to the callback function.

Thread Safety

The factory function esp_isp_new_processor(), esp_isp_del_processor(), esp_isp_new_af_controller(), esp_isp_del_af_controller(), esp_isp_new_af_env_detector(), and esp_isp_del_af_env_detector() are guaranteed to be thread safe by the driver, which means, user can call them from different RTOS tasks without protection by extra locks.

Kconfig Options

IRAM Safe

By default, the ISP interrupt will be deferred when the cache is disabled because of writing or erasing the flash.

There is a Kconfig option CONFIG_ISP_ISR_IRAM_SAFE that:

  • Enables the interrupt being serviced even when the cache is disabled

  • Places all functions that used by the ISR into IRAM

  • Places driver object into DRAM (in case it is mapped to PSRAM by accident)

This allows the interrupt to run while the cache is disabled, but comes at the cost of increased IRAM consumption.

API Reference

Header File

  • components/esp_driver_isp/include/driver/isp.h

  • This header file can be included with:

    #include "driver/isp.h"

  • This header file is a part of the API provided by the esp_driver_isp component. To declare that your component depends on esp_driver_isp, add the following to your CMakeLists.txt:

    REQUIRES esp_driver_isp

    or

    PRIV_REQUIRES esp_driver_isp

Header File

  • components/esp_driver_isp/include/driver/isp_types.h

  • This header file can be included with:

    #include "driver/isp_types.h"

  • This header file is a part of the API provided by the esp_driver_isp component. To declare that your component depends on esp_driver_isp, add the following to your CMakeLists.txt:

    REQUIRES esp_driver_isp

    or

    PRIV_REQUIRES esp_driver_isp

Structures

struct isp_u32_range_t

ISP unsigned integer range type.

备注

Whether the edge value are included depends on the variable itself

Public Members

uint32_t min

Minimum unsigned int value.

uint32_t max

Maximum unsigned int value.

struct isp_float_range_t

ISP float range type.

备注

Whether the edge value are included depends on the variable itself

Public Members

float min

Minimum float value.

float max

Maximum float value.

struct isp_af_result_t

ISP AF result.

Public Members

int definition[ISP_AF_WINDOW_NUM]

Definition, it refers how clear and sharp an image is.

int luminance[ISP_AF_WINDOW_NUM]

Luminance, it refers how luminant an image is.

struct isp_awb_stat_result_t

ISP AWB result.

Public Members

uint32_t white_patch_num

white patch number that counted by AWB in the window

uint32_t sum_r

The sum of R channel of these white patches.

uint32_t sum_g

The sum of G channel of these white patches.

uint32_t sum_b

The sum of B channel of these white patches.

Type Definitions

typedef struct isp_processor_t *isp_proc_handle_t

Type of ISP processor handle.

typedef struct isp_af_controller_t *isp_af_ctlr_t

Type of ISP AF controller handle.

typedef struct isp_awb_controller_t *isp_awb_ctlr_t

Type of ISP AWB controller handle.

Header File

  • components/esp_driver_isp/include/driver/isp_af.h

  • This header file can be included with:

    #include "driver/isp_af.h"

  • This header file is a part of the API provided by the esp_driver_isp component. To declare that your component depends on esp_driver_isp, add the following to your CMakeLists.txt:

    REQUIRES esp_driver_isp

    or

    PRIV_REQUIRES esp_driver_isp

Functions

esp_err_t esp_isp_new_af_controller(isp_proc_handle_t isp_proc, const esp_isp_af_config_t *af_config, isp_af_ctlr_t *ret_hdl)

New an ISP AF controller.

参数

  • isp_proc -- [in] ISP Processor handle

  • af_config -- [in] Pointer to AF config. Refer to esp_isp_af_config_t.

  • ret_hdl -- [out] AF controller handle

返回

  • ESP_OK On success

  • ESP_ERR_INVALID_ARG If the combination of arguments is invalid

  • ESP_ERR_INVALID_STATE Invalid state

  • ESP_ERR_NOT_FOUND No free interrupt found with the specified flags

  • ESP_ERR_NO_MEM If out of memory

esp_err_t esp_isp_del_af_controller(isp_af_ctlr_t af_ctrlr)

Delete an ISP AF controller.

参数

af_ctrlr -- [in] AF controller handle

返回

  • ESP_OK On success

  • ESP_ERR_INVALID_ARG If the combination of arguments is invalid.

  • ESP_ERR_INVALID_STATE Driver state is invalid.

esp_err_t esp_isp_af_controller_enable(isp_af_ctlr_t af_ctrlr)

Enable an ISP AF controller.

参数

af_ctrlr -- [in] AF controller handle

返回

  • ESP_OK On success

  • ESP_ERR_INVALID_ARG If the combination of arguments is invalid.

  • ESP_ERR_INVALID_STATE Driver state is invalid.

esp_err_t esp_isp_af_controller_disable(isp_af_ctlr_t af_ctrlr)

Disable an ISP AF controller.

参数

af_ctrlr -- [in] AF controller handle

返回

  • ESP_OK On success

  • ESP_ERR_INVALID_ARG If the combination of arguments is invalid.

  • ESP_ERR_INVALID_STATE Driver state is invalid.

esp_err_t esp_isp_af_controller_get_oneshot_statistics(isp_af_ctlr_t af_ctrlr, int timeout_ms, isp_af_result_t *out_res)

Trigger AF luminance and definition statistics for one time and get the result.

备注

This function is a synchronous and block function, it only returns when AF luminance and definition statistics is done or timeout. It's a simple method to get the result directly for one time.

参数

  • af_ctrlr -- [in] AF controller handle

  • timeout_ms -- [in] Timeout in millisecond

    • timeout_ms < 0: Won't return until finished

    • timeout_ms = 0: No timeout, trigger one time statistics and return immediately, in this case, the result won't be assigned in this function, but you can get the result in the callback esp_isp_af_env_detector_evt_cbs_t::on_env_statistics_done

    • timeout_ms > 0: Wait for specified milliseconds, if not finished, then return timeout error

  • out_res -- [out] AF luminance and definition statistics result, can be NULL if timeout_ms = 0

返回

  • ESP_OK On success

  • ESP_ERR_TIMEOUT If the waiting time exceeds the specified timeout.

  • ESP_ERR_INVALID_ARG If the combination of arguments is invalid.

  • ESP_ERR_INVALID_STATE Driver state is invalid.

esp_err_t esp_isp_af_controller_start_continuous_statistics(isp_af_ctlr_t af_ctrlr)

Start AF continuous statistics of the luminance and definition in the windows.

备注

This function is an asynchronous and non-block function, it will start the continuous statistics and return immediately. You have to register the AF callback and get the result from the callback event data.

参数

af_ctrlr -- [in] AF controller handle

返回

  • ESP_OK On success

  • ESP_ERR_INVALID_ARG Null pointer

  • ESP_ERR_INVALID_STATE Driver state is invalid.

esp_err_t esp_isp_af_controller_stop_continuous_statistics(isp_af_ctlr_t af_ctrlr)

Stop AF continuous statistics of the luminance and definition in the windows.

参数

af_ctrlr -- [in] AF controller handle

返回

  • ESP_OK On success

  • ESP_ERR_INVALID_ARG Null pointer

  • ESP_ERR_INVALID_STATE Driver state is invalid.

esp_err_t esp_isp_af_controller_set_env_detector(isp_af_ctlr_t af_ctrlr, const esp_isp_af_env_config_t *env_config)

Set ISP AF environment detector.

参数

  • af_ctrlr -- [in] AF controller handle

  • env_config -- [in] AF Env detector configuration

返回

  • ESP_OK On success

  • ESP_ERR_INVALID_ARG If the combination of arguments is invalid.

  • ESP_ERR_INVALID_STATE Driver state is invalid.

esp_err_t esp_isp_af_controller_set_env_detector_threshold(isp_af_ctlr_t af_ctrlr, int definition_thresh, int luminance_thresh)

Set ISP AF environment detector detecting threshold.

参数

  • af_ctrlr -- [in] AF controller handle

  • definition_thresh -- [in] Threshold for definition

  • luminance_thresh -- [in] Threshold for luminance

返回

  • ESP_OK On success

  • ESP_ERR_INVALID_ARG If the combination of arguments is invalid.

  • ESP_ERR_INVALID_STATE Driver state is invalid.

esp_err_t esp_isp_af_env_detector_register_event_callbacks(isp_af_ctlr_t af_ctrlr, const esp_isp_af_env_detector_evt_cbs_t *cbs, void *user_data)

Register AF environment detector event callbacks.

备注

User can deregister a previously registered callback by calling this function and setting the to-be-deregistered callback member in the cbs structure to NULL.

备注

When CONFIG_ISP_ISR_IRAM_SAFE is enabled, the callback itself and functions called by it should be placed in IRAM. Involved variables (including user_data) should be in internal RAM as well.

参数

  • af_ctrlr -- [in] AF controller handle

  • cbs -- [in] Group of callback functions

  • user_data -- [in] User data, which will be delivered to the callback functions directly

返回

  • ESP_OK: On success

  • ESP_ERR_INVALID_ARG: Invalid arguments

  • ESP_ERR_INVALID_STATE: Driver state is invalid, you shouldn't call this API at this moment

Structures

struct esp_isp_af_config_t

AF controller config.

Public Members

isp_window_t window[ISP_AF_WINDOW_NUM]

The sampling windows of AF.

int edge_thresh

Edge threshold, definition higher than this value will be counted as a valid pixel for calculating AF result.

int intr_priority

The interrupt priority, range 0~7, if set to 0, the driver will try to allocate an interrupt with a relative low priority (1,2,3) otherwise the larger the higher, 7 is NMI.

struct esp_isp_af_env_config_t

AF environment detector config.

Public Members

int interval

Interval between environment detection, in frames. i.e., AF controller will trigger the statistic periodically to detect the environment change.

struct esp_isp_af_env_detector_evt_data_t

Event data structure.

Public Members

isp_af_result_t af_result

The AF statistics result

struct esp_isp_af_env_detector_evt_cbs_t

Group of ISP AF Env detector callbacks.

备注

These callbacks are all running in an ISR environment.

备注

When CONFIG_ISP_ISR_IRAM_SAFE is enabled, the callback itself and functions called by it should be placed in IRAM. Involved variables should be in internal RAM as well.

Public Members

esp_isp_af_env_detector_callback_t on_env_statistics_done

Event callback, invoked when environment sample done.

esp_isp_af_env_detector_callback_t on_env_change

Event callback, invoked when environment change happens.

Type Definitions

typedef bool (*esp_isp_af_env_detector_callback_t)(isp_af_ctlr_t af_ctrlr, const esp_isp_af_env_detector_evt_data_t *edata, void *user_data)

Prototype of ISP AF Env detector event callback.

Param af_ctrlr

[in] ISP AF controller handle

Param edata

[in] ISP AF Env detector event data

Param user_data

[in] User registered context, registered when in esp_isp_af_env_detector_register_event_callbacks()

Return

Whether a high priority task is woken up by this function

提供有关此文档的反馈