# Camera Controller Driver

[中文]

## Introduction

ESP32-P4 has the following hardware that is intended for communication with external camera sensor:

- MIPI Camera Serial Interface (MIPI CSI)

- Digital Video Port through ISP module (ISP DVP)

- Digital Video Port through LCD_CAM module(LCD_CAM DVP)

The camera controller driver is designed for this hardware peripheral.

## Functional Overview

- Resource Allocation - covers how to allocate camera controller instances with properly set of configurations. It also covers how to recycle the resources when they are no longer needed.

- Enable and Disable Camera Controller Driver - covers how to enable and disable a camera controller.

- Start and Stop Camera Controller Driver - covers how to start and stop a camera controller.

- Receive from a Camera Sensor- covers how to receive camera signal from a sensor or something else.

- Register Event Callbacks- covers how to hook user specific code to camera controller driver event callback function.

- Thread Safety - lists which APIs are guaranteed to be thread safe by the driver.

- Kconfig Options - lists the supported Kconfig options that can bring different effects to the driver.

- IRAM Safe - describes tips on how to make the CSI interrupt and control functions work better along with a disabled cache.

### Resource Allocation

#### Install Camera Controller Driver

Camera controller driver can be implemented in one of following ways:

- `esp_cam_new_csi_ctlr()`

- `esp_cam_new_isp_dvp_ctlr()`

- `esp_cam_new_lcd_cam_ctlr()`

A camera controller driver can be implemented by the CSI peripheral, which requires the configuration that specified by `esp_cam_ctlr_csi_config_t`.

If the configurations in `esp_cam_ctlr_csi_config_t` is specified, users can call `esp_cam_new_csi_ctlr()` to allocate and initialize a CSI camera controller handle. This function will return an CSI camera controller handle if it runs correctly. You can take following code as reference.

    esp_cam_ctlr_csi_config_t csi_config = {
        .ctlr_id = 0,
        .h_res = MIPI_CSI_DISP_HSIZE,
        .v_res = MIPI_CSI_DISP_VSIZE_640P,
        .lane_bit_rate_mbps = MIPI_CSI_LANE_BITRATE_MBPS,
        .input_data_color_type = CAM_CTLR_COLOR_RAW8,
        .output_data_color_type = CAM_CTLR_COLOR_RGB565,
        .data_lane_num = 2,
        .byte_swap_en = false,
        .queue_items = 1,
    };
    esp_cam_ctlr_handle_t handle = NULL;
    ESP_ERROR_CHECK(esp_cam_new_csi_ctlr(&csi_config, &handle));

A camera controller driver can be implemented by the ISP DVP peripheral, which requires the configuration that specified by `esp_cam_ctlr_isp_dvp_cfg_t`.

If the configurations in `esp_cam_ctlr_isp_dvp_cfg_t` is specified, users can call `esp_cam_new_isp_dvp_ctlr()` to allocate and initialize a ISP DVP camera controller handle. This function will return an ISP DVP camera controller handle if it runs correctly. You can take following code as reference.

Before calling `esp_cam_new_isp_dvp_ctlr()`, you should also call `esp_isp_new_processor()` to create an ISP handle.

    isp_proc_handle_t isp_proc = NULL;
    esp_isp_processor_cfg_t isp_config = {
        .clk_hz = 120 * 1000 * 1000,
        .input_data_source = ISP_INPUT_DATA_SOURCE_DVP,
        .input_data_color_type = ISP_COLOR_RAW8,
        .output_data_color_type = ISP_COLOR_RGB565,
        .has_line_start_packet = false,
        .has_line_end_packet = false,
        .h_res = MIPI_CSI_DISP_HSIZE,
        .v_res = MIPI_CSI_DISP_VSIZE,
    };
    ESP_ERROR_CHECK(esp_isp_new_processor(&isp_config, &isp_proc));

    esp_cam_ctlr_isp_dvp_cfg_t dvp_ctlr_config = {
        .data_width = 8,
        .data_io = {53, 54, 52, 0, 1, 45, 46, 47, -1, -1, -1, -1, -1, -1, -1, -1},
        .pclk_io = 21,
        .hsync_io = 5,
        .vsync_io = 23,
        .de_io = 22,
        .io_flags.vsync_invert = 1,
        .queue_items = 10,
    };
    ESP_ERROR_CHECK(esp_cam_new_isp_dvp_ctlr(isp_proc, &dvp_ctlr_config, &cam_handle));

A camera controller driver can be implemented by the DVP port of LCD_CAM, which requires the configuration that specified by `esp_cam_ctlr_dvp_config_t`.

`esp_cam_ctlr_dvp_config_t::exexternal_xtal`: set this to use externally generated xclk, otherwise the camera driver will generate it internally.

If `esp_cam_ctlr_lcd_cam_cfg_t` is specified, users can call `esp_cam_new_lcd_cam_ctlr()` to allocate and initialize a DVP camera controller handle. This function will return an DVP camera controller handle if it runs correctly. You can take following code as reference.

After calling `esp_cam_new_dvp_ctlr()`, you should allocate a camera buffer that meets the alignment constraints, or call `esp_cam_ctlr_alloc_buffer()` to automatically allocate.

You can call `esp_cam_ctlr_format_conversion()` to configure format conversion. The driver supports the following conversion types:

- YUV to RGB conversion

- RGB to YUV conversion

- YUV to YUV conversion

Color range support: * Full range: 0-255 for both RGB and YUV * Limited range: RGB 16-240, YUV Y:16-240, U-V:16-235

    esp_cam_ctlr_handle_t cam_handle = NULL;
    esp_cam_ctlr_dvp_pin_config_t pin_cfg = {
        .data_width = EXAMPLE_DVP_CAM_DATA_WIDTH,
        .data_io = {
            EXAMPLE_DVP_CAM_D0_IO,
            EXAMPLE_DVP_CAM_D1_IO,
            EXAMPLE_DVP_CAM_D2_IO,
            EXAMPLE_DVP_CAM_D3_IO,
            EXAMPLE_DVP_CAM_D4_IO,
            EXAMPLE_DVP_CAM_D5_IO,
            EXAMPLE_DVP_CAM_D6_IO,
            EXAMPLE_DVP_CAM_D7_IO,
        },
        .vsync_io = EXAMPLE_DVP_CAM_VSYNC_IO,
        .de_io = EXAMPLE_DVP_CAM_DE_IO,
        .pclk_io = EXAMPLE_DVP_CAM_PCLK_IO,
        .xclk_io = EXAMPLE_DVP_CAM_XCLK_IO, // Set XCLK pin to generate XCLK signal
    };
    esp_cam_ctlr_dvp_config_t dvp_config = {
        .ctlr_id = 0,
        .clk_src = CAM_CLK_SRC_DEFAULT,
        .h_res = CONFIG_EXAMPLE_CAM_HRES,
        .v_res = CONFIG_EXAMPLE_CAM_VRES,
        .input_data_color_type = CAM_CTLR_COLOR_RGB565,
        .dma_burst_size = 128,
        .pin = &pin_cfg,
        .bk_buffer_dis = 1,
        .xclk_freq = EXAMPLE_DVP_CAM_XCLK_FREQ_HZ,
    };

    ESP_ERROR_CHECK(esp_cam_new_dvp_ctlr(&dvp_config, &cam_handle));

    const cam_ctlr_format_conv_config_t conv_cfg = {
        .src_format = CAM_CTLR_COLOR_YUV422,      // Source format: YUV422
        .dst_format = CAM_CTLR_COLOR_RGB565,      // Destination format: RGB565
        .conv_std = COLOR_CONV_STD_RGB_YUV_BT601,
        .data_width = 8,
        .input_range = COLOR_RANGE_LIMIT,
        .output_range = COLOR_RANGE_LIMIT,
    };
    ESP_ERROR_CHECK(esp_cam_ctlr_format_conversion(cam_handle, &conv_cfg));

#### Uninstall Camera Controller Driver

If a previously installed camera controller driver is no longer needed, it's recommended to recycle the resource by calling `esp_cam_ctlr_del()`, so that to release the underlying hardware.

### Enable and Disable Camera Controller Driver

Before starting camera controller operation, you need to enable the camera controller driver first, by calling `esp_cam_ctlr_enable()`. This function:

- Switches the driver state from init to enable.

    ESP_ERROR_CHECK(esp_cam_ctlr_enable(handle));

Calling `esp_cam_ctlr_disable()` does the opposite, that is, put the driver back to the init state.

    ESP_ERROR_CHECK(esp_cam_ctlr_disable(handle));

### Start and Stop Camera Controller Driver

Before receiving camera signal from camera sensor, you need to start the camera controller driver first, by calling `esp_cam_ctlr_start()`. This function:

- Switches the driver state from enable to start.

    ESP_ERROR_CHECK(esp_cam_ctlr_start(handle));

Calling `esp_cam_ctlr_stop()` does the opposite, that is, put the driver back to the enable state.

    ESP_ERROR_CHECK(esp_cam_ctlr_stop(handle));

### Receive from a Camera Sensor

Now you can call `esp_cam_ctlr_receive()` to receive from a camera sensor or something else.

    ESP_ERROR_CHECK(esp_cam_ctlr_receive(handle, &my_trans, ESP_CAM_CTLR_MAX_DELAY));

### Register Event Callbacks

After the camera controller driver starts receiving, it can generate a specific event dynamically. If you have some functions that should be called when the event happens, please hook your function to the interrupt service routine by calling `esp_cam_ctlr_register_event_callbacks()`. All supported event callbacks are listed in `esp_cam_ctlr_evt_cbs_t`:

- `esp_cam_ctlr_evt_cbs_t::on_get_new_trans` sets a callback function which will be called after the camera controller driver finishes previous transaction, and tries to get a new transaction descriptor. It will also be called when in `s_ctlr_csi_start()`. If this callback does not get a new transaction descriptor, the camera controller driver will use the internal backup buffer if `bk_buffer_dis` flag is set.

- `esp_cam_ctlr_evt_cbs_t::on_trans_finished` sets a callback function when the camera controller driver finishes a transaction. As this function is called within the ISR context, you must ensure that the function does not attempt to block (e.g., by making sure that only FreeRTOS APIs with `ISR` suffix are called from within the function).

### Thread Safety

The factory functions:

- `esp_cam_new_csi_ctlr()`

- `esp_cam_new_isp_dvp_ctlr()`

- `esp_cam_ctlr_del()`

are guaranteed to be thread safe by the driver, which means, they can be called from different RTOS tasks without protection by extra locks.

### Kconfig Options

The following Kconfig options affect the behavior of the interrupt handler when cache is disabled:

- CONFIG_CAM_CTLR_MIPI_CSI_ISR_CACHE_SAFE, see Thread Safety for more details.

- CONFIG_CAM_CTLR_ISP_DVP_ISR_CACHE_SAFE, see Thread Safety for more details.

### IRAM Safe

By default, the CSI interrupt will be deferred when the cache is disabled because of writing or erasing the flash.

There are Kconfig options

- CONFIG_CAM_CTLR_MIPI_CSI_ISR_CACHE_SAFE

- CONFIG_CAM_CTLR_ISP_DVP_ISR_CACHE_SAFE

that

- Enables the interrupt being serviced even when the cache is disabled

- Places all functions that used by the ISR into IRAM

- Places driver object into DRAM (in case it is mapped to PSRAM by accident)

This allows the interrupt to run while the cache is disabled, but comes at the cost of increased IRAM consumption. So user callbacks need to notice that the code and data inside (the callback) should be IRAM-safe or DRAM-safe, when cache is disabled.

## Application Examples

- peripherals/camera/mipi_isp_dsi demonstrates how to use the `esp_driver_cam` component to capture signals from a MIPI CSI camera sensor via the ISP module and display it on a LCD screen via a DSI interface.

- peripherals/camera/dvp_isp_dsi demonstrates how to use the `esp_driver_cam` component to capture signals from a DVP camera sensor via the ISP module and display it on a LCD screen via a DSI interface.

## API Reference

### Header File

- components/esp_driver_cam/include/esp_cam_ctlr.h

- This header file can be included with:

  >     #include "esp_cam_ctlr.h"

- This header file is a part of the API provided by the `esp_driver_cam` component. To declare that your component depends on `esp_driver_cam`, add the following to your CMakeLists.txt:

  >     REQUIRES esp_driver_cam

  > or

  >     PRIV_REQUIRES esp_driver_cam

### Functions

esp_err_t esp_cam_ctlr_enable(esp_cam_ctlr_handle_t handle)

  Enable ESP CAM controller.

  Parameters:

    handle -- [in] ESP CAM controller handle

  Returns:

    - ESP_OK

    - ESP_ERR_INVALID_ARG: Invalid argument

    - ESP_ERR_INVALID_STATE: Invalid state

esp_err_t esp_cam_ctlr_start(esp_cam_ctlr_handle_t handle)

  Start ESP CAM controller.

  Parameters:

    handle -- [in] ESP CAM controller handle

  Returns:

    - ESP_OK

    - ESP_ERR_INVALID_ARG: Invalid argument

    - ESP_ERR_INVALID_STATE: Invalid state

esp_err_t esp_cam_ctlr_stop(esp_cam_ctlr_handle_t handle)

  Stop ESP CAM controller.

  Parameters:

    handle -- [in] ESP CAM controller handle

  Returns:

    - ESP_OK

    - ESP_ERR_INVALID_ARG: Invalid argument

    - ESP_ERR_INVALID_STATE: Invalid state

esp_err_t esp_cam_ctlr_disable(esp_cam_ctlr_handle_t handle)

  Disable ESP CAM controller.

  Parameters:

    handle -- [in] ESP CAM controller handle

  Returns:

    - ESP_OK

    - ESP_ERR_INVALID_ARG: Invalid argument

    - ESP_ERR_INVALID_STATE: Invalid state

esp_err_t esp_cam_ctlr_receive(esp_cam_ctlr_handle_t handle, esp_cam_ctlr_trans_t *trans, uint32_t timeout_ms)

  Receive data to the given transaction.

  Parameters:

    - handle -- [in] ESP CAM controller handle

    - trans -- [in] ESP CAM controller transaction type

    - timeout_ms -- [in] Timeout in ms

  Returns:

    - ESP_OK

    - ESP_ERR_INVALID_ARG: Invalid argument

    - ESP_ERR_INVALID_STATE: Invalid state

esp_err_t esp_cam_ctlr_del(esp_cam_ctlr_handle_t handle)

  Delete ESP CAM controller handle.

  Parameters:

    handle -- [in] ESP CAM controller handle

  Returns:

    - ESP_OK

    - ESP_ERR_INVALID_ARG: Invalid argument

    - ESP_ERR_INVALID_STATE: Invalid state

esp_err_t esp_cam_ctlr_register_event_callbacks(esp_cam_ctlr_handle_t handle, const esp_cam_ctlr_evt_cbs_t *cbs, void *user_data)

  Register ESP CAM controller event callbacks.

  Parameters:

    - handle -- [in] ESP CAM controller handle

    - cbs -- [in] ESP CAM controller event callbacks

    - user_data -- [in] User data

  Returns:

    - ESP_OK

    - ESP_ERR_INVALID_ARG: Invalid argument

    - ESP_ERR_INVALID_STATE: Invalid state

esp_err_t esp_cam_ctlr_get_frame_buffer(esp_cam_ctlr_handle_t handle, uint32_t fb_num, const void **fb0, ...)

  Get ESP CAM controller internal malloced backup buffer(s) addr.

  Note

  Generally, data in internal buffer is ready when `on_trans_finished` event

  Parameters:

    - handle -- [in] ESP CAM controller handle

    - fb_num -- [in] Number of frame buffer(s) to get. This value must be the same as the number of the followed fbN parameters

    - fb0 -- [out] Address of the frame buffer 0 (first frame buffer)

    - ... -- [out] List of other frame buffers if any

  Returns:

    - ESP_OK

    - ESP_ERR_INVALID_ARG: Invalid argument

    - ESP_ERR_INVALID_STATE: Invalid driver state

esp_err_t esp_cam_ctlr_get_frame_buffer_len(esp_cam_ctlr_handle_t handle, size_t *ret_fb_len)

  Get ESP CAM controller internal backup buffer length.

  Parameters:

    - handle -- [in] ESP CAM controller handle

    - ret_fb_len -- [out] Optional, The size of each frame buffer, in bytes.

  Returns:

    - ESP_OK

    - ESP_ERR_INVALID_ARG: NULL ptr

    - ESP_ERR_INVALID_STATE: Invalid driver state

void *esp_cam_ctlr_alloc_buffer(esp_cam_ctlr_handle_t handle, size_t size, uint32_t buf_caps)

  Allocate camera buffer for ESP CAM controller.

  Note

  This function must be called after esp_cam_new_*_ctlr

  Parameters:

    - handle -- [in] ESP CAM controller handle

    - size -- [in] Buffer size in bytes

    - buf_caps -- [in] Buffer allocation capabilities:

      - MALLOC_CAP_SPIRAM || MALLOC_CAP_DMA: Memory in external SPI RAM

      - MALLOC_CAP_INTERNAL || MALLOC_CAP_DMA: Memory in internal SRAM

  Returns:

    - Buffer pointer on success

    - NULL on failure

esp_err_t esp_cam_ctlr_format_conversion(esp_cam_ctlr_handle_t handle, const cam_ctlr_format_conv_config_t *conv_cfg)

  Configure format conversion.

  Parameters:

    - handle -- [in] ESP CAM controller handle

    - conv_cfg -- [in] Color conversion configuration, contains source and destination formats

  Returns:

    - ESP_OK on success

    - ESP_ERR_INVALID_ARG: Invalid argument

    - ESP_ERR_NOT_SUPPORTED: Format conversion not supported by this controller

### Header File

- components/esp_driver_cam/include/esp_cam_ctlr_types.h

- This header file can be included with:

  >     #include "esp_cam_ctlr_types.h"

- This header file is a part of the API provided by the `esp_driver_cam` component. To declare that your component depends on `esp_driver_cam`, add the following to your CMakeLists.txt:

  >     REQUIRES esp_driver_cam

  > or

  >     PRIV_REQUIRES esp_driver_cam

### Structures

struct esp_cam_ctlr_trans_t

  ESP CAM controller transaction type.

  Public Members

  void *buffer

    Transaction buffer.

  size_t buflen

    Len of the transaction buffer.

  size_t received_size

    Received size, this received_size will be written by the driver, indicating the real received size.

struct esp_cam_ctlr_evt_cbs_t

  ESP CAM controller event callbacks.

  Public Members

  bool (*on_get_new_trans)(esp_cam_ctlr_handle_t handle, esp_cam_ctlr_trans_t *trans, void *user_data)

    On get new transaction callback.

    Param handle:

      [in] ESP CAM controller handle

    Param trans:

      [in] New transaction

    Param user_data:

      [in] User registered data

    Return:

      Whether a high priority task is woken up by this function

  bool (*on_trans_finished)(esp_cam_ctlr_handle_t handle, esp_cam_ctlr_trans_t *trans, void *user_data)

    On transaction finish callback.

    Param handle:

      [in] ESP CAM controller handle

    Param trans:

      [out] Finished transaction

    Param user_data:

      [in] User registered data

    Return:

      Whether a high priority task is woken up by this function

### Macros

ESP_CAM_CTLR_MAX_DELAY

  ESP CAM controller max timeout value.

### Type Definitions

typedef struct esp_cam_ctlr_t *esp_cam_ctlr_handle_t

  ESP CAM controller handle.

### Header File

- components/esp_driver_cam/csi/include/esp_cam_ctlr_csi.h

- This header file can be included with:

  >     #include "esp_cam_ctlr_csi.h"

- This header file is a part of the API provided by the `esp_driver_cam` component. To declare that your component depends on `esp_driver_cam`, add the following to your CMakeLists.txt:

  >     REQUIRES esp_driver_cam

  > or

  >     PRIV_REQUIRES esp_driver_cam

### Functions

esp_err_t esp_cam_new_csi_ctlr(const esp_cam_ctlr_csi_config_t *config, esp_cam_ctlr_handle_t *ret_handle)

  New ESP CAM CSI controller.

  Parameters:

    - config -- [in] CSI controller configurations

    - ret_handle -- [out] Returned ESP CAM controller handle

  Returns:

    - ESP_OK

    - ESP_ERR_INVALID_ARG: Invalid argument

    - ESP_ERR_NO_MEM: Out of memory

    - ESP_ERR_NOT_SUPPORTED: Currently not support modes or types

    - ESP_ERR_NOT_FOUND: CSI is registered already

### Structures

struct esp_cam_ctlr_csi_config_t

  ESP CAM CSI controller configurations.

  Public Members

  int ctlr_id

    CSI controller ID.

  mipi_csi_phy_clock_source_t clk_src

    CSI phy clock source.

  uint32_t h_res

    Input horizontal resolution, i.e. the number of pixels in a line.

  uint32_t v_res

    Input vertical resolution, i.e. the number of lines in a frame.

  uint8_t data_lane_num

    Data lane num.

  int lane_bit_rate_mbps

    Lane bit rate in Mbps.

  cam_ctlr_color_t input_data_color_type

    Input color type.

  cam_ctlr_color_t output_data_color_type

    Output color type.

  int queue_items

    Queue items.

  uint32_t byte_swap_en

    Enable byte swap.

  uint32_t bk_buffer_dis

    Disable backup buffer.

  struct esp_cam_ctlr_csi_config_t

    Boolean Flags.

### Header File

- components/esp_driver_cam/isp_dvp/include/esp_cam_ctlr_isp_dvp.h

- This header file can be included with:

  >     #include "esp_cam_ctlr_isp_dvp.h"

- This header file is a part of the API provided by the `esp_driver_cam` component. To declare that your component depends on `esp_driver_cam`, add the following to your CMakeLists.txt:

  >     REQUIRES esp_driver_cam

  > or

  >     PRIV_REQUIRES esp_driver_cam

### Functions

esp_err_t esp_cam_new_isp_dvp_ctlr(isp_proc_handle_t isp_proc, const esp_cam_ctlr_isp_dvp_cfg_t *ctlr_config, esp_cam_ctlr_handle_t *ret_handle)

  New ESP CAM ISP DVP controller.

  Parameters:

    - isp_proc -- [in] Processor handle

    - ctlr_config -- [in] ISP DVP controller configurations

    - ret_handle -- [out] Returned ESP CAM controller handle

  Returns:

    - ESP_OK

    - ESP_ERR_INVALID_ARG: Invalid argument

    - ESP_ERR_NO_MEM: Out of memory

    - ESP_ERR_NOT_SUPPORTED: Currently not support modes or types

    - ESP_ERR_NOT_FOUND: ISP DVP is registered already

### Structures

struct esp_cam_ctlr_isp_dvp_cfg_t

  ESP CAM ISP DVP controller configurations.

  Public Members

  cam_ctlr_data_width_t data_width

    Number of data lines.

  int data_io[ISP_DVP_DATA_SIG_NUM]

    ISP DVP data-in IO numbers.

  int pclk_io

    ISP DVP pclk IO numbers.

  int hsync_io

    ISP DVP hsync IO numbers.

  int vsync_io

    ISP DVP vsync IO numbers.

  int de_io

    ISP DVP de IO numbers.

  uint32_t pclk_invert

    The pclk is inverted.

  uint32_t hsync_invert

    The hsync signal is inverted (i.e. active low)

  uint32_t vsync_invert

    The vsync signal is inverted (i.e. active high)

  uint32_t de_invert

    The de signal is inverted (i.e. active low)

  struct esp_cam_ctlr_isp_dvp_cfg_t io_flags

    ISP DVP IO flags.

  int queue_items

    Queue items.

  uint32_t byte_swap_en

    Enable byte swap.

  uint32_t bk_buffer_dis

    Disable backup buffer.

Was this page helpful?

- Thank you! We received your feedback.
- If you have any comments, fill in Espressif Documentation Feedback Form.

- We value your feedback.
- Let us know how we can improve this page by filling in Espressif Documentation Feedback Form.