Over The Air Updates (OTA)

[中文]

OTA Process Overview

The OTA update mechanism allows a device to update itself based on data received while the normal firmware is running (for example, over WiFi or Bluetooth.)

OTA requires configuring the Partition Table of the device with at least two “OTA app slot” partitions (ie ota_0 and ota_1) and an “OTA Data Partition”.

The OTA operation functions write a new app firmware image to whichever OTA app slot is not currently being used for booting. Once the image is verified, the OTA Data partition is updated to specify that this image should be used for the next boot.

OTA Data Partition

An OTA data partition (type data, subtype ota) must be included in the Partition Table of any project which uses the OTA functions.

For factory boot settings, the OTA data partition should contain no data (all bytes erased to 0xFF). In this case the esp-idf software bootloader will boot the factory app if it is present in the the partition table. If no factory app is included in the partition table, the first available OTA slot (usually ota_0) is booted.

After the first OTA update, the OTA data partition is updated to specify which OTA app slot partition should be booted next.

The OTA data partition is two flash sectors (0x2000 bytes) in size, to prevent problems if there is a power failure while it is being written. Sectors are independently erased and written with matching data, and if they disagree a counter field is used to determine which sector was written more recently.

App rollback

The main purpose of the application rollback is to keep the device working after the update. This feature allows you to roll back to the previous working application in case a new application has critical errors. When the rollback process is enabled and an OTA update provides a new version of the app, one of three things can happen:

  • The application works fine, esp_ota_mark_app_valid_cancel_rollback() marks the running application with the state ESP_OTA_IMG_VALID. There are no restrictions on booting this application.

  • The application has critical errors and further work is not possible, a rollback to the previous application is required, esp_ota_mark_app_invalid_rollback_and_reboot() marks the running application with the state ESP_OTA_IMG_INVALID and reset. This application will not be selected by the bootloader for boot and will boot the previously working application.

  • If the CONFIG_BOOTLOADER_APP_ROLLBACK_ENABLE option is set, and a reset occurs without calling either function then the application is rolled back.

Note: The state is not written to the binary image of the application it is written to the otadata partition. The partition contains a ota_seq counter which is a pointer to the slot (ota_0, ota_1, …) from which the application will be selected for boot.

App OTA State

States control the process of selecting a boot app:

States

Restriction of selecting a boot app in bootloader

ESP_OTA_IMG_VALID

None restriction. Will be selected.

ESP_OTA_IMG_UNDEFINED

None restriction. Will be selected.

ESP_OTA_IMG_INVALID

Will not be selected.

ESP_OTA_IMG_ABORTED

Will not be selected.

ESP_OTA_IMG_NEW

If CONFIG_BOOTLOADER_APP_ROLLBACK_ENABLE option is set it will be selected only once. In bootloader the state immediately changes to ESP_OTA_IMG_PENDING_VERIFY.

ESP_OTA_IMG_PENDING_VERIFY

If CONFIG_BOOTLOADER_APP_ROLLBACK_ENABLE option is set it will not be selected and the state will change to ESP_OTA_IMG_ABORTED.

If CONFIG_BOOTLOADER_APP_ROLLBACK_ENABLE option is not enabled (by default), then the use of the following functions esp_ota_mark_app_valid_cancel_rollback() and esp_ota_mark_app_invalid_rollback_and_reboot() are optional, and ESP_OTA_IMG_NEW and ESP_OTA_IMG_PENDING_VERIFY states are not used.

An option in Kconfig CONFIG_BOOTLOADER_APP_ROLLBACK_ENABLE allows you to track the first boot of a new application. In this case, the application must confirm its operability by calling esp_ota_mark_app_valid_cancel_rollback() function, otherwise the application will be rolled back upon reboot. It allows you to control the operability of the application during the boot phase. Thus, a new application has only one attempt to boot successfully.

Rollback Process

The description of the rollback process when CONFIG_BOOTLOADER_APP_ROLLBACK_ENABLE option is enabled:

  • The new application successfully downloaded and esp_ota_set_boot_partition() function makes this partition bootable and sets the state ESP_OTA_IMG_NEW. This state means that the application is new and should be monitored for its first boot.

  • Reboot esp_restart().

  • The bootloader checks for the ESP_OTA_IMG_PENDING_VERIFY state if it is set, then it will be written to ESP_OTA_IMG_ABORTED.

  • The bootloader selects a new application to boot so that the state is not set as ESP_OTA_IMG_INVALID or ESP_OTA_IMG_ABORTED.

  • The bootloader checks the selected application for ESP_OTA_IMG_NEW state if it is set, then it will be written to ESP_OTA_IMG_PENDING_VERIFY. This state means that the application requires confirmation of its operability, if this does not happen and a reboot occurs, this state will be overwritten to ESP_OTA_IMG_ABORTED (see above) and this application will no longer be able to start, i.e. there will be a rollback to the previous work application.

  • A new application has started and should make a self-test.

  • If the self-test has completed successfully, then you must call the function esp_ota_mark_app_valid_cancel_rollback() because the application is awaiting confirmation of operability (ESP_OTA_IMG_PENDING_VERIFY state).

  • If the self-test fails then call esp_ota_mark_app_invalid_rollback_and_reboot() function to roll back to the previous working application, while the invalid application is set ESP_OTA_IMG_INVALID state.

  • If the application has not been confirmed, the state remains ESP_OTA_IMG_PENDING_VERIFY, and the next boot it will be changed to ESP_OTA_IMG_ABORTED. That will prevent re-boot of this application. There will be a rollback to the previous working application.

Unexpected Reset

If a power loss or an unexpected crash occurs at the time of the first boot of a new application, it will roll back the application.

Recommendation: Perform the self-test procedure as quickly as possible, to prevent rollback due to power loss.

Only OTA partitions can be rolled back. Factory partition is not rolled back.

Booting invalid/aborted apps

Booting an application which was previously set to ESP_OTA_IMG_INVALID or ESP_OTA_IMG_ABORTED is possible:

To determine if self-tests should be run during startup of an application, call the esp_ota_get_state_partition() function. If result is ESP_OTA_IMG_PENDING_VERIFY then self-testing and subsequent confirmation of operability is required.

Where the states are set

A brief description of where the states are set:

Anti-rollback

Anti-rollback prevents rollback to application with security version lower than one programmed in eFuse of chip.

This function works if set CONFIG_BOOTLOADER_APP_ANTI_ROLLBACK option. In the bootloader, when selecting a bootable application, an additional security version check is added which is on the chip and in the application image. The version in the bootable firmware must be greater than or equal to the version in the chip.

CONFIG_BOOTLOADER_APP_ANTI_ROLLBACK and CONFIG_BOOTLOADER_APP_ROLLBACK_ENABLE options are used together. In this case, rollback is possible only on the security version which is equal or higher than the version in the chip.

A typical anti-rollback scheme is

  • New firmware released with the elimination of vulnerabilities with the previous version of security.

  • After the developer makes sure that this firmware is working. He can increase the security version and release a new firmware.

  • Download new application.

  • To make it bootable, run the function esp_ota_set_boot_partition(). If the security version of the new application is smaller than the version in the chip, the new application will be erased. Update to new firmware is not possible.

  • Reboot.

  • In the bootloader, an application with a security version greater than or equal to the version in the chip will be selected. If otadata is in the initial state, and one firmware was loaded via a serial channel, whose secure version is higher than the chip, then the secure version of efuse will be immediately updated in the bootloader.

  • New application booted. Then the application should perform diagnostics of the operation and if it is completed successfully, you should call esp_ota_mark_app_valid_cancel_rollback() function to mark the running application with the ESP_OTA_IMG_VALID state and update the secure version on chip. Note that if was called esp_ota_mark_app_invalid_rollback_and_reboot() function a rollback may not happend due to the device may not have any bootable apps then it will return ESP_ERR_OTA_ROLLBACK_FAILED error and stay in the ESP_OTA_IMG_PENDING_VERIFY state.

  • The next update of app is possible if a running app is in the ESP_OTA_IMG_VALID state.

Recommendation:

If you want to avoid the download/erase overhead in case of the app from the server has security version lower then running app you have to get new_app_info.secure_version from the first package of an image and compare it with the secure version of efuse. Use esp_efuse_check_secure_version(new_app_info.secure_version) function if it is true then continue downloading otherwise abort.

....
bool image_header_was_checked = false;
while (1) {
    int data_read = esp_http_client_read(client, ota_write_data, BUFFSIZE);
    ...
    if (data_read > 0) {
        if (image_header_was_checked == false) {
            esp_app_desc_t new_app_info;
            if (data_read > sizeof(esp_image_header_t) + sizeof(esp_image_segment_header_t) + sizeof(esp_app_desc_t)) {
                // check current version with downloading
                if (esp_efuse_check_secure_version(new_app_info.secure_version) == false) {
                    ESP_LOGE(TAG, "This a new app can not be downloaded due to a secure version is lower than stored in efuse.");
                    http_cleanup(client);
                    task_fatal_error();
                }

                image_header_was_checked = true;

                esp_ota_begin(update_partition, OTA_SIZE_UNKNOWN, &update_handle);
            }
        }
        esp_ota_write( update_handle, (const void *)ota_write_data, data_read);
    }
}
...

Restrictions:

  • The number of bits in the secure_version field is limited to 32 bits. This means that only 32 times you can do an anti-rollback. You can reduce the length of this efuse field use CONFIG_BOOTLOADER_APP_SEC_VER_SIZE_EFUSE_FIELD option.

  • Anti-rollback only works if the encoding scheme for efuse is set to NONE.

  • The partition table should not have a factory partition, only two of the app.

security_version:

  • In ESP32 it is stored in efuse EFUSE_BLK3_RDATA4_REG. (when a eFuse bit is programmed to 1, it can never be reverted to 0). The number of bits set in this register is the security_version from app.

Secure OTA Updates Without Secure boot

The verification of signed OTA updates can be performed even without enabling hardware secure boot. For doing so, refer Signed App Verification Without Hardware Secure Boot

OTA Tool (otatool.py)

The component app_update provides a tool otatool.py for performing OTA partition-related operations on a target device. The following operations can be performed using the tool:

  • read contents of otadata partition (read_otadata)

  • erase otadata partition, effectively resetting device to factory app (erase_otadata)

  • switch OTA partitions (switch_ota_partition)

  • erasing OTA partition (erase_ota_partition)

  • write to OTA partition (write_ota_partition)

  • read contents of OTA partition (read_ota_partition)

The tool can either be imported and used from another Python script or invoked from shell script for users wanting to perform operation programmatically. This is facilitated by the tool’s Python API and command-line interface, respectively.

Python API

Before anything else, make sure that the otatool module is imported.

import sys
import os

idf_path = os.environ["IDF_PATH"]  # get value of IDF_PATH from environment
otatool_dir = os.path.join(idf_path, "components", "app_update")  # otatool.py lives in $IDF_PATH/components/app_update

sys.path.append(otatool_dir)  # this enables Python to find otatool module
from otatool import *  # import all names inside otatool module

The starting point for using the tool’s Python API to do is create a OtatoolTarget object:

# Create a partool.py target device connected on serial port /dev/ttyUSB1
target = OtatoolTarget("/dev/ttyUSB1")

The created object can now be used to perform operations on the target device:

# Erase otadata, reseting the device to factory app
target.erase_otadata()

# Erase contents of OTA app slot 0
target.erase_ota_partition(0)

# Switch boot partition to that of app slot 1
target.switch_ota_partition(1)

# Read OTA partition 'ota_3' and save contents to a file named 'ota_3.bin'
target.read_ota_partition("ota_3", "ota_3.bin")

The OTA partition to operate on is specified using either the app slot number or the partition name.

More information on the Python API is available in the docstrings for the tool.

Command-line Interface

The command-line interface of otatool.py has the following structure:

otatool.py [command-args] [subcommand] [subcommand-args]

- command-args - these are arguments that are needed for executing the main command (parttool.py), mostly pertaining to the target device
- subcommand - this is the operation to be performed
- subcommand-args - these are arguments that are specific to the chosen operation

# Erase otadata, resetting the device to factory app
otatool.py --port "/dev/ttyUSB1" erase_otadata

# Erase contents of OTA app slot 0
otatool.py --port "/dev/ttyUSB1" erase_ota_partition --slot 0

# Switch boot partition to that of app slot 1
otatool.py --port "/dev/ttyUSB1" switch_ota_partition --slot 1

# Read OTA partition 'ota_3' and save contents to a file named 'ota_3.bin'
otatool.py --port "/dev/ttyUSB1" read_ota_partition --name=ota_3

More information can be obtained by specifying –help as argument:

# Display possible subcommands and show main command argument descriptions
otatool.py --help

# Show descriptions for specific subcommand arguments
otatool.py [subcommand] --help

Application Example

End-to-end example of OTA firmware update workflow: system/ota.

API Reference

Functions

const esp_app_desc_t *esp_ota_get_app_description(void)

Return esp_app_desc structure. This structure includes app version.

Return description for running app.

Return

Pointer to esp_app_desc structure.

int esp_ota_get_app_elf_sha256(char *dst, size_t size)

Fill the provided buffer with SHA256 of the ELF file, formatted as hexadecimal, null-terminated. If the buffer size is not sufficient to fit the entire SHA256 in hex plus a null terminator, the largest possible number of bytes will be written followed by a null.

Return

Number of bytes written to dst (including null terminator)

Parameters

  • dst: Destination buffer

  • size: Size of the buffer

esp_err_t esp_ota_begin(const esp_partition_t *partition, size_t image_size, esp_ota_handle_t *out_handle)

Commence an OTA update writing to the specified partition.

The specified partition is erased to the specified image size.

If image size is not yet known, pass OTA_SIZE_UNKNOWN which will cause the entire partition to be erased.

On success, this function allocates memory that remains in use until esp_ota_end() is called with the returned handle.

Note: If the rollback option is enabled and the running application has the ESP_OTA_IMG_PENDING_VERIFY state then it will lead to the ESP_ERR_OTA_ROLLBACK_INVALID_STATE error. Confirm the running app before to run download a new app, use esp_ota_mark_app_valid_cancel_rollback() function for it (this should be done as early as possible when you first download a new application).

Return

  • ESP_OK: OTA operation commenced successfully.

  • ESP_ERR_INVALID_ARG: partition or out_handle arguments were NULL, or partition doesn’t point to an OTA app partition.

  • ESP_ERR_NO_MEM: Cannot allocate memory for OTA operation.

  • ESP_ERR_OTA_PARTITION_CONFLICT: Partition holds the currently running firmware, cannot update in place.

  • ESP_ERR_NOT_FOUND: Partition argument not found in partition table.

  • ESP_ERR_OTA_SELECT_INFO_INVALID: The OTA data partition contains invalid data.

  • ESP_ERR_INVALID_SIZE: Partition doesn’t fit in configured flash size.

  • ESP_ERR_FLASH_OP_TIMEOUT or ESP_ERR_FLASH_OP_FAIL: Flash write failed.

  • ESP_ERR_OTA_ROLLBACK_INVALID_STATE: If the running app has not confirmed state. Before performing an update, the application must be valid.

Parameters

  • partition: Pointer to info for partition which will receive the OTA update. Required.

  • image_size: Size of new OTA app image. Partition will be erased in order to receive this size of image. If 0 or OTA_SIZE_UNKNOWN, the entire partition is erased.

  • out_handle: On success, returns a handle which should be used for subsequent esp_ota_write() and esp_ota_end() calls.

esp_err_t esp_ota_write(esp_ota_handle_t handle, const void *data, size_t size)

Write OTA update data to partition.

This function can be called multiple times as data is received during the OTA operation. Data is written sequentially to the partition.

Return

  • ESP_OK: Data was written to flash successfully.

  • ESP_ERR_INVALID_ARG: handle is invalid.

  • ESP_ERR_OTA_VALIDATE_FAILED: First byte of image contains invalid app image magic byte.

  • ESP_ERR_FLASH_OP_TIMEOUT or ESP_ERR_FLASH_OP_FAIL: Flash write failed.

  • ESP_ERR_OTA_SELECT_INFO_INVALID: OTA data partition has invalid contents

Parameters

  • handle: Handle obtained from esp_ota_begin

  • data: Data buffer to write

  • size: Size of data buffer in bytes.

esp_err_t esp_ota_write_with_offset(esp_ota_handle_t handle, const void *data, size_t size, uint32_t offset)

Write OTA update data to partition.

This function can write data in non contiguous manner. If flash encryption is enabled, data should be 16 byte aligned.

Note

While performing OTA, if the packets arrive out of order, esp_ota_write_with_offset() can be used to write data in non contiguous manner. Use of esp_ota_write_with_offset() in combination with esp_ota_write() is not recommended.

Return

  • ESP_OK: Data was written to flash successfully.

  • ESP_ERR_INVALID_ARG: handle is invalid.

  • ESP_ERR_OTA_VALIDATE_FAILED: First byte of image contains invalid app image magic byte.

  • ESP_ERR_FLASH_OP_TIMEOUT or ESP_ERR_FLASH_OP_FAIL: Flash write failed.

  • ESP_ERR_OTA_SELECT_INFO_INVALID: OTA data partition has invalid contents

Parameters

  • handle: Handle obtained from esp_ota_begin

  • data: Data buffer to write

  • size: Size of data buffer in bytes

  • offset: Offset in flash partition

esp_err_t esp_ota_end(esp_ota_handle_t handle)

Finish OTA update and validate newly written app image.

Note

After calling esp_ota_end(), the handle is no longer valid and any memory associated with it is freed (regardless of result).

Return

  • ESP_OK: Newly written OTA app image is valid.

  • ESP_ERR_NOT_FOUND: OTA handle was not found.

  • ESP_ERR_INVALID_ARG: Handle was never written to.

  • ESP_ERR_OTA_VALIDATE_FAILED: OTA image is invalid (either not a valid app image, or - if secure boot is enabled - signature failed to verify.)

  • ESP_ERR_INVALID_STATE: If flash encryption is enabled, this result indicates an internal error writing the final encrypted bytes to flash.

Parameters

  • handle: Handle obtained from esp_ota_begin().

esp_err_t esp_ota_set_boot_partition(const esp_partition_t *partition)

Configure OTA data for a new boot partition.

Note

If this function returns ESP_OK, calling esp_restart() will boot the newly configured app partition.

Return

  • ESP_OK: OTA data updated, next reboot will use specified partition.

  • ESP_ERR_INVALID_ARG: partition argument was NULL or didn’t point to a valid OTA partition of type “app”.

  • ESP_ERR_OTA_VALIDATE_FAILED: Partition contained invalid app image. Also returned if secure boot is enabled and signature validation failed.

  • ESP_ERR_NOT_FOUND: OTA data partition not found.

  • ESP_ERR_FLASH_OP_TIMEOUT or ESP_ERR_FLASH_OP_FAIL: Flash erase or write failed.

Parameters

  • partition: Pointer to info for partition containing app image to boot.

const esp_partition_t *esp_ota_get_boot_partition(void)

Get partition info of currently configured boot app.

If esp_ota_set_boot_partition() has been called, the partition which was set by that function will be returned.

If esp_ota_set_boot_partition() has not been called, the result is usually the same as esp_ota_get_running_partition(). The two results are not equal if the configured boot partition does not contain a valid app (meaning that the running partition will be an app that the bootloader chose via fallback).

If the OTA data partition is not present or not valid then the result is the first app partition found in the partition table. In priority order, this means: the factory app, the first OTA app slot, or the test app partition.

Note that there is no guarantee the returned partition is a valid app. Use esp_image_verify(ESP_IMAGE_VERIFY, …) to verify if the returned partition contains a bootable image.

Return

Pointer to info for partition structure, or NULL if partition table is invalid or a flash read operation failed. Any returned pointer is valid for the lifetime of the application.

const esp_partition_t *esp_ota_get_running_partition(void)

Get partition info of currently running app.

This function is different to esp_ota_get_boot_partition() in that it ignores any change of selected boot partition caused by esp_ota_set_boot_partition(). Only the app whose code is currently running will have its partition information returned.

The partition returned by this function may also differ from esp_ota_get_boot_partition() if the configured boot partition is somehow invalid, and the bootloader fell back to a different app partition at boot.

Return

Pointer to info for partition structure, or NULL if no partition is found or flash read operation failed. Returned pointer is valid for the lifetime of the application.

const esp_partition_t *esp_ota_get_next_update_partition(const esp_partition_t *start_from)

Return the next OTA app partition which should be written with a new firmware.

Call this function to find an OTA app partition which can be passed to esp_ota_begin().

Finds next partition round-robin, starting from the current running partition.

Return

Pointer to info for partition which should be updated next. NULL result indicates invalid OTA data partition, or that no eligible OTA app slot partition was found.

Parameters

  • start_from: If set, treat this partition info as describing the current running partition. Can be NULL, in which case esp_ota_get_running_partition() is used to find the currently running partition. The result of this function is never the same as this argument.

esp_err_t esp_ota_get_partition_description(const esp_partition_t *partition, esp_app_desc_t *app_desc)

Returns esp_app_desc structure for app partition. This structure includes app version.

Returns a description for the requested app partition.

Return

  • ESP_OK Successful.

  • ESP_ERR_NOT_FOUND app_desc structure is not found. Magic word is incorrect.

  • ESP_ERR_NOT_SUPPORTED Partition is not application.

  • ESP_ERR_INVALID_ARG Arguments is NULL or if partition’s offset exceeds partition size.

  • ESP_ERR_INVALID_SIZE Read would go out of bounds of the partition.

  • or one of error codes from lower-level flash driver.

Parameters

  • [in] partition: Pointer to app partition. (only app partition)

  • [out] app_desc: Structure of info about app.

esp_err_t esp_ota_mark_app_valid_cancel_rollback(void)

This function is called to indicate that the running app is working well.

Return

  • ESP_OK: if successful.

esp_err_t esp_ota_mark_app_invalid_rollback_and_reboot(void)

This function is called to roll back to the previously workable app with reboot.

If rollback is successful then device will reset else API will return with error code. Checks applications on a flash drive that can be booted in case of rollback. If the flash does not have at least one app (except the running app) then rollback is not possible.

Return

  • ESP_FAIL: if not successful.

  • ESP_ERR_OTA_ROLLBACK_FAILED: The rollback is not possible due to flash does not have any apps.

const esp_partition_t *esp_ota_get_last_invalid_partition(void)

Returns last partition with invalid state (ESP_OTA_IMG_INVALID or ESP_OTA_IMG_ABORTED).

Return

partition.

esp_err_t esp_ota_get_state_partition(const esp_partition_t *partition, esp_ota_img_states_t *ota_state)

Returns state for given partition.

Return

  • ESP_OK: Successful.

  • ESP_ERR_INVALID_ARG: partition or ota_state arguments were NULL.

  • ESP_ERR_NOT_SUPPORTED: partition is not ota.

  • ESP_ERR_NOT_FOUND: Partition table does not have otadata or state was not found for given partition.

Parameters

  • [in] partition: Pointer to partition.

  • [out] ota_state: state of partition (if this partition has a record in otadata).

esp_err_t esp_ota_erase_last_boot_app_partition(void)

Erase previous boot app partition and corresponding otadata select for this partition.

When current app is marked to as valid then you can erase previous app partition.

Return

  • ESP_OK: Successful, otherwise ESP_ERR.

bool esp_ota_check_rollback_is_possible(void)

Checks applications on the slots which can be booted in case of rollback.

These applications should be valid (marked in otadata as not UNDEFINED, INVALID or ABORTED and crc is good) and be able booted, and secure_version of app >= secure_version of efuse (if anti-rollback is enabled).

Return

  • True: Returns true if the slots have at least one app (except the running app).

  • False: The rollback is not possible.

Macros

OTA_SIZE_UNKNOWN

Used for esp_ota_begin() if new image size is unknown

ESP_ERR_OTA_BASE

Base error code for ota_ops api

ESP_ERR_OTA_PARTITION_CONFLICT

Error if request was to write or erase the current running partition

ESP_ERR_OTA_SELECT_INFO_INVALID

Error if OTA data partition contains invalid content

ESP_ERR_OTA_VALIDATE_FAILED

Error if OTA app image is invalid

ESP_ERR_OTA_SMALL_SEC_VER

Error if the firmware has a secure version less than the running firmware.

ESP_ERR_OTA_ROLLBACK_FAILED

Error if flash does not have valid firmware in passive partition and hence rollback is not possible

ESP_ERR_OTA_ROLLBACK_INVALID_STATE

Error if current active firmware is still marked in pending validation state (ESP_OTA_IMG_PENDING_VERIFY), essentially first boot of firmware image post upgrade and hence firmware upgrade is not possible

Type Definitions

typedef uint32_t esp_ota_handle_t

Opaque handle for an application OTA update.

esp_ota_begin() returns a handle which is then used for subsequent calls to esp_ota_write() and esp_ota_end().

Provide feedback about this document