Build System

[中文]

This document explains the implementation of the ESP-IDF build system and the concept of “components”. Read this document if you want to know how to organise and build a new ESP-IDF project or component.

Overview

An ESP-IDF project can be seen as an amalgamation of a number of components. For example, for a webserver that shows the current humidity, there could be:

  • The ESP-IDF base libraries (libc, ROM bindings, etc)

  • The WiFi drivers

  • A TCP/IP stack

  • The FreeRTOS operating system

  • A webserver

  • A driver for the humidity sensor

  • Main code tying it all together

ESP-IDF makes these components explicit and configurable. To do that, when a project is compiled, the build system will look up all the components in the ESP-IDF directories, the project directories and (optionally) in additional custom component directories. It then allows the user to configure the ESP-IDF project using a a text-based menu system to customize each component. After the components in the project are configured, the build system will compile the project.

Concepts

  • A “project” is a directory that contains all the files and configuration to build a single “app” (executable), as well as additional supporting elements such as a partition table, data/filesystem partitions, and a bootloader.

  • “Project configuration” is held in a single file called sdkconfig in the root directory of the project. This configuration file is modified via idf.py menuconfig to customise the configuration of the project. A single project contains exactly one project configuration.

  • An “app” is an executable which is built by ESP-IDF. A single project will usually build two apps - a “project app” (the main executable, ie your custom firmware) and a “bootloader app” (the initial bootloader program which launches the project app).

  • “components” are modular pieces of standalone code which are compiled into static libraries (.a files) and linked into an app. Some are provided by ESP-IDF itself, others may be sourced from other places.

  • “Target” is the hardware for which an application is built. At the moment, ESP-IDF supports esp32 and esp32s2 targets.

Some things are not part of the project:

  • “ESP-IDF” is not part of the project. Instead it is standalone, and linked to the project via the IDF_PATH environment variable which holds the path of the esp-idf directory. This allows the IDF framework to be decoupled from your project.

  • The toolchain for compilation is not part of the project. The toolchain should be installed in the system command line PATH.

Using the Build System

idf.py

The idf.py command line tool provides a front-end for easily managing your project builds. It manages the following tools:

  • CMake, which configures the project to be built

  • A command line build tool (either Ninja build or GNU Make)

  • esptool.py for flashing the target.

The getting started guide contains a brief introduction to how to set up idf.py to configure, build, and flash projects.

idf.py should be run in an ESP-IDF “project” directory, ie one containing a CMakeLists.txt file. Older style projects with a Makefile will not work with idf.py.

Type idf.py --help for a list of commands. Here are a summary of the most useful ones:

  • idf.py set-target <target> sets the target (chip) for which the project is built. See Selecting the Target.

  • idf.py menuconfig runs the “menuconfig” tool to configure the project.

  • idf.py build will build the project found in the current directory. This can involve multiple steps:

    • Create the build directory if needed. The sub-directory build is used to hold build output, although this can be changed with the -B option.

    • Run CMake as necessary to configure the project and generate build files for the main build tool.

    • Run the main build tool (Ninja or GNU Make). By default, the build tool is automatically detected but it can be explicitly set by passing the -G option to idf.py.

    Building is incremental so if no source files or configuration has changed since the last build, nothing will be done.

  • idf.py clean will “clean” the project by deleting build output files from the build directory, forcing a “full rebuild” the next time the project is built. Cleaning doesn’t delete CMake configuration output and some other files.

  • idf.py fullclean will delete the entire “build” directory contents. This includes all CMake configuration output. The next time the project is built, CMake will configure it from scratch. Note that this option recursively deletes all files in the build directory, so use with care. Project configuration is not deleted.

  • idf.py flash will automatically build the project if necessary, and then flash it to the target. The -p and -b options can be used to set serial port name and flasher baud rate, respectively.

  • idf.py monitor will display serial output from the target. The -p option can be used to set the serial port name. Type Ctrl-] to exit the monitor. See IDF Monitor for more details about using the monitor.

Multiple idf.py commands can be combined into one. For example, idf.py -p COM4 clean flash monitor will clean the source tree, then build the project and flash it to the target before running the serial monitor.

For commands that are not known to idf.py an attempt to execute them as a build system target will be made.

Note

The environment variables ESPPORT and ESPBAUD can be used to set default values for the -p and -b options, respectively. Providing these options on the command line overrides the default.

Advanced Commands

  • idf.py app, idf.py bootloader, idf.py partition_table can be used to build only the app, bootloader, or partition table from the project as applicable.

  • There are matching commands idf.py app-flash, etc. to flash only that single part of the project to the target.

  • idf.py -p PORT erase_flash will use esptool.py to erase the target’s entire flash chip.

  • idf.py size prints some size information about the app. size-components and size-files are similar commands which print more detailed per-component or per-source-file information, respectively. If you define variable -DOUTPUT_JSON=1 when running CMake (or idf.py), the output will be formatted as JSON not as human readable text.

  • idf.py reconfigure re-runs CMake even if it doesn’t seem to need re-running. This isn’t necessary during normal usage, but can be useful after adding/removing files from the source tree, or when modifying CMake cache variables. For example, idf.py -DNAME='VALUE' reconfigure can be used to set variable NAME in CMake cache to value VALUE.

  • idf.py python-clean deletes generated Python byte code from the IDF directory which may cause issues when switching between IDF and Python versions. It is advised to run this target after switching versions of Python.

The order of multiple idf.py commands on the same invocation is not important, they will automatically be executed in the correct order for everything to take effect (ie building before flashing, erasing before flashing, etc.).

idf.py options

To list all available root level options, run idf.py --help. To list options that are specific for a subcommand, run idf.py <command> --help, for example idf.py monitor --help. Here is a list of some useful options:

  • -C <dir> allows overriding the project directory from the default current working directory.

  • -B <dir> allows overriding the build directory from the default build subdirectory of the project directory.

  • --ccache flag can be used to enable CCache when compiling source files, if the CCache tool is installed. This can dramatically reduce some build times.

Note that some older versions of CCache may exhibit bugs on some platforms, so if files are not rebuilt as expected then try disabling ccache and build again. CCache can be enabled by default by setting the IDF_CCACHE_ENABLE environment variable to a non-zero value. - -v flag causes both idf.py and the build system to produce verbose build output. This can be useful for debugging build problems. - --cmake-warn-uninitialized (or -w) will cause CMake to print uninitialized variable warnings inside the project directory (not for directories not found inside the project directory). This only controls CMake variable warnings inside CMake itself, not other types of build warnings. This option can also be set permanently by setting the IDF_CMAKE_WARN_UNINITIALIZED environment variable to a non-zero value.

Using CMake Directly

idf.py is a wrapper around CMake for convenience. However, you can also invoke CMake directly if you prefer.

When idf.py does something, it prints each command that it runs for easy reference. For example, the idf.py build command is the same as running these commands in a bash shell (or similar commands for Windows Command Prompt):

mkdir -p build
cd build
cmake .. -G Ninja   # or 'Unix Makefiles'
ninja

In the above list, the cmake command configures the project and generates build files for use with the final build tool. In this case the final build tool is Ninja: running ninja actually builds the project.

It’s not necessary to run cmake more than once. After the first build, you only need to run ninja each time. ninja will automatically re-invoke cmake if the project needs reconfiguration.

If using CMake with ninja or make, there are also targets for more of the idf.py sub-commands - for example running make menuconfig or ninja menuconfig in the build directory will work the same as idf.py menuconfig.

Note

If you’re already familiar with CMake, you may find the ESP-IDF CMake-based build system unusual because it wraps a lot of CMake’s functionality to reduce boilerplate. See writing pure CMake components for some information about writing more “CMake style” components.

Flashing with ninja or make

It’s possible to build and flash directly from ninja or make by running a target like:

ninja flash

Or:

make app-flash

Available targets are: flash, app-flash (app only), bootloader-flash (bootloader only).

When flashing this way, optionally set the ESPPORT and ESPBAUD environment variables to specify the serial port and baud rate. You can set environment variables in your operating system or IDE project. Alternatively, set them directly on the command line:

ESPPORT=/dev/ttyUSB0 ninja flash

Note

Providing environment variables at the start of the command like this is Bash shell Syntax. It will work on Linux and macOS. It won’t work when using Windows Command Prompt, but it will work when using Bash-like shells on Windows.

Or:

make -j3 app-flash ESPPORT=COM4 ESPBAUD=2000000

Note

Providing variables at the end of the command line is make syntax, and works for make on all platforms.

Using CMake in an IDE

You can also use an IDE with CMake integration. The IDE will want to know the path to the project’s CMakeLists.txt file. IDEs with CMake integration often provide their own build tools (CMake calls these “generators”) to build the source files as part of the IDE.

When adding custom non-build steps like “flash” to the IDE, it is recommended to execute idf.py for these “special” commands.

For more detailed information about integrating ESP-IDF with CMake into an IDE, see Build System Metadata.

Setting up the Python Interpreter

ESP-IDF works well with all supported Python versions. It should work out-of-box even if you have a legacy system where the default python interpreter is still Python 2.7, however, it is advised to switch to Python 3 if possible.

idf.py and other Python scripts will run with the default Python interpreter, i.e. python. You can switch to a different one like python3 $IDF_PATH/tools/idf.py ..., or you can set up a shell alias or another script to simplify the command.

If using CMake directly, running cmake -D PYTHON=python3 ... will cause CMake to override the default Python interpreter.

If using an IDE with CMake, setting the PYTHON value as a CMake cache override in the IDE UI will override the default Python interpreter.

To manage the Python version more generally via the command line, check out the tools pyenv or virtualenv. These let you change the default Python version.

Possible issues

The user of idf.py may sometimes experience ImportError described below.

Traceback (most recent call last):
  File "/Users/user_name/e/esp-idf/tools/kconfig_new/confgen.py", line 27, in <module>
    import kconfiglib
ImportError: bad magic number in 'kconfiglib': b'\x03\xf3\r\n'

The exception is often caused by .pyc files generated by different Python versions. To solve the issue run the following command:

idf.py python-clean

Example Project

An example project directory tree might look like this:

- myProject/
             - CMakeLists.txt
             - sdkconfig
             - components/ - component1/ - CMakeLists.txt
                                         - Kconfig
                                         - src1.c
                           - component2/ - CMakeLists.txt
                                         - Kconfig
                                         - src1.c
                                         - include/ - component2.h
             - main/       - CMakeLists.txt
                           - src1.c
                           - src2.c

             - build/

This example “myProject” contains the following elements:

  • A top-level project CMakeLists.txt file. This is the primary file which CMake uses to learn how to build the project; and may set project-wide CMake variables. It includes the file /tools/cmake/project.cmake which implements the rest of the build system. Finally, it sets the project name and defines the project.

  • “sdkconfig” project configuration file. This file is created/updated when idf.py menuconfig runs, and holds configuration for all of the components in the project (including ESP-IDF itself). The “sdkconfig” file may or may not be added to the source control system of the project.

  • Optional “components” directory contains components that are part of the project. A project does not have to contain custom components of this kind, but it can be useful for structuring reusable code or including third party components that aren’t part of ESP-IDF. Alternatively, EXTRA_COMPONENT_DIRS can be set in the top-level CMakeLists.txt to look for components in other places. See the renaming main section for more info. If you have a lot of source files in your project, we recommend grouping most into components instead of putting them all in “main”.

  • “main” directory is a special component that contains source code for the project itself. “main” is a default name, the CMake variable COMPONENT_DIRS includes this component but you can modify this variable.

  • “build” directory is where build output is created. This directory is created by idf.py if it doesn’t already exist. CMake configures the project and generates interim build files in this directory. Then, after the main build process is run, this directory will also contain interim object files and libraries as well as final binary output files. This directory is usually not added to source control or distributed with the project source code.

Component directories each contain a component CMakeLists.txt file. This file contains variable definitions to control the build process of the component, and its integration into the overall project. See Component CMakeLists Files for more details.

Each component may also include a Kconfig file defining the component configuration options that can be set via menuconfig. Some components may also include Kconfig.projbuild and project_include.cmake files, which are special files for overriding parts of the project.

Project CMakeLists File

Each project has a single top-level CMakeLists.txt file that contains build settings for the entire project. By default, the project CMakeLists can be quite minimal.

Minimal Example CMakeLists

Minimal project:

cmake_minimum_required(VERSION 3.5)
include($ENV{IDF_PATH}/tools/cmake/project.cmake)
project(myProject)

Mandatory Parts

The inclusion of these three lines, in the order shown above, is necessary for every project:

  • cmake_minimum_required(VERSION 3.5) tells CMake the minimum version that is required to build the project. ESP-IDF is designed to work with CMake 3.5 or newer. This line must be the first line in the CMakeLists.txt file.

  • include($ENV{IDF_PATH}/tools/cmake/project.cmake) pulls in the rest of the CMake functionality to configure the project, discover all the components, etc.

  • project(myProject) creates the project itself, and specifies the project name. The project name is used for the final binary output files of the app - ie myProject.elf, myProject.bin. Only one project can be defined per CMakeLists file.

Optional Project Variables

These variables all have default values that can be overridden for custom behaviour. Look in /tools/cmake/project.cmake for all of the implementation details.

  • COMPONENT_DIRS, COMPONENTS_DIRS: Directories to search for components. Defaults to IDF_PATH/components, PROJECT_DIR/components, and EXTRA_COMPONENT_DIRS. Override this variable if you don’t want to search for components in these places.

  • EXTRA_COMPONENT_DIRS, EXTRA_COMPONENTS_DIRS: Optional list of additional directories to search for components. Paths can be relative to the project directory, or absolute.

  • COMPONENTS: A list of component names to build into the project. Defaults to all components found in the COMPONENT_DIRS directories. Use this variable to “trim down” the project for faster build times. Note that any component which “requires” another component via the REQUIRES or PRIV_REQUIRES arguments on component registration will automatically have it added to this list, so the COMPONENTS list can be very short.

Any paths in these variables can be absolute paths, or set relative to the project directory.

To set these variables, use the cmake set command ie set(VARIABLE "VALUE"). The set() commands should be placed after the cmake_minimum(...) line but before the include(...) line.

Renaming main component

The build system provides special treatment to the main component. It is a component that gets automatically added to the build provided that it is in the expected location, PROJECT_DIR/main. All other components in the build are also added as its dependencies, saving the user from hunting down dependencies and providing a build that works right out of the box. Renaming the main component causes the loss of these behind-the-scences heavy lifting, requiring the user to specify the location of the newly renamed component and manually specifying its dependencies. Specifically, the steps to renaming main are as follows:

  1. Rename main directory.

  2. Set EXTRA_COMPONENT_DIRS in the project CMakeLists.txt to include the renamed main directory.

  3. Specify the dependencies in the renamed component’s CMakeLists.txt file via REQUIRES or PRIV_REQUIRES arguments on component registration.

Component CMakeLists Files

Each project contains one or more components. Components can be part of ESP-IDF, part of the project’s own components directory, or added from custom component directories (see above).

A component is any directory in the COMPONENT_DIRS list which contains a CMakeLists.txt file.

Searching for Components

The list of directories in COMPONENT_DIRS is searched for the project’s components. Directories in this list can either be components themselves (ie they contain a CMakeLists.txt file), or they can be top-level directories whose sub-directories are components.

When CMake runs to configure the project, it logs the components included in the build. This list can be useful for debugging the inclusion/exclusion of certain components.

Multiple components with the same name

When ESP-IDF is collecting all the components to compile, it will do this in the order specified by COMPONENT_DIRS; by default, this means ESP-IDF’s internal components first, then the project’s components, and finally any components set in EXTRA_COMPONENT_DIRS. If two or more of these directories contain component sub-directories with the same name, the component in the last place searched is used. This allows, for example, overriding ESP-IDF components with a modified version by copying that component from the ESP-IDF components directory to the project components directory and then modifying it there. If used in this way, the ESP-IDF directory itself can remain untouched.

Minimal Component CMakeLists

The minimal component CMakeLists.txt file simply registers the component to the build system using idf_component_register:

idf_component_register(SRCS "foo.c" "bar.c"
                       INCLUDE_DIRS "include"
                       REQUIRES mbedtls)
  • SRCS is a list of source files (*.c, *.cpp, *.cc, *.S). These source files will be compiled into the component library.

  • INCLUDE_DIRS is a list of directories to add to the global include search path for any component which requires this component, and also the main source files.

  • REQUIRES is not actually required, but it is very often required to declare what other components this component will use. See Component Requirements.

A library with the name of the component will be built and linked into the final app. Directories are usually specified relative to the CMakeLists.txt file itself, although they can be absolute.

There are other arguments that can be passed to idf_component_register. These arguments are discussed here.

See example component requirements and example component CMakeLists for more complete component CMakeLists.txt examples.

Preset Component Variables

The following component-specific variables are available for use inside component CMakeLists, but should not be modified:

  • COMPONENT_DIR: The component directory. Evaluates to the absolute path of the directory containing CMakeLists.txt. The component path cannot contain spaces. This is the same as the CMAKE_CURRENT_SOURCE_DIR variable.

  • COMPONENT_NAME: Name of the component. Same as the name of the component directory.

  • COMPONENT_ALIAS: Alias of the library created internally by the build system for the component.

  • COMPONENT_LIB: Name of the library created internally by the build system for the component.

The following variables are set at the project level, but available for use in component CMakeLists:

  • CONFIG_*: Each value in the project configuration has a corresponding variable available in cmake. All names begin with CONFIG_. More information here.

  • ESP_PLATFORM: Set to 1 when the CMake file is processed within ESP-IDF build system.

Build/Project Variables

The following are some project/build variables that are available as build pro