Build System¶
This document explains the implementation of the ESP-IDF build system and the concept of “components”. Read this document if you want to know how to organize and build a new ESP-IDF project or component.
Note
This document describes the CMake-based build system, which is the default since ESP-IDF V4.0. ESP-IDF also supports a legacy build system based on GNU Make, which was the default before ESP-IDF V4.0.
Overview¶
An ESP-IDF project can be seen as an amalgamation of a number of components. For example, for a webserver that shows the current humidity, there could be:
The ESP-IDF base libraries (libc, ROM bindings, etc)
The Wi-Fi drivers
A TCP/IP stack
The FreeRTOS operating system
A webserver
A driver for the humidity sensor
Main code tying it all together
ESP-IDF makes these components explicit and configurable. To do that, when a project is compiled, the build system will look up all the components in the ESP-IDF directories, the project directories and (optionally) in additional custom component directories. It then allows the user to configure the ESP-IDF project using a text-based menu system to customize each component. After the components in the project are configured, the build system will compile the project.
Concepts¶
A “project” is a directory that contains all the files and configuration to build a single “app” (executable), as well as additional supporting elements such as a partition table, data/filesystem partitions, and a bootloader.
“Project configuration” is held in a single file called
sdkconfig
in the root directory of the project. This configuration file is modified viaidf.py menuconfig
to customise the configuration of the project. A single project contains exactly one project configuration.An “app” is an executable which is built by ESP-IDF. A single project will usually build two apps - a “project app” (the main executable, ie your custom firmware) and a “bootloader app” (the initial bootloader program which launches the project app).
“components” are modular pieces of standalone code which are compiled into static libraries (.a files) and linked into an app. Some are provided by ESP-IDF itself, others may be sourced from other places.
“Target” is the hardware for which an application is built. At the moment, ESP-IDF supports
esp32
,esp32s2
andesp32c3
targets.
Some things are not part of the project:
“ESP-IDF” is not part of the project. Instead it is standalone, and linked to the project via the
IDF_PATH
environment variable which holds the path of theesp-idf
directory. This allows the IDF framework to be decoupled from your project.The toolchain for compilation is not part of the project. The toolchain should be installed in the system command line PATH.
Using the Build System¶
idf.py¶
The idf.py
command line tool provides a front-end for easily managing your project builds. It manages the following tools:
CMake, which configures the project to be built
A command line build tool (either Ninja build or GNU Make)
esptool.py for flashing the target.
The getting started guide contains a brief introduction to how to set up idf.py
to configure, build, and flash projects.
idf.py
should be run in an ESP-IDF “project” directory, ie one containing a CMakeLists.txt
file. Older style projects with a Makefile will not work with idf.py
.
Type idf.py --help
for a list of commands. Here are a summary of the most useful ones:
idf.py set-target <target>
sets the target (chip) for which the project is built. See Selecting the Target.idf.py menuconfig
runs the “menuconfig” tool to configure the project.idf.py build
will build the project found in the current directory. This can involve multiple steps:Create the build directory if needed. The sub-directory
build
is used to hold build output, although this can be changed with the-B
option.Run CMake as necessary to configure the project and generate build files for the main build tool.
Run the main build tool (Ninja or GNU Make). By default, the build tool is automatically detected but it can be explicitly set by passing the
-G
option toidf.py
.
Building is incremental so if no source files or configuration has changed since the last build, nothing will be done.
idf.py clean
will “clean” the project by deleting build output files from the build directory, forcing a “full rebuild” the next time the project is built. Cleaning doesn’t delete CMake configuration output and some other files.idf.py fullclean
will delete the entire “build” directory contents. This includes all CMake configuration output. The next time the project is built, CMake will configure it from scratch. Note that this option recursively deletes all files in the build directory, so use with care. Project configuration is not deleted.idf.py flash
will automatically build the project if necessary, and then flash it to the target. The-p
and-b
options can be used to set serial port name and flasher baud rate, respectively.idf.py monitor
will display serial output from the target. The-p
option can be used to set the serial port name. TypeCtrl-]
to exit the monitor. See IDF Monitor for more details about using the monitor.
Multiple idf.py
commands can be combined into one. For example, idf.py -p COM4 clean flash monitor
will clean the source tree, then build the project and flash it to the target before running the serial monitor.
For commands that are not known to idf.py
an attempt to execute them as a build system target will be made.
The command idf.py
supports shell autocompletion for bash, zsh and fish shells.
In order to make shell autocompletion supported, please make sure you have at least Python 3.5 and click 7.1 or newer (see also).
To enable autocompletion for idf.py
use the export
command (see this). Autocompletion is initiated by pressing the TAB key. Type “idf.py -” and press the TAB key to autocomplete options.
The autocomplete support for PowerShell is planned in the future.
Note
The environment variables ESPPORT
and ESPBAUD
can be used to set default values for the -p
and -b
options, respectively. Providing these options on the command line overrides the default.
Advanced Commands¶
idf.py app
,idf.py bootloader
,idf.py partition_table
can be used to build only the app, bootloader, or partition table from the project as applicable.There are matching commands
idf.py app-flash
, etc. to flash only that single part of the project to the target.idf.py -p PORT erase_flash
will use esptool.py to erase the target’s entire flash chip.idf.py size
prints some size information about the app.size-components
andsize-files
are similar commands which print more detailed per-component or per-source-file information, respectively. If you define variable-DOUTPUT_JSON=1
when running CMake (oridf.py
), the output will be formatted as JSON not as human readable text.idf.py reconfigure
re-runs CMake even if it doesn’t seem to need re-running. This isn’t necessary during normal usage, but can be useful after adding/removing files from the source tree, or when modifying CMake cache variables. For example,idf.py -DNAME='VALUE' reconfigure
can be used to set variableNAME
in CMake cache to valueVALUE
.idf.py python-clean
deletes generated Python byte code from the IDF directory which may cause issues when switching between IDF and Python versions. It is advised to run this target after switching versions of Python.
The order of multiple idf.py
commands on the same invocation is not important, they will automatically be executed in the correct order for everything to take effect (ie building before flashing, erasing before flashing, etc.).
idf.py options¶
To list all available root level options, run idf.py --help
. To list options that are specific for a subcommand, run idf.py <command> --help
, for example idf.py monitor --help
. Here is a list of some useful options:
-C <dir>
allows overriding the project directory from the default current working directory.-B <dir>
allows overriding the build directory from the defaultbuild
subdirectory of the project directory.--ccache
flag can be used to enable CCache when compiling source files, if the CCache tool is installed. This can dramatically reduce some build times.
Note that some older versions of CCache may exhibit bugs on some platforms, so if files are not rebuilt as expected then try disabling CCache and build again. CCache can be enabled by default by setting the IDF_CCACHE_ENABLE
environment variable to a non-zero value.
-v
flag causes bothidf.py
and the build system to produce verbose build output. This can be useful for debugging build problems.--cmake-warn-uninitialized
(or-w
) will cause CMake to print uninitialized variable warnings inside the project directory (not for directories not found inside the project directory). This only controls CMake variable warnings inside CMake itself, not other types of build warnings. This option can also be set permanently by setting theIDF_CMAKE_WARN_UNINITIALIZED
environment variable to a non-zero value.
Start a new project¶
Use the command idf.py create-project
for starting a new project. Execute idf.py create-project --help
for more information.
Example:
idf.py create-project --path my_projects my_new_project
This example will create a new project called my_new_project directly into the directory my_projects.
Using CMake Directly¶
idf.py is a wrapper around CMake for convenience. However, you can also invoke CMake directly if you prefer.
When idf.py
does something, it prints each command that it runs for easy reference. For example, the idf.py build
command is the same as running these commands in a bash shell (or similar commands for Windows Command Prompt):
mkdir -p build
cd build
cmake .. -G Ninja # or 'Unix Makefiles'
ninja
In the above list, the cmake
command configures the project and generates build files for use with the final build tool. In this case the final build tool is Ninja: running ninja
actually builds the project.
It’s not necessary to run cmake
more than once. After the first build, you only need to run ninja
each time. ninja
will automatically re-invoke cmake
if the project needs reconfiguration.
If using CMake with ninja
or make
, there are also targets for more of the idf.py
sub-commands - for example running make menuconfig
or ninja menuconfig
in the build directory will work the same as idf.py menuconfig
.
Note
If you’re already familiar with CMake, you may find the ESP-IDF CMake-based build system unusual because it wraps a lot of CMake’s functionality to reduce boilerplate. See writing pure CMake components for some information about writing more “CMake style” components.
Flashing with ninja or make¶
It’s possible to build and flash directly from ninja or make by running a target like:
ninja flash
Or:
make app-flash
Available targets are: flash
, app-flash
(app only), bootloader-flash
(bootloader only).
When flashing this way, optionally set the ESPPORT
and ESPBAUD
environment variables to specify the serial port and baud rate. You can set environment variables in your operating system or IDE project. Alternatively, set them directly on the command line:
ESPPORT=/dev/ttyUSB0 ninja flash
Note
Providing environment variables at the start of the command like this is Bash shell Syntax. It will work on Linux and macOS. It won’t work when using Windows Command Prompt, but it will work when using Bash-like shells on Windows.
Or:
make -j3 app-flash ESPPORT=COM4 ESPBAUD=2000000
Note
Providing variables at the end of the command line is make
syntax, and works for make
on all platforms.
Using CMake in an IDE¶
You can also use an IDE with CMake integration. The IDE will want to know the path to the project’s CMakeLists.txt
file. IDEs with CMake integration often provide their own build tools (CMake calls these “generators”) to build the source files as part of the IDE.
When adding custom non-build steps like “flash” to the IDE, it is recommended to execute idf.py
for these “special” commands.
For more detailed information about integrating ESP-IDF with CMake into an IDE, see Build System Metadata.
Setting up the Python Interpreter¶
ESP-IDF works well with all supported Python versions. It should work out-of-box even if you have a legacy system where the default python
interpreter is still Python 2.7, however, it is advised to switch to Python 3 if possible.
idf.py
and other Python scripts will run with the default Python interpreter, i.e. python
. You can switch to a different one like python3 $IDF_PATH/tools/idf.py ...
, or you can set up a shell alias or another script to simplify the command.
If using CMake directly, running cmake -D PYTHON=python3 ...
will cause CMake to override the default Python interpreter.
If using an IDE with CMake, setting the PYTHON
value as a CMake cache override in the IDE UI will override the default Python interpreter.
To manage the Python version more generally via the command line, check out the tools pyenv or virtualenv. These let you change the default Python version.
Possible issues¶
The user of idf.py
may sometimes experience ImportError
described below.
Traceback (most recent call last):
File "/Users/user_name/e/esp-idf/tools/kconfig_new/confgen.py", line 27, in <module>
import kconfiglib
ImportError: bad magic number in 'kconfiglib': b'\x03\xf3\r\n'
The exception is often caused by .pyc
files generated by different Python versions. To solve the issue run the following command:
idf.py python-clean
Example Project¶
An example project directory tree might look like this:
- myProject/
- CMakeLists.txt
- sdkconfig
- components/ - component1/ - CMakeLists.txt
- Kconfig
- src1.c
- component2/ - CMakeLists.txt
- Kconfig
- src1.c
- include/ - component2.h
- main/ - CMakeLists.txt
- src1.c
- src2.c
- build/
This example “myProject” contains the following elements:
A top-level project CMakeLists.txt file. This is the primary file which CMake uses to learn how to build the project; and may set project-wide CMake variables. It includes the file /tools/cmake/project.cmake which implements the rest of the build system. Finally, it sets the project name and defines the project.
“sdkconfig” project configuration file. This file is created/updated when
idf.py menuconfig
runs, and holds configuration for all of the components in the project (including ESP-IDF itself). The “sdkconfig” file may or may not be added to the source control system of the project.Optional “components” directory contains components that are part of the project. A project does not have to contain custom components of this kind, but it can be useful for structuring reusable code or including third party components that aren’t part of ESP-IDF. Alternatively,
EXTRA_COMPONENT_DIRS
can be set in the top-level CMakeLists.txt to look for components in other places. See the renaming main section for more info. If you have a lot of source files in your project, we recommend grouping most into components instead of putting them all in “main”.“main” directory is a special component that contains source code for the project itself. “main” is a default name, the CMake variable
COMPONENT_DIRS
includes this component but you can modify this variable.“build” directory is where build output is created. This directory is created by
idf.py
if it doesn’t already exist. CMake configures the project and generates interim build files in this directory. Then, after the main build process is run, this directory will also contain interim object files and libraries as well as final binary output files. This directory is usually not added to source control or distributed with the project source code.
Component directories each contain a component CMakeLists.txt
file. This file contains variable definitions to control the build process of the component, and its integration into the overall project. See Component CMakeLists Files for more details.
Each component may also include a Kconfig
file defining the component configuration options that can be set via menuconfig
. Some components may also include Kconfig.projbuild
and project_include.cmake
files, which are special files for overriding parts of the project.
Project CMakeLists File¶
Each project has a single top-level CMakeLists.txt
file that contains build settings for the entire project. By default, the project CMakeLists can be quite minimal.
Minimal Example CMakeLists¶
Minimal project:
cmake_minimum_required(VERSION 3.5)
include($ENV{IDF_PATH}/tools/cmake/project.cmake)
project(myProject)
Mandatory Parts¶
The inclusion of these three lines, in the order shown above, is necessary for every project:
cmake_minimum_required(VERSION 3.5)
tells CMake the minimum version that is required to build the project. ESP-IDF is designed to work with CMake 3.5 or newer. This line must be the first line in the CMakeLists.txt file.include($ENV{IDF_PATH}/tools/cmake/project.cmake)
pulls in the rest of the CMake functionality to configure the project, discover all the components, etc.project(myProject)
creates the project itself, and specifies the project name. The project name is used for the final binary output files of the app - iemyProject.elf
,myProject.bin
. Only one project can be defined per CMakeLists file.
Optional Project Variables¶
These variables all have default values that can be overridden for custom behaviour. Look in /tools/cmake/project.cmake for all of the implementation details.
COMPONENT_DIRS
: Directories to search for components. Defaults toIDF_PATH/components
,PROJECT_DIR/components
, andEXTRA_COMPONENT_DIRS
. Override this variable if you don’t want to search for components in these places.EXTRA_COMPONENT_DIRS
: Optional list of additional directories to search for components. Paths can be relative to the project directory, or absolute.COMPONENTS
: A list of component names to build into the project. Defaults to all components found in theCOMPONENT_DIRS
directories. Use this variable to “trim down” the project for faster build times. Note that any component which “requires” another component via the REQUIRES or PRIV_REQUIRES arguments on component registration will automatically have it added to this list, so theCOMPONENTS
list can be very short.
Any paths in these variables can be absolute paths, or set relative to the project directory.
To set these variables, use the cmake set command ie set(VARIABLE "VALUE")
. The set()
commands should be placed after the cmake_minimum(...)
line but before the include(...)
line.
Renaming main
component¶
The build system provides special treatment to the main
component. It is a component that gets automatically added to the build provided that it is in the expected location, PROJECT_DIR/main. All other components in the build are also added as its dependencies, saving the user from hunting down dependencies and providing a build that works right out of the box. Renaming the main
component causes the loss of these behind-the-scenes heavy lifting, requiring the user to specify the location of the newly renamed component and manually specifying its dependencies. Specifically, the steps to renaming main
are as follows:
Rename
main
directory.Set
EXTRA_COMPONENT_DIRS
in the project CMakeLists.txt to include the renamedmain
directory.Specify the dependencies in the renamed component’s CMakeLists.txt file via REQUIRES or PRIV_REQUIRES arguments on component registration.
Component CMakeLists Files¶
Each project contains one or more components. Components can be part of ESP-IDF, part of the project’s own components directory, or added from custom component directories (see above).
A component is any directory in the COMPONENT_DIRS
list which contains a CMakeLists.txt
file.
Searching for Components¶
The list of directories in COMPONENT_DIRS
is searched for the project’s components. Directories in this list can either be components themselves (ie they contain a CMakeLists.txt file), or they can be top-level directories whose sub-directories are components.
When CMake runs to configure the project, it logs the components included in the build. This list can be useful for debugging the inclusion/exclusion of certain components.
Multiple components with the same name¶
When ESP-IDF is collecting all the components to compile, it will do this in the order specified by COMPONENT_DIRS
; by default, this means ESP-IDF’s internal components first, then the project’s components, and finally any components set in EXTRA_COMPONENT_DIRS
. If two or more of these directories contain component sub-directories with the same name, the component in the last place searched is used. This allows, for example, overriding ESP-IDF components with a modified version by copying that component from the ESP-IDF components directory to the project components directory and then modifying it there. If used in this way, the ESP-IDF directory itself can remain untouched.
Note
If a component is overridden in an existing project by moving it to a new location, the project will not automatically see the new component path. Run idf.py reconfigure
(or delete the project build folder) and then build again.
Minimal Component CMakeLists¶
The minimal component CMakeLists.txt
file simply registers the component to the build system using idf_component_register
:
idf_component_register(SRCS "foo.c" "bar.c"
INCLUDE_DIRS "include"
REQUIRES mbedtls)
SRCS
is a list of source files (*.c
,*.cpp
,*.cc
,*.S
). These source files will be compiled into the component library.INCLUDE_DIRS
is a list of directories to add to the global include search path for any component which requires this component, and also the main source files.REQUIRES
is not actually required, but it is very often required to declare what other components this component will use. See Component Requirements.
A library with the name of the component will be built and linked into the final app.
Directories are usually specified relative to the CMakeLists.txt
file itself, although they can be absolute.
There are other arguments that can be passed to idf_component_register
. These arguments are discussed here.
See example component requirements and example component CMakeLists for more complete component CMakeLists.txt
examples.
Create a new component¶
Use the command idf.py create-component
for creating a new component. The new component will contain set of files necessary for building a component. You may include the component’s header file into your project and use its functionality. For more information execute idf.py create-component --help
.
Example:
idf.py -C components create-component my_component
The example will create a new component in the subdirectory components under the current working directory. For more information about components follow the documentation page see above.
Preset Component Variables¶
The following component-specific variables are available for use inside component CMakeLists, but should not be modified:
COMPONENT_DIR
: The component directory. Evaluates to the absolute path of the directory containingCMakeLists.txt
. The component path cannot contain spaces. This is the same as theCMAKE_CURRENT_SOURCE_DIR
variable.COMPONENT_NAME
: Name of the component. Same as the name of the component directory.COMPONENT_ALIAS
: Alias of the library created internally by the build system for the component.COMPONENT_LIB
: Name of the library created internally by the build system for the component.
The following variables are set at the project level, but available for use in component CMakeLists:
CONFIG_*
: Each value in the project configuration has a corresponding variable available in cmake. All names begin withCONFIG_
. More information here.ESP_PLATFORM
: Set to 1 when the CMake file is processed within ESP-IDF build system.
Build/Project Variables¶
The following are some project/build variables that are available as build properties and whose values can be queried using idf_build_get_property
from the component CMakeLists.txt:
PROJECT_NAME
: Name of the project, as set in project CMakeLists.txt file.PROJECT_DIR
: Absolute path of the project directory containing the project CMakeLists. Same as theCMAKE_SOURCE_DIR
variable.COMPONENTS
: Names of all components that are included in this build, formatted as a semicolon-delimited CMake list.IDF_VER
: Git version of ESP-IDF (produced bygit describe
)IDF_VERSION_MAJOR
,IDF_VERSION_MINOR
,IDF_VERSION_PATCH
: Components of ESP-IDF version, to be used in conditional expressions. Note that this information is less precise than that provided byIDF_VER
variable.v4.0-dev-*
,v4.0-beta1
,v4.0-rc1
andv4.0
will all have the same values ofIDF_VERSION_*
variables, but differentIDF_VER
values.IDF_TARGET
: Name of the target for which the project is being built.PROJECT_VER
: Project version.If CONFIG_APP_PROJECT_VER_FROM_CONFIG option is set, the value of CONFIG_APP_PROJECT_VER will be used.
Else, if
PROJECT_VER
variable is set in project CMakeLists.txt file, its value will be used.Else, if the
PROJECT_DIR/version.txt
exists, its contents will be used asPROJECT_VER
.Else, if the project is located inside a Git repository, the output of git describe will be used.
Otherwise,
PROJECT_VER
will be “1”.
Other build properties are listed here.
Controlling Component Compilation¶
To pass compiler options when compiling source files belonging to a particular component, use the target_compile_options
function:
target_compile_options(${COMPONENT_LIB} PRIVATE -Wno-unused-variable)
To apply the compilation flags to a single source file, use the CMake set_source_files_properties command:
set_source_files_properties(mysrc.c
PROPERTIES COMPILE_FLAGS
-Wno-unused-variable
)
This can be useful if there is upstream code that emits warnings.
When using these commands, place them after the call to idf_component_register
in the component CMakeLists file.
Component Configuration¶
Each component can also have a Kconfig
file, alongside CMakeLists.txt
. This contains configuration settings to add to the configuration menu for this component.
These settings are found under the “Component Settings” menu when menuconfig is run.
To create a component Kconfig file, it is easiest to start with one of the Kconfig files distributed with ESP-IDF.
For an example, see Adding conditional configuration.
Preprocessor Definitions¶
The ESP-IDF build system adds the following C preprocessor definitions on the command line:
ESP_PLATFORM
: Can be used to detect that build happens within ESP-IDF.IDF_VER
: Defined to a git version string. E.g.v2.0
for a tagged release orv1.0-275-g0efaa4f
for an arbitrary commit.
Component Requirements¶
When compiling each component, the ESP-IDF build system recursively evaluates its dependencies. This means each component needs to declare the components that it depends on (“requires”).
When writing a component¶
idf_component_register(...
REQUIRES mbedtls
PRIV_REQUIRES console spiffs)
REQUIRES
should be set to all components whose header files are #included from the public header files of this component.PRIV_REQUIRES
should be set to all components whose header files are #included from any source files in this component, unless already listed inREQUIRES
. Also any component which is required to be linked in order for this component to function correctly.The values of
REQUIRES
andPRIV_REQUIRES
should not depend on any configuration choices (CONFIG_xxx
macros). This is because requirements are expanded before configuration is loaded. Other component variables (like include paths or source files) can depend on configuration choices.Not setting either or both
REQUIRES
variables is fine. If the component has no requirements except for the Common component requirements needed for RTOS, libc, etc.
If a components only supports some target chips (values of IDF_TARGET
) then it can specify REQUIRED_IDF_TARGETS
in the idf_component_register
call to express these requirements. In this case the build system will generate an error if the component is included into the build, but does not support the selected target.
Note
In CMake terms, REQUIRES
& PRIV_REQUIRES
are approximate wrappers around the CMake functions target_link_libraries(... PUBLIC ...)
and target_link_libraries(... PRIVATE ...)
.
Example of component requirements¶
Imagine there is a car
component, which uses the engine
component, which uses the spark_plug
component:
- autoProject/
- CMakeLists.txt
- components/ - car/ - CMakeLists.txt
- car.c
- car.h
- engine/ - CMakeLists.txt
- engine.c
- include/ - engine.h
- spark_plug/ - CMakeLists.txt
- plug.c
- plug.h
Car component¶
The car.h
header file is the public interface for the car
component. This header includes engine.h
directly because it uses some declarations from this header:
/* car.h */
#include "engine.h"
#ifdef ENGINE_IS_HYBRID
#define CAR_MODEL "Hybrid"
#endif
And car.c includes car.h
as well:
/* car.c */
#include "car.h"
This means the car/CMakeLists.txt
file needs to declare that car
requires engine
:
idf_component_register(SRCS "car.c"
INCLUDE_DIRS "."
REQUIRES engine)
SRCS
gives the list of source files in thecar
component.INCLUDE_DIRS
gives the list of public include directories for this component. Because the public interface iscar.h
, the directory containingcar.h
is listed here.REQUIRES
gives the list of components required by the public interface of this component. Becausecar.h
is a public header and includes a header fromengine
, we includeengine
here. This makes sure that any other component which includescar.h
will be able to recursively include the requiredengine.h
also.
Engine component¶
The engine
component also has a public header file include/engine.h
, but this header is simpler:
/* engine.h */
#define ENGINE_IS_HYBRID
void engine_start(void);
The implementation is in engine.c
:
/* engine.c */
#include "engine.h"
#include "spark_plug.h"
...
In this component, engine
depends on spark_plug
but this is a private dependency. spark_plug.h
is needed to compile engine.c
, but not needed to include engine.h
.
This means that the engine/CMakeLists.txt
file can use PRIV_REQUIRES
:
idf_component_register(SRCS "engine.c"
INCLUDE_DIRS "include"
PRIV_REQUIRES spark_plug)
As a result, source files in the car
component don’t need the spark_plug
include directories added to their compiler search path. This can speed up compilation, and stops compiler command lines from becoming longer than necessary.
Spark Plug Component¶
The spark_plug
component doesn’t depend on anything else. It has a public header file spark_plug.h
, but this doesn’t include headers from any other components.
This means that the spark_plug/CMakeLists.txt
file doesn’t need any REQUIRES
or PRIV_REQUIRES
clauses:
idf_component_register(SRCS "spark_plug.c"
INCLUDE_DIRS ".")
Source File Include Directories¶
Each component’s source file is compiled with these include path directories, as specified in the passed arguments to idf_component_register
:
idf_component_register(..
INCLUDE_DIRS "include"
PRIV_INCLUDE_DIRS "other")
The current component’s
INCLUDE_DIRS
andPRIV_INCLUDE_DIRS
.The
INCLUDE_DIRS
belonging to all other components listed in theREQUIRES
andPRIV_REQUIRES
parameters (ie all the current component’s public and private dependencies).Recursively, all of the
INCLUDE_DIRS
of those componentsREQUIRES
lists (ie all public dependencies of this component’s dependencies, recursively expanded).
Main component requirements¶
The component named main
is special because it automatically requires all other components in the build. So it’s not necessary to pass REQUIRES
or PRIV_REQUIRES
to this component. See renaming main for a description of what needs to be changed if no longer using the main
component.
Common component requirements¶
To avoid duplication, every component automatically requires some “common” IDF components even if they are not mentioned explicitly. Headers from these components can always be included.
The list of common components is: freertos, newlib, heap, log, soc, esp_rom, esp_common, xtensa/riscv, cxx.
Including components in the build¶
By default, every component is included in the build.
If you set the
COMPONENTS
variable to a minimal list of components used directly by your project, then the build will expand to also include required components. The full list of components will be:Components mentioned explicitly in
COMPONENTS
.Those components’ requirements (evaluated recursively).
The “common” components that every component depends on.
Setting
COMPONENTS
to the minimal list of required components can significantly reduce compile times.
Requirements in the build system implementation¶
Very early in the CMake configuration process, the script
expand_requirements.cmake
is run. This script does a partial evaluation of all component CMakeLists.txt files and builds a graph of component requirements (this graph may have cycles). The graph is used to generate a filecomponent_depends.cmake
in the build directory.The main CMake process then includes this file and uses it to determine the list of components to include in the build (internal
BUILD_COMPONENTS
variable). TheBUILD_COMPONENTS
variable is sorted so dependencies are listed first, however as the component dependency graph has cycles this cannot be guaranteed for all components. The order should be deterministic given the same set of components and component dependencies.The value of
BUILD_COMPONENTS
is logged by CMake as “Component names: “Configuration is then evaluated for the components included in the build.
Each component is included in the build normally and the CMakeLists.txt file is evaluated again to add the component libraries to the build.
Component Dependency Order¶
The order of components in the BUILD_COMPONENTS
variable determines other orderings during the build:
Order that project_include.cmake files are included into the project.
Order that the list of header paths is generated for compilation (via
-I
argument). (Note that for a given component’s source files, only that component’s dependency’s header paths are passed to the compiler.)
Overriding Parts of the Project¶
project_include.cmake¶
For components that have build requirements which must be evaluated before any component CMakeLists files are evaluated, you can create a file called project_include.cmake
in the component directory. This CMake file is included when project.cmake
is evaluating the entire project.
project_include.cmake
files are used inside ESP-IDF, for defining project-wide build features such as esptool.py
command line arguments and the bootloader
“special app”.
Unlike component CMakeLists.txt
files, when including a project_include.cmake
file the current source directory (CMAKE_CURRENT_SOURCE_DIR
and working directory) is the project directory. Use the variable COMPONENT_DIR
for the absolute directory of the component.
Note that project_include.cmake
isn’t necessary for the most common component uses - such as adding include directories to the project, or LDFLAGS
to the final linking step. These values can be customised via the CMakeLists.txt
file itself. See Optional Project Variables for details.
project_include.cmake
files are included in the order given in BUILD_COMPONENTS
variable (as logged by CMake). This means that a component’s project_include.cmake
file will be included after it’s all dependencies’ project_include.cmake
files, unless both components are part of a dependency cycle. This is important if a project_include.cmake
file relies on variables set by another component. See also above.
Take great care when setting variables or targets in a project_include.cmake
file. As the values are included into the top-level project CMake pass, they can influence or break functionality across all components!
KConfig.projbuild¶
This is an equivalent to project_include.cmake
for Component Configuration KConfig files. If you want to include configuration options at the top-level of menuconfig, rather than inside the “Component Configuration” sub-menu, then these can be defined in the KConfig.projbuild file alongside the CMakeLists.txt
file.
Take care when adding configuration values in this file, as they will be included across the entire project configuration. Where possible, it’s generally better to create a KConfig file for Component Configuration.
project_include.cmake
files are used inside ESP-IDF, for defining project-wide build features such as esptool.py
command line arguments and the bootloader
“special app”.
Configuration-Only Components¶
Special components which contain no source files, only Kconfig.projbuild
and KConfig
, can have a one-line CMakeLists.txt
file which calls the function idf_component_register()
with no arguments specified. This function will include the component in the project build, but no library will be built and no header files will be added to any include paths.
Debugging CMake¶
For full details about CMake and CMake commands, see the CMake v3.5 documentation.
Some tips for debugging the ESP-IDF CMake-based build system:
When CMake runs, it prints quite a lot of diagnostic information including lists of components and component paths.
Running
cmake -DDEBUG=1
will produce more verbose diagnostic output from the IDF build system.Running
cmake
with the--trace
or--trace-expand
options will give a lot of information about control flow. See the cmake command line documentation.
When included from a project CMakeLists file, the project.cmake
file defines some utility modules and global variables and then sets IDF_PATH
if it was not set in the system environment.
It also defines an overridden custom version of the built-in CMake project
function. This function is overridden to add all of the ESP-IDF specific project functionality.
Warning On Undefined Variables¶
By default, idf.py
passes the --warn-uninitialized
flag to CMake so it will print a warning if an undefined variable is referenced in the build. This can be very useful to find buggy CMake files.
If you don’t want this behaviour, it can be disabled by passing --no-warnings
to idf.py
.
Browse the /tools/cmake/project.cmake file and supporting functions in /tools/cmake/ for more details.
Example Component CMakeLists¶
Because the build environment tries to set reasonable defaults that will work most
of the time, component CMakeLists.txt
can be very small or even empty (see Minimal Component CMakeLists). However, overriding component variables is usually required for some functionality.
Here are some more advanced examples of component CMakeLists files.
Adding conditional configuration¶
The configuration system can be used to conditionally compile some files depending on the options selected in the project configuration.
Kconfig
:
config FOO_ENABLE_BAR
bool "Enable the BAR feature."
help
This enables the BAR feature of the FOO component.
CMakeLists.txt
:
set(srcs "foo.c" "more_foo.c")
if(CONFIG_FOO_ENABLE_BAR)
list(APPEND srcs "bar.c")
endif()
idf_component_register(SRCS "${srcs}"
...)
This example makes use of the CMake if function and list APPEND function.
This can also be used to select or stub out an implementation, as such:
Kconfig
:
config ENABLE_LCD_OUTPUT
bool "Enable LCD output."
help
Select this if your board has a LCD.
config ENABLE_LCD_CONSOLE
bool "Output console text to LCD"
depends on ENABLE_LCD_OUTPUT
help
Select this to output debugging output to the lcd
config ENABLE_LCD_PLOT
bool "Output temperature plots to LCD"
depends on ENABLE_LCD_OUTPUT
help
Select this to output temperature plots
CMakeLists.txt
:
if(CONFIG_ENABLE_LCD_OUTPUT)
set(srcs lcd-real.c lcd-spi.c)
else()
set(srcs lcd-dummy.c)
endif()
# We need font if either console or plot is enabled
if(CONFIG_ENABLE_LCD_CONSOLE OR CONFIG_ENABLE_LCD_PLOT)
list(APPEND srcs "font.c")
endif()
idf_component_register(SRCS "${srcs}"
...)
Conditions which depend on the target¶
The current target is available to CMake files via IDF_TARGET
variable.
In addition to that, if target xyz
is used (IDF_TARGET=xyz
), then Kconfig variable CONFIG_IDF_TARGET_XYZ
will be set.
Note that component dependencies may depend on IDF_TARGET
variable, but not on Kconfig variables. Also one can not use Kconfig variables in include
statements in CMake files, but IDF_TARGET
can be used in such context.
Source Code Generation¶
Some components will have a situation where a source file isn’t supplied with the component itself but has to be generated from another file. Say our component has a header file that consists of the converted binary data of a BMP file, converted using a hypothetical tool called bmp2h. The header file is then included in as C source file called graphics_lib.c:
add_custom_command(OUTPUT logo.h
COMMAND bmp2h -i ${COMPONENT_DIR}/logo.bmp -o log.h
DEPENDS ${COMPONENT_DIR}/logo.bmp
VERBATIM)
add_custom_target(logo DEPENDS logo.h)
add_dependencies(${COMPONENT_LIB} logo)
set_property(DIRECTORY "${COMPONENT_DIR}" APPEND PROPERTY
ADDITIONAL_MAKE_CLEAN_FILES logo.h)
This answer is adapted from the CMake FAQ entry, which contains some other examples that will also work with ESP-IDF builds.
In this example, logo.h will be generated in the current directory (the build directory) while logo.bmp comes with the component and resides under the component path. Because logo.h is a generated file, it should be cleaned when the project is cleaned. For this reason it is added to the ADDITIONAL_MAKE_CLEAN_FILES property.
Note
If generating files as part of the project CMakeLists.txt file, not a component CMakeLists.txt, then use build property PROJECT_DIR
instead of ${COMPONENT_DIR}
and ${PROJECT_NAME}.elf
instead of ${COMPONENT_LIB}
.)
If a a source file from another component included logo.h
, then add_dependencies
would need to be called to add a dependency between the two components, to ensure that the component source files were always compiled in the correct order.
Embedding Binary Data¶
Sometimes you have a file with some binary or text data that you’d like to make available to your component - but you don’t want to reformat the file as C source.
You can specify argument EMBED_FILES
in the component registration, giving space-delimited names of the files to embed:
idf_component_register(...
EMBED_FILES server_root_cert.der)
Or if the file is a string, you can use the variable EMBED_TXTFILES
. This will embed the contents of the text file as a null-terminated string:
idf_component_register(...
EMBED_TXTFILES server_root_cert.pem)
The file’s contents will be added to the .rodata section in flash, and are available via symbol names as follows:
extern const uint8_t server_root_cert_pem_start[] asm("_binary_server_root_cert_pem_start");
extern const uint8_t server_root_cert_pem_end[] asm("_binary_server_root_cert_pem_end");
The names are generated from the full name of the file, as given in EMBED_FILES
. Characters /, ., etc. are replaced with underscores. The _binary prefix in the symbol name is added by objcopy and is the same for both text and binary files.
To embed a file into a project, rather than a component, you can call the function target_add_binary_data
like this:
target_add_binary_data(myproject.elf "main/data.bin" TEXT)
Place this line after the project()
line in your project CMakeLists.txt file. Replace myproject.elf
with your project name. The final argument can be TEXT
to embed a null-terminated string, or BINARY
to embed the content as-is.
For an example of using this technique, see the “main” component of the file_serving example protocols/http_server/file_serving/main/CMakeLists.txt - two files are loaded at build time and linked into the firmware.
It is also possible embed a generated file:
add_custom_command(OUTPUT my_processed_file.bin
COMMAND my_process_file_cmd my_unprocessed_file.bin)
target_add_binary_data(my_target "my_processed_file.bin" BINARY)
In the example above, my_processed_file.bin
is generated from my_unprocessed_file.bin
through some command my_process_file_cmd
, then embedded into the target.
To specify a dependence on a target, use the DEPENDS
argument:
add_custom_target(my_process COMMAND ...)
target_add_binary_data(my_target "my_embed_file.bin" BINARY DEPENDS my_process)
The DEPENDS
argument to target_add_binary_data
ensures that the target executes first.
Code and Data Placements¶
ESP-IDF has a feature called linker script generation that enables components to define where its code and data will be placed in memory through linker fragment files. These files are processed by the build system, and is used to augment the linker script used for linking app binary. See Linker Script Generation for a quick start guide as well as a detailed discussion of the mechanism.
Fully Overriding The Component Build Process¶
Obviously, there are cases where all these recipes are insufficient for a certain component, for example when the component is basically a wrapper around another third-party component not originally intended to be compiled under this build system. In that case, it’s possible to forego the ESP-IDF build system entirely by using a CMake feature called ExternalProject. Example component CMakeLists:
# External build process for quirc, runs in source dir and
# produces libquirc.a
externalproject_add(quirc_build
PREFIX ${COMPONENT_DIR}
SOURCE_DIR ${COMPONENT_DIR}/quirc
CONFIGURE_COMMAND ""
BUILD_IN_SOURCE 1
BUILD_COMMAND make CC=${CMAKE_C_COMPILER} libquirc.a
INSTALL_COMMAND ""
)
# Add libquirc.a to the build process
add_library(quirc STATIC IMPORTED GLOBAL)
add_dependencies(quirc quirc_build)
set_target_properties(quirc PROPERTIES IMPORTED_LOCATION
${COMPONENT_DIR}/quirc/libquirc.a)
set_target_properties(quirc PROPERTIES INTERFACE_INCLUDE_DIRECTORIES
${COMPONENT_DIR}/quirc/lib)
set_directory_properties( PROPERTIES ADDITIONAL_MAKE_CLEAN_FILES
"${COMPONENT_DIR}/quirc/libquirc.a")
(The above CMakeLists.txt can be used to create a component named quirc
that builds the quirc project using its own Makefile.)
externalproject_add
defines an external build system.SOURCE_DIR
,CONFIGURE_COMMAND
,BUILD_COMMAND
andINSTALL_COMMAND
should always be set.CONFIGURE_COMMAND
can be set to an empty string if the build system has no “configure” step.INSTALL_COMMAND
will generally be empty for ESP-IDF builds.Setting
BUILD_IN_SOURCE
means the build directory is the same as the source directory. Otherwise you can setBUILD_DIR
.Consult the ExternalProject documentation for more details about
externalproject_add()
The second set of commands adds a library target, which points to the “imported” library file built by the external system. Some properties need to be set in order to add include directories and tell CMake where this file is.
Finally, the generated library is added to ADDITIONAL_MAKE_CLEAN_FILES. This means
make clean
will delete this library. (Note that the other object files from the build won’t be deleted.)
Note
When using an external build process with PSRAM, remember to add -mfix-esp32-psram-cache-issue
to the C compiler arguments. See CONFIG_SPIRAM_CACHE_WORKAROUND for details of this flag.
ExternalProject dependencies, clean builds¶
CMake has some unusual behaviour around external project builds:
ADDITIONAL_MAKE_CLEAN_FILES only works when “make” is used as the build system. If Ninja or an IDE build system is used, it won’t delete these files when cleaning.
However, the ExternalProject configure & build commands will always be re-run after a clean is run.
Therefore, there are two alternative recommended ways to configure the external build command:
Have the external
BUILD_COMMAND
run a full clean compile of all sources. The build command will be run if any of the dependencies passed toexternalproject_add
withDEPENDS
have changed, or if this is a clean build (ie any ofidf.py clean
,ninja clean
, ormake clean
was run.)Have the external
BUILD_COMMAND
be an incremental build command. Pass the parameterBUILD_ALWAYS 1
toexternalproject_add
. This means the external project will be built each time a build is run, regardless of dependencies. This is only recommended if the external project has correct incremental build behaviour, and doesn’t take too long to run.
The best of these approaches for building an external project will depend on the project itself, its build system, and whether you anticipate needing to frequently recompile the project.
Custom sdkconfig defaults¶
For example projects or other projects where you don’t want to specify a full sdkconfig configuration, but you do want to override some key values from the ESP-IDF defaults, it is possible to create a file sdkconfig.defaults
in the project directory. This file will be used when creating a new config from scratch, or when any new config value hasn’t yet been set in the sdkconfig
file.
To override the name of this file or to specify multiple files, set the SDKCONFIG_DEFAULTS
environment variable or set SDKCONFIG_DEFAULTS
in top-level CMakeLists.txt. If specifying multiple files, use semicolon as the list separator. File names not specified as full paths are resolved relative to current project.
Target-dependent sdkconfig defaults¶
In addition to sdkconfig.defaults
file, build system will also load defaults from sdkconfig.defaults.TARGET_NAME
file, where TARGET_NAME
is the value of IDF_TARGET
. For example, for esp32
target, default settings will be taken from sdkconfig.defaults
first, and then from sdkconfig.defaults.esp32
.
If SDKCONFIG_DEFAULTS
is used to override the name of defaults file/files, the name of target-specific defaults file will be derived from SDKCONFIG_DEFAULTS
value/values using the rule above.
Flash arguments¶
There are some scenarios that we want to flash the target board without IDF. For this case we want to save the built binaries, esptool.py and esptool write_flash arguments. It’s simple to write a script to save binaries and esptool.py.
After running a project build, the build directory contains binary output files (.bin
files) for the project and also the following flashing data files:
flash_project_args
contains arguments to flash the entire project (app, bootloader, partition table, PHY data if this is configured).flash_app_args
contains arguments to flash only the app.flash_bootloader_args
contains arguments to flash only the bootloader.
You can pass any of these flasher argument files to esptool.py
as follows:
python esptool.py --chip esp32 write_flash @build/flash_project_args
Alternatively, it is possible to manually copy the parameters from the argument file and pass them on the command line.
The build directory also contains a generated file flasher_args.json
which contains project flash information, in JSON format. This file is used by idf.py
and can also be used by other tools which need information about the project build.
Building the Bootloader¶
The bootloader is built by default as part of idf.py build
, or can be built standalone via idf.py bootloader
.
The bootloader is a special “subproject” inside /components/bootloader/subproject. It has its own project CMakeLists.txt file and builds separate .ELF and .BIN files to the main project. However it shares its configuration and build directory with the main project.
The subproject is inserted as an external project from the top-level project, by the file /components/bootloader/project_include.cmake. The main build process runs CMake for the subproject, which includes discovering components (a subset of the main components) and generating a bootloader-specific config (derived from the main sdkconfig
).
Selecting the Target¶
ESP-IDF supports multiple targets (chips). The identifiers used for each chip are as follows:
esp32
— for ESP32-D0WD, ESP32-D2WD, ESP32-S0WD (ESP-SOLO), ESP32-U4WDH, ESP32-PICO-D4esp32s2
— for ESP32-S2esp32c3
— for ESP32-C3
To select the target before building the project, use idf.py set-target <target>
command, for example:
idf.py set-target esp32s2
Important
idf.py set-target
will clear the build directory and re-generate the sdkconfig
file from scratch. The old sdkconfig
file will be saved as sdkconfig.old
.
Note
The behavior of idf.py set-target
command is equivalent to:
clearing the build directory (
idf.py fullclean
)removing the sdkconfig file (
mv sdkconfig sdkconfig.old
)configuring the project with the new target (
idf.py -DIDF_TARGET=esp32 reconfigure
)
It is also possible to pass the desired IDF_TARGET
as an environment variable (e.g. export IDF_TARGET=esp32s2
) or as a CMake variable (e.g. -DIDF_TARGET=esp32s2
argument to CMake or idf.py). Setting the environment variable is a convenient method if you mostly work with one type of the chip.
To specify the _default_ value of IDF_TARGET
for a given project, add CONFIG_IDF_TARGET
value to sdkconfig.defaults
. For example, CONFIG_IDF_TARGET="esp32s2"
. This value will be used if IDF_TARGET
is not specified by other method: using an environment variable, CMake variable, or idf.py set-target
command.
If the target has not been set by any of these methods, the build system will default to esp32
target.
Writing Pure CMake Components¶
The ESP-IDF build system “wraps” CMake with the concept of “components”, and helper functions to automatically integrate these components into a project build.
However, underneath the concept of “components” is a full CMake build system. It is also possible to make a component which is pure CMake.
Here is an example minimal “pure CMake” component CMakeLists file for a component named json
:
add_library(json STATIC
cJSON/cJSON.c
cJSON/cJSON_Utils.c)
target_include_directories(json PUBLIC cJSON)
This is actually an equivalent declaration to the IDF
json
component /components/json/CMakeLists.txt.This file is quite simple as there are not a lot of source files. For components with a large number of files, the globbing behaviour of ESP-IDF’s component logic can make the component CMakeLists style simpler.)
Any time a component adds a library target with the component name, the ESP-IDF build system will automatically add this to the build, expose public include directories, etc. If a component wants to add a library target with a different name, dependencies will need to be added manually via CMake commands.
Using Third-Party CMake Projects with Components¶
CMake is used for a lot of open-source C and C++ projects — code that users can tap into for their applications. One of the benefits of having a CMake build system is the ability to import these third-party projects, sometimes even without modification! This allows for users to be able to get functionality that may not yet be provided by a component, or use another library for the same functionality.
Importing a library might look like this for a hypothetical library foo
to be used in the main
component:
# Register the component
idf_component_register(...)
# Set values of hypothetical variables that control the build of `foo`
set(FOO_BUILD_STATIC OFF)
set(FOO_BUILD_TESTS OFF)
# Create and import the library targets
add_subdirectory(foo)
# Publicly link `foo` to `main` component
target_link_libraries(main PUBLIC foo)
For an actual example, take a look at build_system/cmake/import_lib. Take note that what needs to be done in order to import the library may vary. It is recommended to read up on the library’s documentation for instructions on how to import it from other projects. Studying the library’s CMakeLists.txt and build structure can also be helpful.
It is also possible to wrap a third-party library to be used as a component in this manner. For example, the mbedtls component is a wrapper for Espressif’s fork of mbedtls. See its component CMakeLists.txt .
The CMake variable ESP_PLATFORM
is set to 1 whenever the ESP-IDF build system is being used. Tests such as if (ESP_PLATFORM)
can be used in generic CMake code if special IDF-specific logic is required.
Using ESP-IDF components from external libraries¶
The above example assumes that the external library foo
(or tinyxml
in the case of the import_lib
example) doesn’t need to use any ESP-IDF APIs apart from common APIs such as libc, libstdc++, etc. If the external library needs to use APIs provided by other ESP-IDF components, this needs to be specified in the external CMakeLists.txt file by adding a dependency on the library target idf::<componentname>
.
For example, in the foo/CMakeLists.txt
file:
add_library(foo bar.c fizz.cpp buzz.cpp)
if(ESP_PLATFORM)
# On ESP-IDF, bar.c needs to include esp_spi_flash.h from the spi_flash component
target_link_libraries(foo PRIVATE idf::spi_flash)
endif()
Using Prebuilt Libraries with Components¶
Another possibility is that you have a prebuilt static library (.a
file), built by some other build process.
The ESP-IDF build system provides a utility function add_prebuilt_library
for users to be able to easily import and use prebuilt libraries:
add_prebuilt_library(target_name lib_path [REQUIRES req1 req2 ...] [PRIV_REQUIRES req1 req2 ...])
where:
target_name
- name that can be used to reference the imported library, such as when linking to other targetslib_path
- path to prebuilt library; may be an absolute or relative path to the component directory
Optional arguments REQUIRES
and PRIV_REQUIRES
specify dependency on other components. These have the same meaning as the arguments for idf_component_register
.
Take note that the prebuilt library must have been compiled for the same target as the consuming project. Configuration relevant to the prebuilt library must also match. If not paid attention to, these two factors may contribute to subtle bugs in the app.
For an example, take a look at build_system/cmake/import_prebuilt.
Using ESP-IDF in Custom CMake Projects¶
ESP-IDF provides a template CMake project for easily creating an application. However, in some instances the user might already have an existing CMake project or may want to create a custom one. In these cases it is desirable to be able to consume IDF components as libraries to be linked to the user’s targets (libraries/ executables).
It is possible to do so by using the build system APIs provided by tools/cmake/idf.cmake. For example:
cmake_minimum_required(VERSION 3.5)
project(my_custom_app C)
# Include CMake file that provides ESP-IDF CMake build system APIs.
include($ENV{IDF_PATH}/tools/cmake/idf.cmake)
# Include ESP-IDF components in the build, may be thought as an equivalent of
# add_subdirectory() but with some additional processing and magic for ESP-IDF build
# specific build processes.
idf_build_process(esp32)
# Create the project executable and plainly link the newlib component to it using
# its alias, idf::newlib.
add_executable(${CMAKE_PROJECT_NAME}.elf main.c)
target_link_libraries(${CMAKE_PROJECT_NAME}.elf idf::newlib)
# Let the build system know what the project executable is to attach more targets, dependencies, etc.
idf_build_executable(${CMAKE_PROJECT_NAME}.elf)
The example in build_system/cmake/idf_as_lib demonstrates the creation of an application equivalent to hello world application using a custom CMake project.
Note
The IDF build system can only set compiler flags for source files that it builds. When an external CMakeLists.txt file is used and PSRAM is enabled, remember to add -mfix-esp32-psram-cache-issue
to the C compiler arguments. See CONFIG_SPIRAM_CACHE_WORKAROUND for details of this flag.
ESP-IDF CMake Build System API¶
idf-build-commands¶
idf_build_get_property(var property [GENERATOR_EXPRESSION])
Retrieve a build property property and store it in var accessible from the current scope. Specifying GENERATOR_EXPRESSION will retrieve the generator expression string for that property, instead of the actual value, which can be used with CMake commands that support generator expressions.
idf_build_set_property(property val [APPEND])
Set a build property property with value val. Specifying APPEND will append the specified value to the current value of the property. If the property does not previously exist or it is currently empty, the specified value becomes the first element/member instead.
idf_build_component(component_dir)
Present a directory component_dir that contains a component to the build system. Relative paths are converted to absolute paths with respect to current directory. All calls to this command must be performed before idf_build_process.
This command does not guarantee that the component will be processed during build (see the COMPONENTS argument description for idf_build_process)
idf_build_process(target
[PROJECT_DIR project_dir]
[PROJECT_VER project_ver]
[PROJECT_NAME project_name]
[SDKCONFIG sdkconfig]
[SDKCONFIG_DEFAULTS sdkconfig_defaults]
[BUILD_DIR build_dir]
[COMPONENTS component1 component2 ...])
Performs the bulk of the behind-the-scenes magic for including ESP-IDF components such as component configuration, libraries creation, dependency expansion and resolution. Among these functions, perhaps the most important from a user’s perspective is the libraries creation by calling each component’s idf_component_register
. This command creates the libraries for each component, which are accessible using aliases in the form idf::component_name.
These aliases can be used to link the components to the user’s own targets, either libraries or executables.
The call requires the target chip to be specified with target argument. Optional arguments for the call include:
PROJECT_DIR - directory of the project; defaults to CMAKE_SOURCE_DIR
PROJECT_NAME - name of the project; defaults to CMAKE_PROJECT_NAME
PROJECT_VER - version/revision of the project; defaults to “1”
SDKCONFIG - output path of generated sdkconfig file; defaults to PROJECT_DIR/sdkconfig or CMAKE_SOURCE_DIR/sdkconfig depending if PROJECT_DIR is set
SDKCONFIG_DEFAULTS - list of files containing default config to use in the build (list must contain full paths); defaults to empty. For each value filename in the list, the config from file filename.target, if it exists, is also loaded.
BUILD_DIR - directory to place ESP-IDF build-related artifacts, such as generated binaries, text files, components; defaults to CMAKE_BINARY_DIR
COMPONENTS - select components to process among the components known by the build system (added via idf_build_component). This argument is used to trim the build. Other components are automatically added if they are required in the dependency chain, i.e. the public and private requirements of the components in this list are automatically added, and in turn the public and private requirements of those requirements, so on and so forth. If not specified, all components known to the build system are processed.
idf_build_executable(executable)
Specify the executable executable for ESP-IDF build. This attaches additional targets such as dependencies related to flashing, generating additional binary files, etc. Should be called after idf_build_process
.
idf_build_get_config(var config [GENERATOR_EXPRESSION])
Get the value of the specified config. Much like build properties, specifying GENERATOR_EXPRESSION will retrieve the generator expression string for that config, instead of the actual value, which can be used with CMake commands that support generator expressions. Actual config values are only known after call to idf_build_process
, however.
idf-build-properties¶
These are properties that describe the build. Values of build properties can be retrieved by using the build command idf_build_get_property
. For example, to get the Python interpreter used for the build:
idf_build_get_property(python PYTHON)
message(STATUS "The Python intepreter is: ${python}")
BUILD_DIR - build directory; set from
idf_build_process
BUILD_DIR argumentBUILD_COMPONENTS - list of components included in the build; set by
idf_build_process
BUILD_COMPONENT_ALIASES - list of library alias of components included in the build; set by
idf_build_process
C_COMPILE_OPTIONS - compile options applied to all components’ C source files
COMPILE_OPTIONS - compile options applied to all components’ source files, regardless of it being C or C++
COMPILE_DEFINITIONS - compile definitions applied to all component source files
CXX_COMPILE_OPTIONS - compile options applied to all components’ C++ source files
EXECUTABLE - project executable; set by call to
idf_build_executable
EXECUTABLE_NAME - name of project executable without extension; set by call to
idf_build_executable
EXECUTABLE_DIR - path containing the output executable
IDF_COMPONENT_MANAGER - the component manager is enabled by default, but if this property is set to
0
it was disabled by the IDF_COMPONENT_MANAGER environment variableIDF_PATH - ESP-IDF path; set from IDF_PATH environment variable, if not, inferred from the location of
idf.cmake
IDF_TARGET - target chip for the build; set from the required target argument for
idf_build_process
IDF_VER - ESP-IDF version; set from either a version file or the Git revision of the IDF_PATH repository
INCLUDE_DIRECTORIES - include directories for all component source files
KCONFIGS - list of Kconfig files found in components in build; set by
idf_build_process
KCONFIG_PROJBUILDS - list of Kconfig.projbuild files found in components in build; set by
idf_build_process
PROJECT_NAME - name of the project; set from
idf_build_process
PROJECT_NAME argumentPROJECT_DIR - directory of the project; set from
idf_build_process
PROJECT_DIR argumentPROJECT_VER - version of the project; set from
idf_build_process
PROJECT_VER argumentPYTHON - Python interpreter used for the build; set from PYTHON environment variable if available, if not “python” is used
SDKCONFIG - full path to output config file; set from
idf_build_process
SDKCONFIG argumentSDKCONFIG_DEFAULTS - list of files containing default config to use in the build; set from
idf_build_process
SDKCONFIG_DEFAULTS argumentSDKCONFIG_HEADER - full path to C/C++ header file containing component configuration; set by
idf_build_process
SDKCONFIG_CMAKE - full path to CMake file containing component configuration; set by
idf_build_process
SDKCONFIG_JSON - full path to JSON file containing component configuration; set by
idf_build_process
SDKCONFIG_JSON_MENUS - full path to JSON file containing config menus; set by
idf_build_process
idf-component-commands¶
idf_component_get_property(var component property [GENERATOR_EXPRESSION])
Retrieve a specified component’s component property, property and store it in var accessible from the current scope. Specifying GENERATOR_EXPRESSION will retrieve the generator expression string for that property, instead of the actual value, which can be used with CMake commands that support generator expressions.
idf_component_set_property(component property val [APPEND])
Set a specified component’s component property, property with value val. Specifying APPEND will append the specified value to the current value of the property. If the property does not previously exist or it is currently empty, the specified value becomes the first element/member instead.
idf_component_register([[SRCS src1 src2 ...] | [[SRC_DIRS dir1 dir2 ...] [EXCLUDE_SRCS src1 src2 ...]]
[INCLUDE_DIRS dir1 dir2 ...]
[PRIV_INCLUDE_DIRS dir1 dir2 ...]
[REQUIRES component1 component2 ...]
[PRIV_REQUIRES component1 component2 ...]
[LDFRAGMENTS ldfragment1 ldfragment2 ...]
[REQUIRED_IDF_TARGETS target1 target2 ...]
[EMBED_FILES file1 file2 ...]
[EMBED_TXTFILES file1 file2 ...]
[KCONFIG kconfig]
[KCONFIG_PROJBUILD kconfig_projbuild])
Register a component to the build system. Much like the project()
CMake command, this should be called from the component’s CMakeLists.txt directly (not through a function or macro) and is recommended to be called before any other command. Here are some guidelines on what commands can not be called before idf_component_register
:
commands that are not valid in CMake script mode
custom commands defined in project_include.cmake
build system API commands except
idf_build_get_property
; although consider whether the property may not have been set yet
Commands that set and operate on variables are generally okay to call before idf_component_register
.
The arguments for idf_component_register
include:
SRCS - component source files used for creating a static library for the component; if not specified, component is a treated as a config-only component and an interface library is created instead.
SRC_DIRS, EXCLUDE_SRCS - used to glob source files (.c, .cpp, .S) by specifying directories, instead of specifying source files manually via SRCS. Note that this is subject to the limitations of globbing in CMake. Source files specified in EXCLUDE_SRCS are removed from the globbed files.
INCLUDE_DIRS - paths, relative to the component directory, which will be added to the include search path for all other components which require the current component
PRIV_INCLUDE_DIRS - directory paths, must be relative to the component directory, which will be added to the include search path for this component’s source files only
REQUIRES - public component requirements for the component
PRIV_REQUIRES - private component requirements for the component; ignored on config-only components
LDFRAGMENTS - component linker fragment files
REQUIRED_IDF_TARGETS - specify the only target the component supports
KCONFIG - override the default Kconfig file
KCONFIG_PROJBUILD - override the default Kconfig.projbuild file
The following are used for embedding data into the component, and is considered as source files when determining if a component is config-only. This means that even if the component does not specify source files, a static library is still created internally for the component if it specifies either:
EMBED_FILES - binary files to be embedded in the component
EMBED_TXTFILES - text files to be embedded in the component
idf-component-properties¶
These are properties that describe a component. Values of component properties can be retrieved by using the build command idf_component_get_property
. For example, to get the directory of the freertos
component:
idf_component_get_property(dir freertos COMPONENT_DIR)
message(STATUS "The 'freertos' component directory is: ${dir}")
COMPONENT_ALIAS - alias for COMPONENT_LIB used for linking the component to external targets; set by
idf_build_component
and alias library itself is created byidf_component_register
COMPONENT_DIR - component directory; set by
idf_build_component
COMPONENT_OVERRIDEN_DIR - contains the directory of the original component if this component overrides another component
COMPONENT_LIB - name for created component static/interface library; set by
idf_build_component
and library itself is created byidf_component_register
COMPONENT_NAME - name of the component; set by
idf_build_component
based on the component directory nameCOMPONENT_TYPE - type of the component, whether LIBRARY or CONFIG_ONLY. A component is of type LIBRARY if it specifies source files or embeds a file
EMBED_FILES - list of files to embed in component; set from
idf_component_register
EMBED_FILES argumentEMBED_TXTFILES - list of text files to embed in component; set from
idf_component_register
EMBED_TXTFILES argumentINCLUDE_DIRS - list of component include directories; set from
idf_component_register
INCLUDE_DIRS argumentKCONFIG - component Kconfig file; set by
idf_build_component
KCONFIG_PROJBUILD - component Kconfig.projbuild; set by
idf_build_component
LDFRAGMENTS - list of component linker fragment files; set from
idf_component_register
LDFRAGMENTS argumentPRIV_INCLUDE_DIRS - list of component private include directories; set from
idf_component_register
PRIV_INCLUDE_DIRS on components of type LIBRARYPRIV_REQUIRES - list of private component dependentices; set from
idf_component_register
PRIV_REQUIRES argumentREQUIRED_IDF_TARGETS - list of targets the component supports; set from
idf_component_register
EMBED_TXTFILES argumentREQUIRES - list of public component dependencies; set from
idf_component_register
REQUIRES argumentSRCS - list of component source files; set from SRCS or SRC_DIRS/EXCLUDE_SRCS argument of
idf_component_register
File Globbing & Incremental Builds¶
The preferred way to include source files in an ESP-IDF component is to list them manually via SRCS argument to idf_component_register
:
idf_component_register(SRCS library/a.c library/b.c platform/platform.c
...)
This preference reflects the CMake best practice of manually listing source files. This could, however, be inconvenient when there are lots of source files to add to the build. The ESP-IDF build system provides an alternative way for specifying source files using SRC_DIRS
:
idf_component_register(SRC_DIRS library platform
...)
This uses globbing behind the scenes to find source files in the specified directories. Be aware, however, that if a new source file is added and this method is used, then CMake won’t know to automatically re-run and this file won’t be added to the build.
The trade-off is acceptable when you’re adding the file yourself, because you can trigger a clean build or run idf.py reconfigure
to manually re-run CMake. However, the problem gets harder when you share your project with others who may check out a new version using a source control tool like Git…
For components which are part of ESP-IDF, we use a third party Git CMake integration module (/tools/cmake/third_party/GetGitRevisionDescription.cmake) which automatically re-runs CMake any time the repository commit changes. This means if you check out a new ESP-IDF version, CMake will automatically rerun.
For project components (not part of ESP-IDF), there are a few different options:
If keeping your project file in Git, ESP-IDF will automatically track the Git revision and re-run CMake if the revision changes.
If some components are kept in a third git repository (not the project repository or ESP-IDF repository), you can add a call to the
git_describe
function in a component CMakeLists file in order to automatically trigger re-runs of CMake when the Git revision changes.If not using Git, remember to manually run
idf.py reconfigure
whenever a source file may change.To avoid this problem entirely, use
SRCS
argument toidf_component_register
to list all source files in project components.
The best option will depend on your particular project and its users.
Build System Metadata¶
For integration into IDEs and other build systems, when CMake runs the build process generates a number of metadata files in the build/
directory. To regenerate these files, run cmake
or idf.py reconfigure
(or any other idf.py
build command).
compile_commands.json
is a standard format JSON file which describes every source file which is compiled in the project. A CMake feature generates this file, and many IDEs know how to parse it.project_description.json
contains some general information about the ESP-IDF project, configured paths, etc.flasher_args.json
contains esptool.py arguments to flash the project’s binary files. There are alsoflash_*_args
files which can be used directly with esptool.py. See Flash arguments.CMakeCache.txt
is the CMake cache file which contains other information about the CMake process, toolchain, etc.config/sdkconfig.json
is a JSON-formatted version of the project configuration values.config/kconfig_menus.json
is a JSON-formatted version of the menus shown in menuconfig, for use in external IDE UIs.
JSON Configuration Server¶
A tool called confserver.py
is provided to allow IDEs to easily integrate with the configuration system logic. confserver.py
is designed to run in the background and interact with a calling process by reading and writing JSON over process stdin & stdout.
You can run confserver.py
from a project via idf.py confserver
or ninja confserver
, or a similar target triggered from a different build generator.
For more information about confserver.py
, see tools/kconfig_new/README.md.
Build System Internals¶
Build Scripts¶
The listfiles for the ESP-IDF build system reside in /tools/cmake. The modules which implement core build system functionality are as follows:
build.cmake - Build related commands i.e. build initialization, retrieving/setting build properties, build processing.
component.cmake - Component related commands i.e. adding components, retrieving/setting component properties, registering components.
kconfig.cmake - Generation of configuration files (sdkconfig, sdkconfig.h, sdkconfig.cmake, etc.) from Kconfig files.
ldgen.cmake - Generation of final linker script from linker fragment files.
target.cmake - Setting build target and toolchain file.
utilities.cmake - Miscellaneous helper commands.
Aside from these files, there are two other important CMake scripts in /tools/cmake:
idf.cmake - Sets up the build and includes the core modules listed above. Included in CMake projects in order to access ESP-IDF build system functionality.
project.cmake - Includes
idf.cmake
and provides a customproject()
command that takes care of all the heavy lifting of building an executable. Included in the top-level CMakeLists.txt of standard ESP-IDF projects.
The rest of the files in /tools/cmake are support or third-party scripts used in the build process.
Build Process¶
This section describes the standard ESP-IDF application build process. The build process can be broken down roughly into four phases:
Initialization¶
This phase sets up necessary parameters for the build.
- Upon inclusion of
idf.cmake
inproject.cmake
, the following steps are performed:
Set
IDF_PATH
from environment variable or inferred from path toproject.cmake
included in the top-level CMakeLists.txt.Add /tools/cmake to
CMAKE_MODULE_PATH
and include core modules plus the various helper/third-party scripts.Set build tools/executables such as default Python interpreter.
Get ESP-IDF git revision and store as
IDF_VER
.Set global build specifications i.e. compile options, compile definitions, include directories for all components in the build.
Add components in components to the build.
- The initial part of the custom
project()
command performs the following steps:
Set
IDF_TARGET
from environment variable or CMake cache and the correspondingCMAKE_TOOLCHAIN_FILE
to be used.Add components in
EXTRA_COMPONENTS_DIRS
to the build.Prepare arguments for calling command
idf_build_process()
from variables such asCOMPONENTS
/EXCLUDE_COMPONENTS
,SDKCONFIG
,SDKCONFIG_DEFAULTS
.The call to
idf_build_process()
command marks the end of this phase.
Enumeration¶
This phase builds a final list of components to be processed in the build, and is performed in the first half of
idf_build_process()
.
Retrieve each component’s public and private requirements. A child process is created which executes each component’s CMakeLists.txt in script mode. The values of
idf_component_register
REQUIRES and PRIV_REQUIRES argument is returned to the parent build process. This is called early expansion. The variableCMAKE_BUILD_EARLY_EXPANSION
is defined during this step.Recursively include components based on public and private requirements.
Processing¶
This phase processes the components in the build, and is the second half of
idf_build_process()
.
Load project configuration from sdkconfig file and generate an sdkconfig.cmake and sdkconfig.h header. These define configuration variables/macros that are accessible from the build scripts and C/C++ source/header files, respectively.
Include each component’s
project_include.cmake
.Add each component as a subdirectory, processing its CMakeLists.txt. The component CMakeLists.txt calls the registration command,
idf_component_register
which adds source files, include directories, creates component library, links dependencies, etc.
Finalization¶
This phase is everything after
idf_build_process()
.
Create executable and link the component libraries to it.
Generate project metadata files such as project_description.json and display relevant information about the project built.
Browse /tools/cmake/project.cmake for more details.
Migrating from ESP-IDF GNU Make System¶
Some aspects of the CMake-based ESP-IDF build system are very similar to the older GNU Make-based system. The developer needs to provide values the include directories, source files etc. There is a syntactical difference, however, as the developer needs to pass these as arguments to the registration command, idf_component_register
.
Automatic Conversion Tool¶
An automatic project conversion tool is available in /tools/cmake/convert_to_cmake.py. Run this command line tool with the path to a project like this:
$IDF_PATH/tools/cmake/convert_to_cmake.py /path/to/project_dir
The project directory must contain a Makefile, and GNU Make (make
) must be installed and available on the PATH.
The tool will convert the project Makefile and any component component.mk
files to their equivalent CMakeLists.txt
files.
It does so by running make
to expand the ESP-IDF build system variables which are set by the build, and then producing equivalent CMakelists files to set the same variables.
Important
When the conversion tool converts a component.mk
file, it doesn’t determine what other components that component depends on. This information needs to be added manually by editing the new component CMakeLists.txt
file and adding REQUIRES
and/or PRIV_REQUIRES
clauses. Otherwise, source files in the component will fail to compile as headers from other components are not found. See Component Requirements.
The conversion tool is not capable of dealing with complex Makefile logic or unusual targets. These will need to be converted by hand.
No Longer Available in CMake¶
Some features are significantly different or removed in the CMake-based system. The following variables no longer exist in the CMake-based build system:
COMPONENT_BUILD_DIR
: UseCMAKE_CURRENT_BINARY_DIR
instead.COMPONENT_LIBRARY
: Defaulted to$(COMPONENT_NAME).a
, but the library name could be overriden by the component. The name of the component library can no longer be overriden by the component.CC
,LD
,AR
,OBJCOPY
: Full paths to each tool from the gcc xtensa cross-toolchain. UseCMAKE_C_COMPILER
,CMAKE_C_LINK_EXECUTABLE
,CMAKE_OBJCOPY
, etc instead. Full list here.HOSTCC
,HOSTLD
,HOSTAR
: Full names of each tool from the host native toolchain. These are no longer provided, external projects should detect any required host toolchain manually.COMPONENT_ADD_LDFLAGS
: Used to override linker flags. Use the CMake target_link_libraries command instead.COMPONENT_ADD_LINKER_DEPS
: List of files that linking should depend on. target_link_libraries will usually infer these dependencies automatically. For linker scripts, use the provided custom CMake functiontarget_linker_scripts
.COMPONENT_SUBMODULES
: No longer used, the build system will automatically enumerate all submodules in the ESP-IDF repository.COMPONENT_EXTRA_INCLUDES
: Used to be an alternative toCOMPONENT_PRIV_INCLUDEDIRS
for absolute paths. UsePRIV_INCLUDE_DIRS
argument toidf_component_register
for all cases now (can be relative or absolute).COMPONENT_OBJS
: Previously, component sources could be specified as a list of object files. Now they can be specified as a list of source files viaSRCS
argument to idf_component_register.COMPONENT_OBJEXCLUDE
: Has been replaced withEXCLUDE_SRCS
argument toidf_component_register
. Specify source files (as absolute paths or relative to component directory), instead.COMPONENT_EXTRA_CLEAN
: Set propertyADDITIONAL_MAKE_CLEAN_FILES
instead but note CMake has some restrictions around this functionality.COMPONENT_OWNBUILDTARGET
&COMPONENT_OWNCLEANTARGET
: Use CMake ExternalProject instead. See Fully Overriding The Component Build Process for full details.COMPONENT_CONFIG_ONLY
: Callidf_component_register
without any arguments instead. See Configuration-Only Components.CFLAGS
,CPPFLAGS
,CXXFLAGS
: Use equivalent CMake commands instead. See Controlling Component Compilation.
No Default Values¶
Unlike in the legacy Make-based build system, the following have no default values:
Source directories (
COMPONENT_SRCDIRS
variable in Make,SRC_DIRS
argument toidf_component_register
in CMake)Include directories (
COMPONENT_ADD_INCLUDEDIRS
variable in Make,INCLUDE_DIRS
argument toidf_component_register
in CMake)
No Longer Necessary¶
In the legacy Make-based build system, it is required to also set
COMPONENT_SRCDIRS
ifCOMPONENT_SRCS
is set. In CMake, the equivalent is not necessary i.e. specifyingSRC_DIRS
toidf_component_register
ifSRCS
is also specified (in fact,SRCS
is ignored ifSRC_DIRS
is specified).
Flashing from make¶
make flash
and similar targets still work to build and flash. However, project sdkconfig
no longer specifies serial port and baud rate. Environment variables can be used to override these. See Flashing with ninja or make for more details.