Find and Build Apps

This chapter mainly explains what’s running behind the two main functions, find_apps and build_apps.

Find Apps

Finding apps is a process that collects all the buildable applications from the specified paths.

To explain the process better, let’s go through an example with two projects test-1 and test-2, under folder /tmp/test/examples. The folder structure looks like this:

/tmp/test/examples
├── test-1
│   ├── CMakeLists.txt
│   ├── main
│   │   ├── CMakeLists.txt
│   │   └── test-1.c
│   ├── sdkconfig.ci.bar
│   ├── sdkconfig.ci.foo
│   ├── sdkconfig.defaults
│   └── sdkconfig.defaults.esp32
└── test-2
    ├── CMakeLists.txt
    └── main
        ├── CMakeLists.txt
        └── test-2.c

The basic command to find all the buildable apps under /tmp/test/examples recursively with target esp32 is:

cd /tmp/test/examples
idf-build-apps find -p . --recursive --target esp32

The output would be:

(cmake) App ./test-1, target esp32, sdkconfig (default), build in ./test-1/build
(cmake) App ./test-2, target esp32, sdkconfig (default), build in ./test-2/build

sdkconfig Files

To customize ESP-IDF projects configurations, developers can run a terminal-based tool idf.py menuconfig interactively, or create sdkconfig.defaults files. For detailed documentation, please refer to ESP-IDF Project Configuration Guide.

Usually in CI, calling idf.py menuconfig interactively is not possible. We use a set of sdkconfig.defaults files instead. Each line of the sdkconfig files is a key-value pair, representing a configuration item. For example, CONFIG_IDF_TARGET_ESP32=y.

The ESP-IDF build system automatically creates thesdkconfig file, which collects all the configuration items, then start the compilation. The build system will first read the settings from the sdkconfig.defaults files, then populate the rest of the settings based on these values, and the default values set in the Kconfig files.

The sdkconfig file is generated in this order:

flowchart TB sdkconfig_defaults(sdkconfig.defaults) sdkconfig_defaults_target(sdkconfig.defaults.TARGET) kconfig(Other Kconfig items) sdkconfig_file(`sdkconfig` file) subgraph pre-set [pre-set Kconfig items] sdkconfig_defaults -- be overriden by --> sdkconfig_defaults_target subgraph "only apply when building with the target" sdkconfig_defaults_target end end pre-set -- configure together with the default value of each Kconfig item --> kconfig -- generates --> sdkconfig_file

Overriding Kconfig Items

idf-build-apps provides CLI options available to globally override kconfig items. Each option is a comma-separated list.

• --override-sdkconfig-items

• --override-sdkconfig-files

Each item is a Kconfig item, as we mentioned earlier. Each file contains multiple lines of Kconfig items, to simplify the CLI usage.

If you use multiple override options together, the sdkconfig file is generated in this order:

flowchart TB sdkconfig_defaults(sdkconfig.defaults) sdkconfig_defaults_target(sdkconfig.defaults.TARGET) kconfig(Other Kconfig items) sdkconfig_file(`sdkconfig` file) override_sdkconfig_items(items defined in --override-sdkconfig-items) override_sdkconfig_files(items defined in --override-sdkconfig-files) subgraph pre-set [pre-set Kconfig items] direction TB sdkconfig_defaults -- be overriden by --> sdkconfig_defaults_target subgraph "only apply when building with the target" sdkconfig_defaults_target end sdkconfig_defaults_target -- be overriden by --> override_sdkconfig_files -- be overriden by --> override_sdkconfig_items end pre-set -- configure together with the default value of each Kconfig item --> kconfig -- generates --> sdkconfig_file

The sequence in each CLI option also matters. The later one would override the previous one. For example,

idf-build-apps find -p test-1 --target esp32 --override-sdkconfig-items CONFIG_A=4,CONFIG_A=5

Will consider CONFIG_A=5, but running with:

idf-build-apps find -p test-1 --target esp32 --override-sdkconfig-items CONFIG_A=5,CONFIG_A=4

Will consider CONFIG_A=4.

Config Rules

In CI, we may want to build the same project with different configurations, to increase our test coverage. idf-build-apps provides a way to do this by using config rules.

Config rule represents a relationship that matches the sdkconfig file pattern and the config name. The syntax is simple: [SDKCONFIG_FILEPATTERN]=[CONFIG_NAME].

• SDKCONFIG_FILEPATTERN: could be a file name, to match a single sdkconfig file, or with one wildcard (*) character, to match multiple sdkconfig files.

• CONFIG_NAME: Name of the corresponding build configuration. or skip setting this value if the value of wildcard is to be used.

For example, in project test-1, the config rules and the corresponding sdkconfig files could be:

Config Rules

Config Rule	Config Name	Explanation	Matched sdkconfig file
=	default	The default value of config name is default
sdkconfig.ci.foo=test	test		sdkconfig.ci.foo
sdkconfig.not_exists=test	default	The config rule doesn’t match any sdkconfig file. Use the default value instead.
sdkconfig.ci.*=	foo bar	The wildcard matches two files. Build two apps based on each sdkconfig file.	sdkconfig.ci.foo sdkconfig.ci.bar

To build the project with the config rules, we could run:

idf-build-apps find -p . --recursive --target esp32 --config [CONFIG_RULE]

For example,

idf-build-apps find -p test-1 --target esp32 --config "sdkconfig.ci.*="

The output is:

(cmake) App test-1, target esp32, sdkconfig sdkconfig.ci.bar, build in test-1/build
(cmake) App test-1, target esp32, sdkconfig sdkconfig.ci.foo, build in test-1/build

You may also use multiple config rules options together:

idf-build-apps find -p test-1 --target esp32 --config "sdkconfig.ci.*=" "sdkconfig.defaults=default"

The output is:

(cmake) App test-1, target esp32, sdkconfig sdkconfig.ci.bar, build in test-1/build
(cmake) App test-1, target esp32, sdkconfig sdkconfig.ci.foo, build in test-1/build
(cmake) App test-1, target esp32, sdkconfig sdkconfig.defaults, build in test-1/build

Note

For each SDKCONFIG_FILEPATTERN, only one wildcard is supported.

Placeholders for Work Directory and Build Directory

As native ESP-IDF does, idf-build-apps builds projects in-place, within the project directory, and generates the binaries under build directory. idf-build-apps also provides a way to customize the work directory and build directory.

Work Directory

Work directory is the directory where the build actually happens. idf-build-apps would first copy the whole project to the work directory, then start the real build process. The benefit of specifying work directory is that you could keep your local build directory and sdkconfig file untouched.

By default, idf-build-apps would use the project directory as the work directory.

Build Directory

Build directory is the directory where the binary files output would be generated. If it is set to a relative path, the full path would be calculated based on the work directory. If it is an absolute path, it would override the work directory settings.

Placeholders

To make the work directory and build directory more flexible, idf-build-apps provides a way to use placeholders in the directory path.

Placeholders are a set of symbols, which could be used when setting work directory and build directory. The placeholders would be replaced while building as follows:

• @t: Would be replaced by the target chip type.

• @w: Would be replaced by the wildcard if exists, otherwise would be replaced by the config name.

• @n: Would be replaced by the project name.

• @f: Would be replaced by the escaped project path (replaced “/” to “_”).

• @v: Would be replaced by the ESP-IDF version like 5_3_0.

• @i: Would be replaced by the build index. (only available in build command)

• @p: Would be replaced by the parallel build index. (default to 1, only available in build command)

For example,

idf-build-apps find -p . --recursive --target esp32 --config "sdkconfig.ci.*=" --build-dir build_@t_@w

The output would be:

(cmake) App ./test-1, target esp32, sdkconfig sdkconfig.ci.bar, build in ./test-1/build_esp32_bar
(cmake) App ./test-1, target esp32, sdkconfig sdkconfig.ci.foo, build in ./test-1/build_esp32_foo
(cmake) App ./test-2, target esp32, sdkconfig (default), build in ./test-2/build_esp32

Another example to set an absolute path with the wildcard symbols as the build directory:

idf-build-apps find -p . --recursive --target esp32 --config "sdkconfig.ci.*=" --build-dir /tmp/build/@n_@t_@w

The output would be:

(cmake) App ./test-1, target esp32, sdkconfig sdkconfig.ci.bar, build in /tmp/build/test-1_esp32_bar
(cmake) App ./test-1, target esp32, sdkconfig sdkconfig.ci.foo, build in /tmp/build/test-1_esp32_foo
(cmake) App ./test-2, target esp32, sdkconfig (default), build in /tmp/build/test-2_esp32

Output in Text File

For find command, you may use --output <file> to output the result to a text file. Each line of the text file represents an app, which is a JSON string that could be deserialized to an App object.

You may reuse the file with python code:

from idf_build_apps import AppDeserializer

with open("output.txt", "r") as f:
    for line in f:
        app = AppDeserializer.from_json(line)

Build Apps

Building apps is a process that build all the applications that are collected by the “find” process.

Note

Almost all CLI options that find supported are also supported in build command. You may call idf-build-apps find -h or idf-build-apps build -h to get a full list of all possible CLI options.

Tips on build CLI Options

Check Build Warnings

You may use --check-warnings to enable this check. If any warning is captured while the building process, the exit code would turn to a non-zero value. Besides, idf-build-apps provides CLI options --ignore-warnings-str and --ignore-warnings-file to let you bypass some false alarms.

Dry Run

It’s useful to call --dry-run with verbose mode -vv to know the whole build process in detail before the build actually happens.

For example:

idf-build-apps build -p . --recursive --target esp32 --dry-run -vv --config "sdkconfig.ci.*="

The output would be:

2023-12-12 13:36:03    DEBUG Looking for CMakeApp apps in . recursively with target esp32
2023-12-12 13:36:03    DEBUG Entering .
2023-12-12 13:36:03    DEBUG Skipping. . is not an app
2023-12-12 13:36:03    DEBUG Entering ./test-1
2023-12-12 13:36:03    DEBUG Use sdkconfig file ./test-1/sdkconfig.defaults
2023-12-12 13:36:03    DEBUG Use sdkconfig file ./test-1/sdkconfig.ci.bar
2023-12-12 13:36:03    DEBUG Found app: (cmake) App ./test-1, target esp32, sdkconfig sdkconfig.ci.bar, build in ./test-1/build
2023-12-12 13:36:03    DEBUG
2023-12-12 13:36:03    DEBUG Use sdkconfig file ./test-1/sdkconfig.defaults
2023-12-12 13:36:03    DEBUG Use sdkconfig file ./test-1/sdkconfig.ci.foo
2023-12-12 13:36:03    DEBUG Found app: (cmake) App ./test-1, target esp32, sdkconfig sdkconfig.ci.foo, build in ./test-1/build
2023-12-12 13:36:03    DEBUG
2023-12-12 13:36:03    DEBUG => Stop iteration sub dirs of ./test-1 since it has apps
2023-12-12 13:36:03    DEBUG Entering ./test-2
2023-12-12 13:36:03    DEBUG sdkconfig file ./test-2/sdkconfig.defaults not exists, skipping...
2023-12-12 13:36:03    DEBUG Found app: (cmake) App ./test-2, target esp32, sdkconfig (default), build in ./test-2/build
2023-12-12 13:36:03    DEBUG
2023-12-12 13:36:03    DEBUG => Stop iteration sub dirs of ./test-2 since it has apps
2023-12-12 13:36:03     INFO Found 3 apps in total
2023-12-12 13:36:03     INFO Total 3 apps. running build for app 1-3
2023-12-12 13:36:03     INFO (1/3) Building app: (cmake) App ./test-1, target esp32, sdkconfig sdkconfig.ci.bar, build in ./test-1/build
2023-12-12 13:36:03     INFO skipped (dry run)
2023-12-12 13:36:03     INFO
2023-12-12 13:36:03     INFO (2/3) Building app: (cmake) App ./test-1, target esp32, sdkconfig sdkconfig.ci.foo, build in ./test-1/build
2023-12-12 13:36:03     INFO skipped (dry run)
2023-12-12 13:36:03     INFO
2023-12-12 13:36:03     INFO (3/3) Building app: (cmake) App ./test-2, target esp32, sdkconfig (default), build in ./test-2/build
2023-12-12 13:36:03     INFO skipped (dry run)
2023-12-12 13:36:03     INFO
Skipped building the following apps:
  (cmake) App ./test-1, target esp32, sdkconfig sdkconfig.ci.bar, build in ./test-1/build, skipped in 0.000176s: dry run
  (cmake) App ./test-1, target esp32, sdkconfig sdkconfig.ci.foo, build in ./test-1/build, skipped in 7.4e-05s: dry run
  (cmake) App ./test-2, target esp32, sdkconfig (default), build in ./test-2/build, skipped in 7.2e-05s: dry run

Logging

All commands provided by idf-build-apps are using the same logger idf_build_apps. By default the logger level is WARNING. You may use -v to increase the logger level to INFO, and -vv to increase the logger level to DEBUG. You may use idf_build_apps.setup_logging to setup the logger level programmatically.

To fully customize the logger, you may get the logger of idf_build_apps with:

import logging

logging.getLogger("idf_build_apps")