Dynamic GitLab Pipelines

idf-ci generates GitLab CI as a small pipeline hierarchy.

It has three parts at minimum:

the parent pipeline prepares inputs and generates the build child pipeline YAML
the build child pipeline builds apps and, when needed, generates the test child pipeline YAML
the test child pipeline runs grouped pytest jobs

What the generated flow looks like

        flowchart LR
    subgraph P["Parent pipeline"]
        direction TD
        PV["pipeline_variables"]
        GBCP["generate_build_child_pipeline"]
    end

    subgraph B["Build child pipeline"]
        direction TD
        BTRA["build_test_related_apps\n(parallel when needed)"]
        BNTRA["build_non_test_related_apps\n(parallel when needed)"]
        GTCP["generate_test_child_pipeline"]
    end

    subgraph TCP["Test child pipeline"]
        direction TD
        TJ1["[target] - [env]\n(parallel when needed)"]
        TJ2["... more grouped test jobs ..."]
    end

    PV --> BTRA
    GBCP --> BTRA
    PV --> BNTRA
    GBCP --> BNTRA
    PV --> GTCP
    BTRA --> GTCP
    GTCP --> TCP
    GTCP --> TJ1
    GTCP --> TJ2

Build child pipeline

The build child pipeline is generated by idf_ci.idf_gitlab.pipeline.build_child_pipeline.

It classifies discovered apps into two buckets:

test-related apps: apps associated with pytest-driven testing
non-test-related apps: apps that still need building but do not feed the generated test pipeline

From those buckets, the generated pipeline can contain up to four jobs.

build_test_related_apps: Builds only apps that feed later pytest execution.
build_non_test_related_apps: Builds the remaining selected apps.
generate_test_child_pipeline: Runs only when there are test-related apps. It depends on build_test_related_apps and the parent pipeline_variables job, then produces test_child_pipeline.yml. The direct parent dependency makes exported variables available during test discovery; a dependency on the build job alone does not pass them through.
test-child-pipeline: Trigger job that starts the downstream test child pipeline from test_child_pipeline.yml.

Both build jobs also declare cross-pipeline needs on:

generate_build_child_pipeline
pipeline_variables

This gives each build job the generated YAML context and exported pipeline variables from the parent pipeline.

If no apps are selected, idf-ci emits a single fake_pass job.

Building with multiple toolchains

Configure the toolchains in .idf_ci.toml (GCC is the only implicit default):

[gitlab]
build_enabled_toolchains = ["gcc", "clang"]
test_enabled_toolchains = ["gcc"]

One invocation of idf-ci gitlab build-child-pipeline discovers apps separately for each toolchain and writes build_child_pipeline.yml for GCC and build_child_pipeline_clang.yml for Clang. It also writes separate app lists (test_related_apps.txt / test_related_apps_clang.txt and their non_test_related_apps counterparts). Include all these files in the parent generation job’s artifacts; no second discovery job is required.

Create one parent trigger job for each desired build YAML, each with an include:artifact pointing at the corresponding file from the same generation job. For example:

generate_build_child_pipeline:
  script: idf-ci gitlab build-child-pipeline
  artifacts:
    paths:
      - build_child_pipeline.yml
      - build_child_pipeline_clang.yml
      - test_related_apps.txt
      - test_related_apps_clang.txt
      - non_test_related_apps.txt
      - non_test_related_apps_clang.txt

build-clang:
  needs: [generate_build_child_pipeline]
  variables:
    PARENT_PIPELINE_ID: "$CI_PIPELINE_ID"
  trigger:
    include:
      - artifact: build_child_pipeline_clang.yml
        job: generate_build_child_pipeline
    strategy: depend

The generated build jobs set IDF_TOOLCHAIN and read only their toolchain’s app lists. Non-GCC jobs also use the toolchain as their IDF_CI_ARTIFACT_NAMESPACE to keep S3 artifacts separate; GCC retains the existing artifact namespace behavior (unnamespaced by default). A caller can set IDF_CI_ARTIFACT_NAMESPACE for another build variant of the same toolchain: its app lists then use that namespace as their filename suffix (for example, test_related_apps_buildv2.txt), and build jobs with the same namespace read those lists. Without a namespace, default GCC lists have no suffix and other toolchains use their toolchain name; an explicit namespace takes precedence even when its value is gcc. Keep the namespace consistent between generation, the build trigger, and test jobs. Only toolchains in test_enabled_toolchains create a test generation/trigger pair. When Clang tests are enabled, the Clang build pipeline generates and triggers test_child_pipeline_clang.yml. The test-generation command can also be called directly with idf-ci gitlab test-child-pipeline --toolchain clang.

What the build jobs contain

Generated build jobs share .default_build_settings.

Build work is sharded with GitLab parallel when the selected app count is large enough. The shard size is controlled by gitlab.build_pipeline.runs_per_job.

The result has two independent build branches:

one branch that feeds tests
one branch that exists only to build the rest of the requested app set

Only the test-related branch continues into test pipeline generation.

Test child pipeline

The test child pipeline is generated by idf_ci.idf_gitlab.pipeline.test_child_pipeline.

Instead of creating one job per test case, it groups pytest cases by:

target selector
environment selector
runner tags

Each generated job is named like <target_selector> - <env_selector> and contains:

job tags derived from the grouped test cases
a nodes variable containing the selected pytest node IDs
optional parallel sharding when a group is large enough

All generated test jobs extend .default_test_settings.

Each test job depends on generate_test_child_pipeline so it can download the build artifacts needed for execution.

If no test cases are selected, the generated test child pipeline also falls back to fake_pass.