Dynamic GitLab Pipelines

idf-ci generates GitLab CI as a small pipeline hierarchy.

It has three parts at minimum:

  • the parent pipeline prepares inputs and generates the build child pipeline YAML

  • the build child pipeline builds apps and, when needed, generates the test child pipeline YAML

  • the test child pipeline runs grouped pytest jobs

What the generated flow looks like

        flowchart LR
    subgraph P["Parent pipeline"]
        direction TD
        PV["pipeline_variables"]
        GBCP["generate_build_child_pipeline"]
    end

    subgraph B["Build child pipeline"]
        direction TD
        BTRA["build_test_related_apps\n(parallel when needed)"]
        BNTRA["build_non_test_related_apps\n(parallel when needed)"]
        GTCP["generate_test_child_pipeline"]
    end

    subgraph TCP["Test child pipeline"]
        direction TD
        TJ1["[target] - [env]\n(parallel when needed)"]
        TJ2["... more grouped test jobs ..."]
    end

    PV --> BTRA
    GBCP --> BTRA
    PV --> BNTRA
    GBCP --> BNTRA
    BTRA --> GTCP
    GTCP --> TCP
    GTCP --> TJ1
    GTCP --> TJ2

Build child pipeline

The build child pipeline is generated by idf_ci.idf_gitlab.pipeline.build_child_pipeline.

It classifies discovered apps into two buckets:

  • test-related apps: apps associated with pytest-driven testing

  • non-test-related apps: apps that still need building but do not feed the generated test pipeline

From those buckets, the generated pipeline can contain up to four jobs.

build_test_related_apps

Builds only apps that feed later pytest execution.

build_non_test_related_apps

Builds the remaining selected apps.

generate_test_child_pipeline

Runs only when there are test-related apps. It depends on build_test_related_apps and produces test_child_pipeline.yml.

test-child-pipeline

Trigger job that starts the downstream test child pipeline from test_child_pipeline.yml.

Both build jobs also declare cross-pipeline needs on:

  • generate_build_child_pipeline

  • pipeline_variables

This gives each build job the generated YAML context and exported pipeline variables from the parent pipeline.

If no apps are selected, idf-ci emits a single fake_pass job.

What the build jobs contain

Generated build jobs share .default_build_settings.

Build work is sharded with GitLab parallel when the selected app count is large enough. The shard size is controlled by gitlab.build_pipeline.runs_per_job.

The result has two independent build branches:

  • one branch that feeds tests

  • one branch that exists only to build the rest of the requested app set

Only the test-related branch continues into test pipeline generation.

Test child pipeline

The test child pipeline is generated by idf_ci.idf_gitlab.pipeline.test_child_pipeline.

Instead of creating one job per test case, it groups pytest cases by:

  • target selector

  • environment selector

  • runner tags

Each generated job is named like <target_selector> - <env_selector> and contains:

  • job tags derived from the grouped test cases

  • a nodes variable containing the selected pytest node IDs

  • optional parallel sharding when a group is large enough

All generated test jobs extend .default_test_settings.

Each test job depends on generate_test_child_pipeline so it can download the build artifacts needed for execution.

If no test cases are selected, the generated test child pipeline also falls back to fake_pass.