ESP LLM Tongyi

[中文]

Introduction

esp_llm_tongyi is the Tongyi adapter for ESP LLM Service. It uses the Alibaba Cloud Model Studio DashScope TLS WebSocket. Synchronize the clock before creating the session. Cloud application setup, message fields, and related documents are in the multimodal interaction protocol.

Features

  • Voice only: duplex conversation with a DashScope voice application. No camera and no display are created

  • Vision: full-screen local camera preview, plus one JPEG sent with esp_llm_service_send_frame(). A prompt in frame.user_data requests a reply. NULL only updates the image in the session. There is no continuous video and no cloud image downlink

How It Works

esp_llm_tongyi_create() reads esp_llm_tongyi_cfg_t. Start from ESP_LLM_TONGYI_CFG_DEFAULT(), then set api_key, workspace_id, and app_id. The structure is in esp_llm_tongyi.h. Upstream, downstream, and client defaults are in esp_tongyi.h.

Voice only keeps cfg.upstream.type at AudioOnly. Vision sets it to ESP_TONGYI_UPSTREAM_TYPE_AUDIO_VIDEO, which is AudioAndVideo. cfg.upstream.mode defaults to duplex, and the audio format defaults to raw-opus. The downlink voice cfg.downstream.voice defaults to longanhuan, and the volume defaults to 80. Sample rate, downlink frame size, downlink bitrate, and uplink bitrate are set in menuconfig under Audio profile. The defaults are 16000 Hz, 60 ms, 32 kbps, and 24000 bit/s. connect_timeout_ms defaults to 30000 ms. An empty task_id is generated by the adapter. Conversation can start immediately, or uplink can stay disabled until a wake word.

esp_llm_tongyi_cfg_t cfg = ESP_LLM_TONGYI_CFG_DEFAULT();
cfg.api_key = CONFIG_TONGYI_API_KEY;
cfg.workspace_id = CONFIG_TONGYI_WORKSPACE_ID;
cfg.app_id = CONFIG_TONGYI_APP_ID;

esp_llm_service_t *llm = NULL;
ESP_ERROR_CHECK(esp_llm_tongyi_create(&cfg, &llm));

Link and start order are in ESP LLM Service.

Example

esp_llm_tongyi_agent_app selects voice only or vision in menuconfig. In vision mode, snap sends one image and snap <prompt> asks about that image.

Provide feedback about this document