Skip to content

feat: support streaming reasoning chunks#6607

Open
WiiliamC wants to merge 1 commit into
FlowiseAI:mainfrom
WiiliamC:feature/support_vllm_qwen3.6_thinking
Open

feat: support streaming reasoning chunks#6607
WiiliamC wants to merge 1 commit into
FlowiseAI:mainfrom
WiiliamC:feature/support_vllm_qwen3.6_thinking

Conversation

@WiiliamC

@WiiliamC WiiliamC commented Jul 9, 2026

Copy link
Copy Markdown

Description

This PR adds support for streaming reasoning/thinking chunks from OpenAI-compatible models such as vLLM Qwen thinking models.

It centralizes streaming chunk extraction so AgentFlow LLM and Agent nodes can separate visible response text from reasoning content, stream reasoning as thinking events, and avoid sending empty token events for reasoning-only chunks.

Changes

  • Add extractStreamingChunkContent to extract visible text and reasoning content from LangChain/OpenAI-compatible streaming chunks.
  • Preserve raw OpenAI-compatible responses in the custom OpenAI chat model so reasoning deltas can be read from raw streaming payloads.
  • Update AgentFlow LLM and Agent streaming handlers to emit reasoning chunks through thinking events and store aggregated reasoning metadata.
  • Add tests for reasoning-only streaming chunks and raw reasoning delta extraction.

Test

  • Manually completed build verification.
  • Manually completed port/runtime testing.
  • Manually completed frontend testing.

before:
image
after:
image

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the streaming chunk content extraction logic across AgentFlow nodes by introducing a centralized extractStreamingChunkContent utility in utils.ts. This utility extracts both visible text and reasoning/thinking content from various LLM chunk formats, including OpenAI-compatible raw responses. The PR also adds corresponding unit tests for the new utility and streaming behavior, and configures ChatOpenAICustom to include raw responses. Feedback on the changes suggests improving the robustness of the array check for chunk.contentBlocks in extractStreamingChunkContent by using Array.isArray instead of checking the length property to prevent potential issues if contentBlocks is a non-array object or string.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

}
}

const reasoningBlocks = chunk.contentBlocks?.length ? chunk.contentBlocks : Array.isArray(chunk.content) ? chunk.content : []

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Using chunk.contentBlocks?.length to check if contentBlocks is a valid array can be problematic if contentBlocks is a non-array object or a string (which also has a length property). It is safer and more robust to explicitly check if it is an array using Array.isArray(chunk.contentBlocks).

    const reasoningBlocks = Array.isArray(chunk.contentBlocks) ? chunk.contentBlocks : Array.isArray(chunk.content) ? chunk.content : []

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant