Add Redpanda Kafka-compatible hosting integration#1410
Open
slang25 wants to merge 13 commits into
Open
Conversation
Contributor
|
🚀 Dogfood this PR with:
curl -fsSL https://raw.githubusercontent.com/CommunityToolkit/Aspire/main/eng/scripts/dogfood-pr.sh | bash -s -- 1410Or
iex "& { $(irm https://raw.githubusercontent.com/CommunityToolkit/Aspire/main/eng/scripts/dogfood-pr.ps1) } 1410" |
Contributor
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Adds a new Aspire hosting integration for Redpanda (Kafka-compatible streaming platform), including the core hosting resource/builder APIs, docs, examples, and tests.
Changes:
- Introduces
AddRedPanda(...)and related builder extensions (console + data persistence) plus exported resource types. - Adds a Redpanda example AppHost project.
- Adds unit + integration tests and links the new integration from the repo README (plus a NuGet audit suppression note).
Reviewed changes
Copilot reviewed 16 out of 16 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/CommunityToolkit.Aspire.Hosting.RedPanda.Tests/ResourceCreationTests.cs | Adds unit tests validating resource creation, endpoints, image annotations, console, and volume mounts. |
| tests/CommunityToolkit.Aspire.Hosting.RedPanda.Tests/AppHostTests.cs | Adds Docker-based integration tests validating Redpanda Admin API and Schema Registry readiness. |
| tests/CommunityToolkit.Aspire.Hosting.RedPanda.Tests/CommunityToolkit.Aspire.Hosting.RedPanda.Tests.csproj | Introduces a new test project referencing the new hosting package + example AppHost. |
| src/CommunityToolkit.Aspire.Hosting.RedPanda/RedPandaServerResource.cs | Adds the primary Redpanda server resource with connection-string expression and exported properties. |
| src/CommunityToolkit.Aspire.Hosting.RedPanda/RedPandaConsoleContainerResource.cs | Adds the Redpanda Console container resource type. |
| src/CommunityToolkit.Aspire.Hosting.RedPanda/RedPandaContainerImageTags.cs | Centralizes pinned image/registry/tag constants for broker and console images. |
| src/CommunityToolkit.Aspire.Hosting.RedPanda/RedPandaBuilderExtensions.cs | Implements AddRedPanda, persistence options, console support, and container configuration (args/env/healthcheck). |
| src/CommunityToolkit.Aspire.Hosting.RedPanda/README.md | Documents usage, endpoints, and pinned upstream images/tags. |
| src/CommunityToolkit.Aspire.Hosting.RedPanda/CommunityToolkit.Aspire.Hosting.RedPanda.csproj | Adds the new hosting package project and exposes internals to the test assembly. |
| examples/redpanda/CommunityToolkit.Aspire.Hosting.RedPanda.AppHost/Program.cs | Adds an example AppHost that provisions Redpanda + Console. |
| examples/redpanda/CommunityToolkit.Aspire.Hosting.RedPanda.AppHost/appsettings.json | Adds logging defaults for the example AppHost. |
| examples/redpanda/CommunityToolkit.Aspire.Hosting.RedPanda.AppHost/Properties/launchSettings.json | Adds launch profiles for the example AppHost. |
| examples/redpanda/CommunityToolkit.Aspire.Hosting.RedPanda.AppHost/CommunityToolkit.Aspire.Hosting.RedPanda.AppHost.csproj | Adds the example AppHost project file referencing the new hosting package. |
| README.md | Adds Redpanda integration entry + NuGet/doc links to the main repository readme table. |
| Directory.Build.props | Adds a repo-wide NuGet audit suppression entry with justification comment. |
| CommunityToolkit.Aspire.slnx | Adds the new Redpanda src/tests projects and the example folder/project to the solution. |
Odonno
approved these changes
Jun 14, 2026
Odonno
left a comment
Odonno
left a comment
Contributor
There was a problem hiding this comment.
LGTM.
Just one question: can Kafbat UI also be supported? (see Kafka integration)
Author
|
@aaronpowell 👋 are you able to take a look when you have some time 🙂 |
Adds CommunityToolkit.Aspire.Hosting.RedPanda, an Aspire hosting package for the Redpanda Kafka-compatible streaming platform. Includes AddRedPanda with Kafka, internal, Schema Registry, and Admin endpoints, WithConsole, WithDataVolume/WithDataBindMount, and WithHostPort extensions, plus unit and Docker integration tests, an example AppHost, README, and solution and NuGet audit wiring. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Addresses PR review feedback: introduces RedPandaServerOptions with CpuCount and Memory, plus an AddRedPanda overload taking a configuration delegate so the Redpanda --smp and --memory limits are no longer hard coded. Adds a unit test for WithDataBindMount and tests covering the default and configured broker arguments. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds an optional WithKafkaUI extension that runs the kafbat/kafka-ui container (the same UI used by the official Aspire Kafka integration) wired to the Redpanda broker's Kafka API and Schema Registry, alongside the existing Redpanda Console. Includes a dedicated resource type, a WithHostPort overload, README example, and unit tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Main bumped AspireVersion to 13.4.3; update the Redpanda example AppHost to use Aspire.AppHost.Sdk/13.4.3 to match the rest of the repo and avoid an Aspire.Hosting.AppHost package downgrade in the test project. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
slang25
force-pushed
the
slang25/redpanda-hosting-package
branch
from
f8bac6e to
14e039d
Compare
aaronpowell
requested changes
Jun 29, 2026
aaronpowell
left a comment
aaronpowell
left a comment
Member
There was a problem hiding this comment.
Couple of changes requested inline. Can you also create a TypeScript app host and an associated test to run it.
My only apprehension is that the sample app is very simplistic and I worry it doesn't give a real indication of how you could use this integration. Would it result in much additional code/complexity to have an app that consumes the Redpanda resource?
Contributor
|
@aaronpowell I will dogfood this once it is released for info. |
- Reference MessagePack explicitly to resolve the CVE instead of suppressing the advisory repo-wide (per maintainer guidance). - Remove the ASPIREATS001 experimental pragmas; the AspireExport API is no longer experimental on the current Aspire version. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Add a TypeScript app host example plus a TypeScriptAppHostTests test that validates it compiles and starts (mirrors the other integrations). - Make the sample app demonstrate real usage: a consumer web app that produces to and consumes from the broker via Aspire.Confluent.Kafka, wired to the resource with WithReference. Adds an example ServiceDefaults project and registers Aspire.Confluent.Kafka in CPM. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Member
|
@Odonno - you should be able to install the dogfood build per #1410 (comment) before it's merged. |
jmezach
reviewed
Jul 7, 2026
Contributor
There was a problem hiding this comment.
Can we add some tests here for the Kafka UI and Redpanda Console UI as well?
Address review feedback from @jmezach: verify the Console and Kafka UI containers start and serve HTTP requests. Enables the Kafka UI in the sample AppHost so both UIs are exercised. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
|
@Odonno I just tried running your PR locally but I'm seeing errors in the consumer logs about not being able to connect. Could we add some kind of health check to the consumer and hook that into Aspire and then add to the tests that it actually becomes healthy eventually? |
Address review feedback from @jmezach: the consumer sample logged connection errors and could never reach the broker. Two root causes: - The external Kafka listener advertised localhost:{targetPort} through Aspire's endpoint proxy, so host clients were told to reconnect on the container port rather than the published host port. Mark the endpoint as not proxied so the advertised address matches the real host port. - MessageConsumer subscribed to the 'messages' topic before it existed, throwing ConsumeException and stopping the host (StopHost behavior). Tolerate the unknown-topic case and retry instead of crashing. Also surface the consumer's Kafka connectivity health checks to Aspire via WithHttpHealthCheck("/health") (mapping the health endpoints unconditionally so the probe works outside Development), enable the Kafka UI in the sample, and add an integration test asserting the consumer becomes healthy once it connects to the broker. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Author
|
@jmezach can you try again with the latest changes, it should be fixed now |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #<ISSUE_NUMBER>
Adds
CommunityToolkit.Aspire.Hosting.RedPanda, an Aspire hosting integration for the Redpanda Kafka-compatible streaming platform.AddRedPandaruns a single-node broker (pinned imageredpandadata/redpanda:v26.1.10) exposing Kafka, internal, Schema Registry, and Admin endpoints with an Admin-API health check, and the resource can be referenced by any Kafka client integration. Optional extensions coverWithConsole(Redpanda Console web UI,redpandadata/console:v3.7.4),WithDataVolume/WithDataBindMountfor persistence, andWithHostPort. The change also adds unit and Docker-based integration tests, an example AppHost, a package README, and wires the projects into the solution, README catalog, and a repo-wide MessagePack NuGet audit suppression.PR Checklist
Other information
The Kafka listener setup mirrors the upstream
Aspire.Hosting.Kafkapattern: the connection string targets the host-facingkafkaendpoint, while a separateinternallistener (advertised as{resource}:29092) is used for container-to-container traffic, including the bundled console.🤖 Generated with Claude Code