(7) threading rewrite

[daniel-noland]

This basically rebuild main in prep for DPDK ACL work to land.

We simply can't manage such an anarchic and nonsensical threading model anymore. Not when we need principled memory mangement, organized shutdown, and an extensible design.

Warning

This step is necessary but in no way sufficient to get all of DPDK to land. This is basically the least drastic thing I could contrive which should still tolerate DPDK ACLs. I'm going to turn around and kick this subsystem much, MUCH harder in the next few weeks as I get DPDK proper to land.

[Copilot]

Pull request overview

This PR introduces a new lifecycle crate and refactors dataplane threading/shutdown to use structured cancellation (CancellationToken) and task tracking (TaskTracker) instead of ad-hoc stop channels and “private” Tokio runtimes per subsystem. It aligns mgmt/BMP/metrics with a caller-owned mgmt runtime and migrates kernel driver threads to std::thread::scope-owned, cancellation-aware workers.

Changes:

• Add dataplane-lifecycle crate providing Shutdown/Subsystem primitives, signal handling, and drain semantics.
• Refactor mgmt/BMP/metrics to spawn onto a provided Tokio runtime and exit via subsystem cancellation tokens.
• Update router RIO and kernel driver workers to observe cancellation tokens and fit the new shutdown model.

Reviewed changes

Copilot reviewed 22 out of 23 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
routing/src/router/rio.rs	RIO loop now exits via subsystem cancellation token; handle triggers cancellation and joins thread.
routing/src/router/mod.rs	Router construction now threads mgmt/router subsystems + runtime handle; BMP spawn updated.
routing/src/router/ctl.rs	Removes RouterCtlMsg::Finish shutdown path in favor of cancellation.
routing/src/frr/test.rs	Updates router construction to pass lifecycle subsystems/runtime handle.
routing/src/bmp/mod.rs	BMP no longer creates/leaks a runtime; spawns on provided handle and observes cancellation.
routing/Cargo.toml	Adds lifecycle dependency.
mgmt/src/tests/mgmt.rs	Updates router construction to pass lifecycle subsystems/runtime handle.
mgmt/src/processor/launch.rs	Replaces dedicated mgmt thread/runtime with run_mgmt spawning tracked tasks on provided runtime.
mgmt/src/lib.rs	Re-exports run_mgmt instead of start_mgmt.
mgmt/Cargo.toml	Adds lifecycle dependency.
lifecycle/src/lib.rs	New lifecycle primitives: Shutdown, Subsystem, drain deadlines, and signal handler install.
lifecycle/Cargo.toml	New crate manifest for lifecycle utilities.
dataplane/src/statistics/mod.rs	Replaces dedicated metrics thread/runtime with spawn_metrics on mgmt runtime + cancellation.
dataplane/src/packet_processor/mod.rs	Threads mgmt/router subsystems + runtime handle into router startup.
dataplane/src/main.rs	Introduces Shutdown and mgmt runtime ownership; unified shutdown signaling and drain flow.
dataplane/src/drivers/tokio_util.rs	Removes helper for local runtime setup (now inlined where needed).
dataplane/src/drivers/mod.rs	Removes tokio_util module reference.
dataplane/src/drivers/kernel/worker.rs	Workers now use scoped threads, build per-thread local runtimes, and observe cancellation in reader loops.
dataplane/src/drivers/kernel/mod.rs	Kernel driver now uses scoped worker threads + supervisor reporting fatal via lifecycle subsystem.
dataplane/docs/threading-invariants.md	Adds threading invariants document used to guide the rewrite.
dataplane/Cargo.toml	Adds lifecycle dependency; adjusts Tokio features for mgmt runtime ownership.
Cargo.toml	Adds lifecycle workspace member and dependency alias.
Cargo.lock	Adds dataplane-lifecycle and updates dependency graph accordingly.

[Copilot]

Pull request overview

Copilot reviewed 22 out of 23 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

dataplane/src/drivers/kernel/mod.rs:63

confidence: 9
tags: [logic]

`spawn_workers_scoped` logs and then ignores worker thread spawn failures, so `DriverKernel::start` can return `Ok(())` with fewer than `num_workers` (including zero) workers running. That leaves the dataplane up but non-functional and may never trigger shutdown.

Propagate the failure (e.g., return an error if any `worker.start(...)` fails or if `workers.len() != num_workers`), or explicitly `report_fatal`/cancel the workers subsystem when a worker cannot be spawned.

        match worker.start(scope, builder, interfaces) {
            Ok(handle) => workers.push(handle),
            Err(e) => {
                error!("Failed to start worker {wid}: {e}");
            }

</details>

[Copilot]

Pull request overview

Copilot reviewed 22 out of 23 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 23 out of 24 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 21 out of 22 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 21 out of 22 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

mgmt/src/processor/launch.rs:182

confidence: 9
tags: [logic]

Same lifetime issue as above: passing `&mgmt.root_token()` across `.await` borrows a temporary `CancellationToken` and is likely to fail to compile. Store `mgmt.root_token()` in a local variable and pass a reference to that instead.

k8s_mgmt_init(&k8s_client, &mgmt.root_token()).await?;

</details>