Fix background worker pruning

[msaroufim]

Summary

• prune finished background worker tasks before counting worker capacity
• keep worker loops alive when a submission task crashes and record failed job status
• add regression coverage for stale worker pruning and early status update failure

Root cause

Production showed queued submissions with busy=0, active=24, and hundreds of enqueued jobs. Finished or failed worker tasks could remain in _workers, so autoscaling believed capacity was full and did not start replacement workers.

Validation

• uv run --with pytest --with pytest-asyncio pytest tests/test_background_submission_manager.py -q
• uv run --with pytest --with pytest-asyncio pytest tests/test_admin_api.py tests/test_background_submission_manager.py -q
• uv run --with ruff ruff check src/libkernelbot/background_submission_manager.py tests/test_background_submission_manager.py --line-length 120
• local Uvicorn smoke test against POST /submission/local-lb/A100/test using the real BackgroundSubmissionManager, with two injected dead worker tasks occupying max_workers; verified submissions reached succeeded and queue drained to zero

CI

• lint: passing
• unit-tests: passing
• integration-tests-github: passing
• integration-tests-modal: passing

[github-actions]

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
src/libkernelbot
background_submission_manager.py					98-99, 109, 182-183, 187-197, 236-243, 255-257, 266-268, 271, 292-293
utils.py
Project Total

This report was generated by python-coverage-comment-action