fix(BA-5499): prevent health checker from recreating deleted etcd keys in Traefik mode

[seedspirit]

Summary

• Add per-circuit asyncio.Lock to CircuitManager to serialize update_circuit_routes() and unload_circuits() for the same circuit, preventing race conditions
• Re-read circuit from DB in propagate_route_updates_to_workers() before etcd write; skip propagation if circuit was deleted (ObjectNotFound)
• Use fresh route_info instead of stale in-memory snapshot to prevent stale backend URLs from persisting in Traefik's weighted service pool

Test plan

• Unit tests pass (pants test tests/unit/appproxy/coordinator::)
• Type checking passes (pants check src/ai/backend/appproxy/coordinator::)
• Lint passes (pants lint --changed-since=origin/main)
• Manual QA: scale-in scenario does not leave stale session service keys in etcd (see DEVIATION.md)

Resolves BA-5499

[Copilot]

Pull request overview

Prevents Traefik-mode health checking from recreating deleted etcd keys by serializing per-circuit updates/unloads and ensuring route propagation uses fresh DB state (and skips deleted circuits).

Changes:

• Add per-circuit asyncio.Lock in CircuitManager to serialize update_circuit_routes() and unload_circuits() per circuit.
• Re-read circuits from DB in propagate_route_updates_to_workers() and skip propagation when the circuit has been deleted.
• Add changelog entry and a deviation report documenting manual verification steps.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.

File	Description
src/ai/backend/appproxy/coordinator/types.py	Introduces per-circuit locks and uses them to serialize route updates and circuit unloads in Traefik mode.
src/ai/backend/appproxy/coordinator/health_checker.py	Re-reads circuit state from DB prior to propagation to avoid writing stale/deleted state.
changes/10666.fix.md	Adds a changelog note for the regression fix.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

unload_circuits() releases the per-circuit lock and then removes it from _circuit_locks. If an update_circuit_routes() call was already queued waiting on that same lock, it will run immediately after unload completes and can still recreate Traefik/etcd state for a circuit you just unloaded/deleted. To make the serialization effective, keep the lock around and add an explicit 'unloaded/deleted' guard checked inside the lock (e.g., a per-circuit tombstone flag/set that makes update_circuit_routes() return early), and only remove the lock when you can guarantee no further work can be queued for that circuit (or during a broader manager shutdown/cleanup).