Notes

Kubernetes controllers support concurrent reconciliation via MaxConcurrentReconciles. Multiple workers can process different resources in parallel, but the workqueue guarantees a single resource is never processed by two workers at the same time. Workqueue Internals The client-go workqueue uses three data structures: queue - ordered list of items waiting to be processed dirty set - all items that need processing (used for deduplication) processing set - items currently being worked on by a worker ┌─────────────────────────────────────────────────┐ │ Workqueue │ │ │ │ dirty set: tracks items needing reconciliation │ │ processing set: tracks items being reconciled │ │ queue: ordered list for workers to pick from │ │ │ └─────────────────────────────────────────────────┘ Adding an Item While It’s Being Processed Yes, you can add an item to the queue while it’s being processed. The workqueue handles it: ...

The network_programming_duration_seconds metric tracks how long it takes for a Pod or Service change to show up in the actual network rules (iptables/ipvs/nftables) on each node. Component Flow ┌─────────────────────────────────────────────────────────────────┐ │ Control Plane │ ├─────────────────────────────────────────────────────────────────┤ │ │ │ 1. Pod becomes Ready (or Service changes) │ │ └─> Timestamp: T0 │ │ │ │ 2. Endpoints Controller detects change │ │ └─> Calculates trigger time from Pod condition │ │ └─> Sets annotation on EndpointSlice: │ │ endpoints.kubernetes.io/last-change-trigger-time = T0 │ │ │ └─────────────────────────────────────────────────────────────────┘ │ │ watch/update ▼ ┌─────────────────────────────────────────────────────────────────┐ │ Worker Node │ ├─────────────────────────────────────────────────────────────────┤ │ │ │ 3. kube-proxy receives updated EndpointSlice │ │ └─> Extracts annotation timestamp: T0 │ │ └─> (EndpointsChangeTracker filters if T0 < trackerStartTime)│ │ │ │ 4. kube-proxy programs iptables/ipvs/nftables rules │ │ └─> Completes at timestamp: T1 │ │ │ │ 5. kube-proxy calculates and emits metric │ │ └─> network_programming_duration_seconds = T1 - T0 │ │ │ └─────────────────────────────────────────────────────────────────┘ Key Points: ...