Scenario 3: All CNO-Managed Operands Restart Together

Objective: Verify that ALL CNO-managed control plane operands restart when restart-date is set, not just CNCC.

Maps to: The fix applies the annotation to ALL rendered workloads

Cluster: brcox-88531-hc on Azure dev mgmt cluster — HCP namespace: clusters-brcox-88531-hc

Release image: registry.build10.ci.openshift.org/ci-ln-1qhpr62/release:latest (CNO PR #3030 + HyperShift PR #8751)

Note: This scenario shares the same annotation event as Scenario 2 (restart-date set at 11:43:15Z).

Overall Scenario 3: PASS
StepCheckResultEvidence
1 Record timestamps for all CNO operands (before) PASS All 4 operands recorded at 11:13 UTC (see evidence below)
2 Set restart-date annotation PASS Set to 2026-06-17T11:43:15Z (same event as Scenario 2)
3 cloud-network-config-controller restarted PASS 5c7997f74f-7hllb76dd98d985-k6zvs (11:45:28Z)
4 multus-admission-controller restarted PASS 6d7b6dfc49-mgxlc5fd559d858-75t5j (11:45:35Z)
5 network-node-identity restarted PASS b5d5546d8-kdgb785d874994d-pd9ks (11:45:50Z)
6 ovnkube-control-plane restarted PASS 6c56d86876-ksfc57dbccb64d4-g8stz (11:45:44Z)
7 All pods reach Running/Ready PASS All 4 pods Running/Ready at 11:47:37Z check

Evidence

Step 1 — All CNO operands before annotation
$ oc get pods -n clusters-brcox-88531-hc \
    -o custom-columns='NAME:.metadata.name,READY:.status.containerStatuses[0].ready,RESTARTS:.status.containerStatuses[0].restartCount,CREATED:.metadata.creationTimestamp' \
    | grep -E "multus|network-node-identity|ovnkube-control-plane|cloud-network-config"
cloud-network-config-controller-5c7997f74f-7hllb     true    0          2026-06-17T11:13:20Z
multus-admission-controller-6d7b6dfc49-mgxlc         true    0          2026-06-17T11:13:27Z
network-node-identity-b5d5546d8-kdgb7                true    0          2026-06-17T11:13:42Z
ovnkube-control-plane-6c56d86876-ksfc5               true    0          2026-06-17T11:13:36Z
Steps 3-7 — All CNO operands after annotation (captured at 11:47 UTC)
$ oc get pods -n clusters-brcox-88531-hc \
    -o custom-columns='NAME:.metadata.name,READY:.status.containerStatuses[0].ready,RESTARTS:.status.containerStatuses[0].restartCount,CREATED:.metadata.creationTimestamp' \
    | grep -E "multus|network-node-identity|ovnkube-control-plane|cloud-network-config"
cloud-network-config-controller-76dd98d985-k6zvs      true    0          2026-06-17T11:45:28Z
multus-admission-controller-5fd559d858-75t5j          true    0          2026-06-17T11:45:35Z
network-node-identity-85d874994d-pd9ks                true    0          2026-06-17T11:45:50Z
ovnkube-control-plane-7dbccb64d4-g8stz                true    0          2026-06-17T11:45:44Z

# All 4 operands have new ReplicaSets and new creation timestamps.
# All rolled out within ~22 seconds of each other (11:45:28Z - 11:45:50Z).
# All pods Running/Ready with 0 restarts.

# Deployment generations all bumped to 2 with restart-date on pod templates:
$ oc get deployment -n clusters-brcox-88531-hc \
    cloud-network-config-controller multus-admission-controller \
    network-node-identity ovnkube-control-plane \
    -o jsonpath='{range .items[*]}Deployment: {.metadata.name}
  generation: {.metadata.generation}
  observedGeneration: {.status.observedGeneration}
  restart-date: {.spec.template.metadata.annotations.hypershift\.openshift\.io/restart-date}
  replicas/ready/updated: {.status.replicas}/{.status.readyReplicas}/{.status.updatedReplicas}
{end}'
Deployment: cloud-network-config-controller
  generation: 2
  observedGeneration: 2
  restart-date: 2026-06-17T11:43:15Z
  replicas/ready/updated: 1/1/1
Deployment: multus-admission-controller
  generation: 2
  observedGeneration: 2
  restart-date: 2026-06-17T11:43:15Z
  replicas/ready/updated: 1/1/1
Deployment: network-node-identity
  generation: 2
  observedGeneration: 2
  restart-date: 2026-06-17T11:43:15Z
  replicas/ready/updated: 1/1/1
Deployment: ovnkube-control-plane
  generation: 2
  observedGeneration: 2
  restart-date: 2026-06-17T11:43:15Z
  replicas/ready/updated: 1/1/1

# ReplicaSet history showing old RS scaled down, new RS active:
$ oc get rs -n clusters-brcox-88531-hc \
    -o custom-columns='NAME:.metadata.name,DESIRED:.spec.replicas,CURRENT:.status.replicas,READY:.status.readyReplicas,CREATED:.metadata.creationTimestamp' \
    | grep -E "multus|network-node-identity|ovnkube-control-plane|cloud-network-config"
cloud-network-config-controller-5c7997f74f      0         0         <none>   2026-06-17T11:13:20Z
cloud-network-config-controller-76dd98d985      1         1         1        2026-06-17T11:45:28Z
multus-admission-controller-5fd559d858          1         1         1        2026-06-17T11:45:35Z
multus-admission-controller-6d7b6dfc49          0         0         <none>   2026-06-17T11:13:27Z
network-node-identity-85d874994d                1         1         1        2026-06-17T11:45:50Z
network-node-identity-b5d5546d8                 0         0         <none>   2026-06-17T11:13:42Z
ovnkube-control-plane-6c56d86876                0         0         <none>   2026-06-17T11:13:36Z
ovnkube-control-plane-7dbccb64d4                1         1         1        2026-06-17T11:45:44Z
← Scenario 2 Scenario 4 →