Log: /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2391/e2e-tests/logs/upgrade-consistency-sharded-tls.log Warning: version difference between client (1.36) and server (1.33) exceeds the supported minor version skew of +/-1 Warning: version difference between client (1.36) and server (1.33) exceeds the supported minor version skew of +/-1 Warning: version difference between client (1.36) and server (1.33) exceeds the supported minor version skew of +/-1 ----------------------------------------------------------------------------------- get and delete old CRDs and RBAC ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io "perconaservermongodbbackups.psmdb.percona.com" deleted customresourcedefinition.apiextensions.k8s.io "perconaservermongodbrestores.psmdb.percona.com" deleted customresourcedefinition.apiextensions.k8s.io "perconaservermongodbs.psmdb.percona.com" deleted grep: warning: stray \ before - grep: warning: stray \ before - error: the server doesn't have a resource type "perconaservermongodbbackups" + kubectl patch perconaservermongodbbackups.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbbackups" error: the server doesn't have a resource type "perconaservermongodbrestores" + kubectl patch perconaservermongodbrestores.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbrestores" error: the server doesn't have a resource type "perconaservermongodbs" + kubectl patch perconaservermongodbs.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbs" clusterrole.rbac.authorization.k8s.io "percona-server-mongodb-operator" deleted clusterrolebinding.rbac.authorization.k8s.io "service-account-percona-server-mongodb-operator" deleted Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found ----------------------------------------------------------------------------------- destroy chaos-mesh ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces psmdb-operator ----------------------------------------------------------------------------------- namespace "stable-resource-version-972" deleted namespace "psmdb-operator" deleted ----------------------------------------------------------------------------------- create namespace psmdb-operator ----------------------------------------------------------------------------------- namespace/psmdb-operator created Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-2391-cbf9fce95-29-cluster10" modified. ----------------------------------------------------------------------------------- start PSMDB operator: docker.io/perconalab/percona-server-mongodb-operator:PR-2391-cbf9fce95 ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io/perconaservermongodbbackups.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbrestores.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbs.psmdb.percona.com serverside-applied clusterrole.rbac.authorization.k8s.io/percona-server-mongodb-operator created serviceaccount/percona-server-mongodb-operator created clusterrolebinding.rbac.authorization.k8s.io/service-account-percona-server-mongodb-operator created deployment.apps/percona-server-mongodb-operator created waiting for pod/percona-server-mongodb-operator-6865988b4d-fv88p to be ready.OK Print operator info from log 2026-06-17T20:04:36.121Z INFO setup Manager starting up {"gitCommit": "cbf9fce9565b428fa52d8f97f62491be3841db28", "gitBranch": "PR-2391-cbf9fce95", "buildTime": "", "goVersion": "go1.26.4", "os": "linux", "arch": "amd64"} ----------------------------------------------------------------------------------- destroy chaos-mesh ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces upgrade-consistency-sharded-tls-9064 ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- create namespace upgrade-consistency-sharded-tls-9064 ----------------------------------------------------------------------------------- namespace/upgrade-consistency-sharded-tls-9064 created Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-2391-cbf9fce95-29-cluster10" modified. ----------------------------------------------------------------------------------- deploy cert manager ----------------------------------------------------------------------------------- namespace/cert-manager created namespace/cert-manager labeled namespace/cert-manager configured customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io created serviceaccount/cert-manager-cainjector created serviceaccount/cert-manager created serviceaccount/cert-manager-webhook created clusterrole.rbac.authorization.k8s.io/cert-manager-cainjector created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-issuers created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificates created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-orders created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-challenges created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created clusterrole.rbac.authorization.k8s.io/cert-manager-cluster-view created clusterrole.rbac.authorization.k8s.io/cert-manager-view created clusterrole.rbac.authorization.k8s.io/cert-manager-edit created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests created clusterrole.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-cainjector created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-issuers created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificates created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-orders created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-challenges created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews created role.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created role.rbac.authorization.k8s.io/cert-manager:leaderelection created role.rbac.authorization.k8s.io/cert-manager-tokenrequest created role.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created rolebinding.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created rolebinding.rbac.authorization.k8s.io/cert-manager:leaderelection created rolebinding.rbac.authorization.k8s.io/cert-manager-tokenrequest created rolebinding.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created service/cert-manager-cainjector created service/cert-manager created service/cert-manager-webhook created deployment.apps/cert-manager-cainjector created deployment.apps/cert-manager created deployment.apps/cert-manager-webhook created mutatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created Warning: resource namespaces/cert-manager is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically. pod/cert-manager-559d798845-w6jts condition met pod/cert-manager-cainjector-64958d9c7c-8mdcx condition met pod/cert-manager-webhook-7fb6f99b56-8f4dq condition met E0617 20:05:35.544034 217983 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/api/v1/namespaces/cert-manager/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dcert-manager-cainjector-64958d9c7c-8mdcx&resourceVersion=1781726735168943012&timeoutSeconds=342&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" ----------------------------------------------------------------------------------- create secrets and start client ----------------------------------------------------------------------------------- secret/some-users created deployment.apps/psmdb-client created role.rbac.authorization.k8s.io/cmctl created serviceaccount/cmctl created rolebinding.rbac.authorization.k8s.io/service-account-cmctl created deployment.apps/cmctl created ----------------------------------------------------------------------------------- create first PSMDB cluster 1.21.2 some-name ----------------------------------------------------------------------------------- perconaservermongodb.psmdb.percona.com/some-name created ----------------------------------------------------------------------------------- check if Pod started ----------------------------------------------------------------------------------- waiting for pod/some-name-rs0-0 to be ready............OK waiting for pod/some-name-rs0-1 to be ready.............OK waiting for pod/some-name-rs0-2 to be ready.............OK Waiting for cluster readyness........................... waiting for pod/some-name-cfg-0 to be ready.OK waiting for pod/some-name-cfg-1 to be ready.OK waiting for pod/some-name-cfg-2 to be ready.OK Waiting for cluster readyness waiting for pod/some-name-mongos-0 to be ready.OK waiting for pod/some-name-mongos-1 to be ready.OK waiting for pod/some-name-mongos-2 to be ready.OK Waiting for cluster readyness waiting for cluster readyness.OK ----------------------------------------------------------------------------------- check if Pod started ----------------------------------------------------------------------------------- waiting for pod/some-name-rs0-0 to be ready.OK waiting for pod/some-name-rs0-1 to be ready.OK waiting for pod/some-name-rs0-2 to be ready.OK Waiting for cluster readyness waiting for pod/some-name-cfg-0 to be ready.OK waiting for pod/some-name-cfg-1 to be ready.OK waiting for pod/some-name-cfg-2 to be ready.OK Waiting for cluster readyness waiting for pod/some-name-mongos-0 to be ready.OK waiting for pod/some-name-mongos-1 to be ready.OK waiting for pod/some-name-mongos-2 to be ready.OK Waiting for cluster readyness waiting for cluster readyness.OK E0617 20:12:34.845566 291012 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781727154357611000&timeoutSeconds=593&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0617 20:12:37.168226 291288 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781727154357611000&timeoutSeconds=545&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0617 20:12:39.521330 291533 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781727159357644000&timeoutSeconds=562&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0617 20:12:41.844933 291771 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781727159357644000&timeoutSeconds=422&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0617 20:12:44.153406 292048 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781727159357644000&timeoutSeconds=482&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0617 20:12:46.438098 292281 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781727164357750000&timeoutSeconds=342&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0617 20:12:48.807089 292487 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781727164357750000&timeoutSeconds=535&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0617 20:12:51.120252 292727 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781727169357765000&timeoutSeconds=531&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0617 20:12:53.464425 292926 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781727169357765000&timeoutSeconds=588&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0617 20:12:55.850376 293150 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781727174357793000&timeoutSeconds=383&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met ----------------------------------------------------------------------------------- renew some-name-ssl ----------------------------------------------------------------------------------- Manually triggered issuance of Certificate upgrade-consistency-sharded-tls-9064/some-name-ssl waiting for pod/some-name-rs0-0 to be ready.OK waiting for pod/some-name-rs0-1 to be ready.OK waiting for pod/some-name-rs0-2 to be ready.OK Waiting for cluster readyness................................................................................................................... waiting for pod/some-name-cfg-0 to be ready.OK waiting for pod/some-name-cfg-1 to be ready.OK waiting for pod/some-name-cfg-2 to be ready.OK Waiting for cluster readyness waiting for pod/some-name-mongos-0 to be ready.OK waiting for pod/some-name-mongos-1 to be ready.OK waiting for pod/some-name-mongos-2 to be ready.OK Waiting for cluster readyness waiting for cluster readyness.OK E0617 20:18:23.585336 321143 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781727499772460000&timeoutSeconds=495&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0617 20:18:25.946787 321357 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781727504772472000&timeoutSeconds=397&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0617 20:18:28.282674 321509 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781727504772472000&timeoutSeconds=489&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0617 20:18:30.619031 321670 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781727509772488000&timeoutSeconds=586&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0617 20:18:32.958124 321893 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781727509772488000&timeoutSeconds=474&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0617 20:18:35.277344 322050 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781727514772506000&timeoutSeconds=553&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0617 20:18:37.579423 322242 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781727514772506000&timeoutSeconds=417&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0617 20:18:39.916578 322470 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781727519772552000&timeoutSeconds=599&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0617 20:18:42.223332 322662 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781727519772552000&timeoutSeconds=565&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0617 20:18:44.575151 322846 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781727519772552000&timeoutSeconds=334&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met ----------------------------------------------------------------------------------- renew some-name-ssl-internal ----------------------------------------------------------------------------------- Manually triggered issuance of Certificate upgrade-consistency-sharded-tls-9064/some-name-ssl-internal waiting for pod/some-name-rs0-0 to be ready.OK waiting for pod/some-name-rs0-1 to be ready.OK waiting for pod/some-name-rs0-2 to be ready.OK Waiting for cluster readyness..................................................................................................................... waiting for pod/some-name-cfg-0 to be ready.OK waiting for pod/some-name-cfg-1 to be ready.OK waiting for pod/some-name-cfg-2 to be ready.OK Waiting for cluster readyness waiting for pod/some-name-mongos-0 to be ready.OK waiting for pod/some-name-mongos-1 to be ready.OK waiting for pod/some-name-mongos-2 to be ready.OK Waiting for cluster readyness waiting for cluster readyness.OK ----------------------------------------------------------------------------------- check if service and statefulset created with expected config ----------------------------------------------------------------------------------- [2026-06-17T20:24:19+0000] compare_kubectl: service/some-name-rs0 OK [2026-06-17T20:24:20+0000] compare_kubectl: service/some-name-cfg OK [2026-06-17T20:24:21+0000] compare_kubectl: statefulset/some-name-rs0 OK [2026-06-17T20:24:22+0000] compare_kubectl: statefulset/some-name-cfg OK ----------------------------------------------------------------------------------- test 1.22.0 ----------------------------------------------------------------------------------- perconaservermongodb.psmdb.percona.com/some-name patched ----------------------------------------------------------------------------------- check if Pod started ----------------------------------------------------------------------------------- waiting for pod/some-name-rs0-0 to be ready.OK waiting for pod/some-name-rs0-1 to be ready.OK waiting for pod/some-name-rs0-2 to be ready.OK Waiting for cluster readyness................................................................................ waiting for pod/some-name-cfg-0 to be ready.OK waiting for pod/some-name-cfg-1 to be ready.OK waiting for pod/some-name-cfg-2 to be ready.OK Waiting for cluster readyness waiting for pod/some-name-mongos-0 to be ready.OK waiting for pod/some-name-mongos-1 to be ready.OK waiting for pod/some-name-mongos-2 to be ready.OK Waiting for cluster readyness waiting for cluster readyness.OK E0617 20:28:26.846640 373400 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781728104062990000&timeoutSeconds=445&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0617 20:28:29.179110 373463 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781728104062990000&timeoutSeconds=459&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0617 20:28:31.480284 373539 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781728109063010000&timeoutSeconds=556&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0617 20:28:33.323066 373636 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781728109063010000&timeoutSeconds=534&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0617 20:28:35.687041 373664 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781728114063480000&timeoutSeconds=445&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0617 20:28:38.003645 373764 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781728114063480000&timeoutSeconds=523&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0617 20:28:40.309414 373841 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781728119063489000&timeoutSeconds=390&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0617 20:28:42.608639 373917 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781728119063489000&timeoutSeconds=350&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0617 20:28:44.903226 373992 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781728124063557000&timeoutSeconds=508&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0617 20:28:47.211340 374100 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781728124063557000&timeoutSeconds=459&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met ----------------------------------------------------------------------------------- renew some-name-ssl ----------------------------------------------------------------------------------- Manually triggered issuance of Certificate upgrade-consistency-sharded-tls-9064/some-name-ssl waiting for pod/some-name-rs0-0 to be ready.OK waiting for pod/some-name-rs0-1 to be ready.OK waiting for pod/some-name-rs0-2 to be ready.OK Waiting for cluster readyness................................................................................................................................................. waiting for pod/some-name-cfg-0 to be ready.OK waiting for pod/some-name-cfg-1 to be ready.OK waiting for pod/some-name-cfg-2 to be ready.OK Waiting for cluster readyness waiting for pod/some-name-mongos-0 to be ready.OK waiting for pod/some-name-mongos-1 to be ready.OK waiting for pod/some-name-mongos-2 to be ready.OK Waiting for cluster readyness waiting for cluster readyness.OK E0617 20:34:30.693229 384498 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781728466187037000&timeoutSeconds=388&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0617 20:34:32.461330 384521 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781728471187263000&timeoutSeconds=504&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0617 20:34:34.220253 384539 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781728471187263000&timeoutSeconds=579&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0617 20:34:35.975555 384562 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781728471187263000&timeoutSeconds=452&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0617 20:34:37.743355 384584 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781728476187284000&timeoutSeconds=348&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0617 20:34:39.510433 384607 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781728476187284000&timeoutSeconds=502&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0617 20:34:41.269715 384627 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781728476187284000&timeoutSeconds=581&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0617 20:34:43.037694 384647 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781728481187386000&timeoutSeconds=349&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0617 20:34:44.797533 384669 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781728481187386000&timeoutSeconds=365&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0617 20:34:46.559830 384692 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781728486187454000&timeoutSeconds=518&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met ----------------------------------------------------------------------------------- renew some-name-ssl-internal ----------------------------------------------------------------------------------- Manually triggered issuance of Certificate upgrade-consistency-sharded-tls-9064/some-name-ssl-internal waiting for pod/some-name-rs0-0 to be ready.OK waiting for pod/some-name-rs0-1 to be ready.OK waiting for pod/some-name-rs0-2 to be ready.OK Waiting for cluster readyness................................................................................................................................................................... waiting for pod/some-name-cfg-0 to be ready.OK waiting for pod/some-name-cfg-1 to be ready.OK waiting for pod/some-name-cfg-2 to be ready.OK Waiting for cluster readyness waiting for pod/some-name-mongos-0 to be ready.OK waiting for pod/some-name-mongos-1 to be ready.OK waiting for pod/some-name-mongos-2 to be ready.OK Waiting for cluster readyness waiting for cluster readyness.OK ----------------------------------------------------------------------------------- check if service and statefulset created with expected config ----------------------------------------------------------------------------------- [2026-06-17T20:40:19+0000] compare_kubectl: service/some-name-rs0 OK [2026-06-17T20:40:19+0000] compare_kubectl: service/some-name-cfg OK [2026-06-17T20:40:20+0000] compare_kubectl: statefulset/some-name-rs0 OK [2026-06-17T20:40:21+0000] compare_kubectl: statefulset/some-name-cfg OK ----------------------------------------------------------------------------------- test 1.23.0 ----------------------------------------------------------------------------------- perconaservermongodb.psmdb.percona.com/some-name patched ----------------------------------------------------------------------------------- check if Pod started ----------------------------------------------------------------------------------- waiting for pod/some-name-rs0-0 to be ready.OK waiting for pod/some-name-rs0-1 to be ready.OK waiting for pod/some-name-rs0-2 to be ready.OK Waiting for cluster readyness........................................................................................................................................................................ waiting for pod/some-name-cfg-0 to be ready.OK waiting for pod/some-name-cfg-1 to be ready.OK waiting for pod/some-name-cfg-2 to be ready.OK Waiting for cluster readyness waiting for pod/some-name-mongos-0 to be ready.OK waiting for pod/some-name-mongos-1 to be ready.OK waiting for pod/some-name-mongos-2 to be ready.OK Waiting for cluster readyness waiting for cluster readyness.OK E0617 20:45:58.228169 394961 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781729155050661000&timeoutSeconds=486&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0617 20:45:59.998740 394981 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781729155050661000&timeoutSeconds=352&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0617 20:46:01.757903 394999 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781729160050669000&timeoutSeconds=542&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0617 20:46:03.523934 395019 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781729160050669000&timeoutSeconds=354&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0617 20:46:05.283472 395037 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781729165050762000&timeoutSeconds=355&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0617 20:46:07.100521 395058 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781729165050762000&timeoutSeconds=319&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0617 20:46:08.863096 395078 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781729165050762000&timeoutSeconds=589&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0617 20:46:10.626201 395096 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781729170050837000&timeoutSeconds=524&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0617 20:46:12.378206 395116 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781729170050837000&timeoutSeconds=409&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0617 20:46:14.136358 395134 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781729170050837000&timeoutSeconds=390&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met ----------------------------------------------------------------------------------- renew some-name-ssl ----------------------------------------------------------------------------------- Manually triggered issuance of Certificate upgrade-consistency-sharded-tls-9064/some-name-ssl waiting for pod/some-name-rs0-0 to be ready.OK waiting for pod/some-name-rs0-1 to be ready.OK waiting for pod/some-name-rs0-2 to be ready.OK Waiting for cluster readyness.............................................................................................................................................................................. waiting for pod/some-name-cfg-0 to be ready.OK waiting for pod/some-name-cfg-1 to be ready.OK waiting for pod/some-name-cfg-2 to be ready.OK Waiting for cluster readyness waiting for pod/some-name-mongos-0 to be ready.OK waiting for pod/some-name-mongos-1 to be ready.OK waiting for pod/some-name-mongos-2 to be ready.OK Waiting for cluster readyness waiting for cluster readyness.OK E0617 20:52:05.014217 400326 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781729524664953000&timeoutSeconds=396&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0617 20:52:06.742664 400348 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781729524664953000&timeoutSeconds=513&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0617 20:52:08.505091 400366 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781729524664953000&timeoutSeconds=472&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0617 20:52:10.266579 400388 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781729529664989000&timeoutSeconds=596&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0617 20:52:12.016689 400406 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781729529664989000&timeoutSeconds=447&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0617 20:52:13.783099 400428 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781729529664989000&timeoutSeconds=463&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0617 20:52:15.521639 400448 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781729534665294000&timeoutSeconds=464&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0617 20:52:17.243643 400466 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781729534665294000&timeoutSeconds=356&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0617 20:52:18.997646 400488 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781729534665294000&timeoutSeconds=354&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0617 20:52:20.756705 400506 reflector.go:227] "Failed to watch" err="Get \"https://34.28.158.181/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-9064/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781729539665305000&timeoutSeconds=505&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met ----------------------------------------------------------------------------------- renew some-name-ssl-internal ----------------------------------------------------------------------------------- Manually triggered issuance of Certificate upgrade-consistency-sharded-tls-9064/some-name-ssl-internal waiting for pod/some-name-rs0-0 to be ready.OK waiting for pod/some-name-rs0-1 to be ready.OK waiting for pod/some-name-rs0-2 to be ready.OK Waiting for cluster readyness........................................................................................................................................................................ waiting for pod/some-name-cfg-0 to be ready.OK waiting for pod/some-name-cfg-1 to be ready.OK waiting for pod/some-name-cfg-2 to be ready.OK Waiting for cluster readyness waiting for pod/some-name-mongos-0 to be ready.OK waiting for pod/some-name-mongos-1 to be ready.OK waiting for pod/some-name-mongos-2 to be ready.OK Waiting for cluster readyness waiting for cluster readyness.OK ----------------------------------------------------------------------------------- check if service and statefulset created with expected config ----------------------------------------------------------------------------------- [2026-06-17T20:58:02+0000] compare_kubectl: service/some-name-rs0 OK [2026-06-17T20:58:02+0000] compare_kubectl: service/some-name-cfg OK [2026-06-17T20:58:03+0000] compare_kubectl: statefulset/some-name-rs0 OK [2026-06-17T20:58:04+0000] compare_kubectl: statefulset/some-name-cfg OK ----------------------------------------------------------------------------------- destroy cluster/operator and all other resources ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- Delete psmdb-backup ----------------------------------------------------------------------------------- No resources found in upgrade-consistency-sharded-tls-9064 namespace. ----------------------------------------------------------------------------------- get and delete old CRDs and RBAC ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io "perconaservermongodbbackups.psmdb.percona.com" deleted customresourcedefinition.apiextensions.k8s.io "perconaservermongodbrestores.psmdb.percona.com" deleted customresourcedefinition.apiextensions.k8s.io "perconaservermongodbs.psmdb.percona.com" deleted grep: warning: stray \ before - grep: warning: stray \ before - error: the server doesn't have a resource type "perconaservermongodbbackups" + kubectl patch perconaservermongodbbackups.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbbackups" error: the server doesn't have a resource type "perconaservermongodbrestores" + kubectl patch perconaservermongodbrestores.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbrestores" + kubectl patch perconaservermongodbs.psmdb.percona.com -n upgrade-consistency-sharded-tls-9064 some-name --type=merge -p '{"metadata":{"finalizers":[]}}' perconaservermongodb.psmdb.percona.com/some-name patched customresourcedefinition.apiextensions.k8s.io/perconaservermongodbs.psmdb.percona.com condition met clusterrole.rbac.authorization.k8s.io "percona-server-mongodb-operator" deleted clusterrolebinding.rbac.authorization.k8s.io "service-account-percona-server-mongodb-operator" deleted namespace "cert-manager" deleted customresourcedefinition.apiextensions.k8s.io "challenges.acme.cert-manager.io" deleted customresourcedefinition.apiextensions.k8s.io "orders.acme.cert-manager.io" deleted customresourcedefinition.apiextensions.k8s.io "certificaterequests.cert-manager.io" deleted customresourcedefinition.apiextensions.k8s.io "certificates.cert-manager.io" deleted customresourcedefinition.apiextensions.k8s.io "clusterissuers.cert-manager.io" deleted customresourcedefinition.apiextensions.k8s.io "issuers.cert-manager.io" deleted serviceaccount "cert-manager-cainjector" deleted from cert-manager namespace serviceaccount "cert-manager" deleted from cert-manager namespace serviceaccount "cert-manager-webhook" deleted from cert-manager namespace clusterrole.rbac.authorization.k8s.io "cert-manager-cainjector" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-controller-issuers" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-controller-clusterissuers" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-controller-certificates" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-controller-orders" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-controller-challenges" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-controller-ingress-shim" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-cluster-view" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-view" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-edit" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-controller-approve:cert-manager-io" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-controller-certificatesigningrequests" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-webhook:subjectaccessreviews" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-cainjector" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-issuers" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-clusterissuers" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-certificates" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-orders" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-challenges" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-ingress-shim" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-approve:cert-manager-io" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-certificatesigningrequests" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-webhook:subjectaccessreviews" deleted role.rbac.authorization.k8s.io "cert-manager-cainjector:leaderelection" deleted from kube-system namespace role.rbac.authorization.k8s.io "cert-manager:leaderelection" deleted from kube-system namespace role.rbac.authorization.k8s.io "cert-manager-tokenrequest" deleted from cert-manager namespace role.rbac.authorization.k8s.io "cert-manager-webhook:dynamic-serving" deleted from cert-manager namespace rolebinding.rbac.authorization.k8s.io "cert-manager-cainjector:leaderelection" deleted from kube-system namespace rolebinding.rbac.authorization.k8s.io "cert-manager:leaderelection" deleted from kube-system namespace rolebinding.rbac.authorization.k8s.io "cert-manager-tokenrequest" deleted from cert-manager namespace rolebinding.rbac.authorization.k8s.io "cert-manager-webhook:dynamic-serving" deleted from cert-manager namespace service "cert-manager-cainjector" deleted from cert-manager namespace service "cert-manager" deleted from cert-manager namespace service "cert-manager-webhook" deleted from cert-manager namespace deployment.apps "cert-manager-cainjector" deleted from cert-manager namespace deployment.apps "cert-manager" deleted from cert-manager namespace deployment.apps "cert-manager-webhook" deleted from cert-manager namespace mutatingwebhookconfiguration.admissionregistration.k8s.io "cert-manager-webhook" deleted validatingwebhookconfiguration.admissionregistration.k8s.io "cert-manager-webhook" deleted ----------------------------------------------------------------------------------- test passed -----------------------------------------------------------------------------------