Log: /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2328/e2e-tests/logs/upgrade-consistency-sharded-tls.log Warning: version difference between client (1.36) and server (1.33) exceeds the supported minor version skew of +/-1 Warning: version difference between client (1.36) and server (1.33) exceeds the supported minor version skew of +/-1 Warning: version difference between client (1.36) and server (1.33) exceeds the supported minor version skew of +/-1 ----------------------------------------------------------------------------------- get and delete old CRDs and RBAC ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io "perconaservermongodbbackups.psmdb.percona.com" deleted customresourcedefinition.apiextensions.k8s.io "perconaservermongodbrestores.psmdb.percona.com" deleted customresourcedefinition.apiextensions.k8s.io "perconaservermongodbs.psmdb.percona.com" deleted grep: warning: stray \ before - grep: warning: stray \ before - error: the server doesn't have a resource type "perconaservermongodbbackups" + kubectl patch perconaservermongodbbackups.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbbackups" error: the server doesn't have a resource type "perconaservermongodbrestores" + kubectl patch perconaservermongodbrestores.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbrestores" error: the server doesn't have a resource type "perconaservermongodbs" + kubectl patch perconaservermongodbs.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbs" clusterrole.rbac.authorization.k8s.io "percona-server-mongodb-operator" deleted clusterrolebinding.rbac.authorization.k8s.io "service-account-percona-server-mongodb-operator" deleted Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found ----------------------------------------------------------------------------------- destroy chaos-mesh ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces psmdb-operator ----------------------------------------------------------------------------------- namespace "stable-resource-version-862" deleted namespace "psmdb-operator" deleted ----------------------------------------------------------------------------------- create namespace psmdb-operator ----------------------------------------------------------------------------------- namespace/psmdb-operator created Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-2328-37c432905-3-cluster1" modified. ----------------------------------------------------------------------------------- start PSMDB operator: docker.io/perconalab/percona-server-mongodb-operator:PR-2328-37c432905 ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io/perconaservermongodbbackups.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbrestores.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbs.psmdb.percona.com serverside-applied clusterrole.rbac.authorization.k8s.io/percona-server-mongodb-operator created serviceaccount/percona-server-mongodb-operator created clusterrolebinding.rbac.authorization.k8s.io/service-account-percona-server-mongodb-operator created deployment.apps/percona-server-mongodb-operator created waiting for pod/percona-server-mongodb-operator-7dbf5fc47b-tc6zg to be ready.OK Print operator info from log 2026-05-21T09:21:51.271Z INFO setup Manager starting up {"gitCommit": "37c432905eccde48178453d4542cff5f669efa6b", "gitBranch": "PR-2328-37c432905", "buildTime": "", "goVersion": "go1.26.3", "os": "linux", "arch": "amd64"} ----------------------------------------------------------------------------------- destroy chaos-mesh ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces upgrade-consistency-sharded-tls-5094 ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- create namespace upgrade-consistency-sharded-tls-5094 ----------------------------------------------------------------------------------- namespace/upgrade-consistency-sharded-tls-5094 created Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-2328-37c432905-3-cluster1" modified. ----------------------------------------------------------------------------------- deploy cert manager ----------------------------------------------------------------------------------- namespace/cert-manager created namespace/cert-manager labeled namespace/cert-manager configured customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io created serviceaccount/cert-manager-cainjector created serviceaccount/cert-manager created serviceaccount/cert-manager-webhook created clusterrole.rbac.authorization.k8s.io/cert-manager-cainjector created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-issuers created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificates created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-orders created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-challenges created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created clusterrole.rbac.authorization.k8s.io/cert-manager-cluster-view created clusterrole.rbac.authorization.k8s.io/cert-manager-view created clusterrole.rbac.authorization.k8s.io/cert-manager-edit created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests created clusterrole.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-cainjector created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-issuers created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificates created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-orders created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-challenges created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews created role.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created role.rbac.authorization.k8s.io/cert-manager:leaderelection created role.rbac.authorization.k8s.io/cert-manager-tokenrequest created role.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created rolebinding.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created rolebinding.rbac.authorization.k8s.io/cert-manager:leaderelection created rolebinding.rbac.authorization.k8s.io/cert-manager-tokenrequest created rolebinding.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created service/cert-manager-cainjector created service/cert-manager created service/cert-manager-webhook created deployment.apps/cert-manager-cainjector created deployment.apps/cert-manager created deployment.apps/cert-manager-webhook created mutatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created Warning: resource namespaces/cert-manager is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically. pod/cert-manager-559d798845-lfjqk condition met pod/cert-manager-cainjector-64958d9c7c-44c9d condition met pod/cert-manager-webhook-7fb6f99b56-rzs94 condition met E0521 09:22:49.001372 1260728 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/api/v1/namespaces/cert-manager/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dcert-manager-cainjector-64958d9c7c-44c9d&resourceVersion=1779355368749215010&timeoutSeconds=350&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" ----------------------------------------------------------------------------------- create secrets and start client ----------------------------------------------------------------------------------- secret/some-users created deployment.apps/psmdb-client created role.rbac.authorization.k8s.io/cmctl created serviceaccount/cmctl created rolebinding.rbac.authorization.k8s.io/service-account-cmctl created deployment.apps/cmctl created ----------------------------------------------------------------------------------- create first PSMDB cluster 1.21.2 some-name ----------------------------------------------------------------------------------- perconaservermongodb.psmdb.percona.com/some-name created ----------------------------------------------------------------------------------- check if Pod started ----------------------------------------------------------------------------------- waiting for pod/some-name-rs0-0 to be ready..............OK waiting for pod/some-name-rs0-1 to be ready.............OK waiting for pod/some-name-rs0-2 to be ready..............OK Waiting for cluster readyness.................................. waiting for pod/some-name-cfg-0 to be ready.OK waiting for pod/some-name-cfg-1 to be ready.OK waiting for pod/some-name-cfg-2 to be ready.OK Waiting for cluster readyness waiting for pod/some-name-mongos-0 to be ready.OK waiting for pod/some-name-mongos-1 to be ready.OK waiting for pod/some-name-mongos-2 to be ready.OK Waiting for cluster readyness waiting for cluster readyness.OK ----------------------------------------------------------------------------------- check if Pod started ----------------------------------------------------------------------------------- waiting for pod/some-name-rs0-0 to be ready.OK waiting for pod/some-name-rs0-1 to be ready.OK waiting for pod/some-name-rs0-2 to be ready.OK Waiting for cluster readyness waiting for pod/some-name-cfg-0 to be ready.OK waiting for pod/some-name-cfg-1 to be ready.OK waiting for pod/some-name-cfg-2 to be ready.OK Waiting for cluster readyness waiting for pod/some-name-mongos-0 to be ready.OK waiting for pod/some-name-mongos-1 to be ready.OK waiting for pod/some-name-mongos-2 to be ready.OK Waiting for cluster readyness waiting for cluster readyness.OK E0521 09:29:55.484538 1337146 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1779355791743225000&timeoutSeconds=408&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0521 09:29:57.688542 1337520 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1779355796743269000&timeoutSeconds=455&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0521 09:29:59.920724 1337785 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1779355796743269000&timeoutSeconds=408&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0521 09:30:02.125040 1337978 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1779355801744005000&timeoutSeconds=596&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0521 09:30:04.313832 1338214 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1779355801744005000&timeoutSeconds=395&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0521 09:30:06.548291 1338472 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1779355801744005000&timeoutSeconds=480&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0521 09:30:08.755112 1338835 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1779355806744034000&timeoutSeconds=585&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0521 09:30:10.966044 1339068 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1779355806744034000&timeoutSeconds=396&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0521 09:30:13.160158 1339300 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1779355811744068000&timeoutSeconds=405&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0521 09:30:15.395811 1339680 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1779355811744068000&timeoutSeconds=490&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met ----------------------------------------------------------------------------------- renew some-name-ssl ----------------------------------------------------------------------------------- Manually triggered issuance of Certificate upgrade-consistency-sharded-tls-5094/some-name-ssl waiting for pod/some-name-rs0-0 to be ready.OK waiting for pod/some-name-rs0-1 to be ready.OK waiting for pod/some-name-rs0-2 to be ready.OK Waiting for cluster readyness........................................................................................................................................ waiting for pod/some-name-cfg-0 to be ready.OK waiting for pod/some-name-cfg-1 to be ready.OK waiting for pod/some-name-cfg-2 to be ready.OK Waiting for cluster readyness waiting for pod/some-name-mongos-0 to be ready.OK waiting for pod/some-name-mongos-1 to be ready.OK waiting for pod/some-name-mongos-2 to be ready.OK Waiting for cluster readyness waiting for cluster readyness.OK E0521 09:36:18.454275 1392749 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1779356175316521000&timeoutSeconds=498&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0521 09:36:20.614916 1393025 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1779356180316772000&timeoutSeconds=303&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0521 09:36:22.794213 1393186 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1779356180316772000&timeoutSeconds=456&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0521 09:36:24.981914 1393371 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1779356180316772000&timeoutSeconds=323&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0521 09:36:27.181993 1393592 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1779356185317213000&timeoutSeconds=409&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0521 09:36:29.406539 1393800 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1779356185317213000&timeoutSeconds=568&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0521 09:36:31.608459 1394013 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1779356190317238000&timeoutSeconds=534&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0521 09:36:33.824505 1394243 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1779356190317238000&timeoutSeconds=596&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0521 09:36:36.042922 1394542 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1779356195317394000&timeoutSeconds=370&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0521 09:36:38.258979 1394706 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1779356195317394000&timeoutSeconds=532&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met ----------------------------------------------------------------------------------- renew some-name-ssl-internal ----------------------------------------------------------------------------------- Manually triggered issuance of Certificate upgrade-consistency-sharded-tls-5094/some-name-ssl-internal waiting for pod/some-name-rs0-0 to be ready.OK waiting for pod/some-name-rs0-1 to be ready.OK waiting for pod/some-name-rs0-2 to be ready.OK Waiting for cluster readyness.............................................................................................................................................. waiting for pod/some-name-cfg-0 to be ready.OK waiting for pod/some-name-cfg-1 to be ready.OK waiting for pod/some-name-cfg-2 to be ready.OK Waiting for cluster readyness waiting for pod/some-name-mongos-0 to be ready.OK waiting for pod/some-name-mongos-1 to be ready.OK waiting for pod/some-name-mongos-2 to be ready.OK Waiting for cluster readyness waiting for cluster readyness.OK ----------------------------------------------------------------------------------- check if service and statefulset created with expected config ----------------------------------------------------------------------------------- [2026-05-21T09:42:47+0000] compare_kubectl: service/some-name-rs0 OK [2026-05-21T09:42:48+0000] compare_kubectl: service/some-name-cfg OK [2026-05-21T09:42:48+0000] compare_kubectl: statefulset/some-name-rs0 OK [2026-05-21T09:42:49+0000] compare_kubectl: statefulset/some-name-cfg OK ----------------------------------------------------------------------------------- test 1.22.0 ----------------------------------------------------------------------------------- perconaservermongodb.psmdb.percona.com/some-name patched ----------------------------------------------------------------------------------- check if Pod started ----------------------------------------------------------------------------------- waiting for pod/some-name-rs0-0 to be ready.OK waiting for pod/some-name-rs0-1 to be ready.OK waiting for pod/some-name-rs0-2 to be ready.OK Waiting for cluster readyness................................................................................................... waiting for pod/some-name-cfg-0 to be ready.OK waiting for pod/some-name-cfg-1 to be ready.OK waiting for pod/some-name-cfg-2 to be ready.OK Waiting for cluster readyness waiting for pod/some-name-mongos-0 to be ready.OK waiting for pod/some-name-mongos-1 to be ready.OK waiting for pod/some-name-mongos-2 to be ready.OK Waiting for cluster readyness waiting for cluster readyness.OK E0521 09:47:26.538130 1461731 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1779356843705718000&timeoutSeconds=404&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0521 09:47:28.900094 1461881 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1779356848705734000&timeoutSeconds=471&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0521 09:47:31.065030 1462034 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1779356848705734000&timeoutSeconds=494&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0521 09:47:33.222721 1462186 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1779356848705734000&timeoutSeconds=562&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0521 09:47:35.382009 1462305 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1779356853705753000&timeoutSeconds=504&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0521 09:47:37.560368 1462451 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1779356853705753000&timeoutSeconds=373&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0521 09:47:39.757245 1462567 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1779356858705832000&timeoutSeconds=503&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0521 09:47:41.951845 1462720 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1779356858705832000&timeoutSeconds=439&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0521 09:47:44.194789 1462839 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1779356863705901000&timeoutSeconds=371&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0521 09:47:46.442761 1462963 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1779356863705901000&timeoutSeconds=588&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met ----------------------------------------------------------------------------------- renew some-name-ssl ----------------------------------------------------------------------------------- Manually triggered issuance of Certificate upgrade-consistency-sharded-tls-5094/some-name-ssl waiting for pod/some-name-rs0-0 to be ready.OK waiting for pod/some-name-rs0-1 to be ready.OK waiting for pod/some-name-rs0-2 to be ready.OK Waiting for cluster readyness.................................................................................................................................................... waiting for pod/some-name-cfg-0 to be ready.OK waiting for pod/some-name-cfg-1 to be ready.OK waiting for pod/some-name-cfg-2 to be ready.OK Waiting for cluster readyness waiting for pod/some-name-mongos-0 to be ready.OK waiting for pod/some-name-mongos-1 to be ready.OK waiting for pod/some-name-mongos-2 to be ready.OK Waiting for cluster readyness waiting for cluster readyness.OK E0521 09:54:04.159109 1489932 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1779357241274559000&timeoutSeconds=446&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0521 09:54:06.342345 1489980 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1779357241274559000&timeoutSeconds=441&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0521 09:54:08.506243 1490103 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1779357246274628000&timeoutSeconds=597&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0521 09:54:10.699367 1490298 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1779357246274628000&timeoutSeconds=484&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0521 09:54:12.874958 1490408 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1779357251275188000&timeoutSeconds=333&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0521 09:54:15.061826 1490557 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1779357251275188000&timeoutSeconds=397&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0521 09:54:17.250152 1490719 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1779357256275478000&timeoutSeconds=444&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0521 09:54:19.456589 1490895 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1779357256275478000&timeoutSeconds=366&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0521 09:54:21.653193 1491040 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1779357261275496000&timeoutSeconds=583&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0521 09:54:23.827177 1491156 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1779357261275496000&timeoutSeconds=588&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met ----------------------------------------------------------------------------------- renew some-name-ssl-internal ----------------------------------------------------------------------------------- Manually triggered issuance of Certificate upgrade-consistency-sharded-tls-5094/some-name-ssl-internal waiting for pod/some-name-rs0-0 to be ready.OK waiting for pod/some-name-rs0-1 to be ready.OK waiting for pod/some-name-rs0-2 to be ready.OK Waiting for cluster readyness.......................................................................................................................................................... waiting for pod/some-name-cfg-0 to be ready.OK waiting for pod/some-name-cfg-1 to be ready.OK waiting for pod/some-name-cfg-2 to be ready.OK Waiting for cluster readyness waiting for pod/some-name-mongos-0 to be ready.OK waiting for pod/some-name-mongos-1 to be ready.OK waiting for pod/some-name-mongos-2 to be ready.OK Waiting for cluster readyness waiting for cluster readyness.OK ----------------------------------------------------------------------------------- check if service and statefulset created with expected config ----------------------------------------------------------------------------------- [2026-05-21T10:00:27+0000] compare_kubectl: service/some-name-rs0 OK [2026-05-21T10:00:28+0000] compare_kubectl: service/some-name-cfg OK [2026-05-21T10:00:28+0000] compare_kubectl: statefulset/some-name-rs0 OK [2026-05-21T10:00:29+0000] compare_kubectl: statefulset/some-name-cfg OK ----------------------------------------------------------------------------------- test 1.23.0 ----------------------------------------------------------------------------------- perconaservermongodb.psmdb.percona.com/some-name patched ----------------------------------------------------------------------------------- check if Pod started ----------------------------------------------------------------------------------- waiting for pod/some-name-rs0-0 to be ready.OK waiting for pod/some-name-rs0-1 to be ready.OK waiting for pod/some-name-rs0-2 to be ready.OK Waiting for cluster readyness waiting for pod/some-name-cfg-0 to be ready.OK waiting for pod/some-name-cfg-1 to be ready.OK waiting for pod/some-name-cfg-2 to be ready.OK Waiting for cluster readyness waiting for pod/some-name-mongos-0 to be ready.OK waiting for pod/some-name-mongos-1 to be ready.OK waiting for pod/some-name-mongos-2 to be ready.OK Waiting for cluster readyness waiting for cluster readyness.OK E0521 10:01:42.381838 1509904 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1779357698132417000&timeoutSeconds=457&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0521 10:01:44.630422 1509951 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1779357703132481000&timeoutSeconds=529&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0521 10:01:46.353521 1509985 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1779357703132481000&timeoutSeconds=583&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0521 10:01:48.065440 1510034 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1779357703132481000&timeoutSeconds=382&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0521 10:01:50.267111 1510062 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1779357708132519000&timeoutSeconds=425&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0521 10:01:51.979336 1510098 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1779357708132519000&timeoutSeconds=508&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0521 10:01:53.701600 1510119 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1779357713132528000&timeoutSeconds=529&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0521 10:01:55.418507 1510141 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1779357713132528000&timeoutSeconds=525&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0521 10:01:57.127136 1510166 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1779357713132528000&timeoutSeconds=406&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0521 10:01:59.320127 1510242 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1779357718132550000&timeoutSeconds=561&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met ----------------------------------------------------------------------------------- renew some-name-ssl ----------------------------------------------------------------------------------- Manually triggered issuance of Certificate upgrade-consistency-sharded-tls-5094/some-name-ssl waiting for pod/some-name-rs0-0 to be ready.OK waiting for pod/some-name-rs0-1 to be ready.OK waiting for pod/some-name-rs0-2 to be ready.OK Waiting for cluster readyness......................................................................................................................................................................................... waiting for pod/some-name-cfg-0 to be ready.OK waiting for pod/some-name-cfg-1 to be ready.OK waiting for pod/some-name-cfg-2 to be ready.OK Waiting for cluster readyness waiting for pod/some-name-mongos-0 to be ready.OK waiting for pod/some-name-mongos-1 to be ready.OK waiting for pod/some-name-mongos-2 to be ready.OK Waiting for cluster readyness waiting for cluster readyness.OK E0521 10:08:17.022527 1518819 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1779358093360399000&timeoutSeconds=548&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0521 10:08:19.235970 1518869 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1779358098360445000&timeoutSeconds=513&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0521 10:08:20.929672 1518905 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1779358098360445000&timeoutSeconds=518&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0521 10:08:22.635077 1518929 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1779358098360445000&timeoutSeconds=423&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0521 10:08:24.331694 1518947 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1779358103360485000&timeoutSeconds=447&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0521 10:08:26.042062 1518971 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1779358103360485000&timeoutSeconds=494&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0521 10:08:27.732042 1519026 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1779358103360485000&timeoutSeconds=501&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0521 10:08:29.885332 1519072 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1779358108360650000&timeoutSeconds=304&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0521 10:08:31.586546 1519107 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1779358108360650000&timeoutSeconds=527&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0521 10:08:33.289645 1519153 reflector.go:227] "Failed to watch" err="Get \"https://34.30.188.147/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-5094/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1779358108360650000&timeoutSeconds=565&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met ----------------------------------------------------------------------------------- renew some-name-ssl-internal ----------------------------------------------------------------------------------- Manually triggered issuance of Certificate upgrade-consistency-sharded-tls-5094/some-name-ssl-internal waiting for pod/some-name-rs0-0 to be ready.OK waiting for pod/some-name-rs0-1 to be ready.OK waiting for pod/some-name-rs0-2 to be ready.OK Waiting for cluster readyness................................................................................................................................................................................ waiting for pod/some-name-cfg-0 to be ready.OK waiting for pod/some-name-cfg-1 to be ready.OK waiting for pod/some-name-cfg-2 to be ready.OK Waiting for cluster readyness waiting for pod/some-name-mongos-0 to be ready.OK waiting for pod/some-name-mongos-1 to be ready.OK waiting for pod/some-name-mongos-2 to be ready.OK Waiting for cluster readyness waiting for cluster readyness.OK ----------------------------------------------------------------------------------- check if service and statefulset created with expected config ----------------------------------------------------------------------------------- [2026-05-21T10:14:33+0000] compare_kubectl: service/some-name-rs0 OK [2026-05-21T10:14:33+0000] compare_kubectl: service/some-name-cfg OK [2026-05-21T10:14:34+0000] compare_kubectl: statefulset/some-name-rs0 OK [2026-05-21T10:14:34+0000] compare_kubectl: statefulset/some-name-cfg OK ----------------------------------------------------------------------------------- destroy cluster/operator and all other resources ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- Delete psmdb-backup ----------------------------------------------------------------------------------- No resources found in upgrade-consistency-sharded-tls-5094 namespace. ----------------------------------------------------------------------------------- get and delete old CRDs and RBAC ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io "perconaservermongodbbackups.psmdb.percona.com" deleted customresourcedefinition.apiextensions.k8s.io "perconaservermongodbrestores.psmdb.percona.com" deleted customresourcedefinition.apiextensions.k8s.io "perconaservermongodbs.psmdb.percona.com" deleted grep: warning: stray \ before - grep: warning: stray \ before - error: the server doesn't have a resource type "perconaservermongodbbackups" + kubectl patch perconaservermongodbbackups.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbbackups" error: the server doesn't have a resource type "perconaservermongodbrestores" + kubectl patch perconaservermongodbrestores.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbrestores" + kubectl patch perconaservermongodbs.psmdb.percona.com -n upgrade-consistency-sharded-tls-5094 some-name --type=merge -p '{"metadata":{"finalizers":[]}}' perconaservermongodb.psmdb.percona.com/some-name patched customresourcedefinition.apiextensions.k8s.io/perconaservermongodbs.psmdb.percona.com condition met clusterrole.rbac.authorization.k8s.io "percona-server-mongodb-operator" deleted clusterrolebinding.rbac.authorization.k8s.io "service-account-percona-server-mongodb-operator" deleted namespace "cert-manager" deleted customresourcedefinition.apiextensions.k8s.io "challenges.acme.cert-manager.io" deleted customresourcedefinition.apiextensions.k8s.io "orders.acme.cert-manager.io" deleted customresourcedefinition.apiextensions.k8s.io "certificaterequests.cert-manager.io" deleted customresourcedefinition.apiextensions.k8s.io "certificates.cert-manager.io" deleted customresourcedefinition.apiextensions.k8s.io "clusterissuers.cert-manager.io" deleted customresourcedefinition.apiextensions.k8s.io "issuers.cert-manager.io" deleted serviceaccount "cert-manager-cainjector" deleted from cert-manager namespace serviceaccount "cert-manager" deleted from cert-manager namespace serviceaccount "cert-manager-webhook" deleted from cert-manager namespace clusterrole.rbac.authorization.k8s.io "cert-manager-cainjector" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-controller-issuers" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-controller-clusterissuers" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-controller-certificates" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-controller-orders" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-controller-challenges" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-controller-ingress-shim" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-cluster-view" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-view" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-edit" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-controller-approve:cert-manager-io" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-controller-certificatesigningrequests" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-webhook:subjectaccessreviews" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-cainjector" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-issuers" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-clusterissuers" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-certificates" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-orders" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-challenges" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-ingress-shim" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-approve:cert-manager-io" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-certificatesigningrequests" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-webhook:subjectaccessreviews" deleted role.rbac.authorization.k8s.io "cert-manager-cainjector:leaderelection" deleted from kube-system namespace role.rbac.authorization.k8s.io "cert-manager:leaderelection" deleted from kube-system namespace role.rbac.authorization.k8s.io "cert-manager-tokenrequest" deleted from cert-manager namespace role.rbac.authorization.k8s.io "cert-manager-webhook:dynamic-serving" deleted from cert-manager namespace rolebinding.rbac.authorization.k8s.io "cert-manager-cainjector:leaderelection" deleted from kube-system namespace rolebinding.rbac.authorization.k8s.io "cert-manager:leaderelection" deleted from kube-system namespace rolebinding.rbac.authorization.k8s.io "cert-manager-tokenrequest" deleted from cert-manager namespace rolebinding.rbac.authorization.k8s.io "cert-manager-webhook:dynamic-serving" deleted from cert-manager namespace service "cert-manager-cainjector" deleted from cert-manager namespace service "cert-manager" deleted from cert-manager namespace service "cert-manager-webhook" deleted from cert-manager namespace deployment.apps "cert-manager-cainjector" deleted from cert-manager namespace deployment.apps "cert-manager" deleted from cert-manager namespace deployment.apps "cert-manager-webhook" deleted from cert-manager namespace mutatingwebhookconfiguration.admissionregistration.k8s.io "cert-manager-webhook" deleted validatingwebhookconfiguration.admissionregistration.k8s.io "cert-manager-webhook" deleted ----------------------------------------------------------------------------------- test passed -----------------------------------------------------------------------------------