Log: /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2280/e2e-tests/logs/upgrade-consistency-sharded-tls.log Warning: version difference between client (1.36) and server (1.33) exceeds the supported minor version skew of +/-1 Warning: version difference between client (1.36) and server (1.33) exceeds the supported minor version skew of +/-1 Warning: version difference between client (1.36) and server (1.33) exceeds the supported minor version skew of +/-1 ----------------------------------------------------------------------------------- get and delete old CRDs and RBAC ----------------------------------------------------------------------------------- grep: warning: stray \ before - grep: warning: stray \ before - error: the server doesn't have a resource type "perconaservermongodbbackups" + kubectl patch perconaservermongodbbackups.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbbackups" error: the server doesn't have a resource type "perconaservermongodbrestores" + kubectl patch perconaservermongodbrestores.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbrestores" error: the server doesn't have a resource type "perconaservermongodbs" + kubectl patch perconaservermongodbs.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbs" Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found ----------------------------------------------------------------------------------- destroy chaos-mesh ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces psmdb-operator ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- create namespace psmdb-operator ----------------------------------------------------------------------------------- namespace/psmdb-operator created Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-2280-43b99c0df-27-cluster8" modified. ----------------------------------------------------------------------------------- start PSMDB operator: docker.io/perconalab/percona-server-mongodb-operator:PR-2280-43b99c0df ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io/perconaservermongodbbackups.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbrestores.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbs.psmdb.percona.com serverside-applied clusterrole.rbac.authorization.k8s.io/percona-server-mongodb-operator created serviceaccount/percona-server-mongodb-operator created clusterrolebinding.rbac.authorization.k8s.io/service-account-percona-server-mongodb-operator created deployment.apps/percona-server-mongodb-operator created waiting for pod/percona-server-mongodb-operator-cb7c4758f-tm9cs to be ready.OK Print operator info from log 2026-06-10T21:42:20.946Z INFO setup Manager starting up {"gitCommit": "43b99c0df015bb8dc57a7c8cdb0b358bb38ba1b1", "gitBranch": "PR-2280-43b99c0df", "buildTime": "", "goVersion": "go1.26.4", "os": "linux", "arch": "amd64"} ----------------------------------------------------------------------------------- destroy chaos-mesh ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces upgrade-consistency-sharded-tls-24035 ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- create namespace upgrade-consistency-sharded-tls-24035 ----------------------------------------------------------------------------------- namespace/upgrade-consistency-sharded-tls-24035 created Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-2280-43b99c0df-27-cluster8" modified. ----------------------------------------------------------------------------------- deploy cert manager ----------------------------------------------------------------------------------- namespace/cert-manager created namespace/cert-manager labeled namespace/cert-manager configured customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io created serviceaccount/cert-manager-cainjector created serviceaccount/cert-manager created serviceaccount/cert-manager-webhook created clusterrole.rbac.authorization.k8s.io/cert-manager-cainjector created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-issuers created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificates created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-orders created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-challenges created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created clusterrole.rbac.authorization.k8s.io/cert-manager-cluster-view created clusterrole.rbac.authorization.k8s.io/cert-manager-view created clusterrole.rbac.authorization.k8s.io/cert-manager-edit created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests created clusterrole.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-cainjector created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-issuers created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificates created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-orders created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-challenges created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews created role.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created role.rbac.authorization.k8s.io/cert-manager:leaderelection created role.rbac.authorization.k8s.io/cert-manager-tokenrequest created role.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created rolebinding.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created rolebinding.rbac.authorization.k8s.io/cert-manager:leaderelection created rolebinding.rbac.authorization.k8s.io/cert-manager-tokenrequest created rolebinding.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created service/cert-manager-cainjector created service/cert-manager created service/cert-manager-webhook created deployment.apps/cert-manager-cainjector created deployment.apps/cert-manager created deployment.apps/cert-manager-webhook created mutatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created Warning: resource namespaces/cert-manager is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically. pod/cert-manager-559d798845-w8r4w condition met pod/cert-manager-cainjector-64958d9c7c-lrfkp condition met pod/cert-manager-webhook-7fb6f99b56-bdgj8 condition met E0610 21:43:17.027753 3322681 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/api/v1/namespaces/cert-manager/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dcert-manager-559d798845-w8r4w&resourceVersion=1781127796680975020&timeoutSeconds=311&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" E0610 21:43:17.329568 3322681 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/api/v1/namespaces/cert-manager/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dcert-manager-cainjector-64958d9c7c-lrfkp&resourceVersion=1781127796680975020&timeoutSeconds=593&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" ----------------------------------------------------------------------------------- create secrets and start client ----------------------------------------------------------------------------------- secret/some-users created deployment.apps/psmdb-client created role.rbac.authorization.k8s.io/cmctl created serviceaccount/cmctl created rolebinding.rbac.authorization.k8s.io/service-account-cmctl created deployment.apps/cmctl created ----------------------------------------------------------------------------------- create first PSMDB cluster 1.21.2 some-name ----------------------------------------------------------------------------------- perconaservermongodb.psmdb.percona.com/some-name created ----------------------------------------------------------------------------------- check if Pod started ----------------------------------------------------------------------------------- waiting for pod/some-name-rs0-0 to be ready..............OK waiting for pod/some-name-rs0-1 to be ready................OK waiting for pod/some-name-rs0-2 to be ready.............OK Waiting for cluster readyness.......................................... waiting for pod/some-name-cfg-0 to be ready.OK waiting for pod/some-name-cfg-1 to be ready.OK waiting for pod/some-name-cfg-2 to be ready.OK Waiting for cluster readyness waiting for pod/some-name-mongos-0 to be ready.OK waiting for pod/some-name-mongos-1 to be ready.OK waiting for pod/some-name-mongos-2 to be ready.OK Waiting for cluster readyness waiting for cluster readyness.OK ----------------------------------------------------------------------------------- check if Pod started ----------------------------------------------------------------------------------- waiting for pod/some-name-rs0-0 to be ready.OK waiting for pod/some-name-rs0-1 to be ready.OK waiting for pod/some-name-rs0-2 to be ready.OK Waiting for cluster readyness waiting for pod/some-name-cfg-0 to be ready.OK waiting for pod/some-name-cfg-1 to be ready.OK waiting for pod/some-name-cfg-2 to be ready.OK Waiting for cluster readyness waiting for pod/some-name-mongos-0 to be ready.OK waiting for pod/some-name-mongos-1 to be ready.OK waiting for pod/some-name-mongos-2 to be ready.OK Waiting for cluster readyness waiting for cluster readyness.OK E0610 21:50:46.007510 3372017 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781128245253355000&timeoutSeconds=591&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0610 21:50:48.260650 3372381 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781128245253355000&timeoutSeconds=541&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0610 21:50:50.468417 3372699 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781128250253372000&timeoutSeconds=322&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0610 21:50:52.794575 3372915 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781128250253372000&timeoutSeconds=474&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0610 21:50:54.532281 3373195 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781128250253372000&timeoutSeconds=381&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0610 21:50:56.735382 3373389 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781128255253421000&timeoutSeconds=349&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0610 21:50:58.998155 3373583 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781128255253421000&timeoutSeconds=378&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0610 21:51:00.720624 3373836 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781128260253435000&timeoutSeconds=390&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0610 21:51:03.039424 3373986 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781128260253435000&timeoutSeconds=527&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0610 21:51:05.282040 3374178 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781128260253435000&timeoutSeconds=358&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met ----------------------------------------------------------------------------------- renew some-name-ssl ----------------------------------------------------------------------------------- Manually triggered issuance of Certificate upgrade-consistency-sharded-tls-24035/some-name-ssl waiting for pod/some-name-rs0-0 to be ready.OK waiting for pod/some-name-rs0-1 to be ready.OK waiting for pod/some-name-rs0-2 to be ready.OK Waiting for cluster readyness.............................................................................................................................................. waiting for pod/some-name-cfg-0 to be ready.OK waiting for pod/some-name-cfg-1 to be ready.OK waiting for pod/some-name-cfg-2 to be ready.OK Waiting for cluster readyness waiting for pod/some-name-mongos-0 to be ready.OK waiting for pod/some-name-mongos-1 to be ready.OK waiting for pod/some-name-mongos-2 to be ready.OK Waiting for cluster readyness waiting for cluster readyness.OK E0610 21:57:20.496218 3408043 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781128635428903000&timeoutSeconds=380&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0610 21:57:22.207454 3408140 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781128640428965000&timeoutSeconds=373&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0610 21:57:24.484907 3408215 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781128640428965000&timeoutSeconds=357&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0610 21:57:26.769094 3408332 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781128645429004000&timeoutSeconds=544&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0610 21:57:29.008254 3408456 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781128645429004000&timeoutSeconds=359&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0610 21:57:31.282298 3408659 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781128650429021000&timeoutSeconds=484&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0610 21:57:33.000074 3408834 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781128650429021000&timeoutSeconds=384&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0610 21:57:35.349790 3408932 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781128650429021000&timeoutSeconds=553&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0610 21:57:37.678202 3409054 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781128655429075000&timeoutSeconds=353&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0610 21:57:39.999878 3409257 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781128655429075000&timeoutSeconds=379&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met ----------------------------------------------------------------------------------- renew some-name-ssl-internal ----------------------------------------------------------------------------------- Manually triggered issuance of Certificate upgrade-consistency-sharded-tls-24035/some-name-ssl-internal waiting for pod/some-name-rs0-0 to be ready.OK waiting for pod/some-name-rs0-1 to be ready.OK waiting for pod/some-name-rs0-2 to be ready.OK Waiting for cluster readyness...................................................................................................................................... waiting for pod/some-name-cfg-0 to be ready.OK waiting for pod/some-name-cfg-1 to be ready.OK waiting for pod/some-name-cfg-2 to be ready.OK Waiting for cluster readyness waiting for pod/some-name-mongos-0 to be ready.OK waiting for pod/some-name-mongos-1 to be ready.OK waiting for pod/some-name-mongos-2 to be ready.OK Waiting for cluster readyness waiting for cluster readyness.OK ----------------------------------------------------------------------------------- check if service and statefulset created with expected config ----------------------------------------------------------------------------------- [2026-06-10T22:03:41+0000] compare_kubectl: service/some-name-rs0 OK [2026-06-10T22:03:42+0000] compare_kubectl: service/some-name-cfg OK [2026-06-10T22:03:43+0000] compare_kubectl: statefulset/some-name-rs0 OK [2026-06-10T22:03:44+0000] compare_kubectl: statefulset/some-name-cfg OK ----------------------------------------------------------------------------------- test 1.22.0 ----------------------------------------------------------------------------------- perconaservermongodb.psmdb.percona.com/some-name patched ----------------------------------------------------------------------------------- check if Pod started ----------------------------------------------------------------------------------- waiting for pod/some-name-rs0-0 to be ready.OK waiting for pod/some-name-rs0-1 to be ready.OK waiting for pod/some-name-rs0-2 to be ready.OK Waiting for cluster readyness.................................................................................................... waiting for pod/some-name-cfg-0 to be ready.OK waiting for pod/some-name-cfg-1 to be ready.OK waiting for pod/some-name-cfg-2 to be ready.OK Waiting for cluster readyness waiting for pod/some-name-mongos-0 to be ready.OK waiting for pod/some-name-mongos-1 to be ready.OK waiting for pod/some-name-mongos-2 to be ready.OK Waiting for cluster readyness waiting for cluster readyness.OK E0610 22:08:12.755245 3454331 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781129289141440000&timeoutSeconds=363&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0610 22:08:15.065929 3454412 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781129294141481000&timeoutSeconds=536&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0610 22:08:17.385574 3454529 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781129294141481000&timeoutSeconds=504&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0610 22:08:19.681240 3454635 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781129299141535000&timeoutSeconds=454&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0610 22:08:21.974649 3454713 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781129299141535000&timeoutSeconds=464&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0610 22:08:23.718992 3454832 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781129299141535000&timeoutSeconds=436&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0610 22:08:25.997143 3454864 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781129304141577000&timeoutSeconds=412&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0610 22:08:28.289309 3454988 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781129304141577000&timeoutSeconds=485&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0610 22:08:30.039593 3455028 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781129309141609000&timeoutSeconds=325&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0610 22:08:31.800357 3455058 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781129309141609000&timeoutSeconds=404&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met ----------------------------------------------------------------------------------- renew some-name-ssl ----------------------------------------------------------------------------------- Manually triggered issuance of Certificate upgrade-consistency-sharded-tls-24035/some-name-ssl waiting for pod/some-name-rs0-0 to be ready.OK waiting for pod/some-name-rs0-1 to be ready.OK waiting for pod/some-name-rs0-2 to be ready.OK Waiting for cluster readyness............................................................................................................................................................... waiting for pod/some-name-cfg-0 to be ready.OK waiting for pod/some-name-cfg-1 to be ready.OK waiting for pod/some-name-cfg-2 to be ready.OK Waiting for cluster readyness waiting for pod/some-name-mongos-0 to be ready.OK waiting for pod/some-name-mongos-1 to be ready.OK waiting for pod/some-name-mongos-2 to be ready.OK Waiting for cluster readyness waiting for cluster readyness.OK E0610 22:14:32.529161 3467961 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781129670894414000&timeoutSeconds=303&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0610 22:14:34.815757 3468023 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781129670894414000&timeoutSeconds=396&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0610 22:14:37.104288 3468097 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781129675894427000&timeoutSeconds=485&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0610 22:14:38.845142 3468230 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781129675894427000&timeoutSeconds=472&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0610 22:14:41.153416 3468295 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781129680894441000&timeoutSeconds=445&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0610 22:14:43.397940 3468374 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781129680894441000&timeoutSeconds=551&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0610 22:14:45.720856 3468481 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781129680894441000&timeoutSeconds=432&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0610 22:14:47.996242 3468596 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781129685894544000&timeoutSeconds=566&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0610 22:14:50.288041 3468677 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781129685894544000&timeoutSeconds=453&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0610 22:14:52.596793 3468752 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781129690894650000&timeoutSeconds=345&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met ----------------------------------------------------------------------------------- renew some-name-ssl-internal ----------------------------------------------------------------------------------- Manually triggered issuance of Certificate upgrade-consistency-sharded-tls-24035/some-name-ssl-internal waiting for pod/some-name-rs0-0 to be ready.OK waiting for pod/some-name-rs0-1 to be ready.OK waiting for pod/some-name-rs0-2 to be ready.OK Waiting for cluster readyness............................................................................................................................................................... waiting for pod/some-name-cfg-0 to be ready.OK waiting for pod/some-name-cfg-1 to be ready.OK waiting for pod/some-name-cfg-2 to be ready.OK Waiting for cluster readyness waiting for pod/some-name-mongos-0 to be ready.OK waiting for pod/some-name-mongos-1 to be ready.OK waiting for pod/some-name-mongos-2 to be ready.OK Waiting for cluster readyness waiting for cluster readyness.OK ----------------------------------------------------------------------------------- check if service and statefulset created with expected config ----------------------------------------------------------------------------------- [2026-06-10T22:21:10+0000] compare_kubectl: service/some-name-rs0 OK [2026-06-10T22:21:11+0000] compare_kubectl: service/some-name-cfg OK [2026-06-10T22:21:11+0000] compare_kubectl: statefulset/some-name-rs0 OK [2026-06-10T22:21:12+0000] compare_kubectl: statefulset/some-name-cfg OK ----------------------------------------------------------------------------------- test 1.23.0 ----------------------------------------------------------------------------------- perconaservermongodb.psmdb.percona.com/some-name patched ----------------------------------------------------------------------------------- check if Pod started ----------------------------------------------------------------------------------- waiting for pod/some-name-rs0-0 to be ready.OK waiting for pod/some-name-rs0-1 to be ready.OK waiting for pod/some-name-rs0-2 to be ready.OK Waiting for cluster readyness waiting for pod/some-name-cfg-0 to be ready.OK waiting for pod/some-name-cfg-1 to be ready.OK waiting for pod/some-name-cfg-2 to be ready.OK Waiting for cluster readyness waiting for pod/some-name-mongos-0 to be ready.OK waiting for pod/some-name-mongos-1 to be ready.OK waiting for pod/some-name-mongos-2 to be ready.OK Waiting for cluster readyness waiting for cluster readyness.OK E0610 22:22:30.339136 3485950 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781130147611076000&timeoutSeconds=423&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0610 22:22:32.629383 3486052 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781130147611076000&timeoutSeconds=338&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0610 22:22:34.933950 3486170 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781130152611114000&timeoutSeconds=468&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0610 22:22:37.255082 3486248 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781130152611114000&timeoutSeconds=430&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0610 22:22:38.996289 3486364 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781130157611176000&timeoutSeconds=502&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0610 22:22:41.302439 3486390 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781130157611176000&timeoutSeconds=431&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0610 22:22:43.613419 3486486 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781130162611187000&timeoutSeconds=528&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0610 22:22:45.918874 3486588 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781130162611187000&timeoutSeconds=471&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0610 22:22:48.220552 3486702 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781130167611198000&timeoutSeconds=584&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0610 22:22:50.514904 3486800 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1781130167611198000&timeoutSeconds=577&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met ----------------------------------------------------------------------------------- renew some-name-ssl ----------------------------------------------------------------------------------- Manually triggered issuance of Certificate upgrade-consistency-sharded-tls-24035/some-name-ssl waiting for pod/some-name-rs0-0 to be ready.OK waiting for pod/some-name-rs0-1 to be ready.OK waiting for pod/some-name-rs0-2 to be ready.OK Waiting for cluster readyness................................................................................................................................................................... waiting for pod/some-name-cfg-0 to be ready.OK waiting for pod/some-name-cfg-1 to be ready.OK waiting for pod/some-name-cfg-2 to be ready.OK Waiting for cluster readyness waiting for pod/some-name-mongos-0 to be ready.OK waiting for pod/some-name-mongos-1 to be ready.OK waiting for pod/some-name-mongos-2 to be ready.OK Waiting for cluster readyness waiting for cluster readyness.OK E0610 22:28:53.776740 3499440 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781130530063614000&timeoutSeconds=422&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0610 22:28:55.637836 3499464 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781130535063634000&timeoutSeconds=469&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0610 22:28:57.381605 3499510 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781130535063634000&timeoutSeconds=409&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0610 22:28:59.126014 3499530 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781130535063634000&timeoutSeconds=537&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0610 22:29:00.870302 3499558 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781130540063662000&timeoutSeconds=589&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0610 22:29:02.608377 3499582 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781130540063662000&timeoutSeconds=548&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0610 22:29:04.370079 3499608 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781130540063662000&timeoutSeconds=432&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0610 22:29:06.116861 3499636 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781130545063902000&timeoutSeconds=429&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0610 22:29:07.884746 3499703 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781130545063902000&timeoutSeconds=598&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0610 22:29:09.632602 3499730 reflector.go:227] "Failed to watch" err="Get \"https://34.66.74.177/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-24035/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1781130545063902000&timeoutSeconds=508&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met ----------------------------------------------------------------------------------- renew some-name-ssl-internal ----------------------------------------------------------------------------------- Manually triggered issuance of Certificate upgrade-consistency-sharded-tls-24035/some-name-ssl-internal waiting for pod/some-name-rs0-0 to be ready.OK waiting for pod/some-name-rs0-1 to be ready.OK waiting for pod/some-name-rs0-2 to be ready.OK Waiting for cluster readyness.................................................................................................................................................................................... waiting for pod/some-name-cfg-0 to be ready.OK waiting for pod/some-name-cfg-1 to be ready.OK waiting for pod/some-name-cfg-2 to be ready.OK Waiting for cluster readyness waiting for pod/some-name-mongos-0 to be ready.OK waiting for pod/some-name-mongos-1 to be ready.OK waiting for pod/some-name-mongos-2 to be ready.OK Waiting for cluster readyness waiting for cluster readyness.OK ----------------------------------------------------------------------------------- check if service and statefulset created with expected config ----------------------------------------------------------------------------------- [2026-06-10T22:35:04+0000] compare_kubectl: service/some-name-rs0 OK [2026-06-10T22:35:05+0000] compare_kubectl: service/some-name-cfg OK [2026-06-10T22:35:05+0000] compare_kubectl: statefulset/some-name-rs0 OK [2026-06-10T22:35:06+0000] compare_kubectl: statefulset/some-name-cfg OK ----------------------------------------------------------------------------------- destroy cluster/operator and all other resources ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- Delete psmdb-backup ----------------------------------------------------------------------------------- No resources found in upgrade-consistency-sharded-tls-24035 namespace. ----------------------------------------------------------------------------------- get and delete old CRDs and RBAC ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io "perconaservermongodbbackups.psmdb.percona.com" deleted customresourcedefinition.apiextensions.k8s.io "perconaservermongodbrestores.psmdb.percona.com" deleted customresourcedefinition.apiextensions.k8s.io "perconaservermongodbs.psmdb.percona.com" deleted grep: warning: stray \ before - grep: warning: stray \ before - error: the server doesn't have a resource type "perconaservermongodbbackups" + kubectl patch perconaservermongodbbackups.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbbackups" error: the server doesn't have a resource type "perconaservermongodbrestores" + kubectl patch perconaservermongodbrestores.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbrestores" + kubectl patch perconaservermongodbs.psmdb.percona.com -n upgrade-consistency-sharded-tls-24035 some-name --type=merge -p '{"metadata":{"finalizers":[]}}' perconaservermongodb.psmdb.percona.com/some-name patched customresourcedefinition.apiextensions.k8s.io/perconaservermongodbs.psmdb.percona.com condition met clusterrole.rbac.authorization.k8s.io "percona-server-mongodb-operator" deleted clusterrolebinding.rbac.authorization.k8s.io "service-account-percona-server-mongodb-operator" deleted namespace "cert-manager" deleted customresourcedefinition.apiextensions.k8s.io "challenges.acme.cert-manager.io" deleted customresourcedefinition.apiextensions.k8s.io "orders.acme.cert-manager.io" deleted customresourcedefinition.apiextensions.k8s.io "certificaterequests.cert-manager.io" deleted customresourcedefinition.apiextensions.k8s.io "certificates.cert-manager.io" deleted customresourcedefinition.apiextensions.k8s.io "clusterissuers.cert-manager.io" deleted customresourcedefinition.apiextensions.k8s.io "issuers.cert-manager.io" deleted serviceaccount "cert-manager-cainjector" deleted from cert-manager namespace serviceaccount "cert-manager" deleted from cert-manager namespace serviceaccount "cert-manager-webhook" deleted from cert-manager namespace clusterrole.rbac.authorization.k8s.io "cert-manager-cainjector" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-controller-issuers" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-controller-clusterissuers" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-controller-certificates" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-controller-orders" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-controller-challenges" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-controller-ingress-shim" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-cluster-view" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-view" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-edit" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-controller-approve:cert-manager-io" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-controller-certificatesigningrequests" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-webhook:subjectaccessreviews" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-cainjector" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-issuers" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-clusterissuers" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-certificates" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-orders" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-challenges" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-ingress-shim" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-approve:cert-manager-io" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-certificatesigningrequests" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-webhook:subjectaccessreviews" deleted role.rbac.authorization.k8s.io "cert-manager-cainjector:leaderelection" deleted from kube-system namespace role.rbac.authorization.k8s.io "cert-manager:leaderelection" deleted from kube-system namespace role.rbac.authorization.k8s.io "cert-manager-tokenrequest" deleted from cert-manager namespace role.rbac.authorization.k8s.io "cert-manager-webhook:dynamic-serving" deleted from cert-manager namespace rolebinding.rbac.authorization.k8s.io "cert-manager-cainjector:leaderelection" deleted from kube-system namespace rolebinding.rbac.authorization.k8s.io "cert-manager:leaderelection" deleted from kube-system namespace rolebinding.rbac.authorization.k8s.io "cert-manager-tokenrequest" deleted from cert-manager namespace rolebinding.rbac.authorization.k8s.io "cert-manager-webhook:dynamic-serving" deleted from cert-manager namespace service "cert-manager-cainjector" deleted from cert-manager namespace service "cert-manager" deleted from cert-manager namespace service "cert-manager-webhook" deleted from cert-manager namespace deployment.apps "cert-manager-cainjector" deleted from cert-manager namespace deployment.apps "cert-manager" deleted from cert-manager namespace deployment.apps "cert-manager-webhook" deleted from cert-manager namespace mutatingwebhookconfiguration.admissionregistration.k8s.io "cert-manager-webhook" deleted validatingwebhookconfiguration.admissionregistration.k8s.io "cert-manager-webhook" deleted ----------------------------------------------------------------------------------- test passed -----------------------------------------------------------------------------------