Log: /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2397/e2e-tests/logs/upgrade-consistency-sharded-tls.log Warning: version difference between client (1.36) and server (1.33) exceeds the supported minor version skew of +/-1 Warning: version difference between client (1.36) and server (1.33) exceeds the supported minor version skew of +/-1 Warning: version difference between client (1.36) and server (1.33) exceeds the supported minor version skew of +/-1 ----------------------------------------------------------------------------------- get and delete old CRDs and RBAC ----------------------------------------------------------------------------------- grep: warning: stray \ before - grep: warning: stray \ before - error: the server doesn't have a resource type "perconaservermongodbbackups" + kubectl patch perconaservermongodbbackups.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbbackups" error: the server doesn't have a resource type "perconaservermongodbclustersyncs" + kubectl patch perconaservermongodbclustersyncs.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbclustersyncs" error: the server doesn't have a resource type "perconaservermongodbrestores" + kubectl patch perconaservermongodbrestores.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbrestores" error: the server doesn't have a resource type "perconaservermongodbs" + kubectl patch perconaservermongodbs.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbs" Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found ----------------------------------------------------------------------------------- destroy chaos-mesh ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces psmdb-operator ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- create namespace psmdb-operator ----------------------------------------------------------------------------------- namespace/psmdb-operator created Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-2397-b574baaaa-19-cluster9" modified. ----------------------------------------------------------------------------------- start PSMDB operator: docker.io/perconalab/percona-server-mongodb-operator:PR-2397-b574baaaa ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io/perconaservermongodbbackups.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbclustersyncs.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbrestores.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbs.psmdb.percona.com serverside-applied clusterrole.rbac.authorization.k8s.io/percona-server-mongodb-operator created serviceaccount/percona-server-mongodb-operator created clusterrolebinding.rbac.authorization.k8s.io/service-account-percona-server-mongodb-operator created deployment.apps/percona-server-mongodb-operator created waiting for pod/percona-server-mongodb-operator-8498c6c4f4-kxbdh to be ready.OK Print operator info from log 2026-06-23T00:45:25.067Z INFO setup Manager starting up {"gitCommit": "b574baaaab6a4ead9617911a3da4490a404d532c", "gitBranch": "PR-2397-b574baaaa", "buildTime": "", "goVersion": "go1.26.4", "os": "linux", "arch": "amd64"} ----------------------------------------------------------------------------------- destroy chaos-mesh ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces upgrade-consistency-sharded-tls-7118 ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- create namespace upgrade-consistency-sharded-tls-7118 ----------------------------------------------------------------------------------- namespace/upgrade-consistency-sharded-tls-7118 created Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-2397-b574baaaa-19-cluster9" modified. ----------------------------------------------------------------------------------- deploy cert manager ----------------------------------------------------------------------------------- namespace/cert-manager created namespace/cert-manager labeled namespace/cert-manager configured customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io created serviceaccount/cert-manager-cainjector created serviceaccount/cert-manager created serviceaccount/cert-manager-webhook created clusterrole.rbac.authorization.k8s.io/cert-manager-cainjector created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-issuers created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificates created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-orders created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-challenges created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created clusterrole.rbac.authorization.k8s.io/cert-manager-cluster-view created clusterrole.rbac.authorization.k8s.io/cert-manager-view created clusterrole.rbac.authorization.k8s.io/cert-manager-edit created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests created clusterrole.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-cainjector created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-issuers created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificates created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-orders created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-challenges created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews created role.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created role.rbac.authorization.k8s.io/cert-manager:leaderelection created role.rbac.authorization.k8s.io/cert-manager-tokenrequest created role.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created rolebinding.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created rolebinding.rbac.authorization.k8s.io/cert-manager:leaderelection created rolebinding.rbac.authorization.k8s.io/cert-manager-tokenrequest created rolebinding.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created service/cert-manager-cainjector created service/cert-manager created service/cert-manager-webhook created deployment.apps/cert-manager-cainjector created deployment.apps/cert-manager created deployment.apps/cert-manager-webhook created mutatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created Warning: resource namespaces/cert-manager is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically. pod/cert-manager-559d798845-fv7fd condition met pod/cert-manager-cainjector-64958d9c7c-hj66t condition met pod/cert-manager-webhook-7fb6f99b56-f6zwm condition met E0623 00:46:22.774079 3837921 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/api/v1/namespaces/cert-manager/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dcert-manager-cainjector-64958d9c7c-hj66t&resourceVersion=1782175582401663022&timeoutSeconds=551&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" ----------------------------------------------------------------------------------- create secrets and start client ----------------------------------------------------------------------------------- secret/some-users created deployment.apps/psmdb-client created role.rbac.authorization.k8s.io/cmctl created serviceaccount/cmctl created rolebinding.rbac.authorization.k8s.io/service-account-cmctl created deployment.apps/cmctl created ----------------------------------------------------------------------------------- create first PSMDB cluster 1.21.2 some-name ----------------------------------------------------------------------------------- perconaservermongodb.psmdb.percona.com/some-name created ----------------------------------------------------------------------------------- check if Pod started ----------------------------------------------------------------------------------- waiting for pod/some-name-rs0-0 to be ready.............OK waiting for pod/some-name-rs0-1 to be ready............OK waiting for pod/some-name-rs0-2 to be ready...........OK Waiting for cluster readyness.................................. waiting for pod/some-name-cfg-0 to be ready.OK waiting for pod/some-name-cfg-1 to be ready.OK waiting for pod/some-name-cfg-2 to be ready.OK Waiting for cluster readyness waiting for pod/some-name-mongos-0 to be ready.OK waiting for pod/some-name-mongos-1 to be ready.OK waiting for pod/some-name-mongos-2 to be ready.OK Waiting for cluster readyness waiting for cluster readyness.OK ----------------------------------------------------------------------------------- check if Pod started ----------------------------------------------------------------------------------- waiting for pod/some-name-rs0-0 to be ready.OK waiting for pod/some-name-rs0-1 to be ready.OK waiting for pod/some-name-rs0-2 to be ready.OK Waiting for cluster readyness waiting for pod/some-name-cfg-0 to be ready.OK waiting for pod/some-name-cfg-1 to be ready.OK waiting for pod/some-name-cfg-2 to be ready.OK Waiting for cluster readyness waiting for pod/some-name-mongos-0 to be ready.OK waiting for pod/some-name-mongos-1 to be ready.OK waiting for pod/some-name-mongos-2 to be ready.OK Waiting for cluster readyness waiting for cluster readyness.OK E0623 00:53:25.933795 3911893 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1782176005705837000&timeoutSeconds=368&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0623 00:53:28.194015 3912159 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1782176005705837000&timeoutSeconds=594&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0623 00:53:30.536183 3912433 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1782176005705837000&timeoutSeconds=562&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0623 00:53:32.770805 3912736 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1782176010705880000&timeoutSeconds=388&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0623 00:53:35.064578 3913048 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1782176010705880000&timeoutSeconds=404&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0623 00:53:37.315750 3913330 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1782176015705978000&timeoutSeconds=554&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0623 00:53:39.538900 3913636 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1782176015705978000&timeoutSeconds=487&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0623 00:53:41.838475 3913827 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1782176020706173000&timeoutSeconds=409&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0623 00:53:44.113714 3914163 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1782176020706173000&timeoutSeconds=553&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0623 00:53:46.371572 3914452 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1782176025706192000&timeoutSeconds=389&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met ----------------------------------------------------------------------------------- renew some-name-ssl ----------------------------------------------------------------------------------- Manually triggered issuance of Certificate upgrade-consistency-sharded-tls-7118/some-name-ssl waiting for pod/some-name-rs0-0 to be ready.OK waiting for pod/some-name-rs0-1 to be ready.OK waiting for pod/some-name-rs0-2 to be ready.OK Waiting for cluster readyness...................................................................................................................... waiting for pod/some-name-cfg-0 to be ready.OK waiting for pod/some-name-cfg-1 to be ready.OK waiting for pod/some-name-cfg-2 to be ready.OK Waiting for cluster readyness waiting for pod/some-name-mongos-0 to be ready.OK waiting for pod/some-name-mongos-1 to be ready.OK waiting for pod/some-name-mongos-2 to be ready.OK Waiting for cluster readyness waiting for cluster readyness.OK E0623 00:59:18.589775 3952291 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1782176355726003000&timeoutSeconds=575&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0623 00:59:20.893800 3952460 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1782176360726179000&timeoutSeconds=502&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0623 00:59:23.172711 3952780 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1782176360726179000&timeoutSeconds=328&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0623 00:59:25.427433 3953003 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1782176360726179000&timeoutSeconds=375&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0623 00:59:27.711422 3953222 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1782176365726187000&timeoutSeconds=351&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0623 00:59:30.026138 3953411 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1782176365726187000&timeoutSeconds=497&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0623 00:59:32.311830 3953616 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1782176370726552000&timeoutSeconds=514&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0623 00:59:34.593120 3953763 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1782176370726552000&timeoutSeconds=521&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0623 00:59:36.857862 3953941 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1782176375726608000&timeoutSeconds=431&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0623 00:59:39.137433 3954203 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1782176375726608000&timeoutSeconds=341&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met ----------------------------------------------------------------------------------- renew some-name-ssl-internal ----------------------------------------------------------------------------------- Manually triggered issuance of Certificate upgrade-consistency-sharded-tls-7118/some-name-ssl-internal waiting for pod/some-name-rs0-0 to be ready.OK waiting for pod/some-name-rs0-1 to be ready.OK waiting for pod/some-name-rs0-2 to be ready.OK Waiting for cluster readyness............................................................................................................................ waiting for pod/some-name-cfg-0 to be ready.OK waiting for pod/some-name-cfg-1 to be ready.OK waiting for pod/some-name-cfg-2 to be ready.OK Waiting for cluster readyness waiting for pod/some-name-mongos-0 to be ready.OK waiting for pod/some-name-mongos-1 to be ready.OK waiting for pod/some-name-mongos-2 to be ready.OK Waiting for cluster readyness waiting for cluster readyness.OK ----------------------------------------------------------------------------------- check if service and statefulset created with expected config ----------------------------------------------------------------------------------- [2026-06-23T01:05:25+0000] compare_kubectl: service/some-name-rs0 OK [2026-06-23T01:05:26+0000] compare_kubectl: service/some-name-cfg OK [2026-06-23T01:05:26+0000] compare_kubectl: statefulset/some-name-rs0 OK [2026-06-23T01:05:28+0000] compare_kubectl: statefulset/some-name-cfg OK ----------------------------------------------------------------------------------- test 1.22.0 ----------------------------------------------------------------------------------- perconaservermongodb.psmdb.percona.com/some-name patched ----------------------------------------------------------------------------------- check if Pod started ----------------------------------------------------------------------------------- waiting for pod/some-name-rs0-0 to be ready.OK waiting for pod/some-name-rs0-1 to be ready.OK waiting for pod/some-name-rs0-2 to be ready.OK Waiting for cluster readyness.................................................................................. waiting for pod/some-name-cfg-0 to be ready.OK waiting for pod/some-name-cfg-1 to be ready.OK waiting for pod/some-name-cfg-2 to be ready.OK Waiting for cluster readyness waiting for pod/some-name-mongos-0 to be ready.OK waiting for pod/some-name-mongos-1 to be ready.OK waiting for pod/some-name-mongos-2 to be ready.OK Waiting for cluster readyness waiting for cluster readyness.OK E0623 01:09:36.266027 4003836 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1782176974628360000&timeoutSeconds=559&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0623 01:09:38.555188 4003946 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1782176974628360000&timeoutSeconds=345&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0623 01:09:40.867870 4004065 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1782176979628423000&timeoutSeconds=578&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0623 01:09:43.176589 4004165 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1782176979628423000&timeoutSeconds=455&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0623 01:09:45.468814 4004243 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1782176984628441000&timeoutSeconds=429&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0623 01:09:47.740453 4004377 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1782176984628441000&timeoutSeconds=396&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0623 01:09:50.075597 4004532 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1782176989628457000&timeoutSeconds=496&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0623 01:09:52.352894 4004616 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1782176989628457000&timeoutSeconds=485&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0623 01:09:54.643112 4004723 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1782176989628457000&timeoutSeconds=589&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0623 01:09:56.936809 4004808 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1782176994628462000&timeoutSeconds=566&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met ----------------------------------------------------------------------------------- renew some-name-ssl ----------------------------------------------------------------------------------- Manually triggered issuance of Certificate upgrade-consistency-sharded-tls-7118/some-name-ssl waiting for pod/some-name-rs0-0 to be ready.OK waiting for pod/some-name-rs0-1 to be ready.OK waiting for pod/some-name-rs0-2 to be ready.OK Waiting for cluster readyness............................................................................................................................................ waiting for pod/some-name-cfg-0 to be ready.OK waiting for pod/some-name-cfg-1 to be ready.OK waiting for pod/some-name-cfg-2 to be ready.OK Waiting for cluster readyness waiting for pod/some-name-mongos-0 to be ready.OK waiting for pod/some-name-mongos-1 to be ready.OK waiting for pod/some-name-mongos-2 to be ready.OK Waiting for cluster readyness waiting for cluster readyness.OK E0623 01:15:45.813653 4019818 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1782177341408499000&timeoutSeconds=463&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0623 01:15:47.554307 4019842 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1782177346408509000&timeoutSeconds=587&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0623 01:15:49.264966 4019864 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1782177346408509000&timeoutSeconds=349&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0623 01:15:51.002956 4019888 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1782177346408509000&timeoutSeconds=575&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0623 01:15:52.707389 4019910 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1782177351408606000&timeoutSeconds=498&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0623 01:15:54.449194 4019935 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1782177351408606000&timeoutSeconds=540&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0623 01:15:56.192170 4019957 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1782177351408606000&timeoutSeconds=476&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0623 01:15:57.938971 4019983 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1782177356408633000&timeoutSeconds=324&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0623 01:15:59.692778 4020008 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1782177356408633000&timeoutSeconds=563&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0623 01:16:01.425859 4020028 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1782177356408633000&timeoutSeconds=549&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met ----------------------------------------------------------------------------------- renew some-name-ssl-internal ----------------------------------------------------------------------------------- Manually triggered issuance of Certificate upgrade-consistency-sharded-tls-7118/some-name-ssl-internal waiting for pod/some-name-rs0-0 to be ready.OK waiting for pod/some-name-rs0-1 to be ready.OK waiting for pod/some-name-rs0-2 to be ready.OK Waiting for cluster readyness............................................................................................................................................................ waiting for pod/some-name-cfg-0 to be ready.OK waiting for pod/some-name-cfg-1 to be ready.OK waiting for pod/some-name-cfg-2 to be ready.OK Waiting for cluster readyness waiting for pod/some-name-mongos-0 to be ready.OK waiting for pod/some-name-mongos-1 to be ready.OK waiting for pod/some-name-mongos-2 to be ready.OK Waiting for cluster readyness waiting for cluster readyness.OK ----------------------------------------------------------------------------------- check if service and statefulset created with expected config ----------------------------------------------------------------------------------- [2026-06-23T01:21:33+0000] compare_kubectl: service/some-name-rs0 OK [2026-06-23T01:21:33+0000] compare_kubectl: service/some-name-cfg OK [2026-06-23T01:21:34+0000] compare_kubectl: statefulset/some-name-rs0 OK [2026-06-23T01:21:34+0000] compare_kubectl: statefulset/some-name-cfg OK ----------------------------------------------------------------------------------- test 1.23.0 ----------------------------------------------------------------------------------- perconaservermongodb.psmdb.percona.com/some-name patched ----------------------------------------------------------------------------------- check if Pod started ----------------------------------------------------------------------------------- waiting for pod/some-name-rs0-0 to be ready.OK waiting for pod/some-name-rs0-1 to be ready.OK waiting for pod/some-name-rs0-2 to be ready.OK Waiting for cluster readyness.............................................................................................................................................................. waiting for pod/some-name-cfg-0 to be ready.OK waiting for pod/some-name-cfg-1 to be ready.OK waiting for pod/some-name-cfg-2 to be ready.OK Waiting for cluster readyness waiting for pod/some-name-mongos-0 to be ready.OK waiting for pod/some-name-mongos-1 to be ready.OK waiting for pod/some-name-mongos-2 to be ready.OK Waiting for cluster readyness waiting for cluster readyness.OK E0623 01:26:56.043756 4033562 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1782178014681110000&timeoutSeconds=335&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0623 01:26:57.770224 4033587 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1782178014681110000&timeoutSeconds=342&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0623 01:26:59.510971 4033607 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1782178014681110000&timeoutSeconds=555&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0623 01:27:01.243525 4033629 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1782178019681133000&timeoutSeconds=360&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0623 01:27:02.990533 4033649 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1782178019681133000&timeoutSeconds=502&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0623 01:27:04.734490 4033673 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1782178019681133000&timeoutSeconds=456&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0623 01:27:06.450945 4033693 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1782178024681161000&timeoutSeconds=438&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0623 01:27:08.174144 4033715 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1782178024681161000&timeoutSeconds=399&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0623 01:27:09.918901 4033737 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1782178029681195000&timeoutSeconds=496&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met E0623 01:27:11.643468 4033755 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl&resourceVersion=1782178029681195000&timeoutSeconds=397&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl condition met ----------------------------------------------------------------------------------- renew some-name-ssl ----------------------------------------------------------------------------------- Manually triggered issuance of Certificate upgrade-consistency-sharded-tls-7118/some-name-ssl waiting for pod/some-name-rs0-0 to be ready.OK waiting for pod/some-name-rs0-1 to be ready.OK waiting for pod/some-name-rs0-2 to be ready.OK Waiting for cluster readyness............................................................................................................................................................... waiting for pod/some-name-cfg-0 to be ready.OK waiting for pod/some-name-cfg-1 to be ready.OK waiting for pod/some-name-cfg-2 to be ready.OK Waiting for cluster readyness waiting for pod/some-name-mongos-0 to be ready.OK waiting for pod/some-name-mongos-1 to be ready.OK waiting for pod/some-name-mongos-2 to be ready.OK Waiting for cluster readyness waiting for cluster readyness.OK E0623 01:32:34.901534 4038589 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1782178350346231000&timeoutSeconds=487&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0623 01:32:36.638069 4038607 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1782178355346243000&timeoutSeconds=548&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0623 01:32:38.388411 4038627 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1782178355346243000&timeoutSeconds=365&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0623 01:32:40.125827 4038648 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1782178355346243000&timeoutSeconds=515&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0623 01:32:41.883287 4038666 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1782178360346252000&timeoutSeconds=510&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0623 01:32:43.625705 4038688 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1782178360346252000&timeoutSeconds=439&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0623 01:32:45.378671 4038706 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1782178360346252000&timeoutSeconds=510&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0623 01:32:47.205352 4038726 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1782178365346279000&timeoutSeconds=304&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0623 01:32:48.915845 4038746 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1782178365346279000&timeoutSeconds=543&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met E0623 01:32:50.673146 4038764 reflector.go:227] "Failed to watch" err="Get \"https://34.56.161.23/apis/cert-manager.io/v1/namespaces/upgrade-consistency-sharded-tls-7118/certificates?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-ssl-internal&resourceVersion=1782178370346804000&timeoutSeconds=576&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" certificate.cert-manager.io/some-name-ssl-internal condition met ----------------------------------------------------------------------------------- renew some-name-ssl-internal ----------------------------------------------------------------------------------- Manually triggered issuance of Certificate upgrade-consistency-sharded-tls-7118/some-name-ssl-internal waiting for pod/some-name-rs0-0 to be ready.OK waiting for pod/some-name-rs0-1 to be ready.OK waiting for pod/some-name-rs0-2 to be ready.OK Waiting for cluster readyness............................................................................................................................................................................. waiting for pod/some-name-cfg-0 to be ready.OK waiting for pod/some-name-cfg-1 to be ready.OK waiting for pod/some-name-cfg-2 to be ready.OK Waiting for cluster readyness waiting for pod/some-name-mongos-0 to be ready.OK waiting for pod/some-name-mongos-1 to be ready.OK waiting for pod/some-name-mongos-2 to be ready.OK Waiting for cluster readyness waiting for cluster readyness.OK ----------------------------------------------------------------------------------- check if service and statefulset created with expected config ----------------------------------------------------------------------------------- [2026-06-23T01:38:35+0000] compare_kubectl: service/some-name-rs0 OK [2026-06-23T01:38:35+0000] compare_kubectl: service/some-name-cfg OK [2026-06-23T01:38:36+0000] compare_kubectl: statefulset/some-name-rs0 OK [2026-06-23T01:38:36+0000] compare_kubectl: statefulset/some-name-cfg OK ----------------------------------------------------------------------------------- destroy cluster/operator and all other resources ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- Delete psmdb-backup ----------------------------------------------------------------------------------- No resources found in upgrade-consistency-sharded-tls-7118 namespace. ----------------------------------------------------------------------------------- get and delete old CRDs and RBAC ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io "perconaservermongodbbackups.psmdb.percona.com" deleted customresourcedefinition.apiextensions.k8s.io "perconaservermongodbclustersyncs.psmdb.percona.com" deleted customresourcedefinition.apiextensions.k8s.io "perconaservermongodbrestores.psmdb.percona.com" deleted customresourcedefinition.apiextensions.k8s.io "perconaservermongodbs.psmdb.percona.com" deleted grep: warning: stray \ before - grep: warning: stray \ before - error: the server doesn't have a resource type "perconaservermongodbbackups" + kubectl patch perconaservermongodbbackups.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbbackups" error: the server doesn't have a resource type "perconaservermongodbclustersyncs" + kubectl patch perconaservermongodbclustersyncs.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbclustersyncs" error: the server doesn't have a resource type "perconaservermongodbrestores" + kubectl patch perconaservermongodbrestores.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbrestores" + kubectl patch perconaservermongodbs.psmdb.percona.com -n upgrade-consistency-sharded-tls-7118 some-name --type=merge -p '{"metadata":{"finalizers":[]}}' perconaservermongodb.psmdb.percona.com/some-name patched customresourcedefinition.apiextensions.k8s.io/perconaservermongodbs.psmdb.percona.com condition met clusterrole.rbac.authorization.k8s.io "percona-server-mongodb-operator" deleted clusterrolebinding.rbac.authorization.k8s.io "service-account-percona-server-mongodb-operator" deleted namespace "cert-manager" deleted customresourcedefinition.apiextensions.k8s.io "challenges.acme.cert-manager.io" deleted customresourcedefinition.apiextensions.k8s.io "orders.acme.cert-manager.io" deleted customresourcedefinition.apiextensions.k8s.io "certificaterequests.cert-manager.io" deleted customresourcedefinition.apiextensions.k8s.io "certificates.cert-manager.io" deleted customresourcedefinition.apiextensions.k8s.io "clusterissuers.cert-manager.io" deleted customresourcedefinition.apiextensions.k8s.io "issuers.cert-manager.io" deleted serviceaccount "cert-manager-cainjector" deleted from cert-manager namespace serviceaccount "cert-manager" deleted from cert-manager namespace serviceaccount "cert-manager-webhook" deleted from cert-manager namespace clusterrole.rbac.authorization.k8s.io "cert-manager-cainjector" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-controller-issuers" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-controller-clusterissuers" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-controller-certificates" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-controller-orders" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-controller-challenges" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-controller-ingress-shim" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-cluster-view" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-view" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-edit" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-controller-approve:cert-manager-io" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-controller-certificatesigningrequests" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-webhook:subjectaccessreviews" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-cainjector" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-issuers" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-clusterissuers" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-certificates" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-orders" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-challenges" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-ingress-shim" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-approve:cert-manager-io" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-certificatesigningrequests" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-webhook:subjectaccessreviews" deleted role.rbac.authorization.k8s.io "cert-manager-cainjector:leaderelection" deleted from kube-system namespace role.rbac.authorization.k8s.io "cert-manager:leaderelection" deleted from kube-system namespace role.rbac.authorization.k8s.io "cert-manager-tokenrequest" deleted from cert-manager namespace role.rbac.authorization.k8s.io "cert-manager-webhook:dynamic-serving" deleted from cert-manager namespace rolebinding.rbac.authorization.k8s.io "cert-manager-cainjector:leaderelection" deleted from kube-system namespace rolebinding.rbac.authorization.k8s.io "cert-manager:leaderelection" deleted from kube-system namespace rolebinding.rbac.authorization.k8s.io "cert-manager-tokenrequest" deleted from cert-manager namespace rolebinding.rbac.authorization.k8s.io "cert-manager-webhook:dynamic-serving" deleted from cert-manager namespace service "cert-manager-cainjector" deleted from cert-manager namespace service "cert-manager" deleted from cert-manager namespace service "cert-manager-webhook" deleted from cert-manager namespace deployment.apps "cert-manager-cainjector" deleted from cert-manager namespace deployment.apps "cert-manager" deleted from cert-manager namespace deployment.apps "cert-manager-webhook" deleted from cert-manager namespace mutatingwebhookconfiguration.admissionregistration.k8s.io "cert-manager-webhook" deleted validatingwebhookconfiguration.admissionregistration.k8s.io "cert-manager-webhook" deleted ----------------------------------------------------------------------------------- test passed -----------------------------------------------------------------------------------