Log: /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2289/e2e-tests/logs/demand-backup-snapshot.log Warning: version difference between client (1.35) and server (1.32) exceeds the supported minor version skew of +/-1 Warning: version difference between client (1.35) and server (1.32) exceeds the supported minor version skew of +/-1 Warning: version difference between client (1.35) and server (1.32) exceeds the supported minor version skew of +/-1 volumesnapshotclass.snapshot.storage.k8s.io/gke-snapshot-class created ----------------------------------------------------------------------------------- get and delete old CRDs and RBAC ----------------------------------------------------------------------------------- grep: warning: stray \ before - grep: warning: stray \ before - error: the server doesn't have a resource type "perconaservermongodbbackups" + kubectl patch perconaservermongodbbackups.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbbackups" error: the server doesn't have a resource type "perconaservermongodbrestores" + kubectl patch perconaservermongodbrestores.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbrestores" error: the server doesn't have a resource type "perconaservermongodbs" + kubectl patch perconaservermongodbs.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbs" Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found ----------------------------------------------------------------------------------- destroy chaos-mesh ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces psmdb-operator ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- create namespace psmdb-operator ----------------------------------------------------------------------------------- namespace/psmdb-operator created Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-2289-3781da70-8-cluster3" modified. ----------------------------------------------------------------------------------- start PSMDB operator: docker.io/perconalab/percona-server-mongodb-operator:PR-2289-3781da70 ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io/perconaservermongodbbackups.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbrestores.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbs.psmdb.percona.com serverside-applied clusterrole.rbac.authorization.k8s.io/percona-server-mongodb-operator created serviceaccount/percona-server-mongodb-operator created clusterrolebinding.rbac.authorization.k8s.io/service-account-percona-server-mongodb-operator created deployment.apps/percona-server-mongodb-operator created waiting for pod/percona-server-mongodb-operator-78d6997565-jcnhs to be ready.OK Print operator info from log 2026-03-31T23:23:49.592Z INFO setup Manager starting up {"gitCommit": "3781da704f1e66ba2322739211bd735215c34940", "gitBranch": "PR-2289-3781da70", "buildTime": "", "goVersion": "go1.25.8", "os": "linux", "arch": "amd64"} ----------------------------------------------------------------------------------- destroy chaos-mesh ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces demand-backup-snapshot-28814 ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- create namespace demand-backup-snapshot-28814 ----------------------------------------------------------------------------------- namespace/demand-backup-snapshot-28814 created Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-2289-3781da70-8-cluster3" modified. ----------------------------------------------------------------------------------- install MinIO: minio-service ----------------------------------------------------------------------------------- "minio" has been removed from your repositories "minio" has been added to your repositories NAME: minio-service LAST DEPLOYED: Tue Mar 31 23:24:19 2026 NAMESPACE: demand-backup-snapshot-28814 STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: MinIO can be accessed via port 9000 on the following DNS name from within your cluster: minio-service.demand-backup-snapshot-28814.cluster.local To access MinIO from localhost, run the below commands: 1. export POD_NAME=$(kubectl get pods --namespace demand-backup-snapshot-28814 -l "release=minio-service" -o jsonpath="{.items[0].metadata.name}") 2. kubectl port-forward $POD_NAME 9000 --namespace demand-backup-snapshot-28814 Read more about port forwarding here: http://kubernetes.io/docs/user-guide/kubectl/kubectl_port-forward/ You can now access MinIO server on http://localhost:9000. Follow the below steps to connect to MinIO server with mc client: 1. Download the MinIO mc client - https://min.io/docs/minio/linux/reference/minio-mc.html#quickstart 2. export MC_HOST_minio-service-local=http://$(kubectl get secret --namespace demand-backup-snapshot-28814 minio-service -o jsonpath="{.data.rootUser}" | base64 --decode):$(kubectl get secret --namespace demand-backup-snapshot-28814 minio-service -o jsonpath="{.data.rootPassword}" | base64 --decode)@localhost:9000 3. mc ls minio-service-local waiting for pod/minio-service-6d5f646cdc-wg98l to be ready.OK service/minio-service created pod "aws-cli" deleted from demand-backup-snapshot-28814 namespace All commands and output from this session will be recorded in container logs, including credentials and sensitive information passed through the command prompt. If you don't see a command prompt, try pressing enter. ----------------------------------------------------------------------------------- create secrets for cloud storages ----------------------------------------------------------------------------------- secret/minio-secret created secret/aws-s3-secret created secret/gcp-cs-secret created secret/azure-secret created secret/gcp-cs-sa-key-secret created ----------------------------------------------------------------------------------- Testing snapshot backup/restore on unencrypted cluster ----------------------------------------------------------------------------------- secret/some-users created perconaservermongodb.psmdb.percona.com/some-name created deployment.apps/psmdb-client created check if all pods started waiting for pod/some-name-rs0-0 to be ready.....................OK waiting for pod/some-name-rs0-1 to be ready....................OK waiting for pod/some-name-rs0-2 to be ready...................OK Waiting for cluster readyness....... waiting for cluster readyness.OK waiting for PBM operation to finish.OK Writing test data Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://some-name-rs0-2.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local:27017,some-name-rs0-1.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local:27017,some-name-rs0-0.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb&replicaSet=rs0&ssl=true {"t":{"$date":"2026-03-31T23:28:52.513Z"},"s":"I", "c":"NETWORK", "id":5490002, "ctx":"thread1","msg":"Started a new thread for the timer service"} Implicit session: session { "id" : UUID("9fe42974-156b-44a3-a154-3ceedafba767") } Percona Server for MongoDB server version: v8.0.19-7 WARNING: shell and server versions do not match Successfully added user: { "user" : "myApp", "roles" : [ { "db" : "myApp", "role" : "readWrite" } ] } bye Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://some-name-rs0-0.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local:27017,some-name-rs0-2.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local:27017,some-name-rs0-1.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb&replicaSet=rs0&ssl=true {"t":{"$date":"2026-03-31T23:28:55.506Z"},"s":"I", "c":"NETWORK", "id":5490002, "ctx":"thread1","msg":"Started a new thread for the timer service"} Implicit session: session { "id" : UUID("4aa12ebe-15fd-4cb3-a864-1154d2209a06") } Percona Server for MongoDB server version: v8.0.19-7 WARNING: shell and server versions do not match switched to db myApp WriteResult({ "nInserted" : 1 }) bye [2026-03-31T23:29:00+0000] running db.test.find() in myApp [2026-03-31T23:29:02+0000] running db.test.find() in myApp [2026-03-31T23:29:04+0000] running db.test.find() in myApp Running snapshot backup (unencrypted) [2026-03-31T23:29:06+0000] running snapshot backup backup-snapshot perconaservermongodbbackup.psmdb.percona.com/backup-snapshot created waiting for backup-snapshot to reach ready state.........................OK Drop collection and restore from snapshot (unencrypted) Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://some-name-rs0-2.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local:27017,some-name-rs0-1.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local:27017,some-name-rs0-0.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb&replicaSet=rs0&ssl=true {"t":{"$date":"2026-03-31T23:29:52.434Z"},"s":"I", "c":"NETWORK", "id":5490002, "ctx":"thread1","msg":"Started a new thread for the timer service"} Implicit session: session { "id" : UUID("e6751db9-e8a5-4e66-9e8d-1dff0d5bd495") } Percona Server for MongoDB server version: v8.0.19-7 WARNING: shell and server versions do not match switched to db myApp true bye [2026-03-31T23:29:52+0000] running restore restore-backup-snapshot perconaservermongodbrestore.psmdb.percona.com/restore-backup-snapshot created Waiting for the psmdb-restore/restore-backup-snapshot object to be created.OK Waiting psmdb-restore/restore-backup-snapshot to reach state "ready" ..2026-03-31T23:29:54.229Z DEBUG checking if restore is allowed {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-snapshot","namespace":"demand-backup-snapshot-28814"}, "namespace": "demand-backup-snapshot-28814", "name": "restore-backup-snapshot", "reconcileID": "e01f2efb-7434-46ba-9814-1bcd492d5697", "cluster": "some-name", "namespace": "demand-backup-snapshot-28814"} 2026-03-31T23:29:54.230Z INFO Restore state changed {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-snapshot","namespace":"demand-backup-snapshot-28814"}, "namespace": "demand-backup-snapshot-28814", "name": "restore-backup-snapshot", "reconcileID": "e01f2efb-7434-46ba-9814-1bcd492d5697", "previous": "", "current": "waiting"} 2026-03-31T23:29:54.775Z DEBUG checking if restore is allowed {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-snapshot","namespace":"demand-backup-snapshot-28814"}, "namespace": "demand-backup-snapshot-28814", "name": "restore-backup-snapshot", "reconcileID": "9d3dc609-6fb5-4d9e-808a-1dc0e2fd38e8", "cluster": "some-name", "namespace": "demand-backup-snapshot-28814"} 2026-03-31T23:29:54.834Z INFO Preparing statefulset for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-snapshot","namespace":"demand-backup-snapshot-28814"}, "namespace": "demand-backup-snapshot-28814", "name": "restore-backup-snapshot", "reconcileID": "9d3dc609-6fb5-4d9e-808a-1dc0e2fd38e8", "name": "some-name-rs0"} 2026-03-31T23:29:54.890Z INFO Updated statefulset {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-snapshot","namespace":"demand-backup-snapshot-28814"}, "namespace": "demand-backup-snapshot-28814", "name": "restore-backup-snapshot", "reconcileID": "9d3dc609-6fb5-4d9e-808a-1dc0e2fd38e8", "name": "some-name-rs0"} 2026-03-31T23:29:54.890Z INFO Waiting for statefulsets to be ready before restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-snapshot","namespace":"demand-backup-snapshot-28814"}, "namespace": "demand-backup-snapshot-28814", "name": "restore-backup-snapshot", "reconcileID": "9d3dc609-6fb5-4d9e-808a-1dc0e2fd38e8", "ready": false} 2026-03-31T23:29:59.890Z DEBUG checking if restore is allowed {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-snapshot","namespace":"demand-backup-snapshot-28814"}, "namespace": "demand-backup-snapshot-28814", "name": "restore-backup-snapshot", "reconcileID": "d97d959f-9252-4f7c-9624-948b1be5dd0e", "cluster": "some-name", "namespace": "demand-backup-snapshot-28814"} 2026-03-31T23:29:59.932Z INFO Waiting for statefulsets to be ready before restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-snapshot","namespace":"demand-backup-snapshot-28814"}, "namespace": "demand-backup-snapshot-28814", "name": "restore-backup-snapshot", "reconcileID": "d97d959f-9252-4f7c-9624-948b1be5dd0e", "ready": false} 2026-03-31T23:30:04.933Z DEBUG checking if restore is allowed {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-snapshot","namespace":"demand-backup-snapshot-28814"}, "namespace": "demand-backup-snapshot-28814", "name": "restore-backup-snapshot", "reconcileID": "491741e4-fb11-47b5-97d3-9fb6108fed32", "cluster": "some-name", "namespace": "demand-backup-snapshot-28814"} 2026-03-31T23:30:05.019Z INFO Waiting for statefulsets to be ready before restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-snapshot","namespace":"demand-backup-snapshot-28814"}, "namespace": "demand-backup-snapshot-28814", "name": "restore-backup-snapshot", "reconcileID": "491741e4-fb11-47b5-97d3-9fb6108fed32", "ready": false} 2026-03-31T23:30:10.020Z DEBUG checking if restore is allowed {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-snapshot","namespace":"demand-backup-snapshot-28814"}, "namespace": "demand-backup-snapshot-28814", "name": "restore-backup-snapshot", "reconcileID": "87a28645-ef4a-4d7e-a013-50f27d67a274", "cluster": "some-name", "namespace": "demand-backup-snapshot-28814"} 2026-03-31T23:30:10.061Z INFO Waiting for statefulsets to be ready before restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-snapshot","namespace":"demand-backup-snapshot-28814"}, "namespace": "demand-backup-snapshot-28814", "name": "restore-backup-snapshot", "reconcileID": "87a28645-ef4a-4d7e-a013-50f27d67a274", "ready": false} 2026-03-31T23:30:15.062Z DEBUG checking if restore is allowed {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-snapshot","namespace":"demand-backup-snapshot-28814"}, "namespace": "demand-backup-snapshot-28814", "name": "restore-backup-snapshot", "reconcileID": "71d7c63d-6df5-4970-afa2-2837efdcf5c2", "cluster": "some-name", "namespace": "demand-backup-snapshot-28814"} 2026-03-31T23:30:15.112Z INFO Waiting for statefulsets to be ready before restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-snapshot","namespace":"demand-backup-snapshot-28814"}, "namespace": "demand-backup-snapshot-28814", "name": "restore-backup-snapshot", "reconcileID": "71d7c63d-6df5-4970-afa2-2837efdcf5c2", "ready": false} 2026-03-31T23:30:20.113Z DEBUG checking if restore is allowed {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-snapshot","namespace":"demand-backup-snapshot-28814"}, "namespace": "demand-backup-snapshot-28814", "name": "restore-backup-snapshot", "reconcileID": "05c82ef9-9222-48fe-9fa7-e1bba8ad34e3", "cluster": "some-name", "namespace": "demand-backup-snapshot-28814"} 2026-03-31T23:30:20.149Z INFO Waiting for statefulsets to be ready before restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-snapshot","namespace":"demand-backup-snapshot-28814"}, "namespace": "demand-backup-snapshot-28814", "name": "restore-backup-snapshot", "reconcileID": "05c82ef9-9222-48fe-9fa7-e1bba8ad34e3", "ready": false} 2026-03-31T23:30:25.149Z DEBUG checking if restore is allowed {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-snapshot","namespace":"demand-backup-snapshot-28814"}, "namespace": "demand-backup-snapshot-28814", "name": "restore-backup-snapshot", "reconcileID": "8c6ea4db-b11f-4a50-a9a8-4108b08161ee", "cluster": "some-name", "namespace": "demand-backup-snapshot-28814"} 2026-03-31T23:30:25.184Z INFO Waiting for statefulsets to be ready before restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-snapshot","namespace":"demand-backup-snapshot-28814"}, "namespace": "demand-backup-snapshot-28814", "name": "restore-backup-snapshot", "reconcileID": "8c6ea4db-b11f-4a50-a9a8-4108b08161ee", "ready": false} 2026-03-31T23:30:30.185Z DEBUG checking if restore is allowed {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-snapshot","namespace":"demand-backup-snapshot-28814"}, "namespace": "demand-backup-snapshot-28814", "name": "restore-backup-snapshot", "reconcileID": "741efdbe-6f44-4c6e-bfa8-6cac15a74457", "cluster": "some-name", "namespace": "demand-backup-snapshot-28814"} 2026-03-31T23:30:30.225Z INFO Waiting for statefulsets to be ready before restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-snapshot","namespace":"demand-backup-snapshot-28814"}, "namespace": "demand-backup-snapshot-28814", "name": "restore-backup-snapshot", "reconcileID": "741efdbe-6f44-4c6e-bfa8-6cac15a74457", "ready": false} 2026-03-31T23:30:35.226Z DEBUG checking if restore is allowed {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-snapshot","namespace":"demand-backup-snapshot-28814"}, "namespace": "demand-backup-snapshot-28814", "name": "restore-backup-snapshot", "reconcileID": "b92c7a75-83c0-4e15-8d9c-a0f907717352", "cluster": "some-name", "namespace": "demand-backup-snapshot-28814"} 2026-03-31T23:30:35.263Z INFO Waiting for statefulsets to be ready before restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-snapshot","namespace":"demand-backup-snapshot-28814"}, "namespace": "demand-backup-snapshot-28814", "name": "restore-backup-snapshot", "reconcileID": "b92c7a75-83c0-4e15-8d9c-a0f907717352", "ready": false} 2026-03-31T23:30:40.263Z DEBUG checking if restore is allowed {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-snapshot","namespace":"demand-backup-snapshot-28814"}, "namespace": "demand-backup-snapshot-28814", "name": "restore-backup-snapshot", "reconcileID": "42ce927e-4c23-4816-872b-efe707f64c28", "cluster": "some-name", "namespace": "demand-backup-snapshot-28814"} 2026-03-31T23:30:40.306Z INFO Waiting for statefulsets to be ready before restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-snapshot","namespace":"demand-backup-snapshot-28814"}, "namespace": "demand-backup-snapshot-28814", "name": "restore-backup-snapshot", "reconcileID": "42ce927e-4c23-4816-872b-efe707f64c28", "ready": false} 2026-03-31T23:30:45.307Z DEBUG checking if restore is allowed {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-snapshot","namespace":"demand-backup-snapshot-28814"}, "namespace": "demand-backup-snapshot-28814", "name": "restore-backup-snapshot", "reconcileID": "2d53a234-78cf-4de1-8e1b-971c7f70d7c6", "cluster": "some-name", "namespace": "demand-backup-snapshot-28814"} 2026-03-31T23:30:45.347Z DEBUG Pod is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-snapshot","namespace":"demand-backup-snapshot-28814"}, "namespace": "demand-backup-snapshot-28814", "name": "restore-backup-snapshot", "reconcileID": "2d53a234-78cf-4de1-8e1b-971c7f70d7c6", "pod": "some-name-rs0-2"} 2026-03-31T23:30:45.347Z INFO Waiting for statefulsets to be ready before restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-snapshot","namespace":"demand-backup-snapshot-28814"}, "namespace": "demand-backup-snapshot-28814", "name": "restore-backup-snapshot", "reconcileID": "2d53a234-78cf-4de1-8e1b-971c7f70d7c6", "ready": false} 2026-03-31T23:30:50.347Z DEBUG checking if restore is allowed {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-snapshot","namespace":"demand-backup-snapshot-28814"}, "namespace": "demand-backup-snapshot-28814", "name": "restore-backup-snapshot", "reconcileID": "95de3757-3acb-4afa-8a8f-ad9faa3f3971", "cluster": "some-name", "namespace": "demand-backup-snapshot-28814"} 2026-03-31T23:30:50.382Z DEBUG Pod is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-snapshot","namespace":"demand-backup-snapshot-28814"}, "namespace": "demand-backup-snapshot-28814", "name": "restore-backup-snapshot", "reconcileID": "95de3757-3acb-4afa-8a8f-ad9faa3f3971", "pod": "some-name-rs0-2"} 2026-03-31T23:30:50.382Z INFO Waiting for statefulsets to be ready before restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-snapshot","namespace":"demand-backup-snapshot-28814"}, "namespace": "demand-backup-snapshot-28814", "name": "restore-backup-snapshot", "reconcileID": "95de3757-3acb-4afa-8a8f-ad9faa3f3971", "ready": false} 2026-03-31T23:30:55.383Z DEBUG checking if restore is allowed {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-snapshot","namespace":"demand-backup-snapshot-28814"}, "namespace": "demand-backup-snapshot-28814", "name": "restore-backup-snapshot", "reconcileID": "792923e7-35de-479c-92f8-e893878bab36", "cluster": "some-name", "namespace": "demand-backup-snapshot-28814"} 2026-03-31T23:30:55.434Z INFO Waiting for statefulsets to be ready before restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-snapshot","namespace":"demand-backup-snapshot-28814"}, "namespace": "demand-backup-snapshot-28814", "name": "restore-backup-snapshot", "reconcileID": "792923e7-35de-479c-92f8-e893878bab36", "ready": false} 2026-03-31T23:31:00.435Z DEBUG checking if restore is allowed {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-snapshot","namespace":"demand-backup-snapshot-28814"}, "namespace": "demand-backup-snapshot-28814", "name": "restore-backup-snapshot", "reconcileID": "974b4fe8-03b3-42f2-98a4-e9f403798ae4", "cluster": "some-name", "namespace": "demand-backup-snapshot-28814"} 2026-03-31T23:31:00.472Z INFO Waiting for statefulsets to be ready before restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-snapshot","namespace":"demand-backup-snapshot-28814"}, "namespace": "demand-backup-snapshot-28814", "name": "restore-backup-snapshot", "reconcileID": "974b4fe8-03b3-42f2-98a4-e9f403798ae4", "ready": false} 2026-03-31T23:31:05.473Z DEBUG checking if restore is allowed {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-snapshot","namespace":"demand-backup-snapshot-28814"}, "namespace": "demand-backup-snapshot-28814", "name": "restore-backup-snapshot", "reconcileID": "418fc9be-16f2-493f-8ab4-8f24c172a4f8", "cluster": "some-name", "namespace": "demand-backup-snapshot-28814"} 2026-03-31T23:31:05.509Z INFO Waiting for statefulsets to be ready before restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-snapshot","namespace":"demand-backup-snapshot-28814"}, "namespace": "demand-backup-snapshot-28814", "name": "restore-backup-snapshot", "reconcileID": "418fc9be-16f2-493f-8ab4-8f24c172a4f8", "ready": false} 2026-03-31T23:31:10.510Z DEBUG checking if restore is allowed {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-snapshot","namespace":"demand-backup-snapshot-28814"}, "namespace": "demand-backup-snapshot-28814", "name": "restore-backup-snapshot", "reconcileID": "0f6399b2-b02f-4e52-8c4d-a147ea486e40", "cluster": "some-name", "namespace": "demand-backup-snapshot-28814"} 2026-03-31T23:31:10.548Z INFO Waiting for statefulsets to be ready before restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-snapshot","namespace":"demand-backup-snapshot-28814"}, "namespace": "demand-backup-snapshot-28814", "name": "restore-backup-snapshot", "reconcileID": "0f6399b2-b02f-4e52-8c4d-a147ea486e40", "ready": false} 2026-03-31T23:31:15.548Z DEBUG checking if restore is allowed {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-snapshot","namespace":"demand-backup-snapshot-28814"}, "namespace": "demand-backup-snapshot-28814", "name": "restore-backup-snapshot", "reconcileID": "e48259fa-96ca-4de9-b660-f66855c8c767", "cluster": "some-name", "namespace": "demand-backup-snapshot-28814"} 2026-03-31T23:31:15.593Z DEBUG Pod is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-snapshot","namespace":"demand-backup-snapshot-28814"}, "namespace": "demand-backup-snapshot-28814", "name": "restore-backup-snapshot", "reconcileID": "e48259fa-96ca-4de9-b660-f66855c8c767", "pod": "some-name-rs0-2"} 2026-03-31T23:31:15.593Z INFO Waiting for statefulsets to be ready before restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-snapshot","namespace":"demand-backup-snapshot-28814"}, "namespace": "demand-backup-snapshot-28814", "name": "restore-backup-snapshot", "reconcileID": "e48259fa-96ca-4de9-b660-f66855c8c767", "ready": false} 2026-03-31T23:31:20.595Z DEBUG checking if restore is allowed {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-snapshot","namespace":"demand-backup-snapshot-28814"}, "namespace": "demand-backup-snapshot-28814", "name": "restore-backup-snapshot", "reconcileID": "d4c8b1a9-2683-4537-9397-0cb3a9e81237", "cluster": "some-name", "namespace": "demand-backup-snapshot-28814"} 2026-03-31T23:31:20.629Z INFO Waiting for statefulsets to be ready before restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-snapshot","namespace":"demand-backup-snapshot-28814"}, "namespace": "demand-backup-snapshot-28814", "name": "restore-backup-snapshot", "reconcileID": "d4c8b1a9-2683-4537-9397-0cb3a9e81237", "ready": false} 2026-03-31T23:31:25.630Z DEBUG checking if restore is allowed {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-snapshot","namespace":"demand-backup-snapshot-28814"}, "namespace": "demand-backup-snapshot-28814", "name": "restore-backup-snapshot", "reconcileID": "fbe3bb4d-8c9d-4fb1-ac4b-14554daac37a", "cluster": "some-name", "namespace": "demand-backup-snapshot-28814"} 2026-03-31T23:31:55.633Z ERROR failed to make restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-snapshot","namespace":"demand-backup-snapshot-28814"}, "namespace": "demand-backup-snapshot-28814", "name": "restore-backup-snapshot", "reconcileID": "fbe3bb4d-8c9d-4fb1-ac4b-14554daac37a", "restore": "restore-backup-snapshot", "backup": "backup-snapshot", "error": "update PBM config secret: new PBM connection: create PBM connection to some-name-rs0-0.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local:27017,some-name-rs0-1.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local:27017,some-name-rs0-2.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local:27017: create mongo connection: ping: server selection error: server selection timeout, current topology: { Type: ReplicaSetNoPrimary, Servers: [{ Addr: some-name-rs0-0.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local:27017, Type: Unknown, Last error: dial tcp 10.148.242.8:27017: i/o timeout }, { Addr: some-name-rs0-1.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local:27017, Type: RSSecondary, Tag sets: podName=some-name-rs0-1,serviceName=some-name,region=us-central1,nodeName=gke-jen-psmdb-2289-3781d-default-pool-1df743e5-zl8m,zone=us-central1-a, Average RTT: 1718669 }, { Addr: some-name-rs0-2.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local:27017, Type: Unknown, Last error: dial tcp: lookup some-name-rs0-2.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local: i/o timeout }, ] }", "errorVerbose": "server selection error: server selection timeout, current topology: { Type: ReplicaSetNoPrimary, Servers: [{ Addr: some-name-rs0-0.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local:27017, Type: Unknown, Last error: dial tcp 10.148.242.8:27017: i/o timeout }, { Addr: some-name-rs0-1.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local:27017, Type: RSSecondary, Tag sets: podName=some-name-rs0-1,serviceName=some-name,region=us-central1,nodeName=gke-jen-psmdb-2289-3781d-default-pool-1df743e5-zl8m,zone=us-central1-a, Average RTT: 1718669 }, { Addr: some-name-rs0-2.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local:27017, Type: Unknown, Last error: dial tcp: lookup some-name-rs0-2.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local: i/o timeout }, ] }\nping\ncreate mongo connection\ncreate PBM connection to some-name-rs0-0.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local:27017,some-name-rs0-1.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local:27017,some-name-rs0-2.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local:27017\ngithub.com/percona/percona-server-mongodb-operator/pkg/psmdb/backup.NewPBM\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/psmdb/backup/pbm.go:223\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).updatePBMConfigSecret\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/physical.go:864\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).reconcileSnapshotWaiting\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/snapshots.go:71\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).reconcileExternalSnapshotRestore\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/snapshots.go:43\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/psmdb_restore_controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:222\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:479\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:313\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1693\nnew PBM connection\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).updatePBMConfigSecret\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/physical.go:866\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).reconcileSnapshotWaiting\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/snapshots.go:71\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).reconcileExternalSnapshotRestore\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/snapshots.go:43\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/psmdb_restore_controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:222\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:479\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:313\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1693\nupdate PBM config secret\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).reconcileSnapshotWaiting\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/snapshots.go:72\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).reconcileExternalSnapshotRestore\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/snapshots.go:43\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/psmdb_restore_controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:222\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:479\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:313\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1693"} 2026-03-31T23:31:55.634Z INFO Restore state changed {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-snapshot","namespace":"demand-backup-snapshot-28814"}, "namespace": "demand-backup-snapshot-28814", "name": "restore-backup-snapshot", "reconcileID": "fbe3bb4d-8c9d-4fb1-ac4b-14554daac37a", "previous": "waiting", "current": "error"} 2026-03-31T23:31:56.211Z INFO Warning: Reconciler returned both a result with either RequeueAfter or Requeue set and a non-nil error. RequeueAfter and Requeue will always be ignored if the error is non-nil. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-snapshot","namespace":"demand-backup-snapshot-28814"}, "namespace": "demand-backup-snapshot-28814", "name": "restore-backup-snapshot", "reconcileID": "fbe3bb4d-8c9d-4fb1-ac4b-14554daac37a"} 2026-03-31T23:31:56.211Z ERROR Reconciler error {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-snapshot","namespace":"demand-backup-snapshot-28814"}, "namespace": "demand-backup-snapshot-28814", "name": "restore-backup-snapshot", "reconcileID": "fbe3bb4d-8c9d-4fb1-ac4b-14554daac37a", "error": "reconcile external snapshot restore: update PBM config secret: new PBM connection: create PBM connection to some-name-rs0-0.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local:27017,some-name-rs0-1.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local:27017,some-name-rs0-2.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local:27017: create mongo connection: ping: server selection error: server selection timeout, current topology: { Type: ReplicaSetNoPrimary, Servers: [{ Addr: some-name-rs0-0.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local:27017, Type: Unknown, Last error: dial tcp 10.148.242.8:27017: i/o timeout }, { Addr: some-name-rs0-1.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local:27017, Type: RSSecondary, Tag sets: podName=some-name-rs0-1,serviceName=some-name,region=us-central1,nodeName=gke-jen-psmdb-2289-3781d-default-pool-1df743e5-zl8m,zone=us-central1-a, Average RTT: 1718669 }, { Addr: some-name-rs0-2.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local:27017, Type: Unknown, Last error: dial tcp: lookup some-name-rs0-2.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local: i/o timeout }, ] }", "errorVerbose": "server selection error: server selection timeout, current topology: { Type: ReplicaSetNoPrimary, Servers: [{ Addr: some-name-rs0-0.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local:27017, Type: Unknown, Last error: dial tcp 10.148.242.8:27017: i/o timeout }, { Addr: some-name-rs0-1.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local:27017, Type: RSSecondary, Tag sets: podName=some-name-rs0-1,serviceName=some-name,region=us-central1,nodeName=gke-jen-psmdb-2289-3781d-default-pool-1df743e5-zl8m,zone=us-central1-a, Average RTT: 1718669 }, { Addr: some-name-rs0-2.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local:27017, Type: Unknown, Last error: dial tcp: lookup some-name-rs0-2.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local: i/o timeout }, ] }\nping\ncreate mongo connection\ncreate PBM connection to some-name-rs0-0.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local:27017,some-name-rs0-1.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local:27017,some-name-rs0-2.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local:27017\ngithub.com/percona/percona-server-mongodb-operator/pkg/psmdb/backup.NewPBM\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/psmdb/backup/pbm.go:223\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).updatePBMConfigSecret\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/physical.go:864\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).reconcileSnapshotWaiting\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/snapshots.go:71\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).reconcileExternalSnapshotRestore\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/snapshots.go:43\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/psmdb_restore_controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:222\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:479\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:313\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1693\nnew PBM connection\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).updatePBMConfigSecret\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/physical.go:866\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).reconcileSnapshotWaiting\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/snapshots.go:71\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).reconcileExternalSnapshotRestore\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/snapshots.go:43\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/psmdb_restore_controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:222\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:479\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:313\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1693\nupdate PBM config secret\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).reconcileSnapshotWaiting\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/snapshots.go:72\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).reconcileExternalSnapshotRestore\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/snapshots.go:43\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/psmdb_restore_controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:222\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:479\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:313\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1693\nreconcile external snapshot restore\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/psmdb_restore_controller.go:268\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:222\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:479\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:313\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1693"} apiVersion: psmdb.percona.com/v1 kind: PerconaServerMongoDBRestore metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"psmdb.percona.com/v1","kind":"PerconaServerMongoDBRestore","metadata":{"annotations":{},"name":"restore-backup-snapshot","namespace":"demand-backup-snapshot-28814"},"spec":{"backupName":"backup-snapshot","clusterName":"some-name"}} creationTimestamp: "2026-03-31T23:29:54Z" generation: 1 name: restore-backup-snapshot namespace: demand-backup-snapshot-28814 resourceVersion: "1774999915651183012" uid: a0538640-bc54-4418-bded-53d927b33a41 spec: backupName: backup-snapshot clusterName: some-name status: error: 'update PBM config secret: new PBM connection: create PBM connection to some-name-rs0-0.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local:27017,some-name-rs0-1.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local:27017,some-name-rs0-2.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local:27017: create mongo connection: ping: server selection error: server selection timeout, current topology: { Type: ReplicaSetNoPrimary, Servers: [{ Addr: some-name-rs0-0.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local:27017, Type: Unknown, Last error: dial tcp 10.148.242.8:27017: i/o timeout }, { Addr: some-name-rs0-1.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local:27017, Type: RSSecondary, Tag sets: podName=some-name-rs0-1,serviceName=some-name,region=us-central1,nodeName=gke-jen-psmdb-2289-3781d-default-pool-1df743e5-zl8m,zone=us-central1-a, Average RTT: 1718669 }, { Addr: some-name-rs0-2.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local:27017, Type: Unknown, Last error: dial tcp: lookup some-name-rs0-2.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local: i/o timeout }, ] }' state: error Name: restore-backup-snapshot Namespace: demand-backup-snapshot-28814 Labels: Annotations: API Version: psmdb.percona.com/v1 Kind: PerconaServerMongoDBRestore Metadata: Creation Timestamp: 2026-03-31T23:29:54Z Generation: 1 Resource Version: 1774999915651183012 UID: a0538640-bc54-4418-bded-53d927b33a41 Spec: Backup Name: backup-snapshot Cluster Name: some-name Status: Error: update PBM config secret: new PBM connection: create PBM connection to some-name-rs0-0.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local:27017,some-name-rs0-1.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local:27017,some-name-rs0-2.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local:27017: create mongo connection: ping: server selection error: server selection timeout, current topology: { Type: ReplicaSetNoPrimary, Servers: [{ Addr: some-name-rs0-0.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local:27017, Type: Unknown, Last error: dial tcp 10.148.242.8:27017: i/o timeout }, { Addr: some-name-rs0-1.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local:27017, Type: RSSecondary, Tag sets: podName=some-name-rs0-1,serviceName=some-name,region=us-central1,nodeName=gke-jen-psmdb-2289-3781d-default-pool-1df743e5-zl8m,zone=us-central1-a, Average RTT: 1718669 }, { Addr: some-name-rs0-2.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local:27017, Type: Unknown, Last error: dial tcp: lookup some-name-rs0-2.some-name-rs0.demand-backup-snapshot-28814.svc.cluster.local: i/o timeout }, ] } State: error Events: Restore object restore-backup-snapshot is in error state after 2 minutes. something went wrong with operator or kubernetes cluster