Log: /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2272/e2e-tests/logs/demand-backup-incremental-gcp-s3.log Warning: version difference between client (1.35) and server (1.32) exceeds the supported minor version skew of +/-1 Warning: version difference between client (1.35) and server (1.32) exceeds the supported minor version skew of +/-1 Warning: version difference between client (1.35) and server (1.32) exceeds the supported minor version skew of +/-1 ----------------------------------------------------------------------------------- get and delete old CRDs and RBAC ----------------------------------------------------------------------------------- grep: warning: stray \ before - grep: warning: stray \ before - error: the server doesn't have a resource type "perconaservermongodbbackups" + kubectl patch perconaservermongodbbackups.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbbackups" error: the server doesn't have a resource type "perconaservermongodbrestores" + kubectl patch perconaservermongodbrestores.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbrestores" error: the server doesn't have a resource type "perconaservermongodbs" + kubectl patch perconaservermongodbs.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbs" Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found ----------------------------------------------------------------------------------- destroy chaos-mesh ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces psmdb-operator ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- create namespace psmdb-operator ----------------------------------------------------------------------------------- namespace/psmdb-operator created Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-2272-ae4e3cbc-4-cluster6" modified. ----------------------------------------------------------------------------------- start PSMDB operator: docker.io/perconalab/percona-server-mongodb-operator:PR-2272-ae4e3cbc ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io/perconaservermongodbbackups.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbrestores.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbs.psmdb.percona.com serverside-applied clusterrole.rbac.authorization.k8s.io/percona-server-mongodb-operator created serviceaccount/percona-server-mongodb-operator created clusterrolebinding.rbac.authorization.k8s.io/service-account-percona-server-mongodb-operator created deployment.apps/percona-server-mongodb-operator created waiting for pod/percona-server-mongodb-operator-6455bb5fb4-xm72t to be ready.OK Print operator info from log 2026-03-26T08:08:46.339Z INFO setup Manager starting up {"gitCommit": "ae4e3cbc053c422311418c8c6083b24139fc7e69", "gitBranch": "PR-2272-ae4e3cbc", "buildTime": "", "goVersion": "go1.25.8", "os": "linux", "arch": "amd64"} ----------------------------------------------------------------------------------- destroy chaos-mesh ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces demand-backup-incremental-gcp-s3-9166 ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- create namespace demand-backup-incremental-gcp-s3-9166 ----------------------------------------------------------------------------------- namespace/demand-backup-incremental-gcp-s3-9166 created Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-2272-ae4e3cbc-4-cluster6" modified. ----------------------------------------------------------------------------------- create secrets for cloud storages ----------------------------------------------------------------------------------- secret/minio-secret created secret/aws-s3-secret created secret/gcp-cs-secret created secret/azure-secret created secret/gcp-cs-sa-key-secret created ----------------------------------------------------------------------------------- Testing on not sharded cluster ----------------------------------------------------------------------------------- [2026-03-26T08:09:24+0000] Creating PSMDB cluster secret/some-users created perconaservermongodb.psmdb.percona.com/some-name created deployment.apps/psmdb-client created [2026-03-26T08:09:30+0000] check if all pods started waiting for pod/some-name-rs0-0 to be ready........OK waiting for pod/some-name-rs0-1 to be ready........OK waiting for pod/some-name-rs0-2 to be ready......OK Waiting for cluster readyness..... waiting for cluster readyness.OK [2026-03-26T08:11:14+0000] writing test data Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://some-name-rs0-2.some-name-rs0.demand-backup-incremental-gcp-s3-9166.svc.cluster.local:27017,some-name-rs0-0.some-name-rs0.demand-backup-incremental-gcp-s3-9166.svc.cluster.local:27017,some-name-rs0-1.some-name-rs0.demand-backup-incremental-gcp-s3-9166.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb&replicaSet=rs0&ssl=false Implicit session: session { "id" : UUID("59a057ea-ebf5-4be9-a25c-823d2505810f") } Percona Server for MongoDB server version: v8.0.19-7 WARNING: shell and server versions do not match Successfully added user: { "user" : "myApp", "roles" : [ { "db" : "myApp", "role" : "readWrite" } ] } bye Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://some-name-rs0-1.some-name-rs0.demand-backup-incremental-gcp-s3-9166.svc.cluster.local:27017,some-name-rs0-2.some-name-rs0.demand-backup-incremental-gcp-s3-9166.svc.cluster.local:27017,some-name-rs0-0.some-name-rs0.demand-backup-incremental-gcp-s3-9166.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb&replicaSet=rs0&ssl=false Implicit session: session { "id" : UUID("dc0fed87-3aea-4fa4-828c-79976d7fcb12") } Percona Server for MongoDB server version: v8.0.19-7 WARNING: shell and server versions do not match switched to db myApp WriteResult({ "nInserted" : 1 }) bye [2026-03-26T08:11:25+0000] running db.test.find() in myApp [2026-03-26T08:11:28+0000] running db.test.find() in myApp [2026-03-26T08:11:30+0000] running db.test.find() in myApp [2026-03-26T08:11:32+0000] running backup [2026-03-26T08:11:32+0000] running backup backup-gcp-cs-s3 perconaservermongodbbackup.psmdb.percona.com/backup-gcp-cs-s3 created waiting for backup-gcp-cs-s3 to reach ready state...............OK Setting up GCS credentials from K8s secret: gcp-cs-secret GCS credentials configured successfully Backup gs://operator-testing/psmdb-demand-backup-incremental-gcp-s3/2026-03-26T08:11:34Z/rs0/filelist.pbm found in gcp-cs [2026-03-26T08:12:08+0000] drop collection Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://some-name-rs0-1.some-name-rs0.demand-backup-incremental-gcp-s3-9166.svc.cluster.local:27017,some-name-rs0-2.some-name-rs0.demand-backup-incremental-gcp-s3-9166.svc.cluster.local:27017,some-name-rs0-0.some-name-rs0.demand-backup-incremental-gcp-s3-9166.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb&replicaSet=rs0&ssl=false Implicit session: session { "id" : UUID("d0147725-b00e-4d83-842d-ddaebc2cbc44") } Percona Server for MongoDB server version: v8.0.19-7 WARNING: shell and server versions do not match switched to db myApp true bye [2026-03-26T08:12:11+0000] check backup and restore -- [2026-03-26T08:12:11+0000] running restore [2026-03-26T08:12:11+0000] running restore restore-backup-gcp-cs-s3 perconaservermongodbrestore.psmdb.percona.com/restore-backup-gcp-cs-s3 created Waiting for the psmdb-restore/restore-backup-gcp-cs-s3 object to be created.OK Waiting psmdb-restore/restore-backup-gcp-cs-s3 to reach state "requested" ...OK after 2 minutes [2026-03-26T08:14:18+0000] compare_kubectl: statefulset/some-name-rs0 OK Waiting for the psmdb-restore/restore-backup-gcp-cs-s3 object to be created.OK Waiting psmdb-restore/restore-backup-gcp-cs-s3 to reach state "ready" ..2026-03-26T08:14:22.938Z DEBUG Statefulset is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "4400f92b-a8fa-4ec5-b1a3-675990dcffec", "sts": "some-name-rs0", "replset": "rs0"} 2026-03-26T08:14:22.938Z DEBUG Check restore status {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "4400f92b-a8fa-4ec5-b1a3-675990dcffec", "command": ["/opt/percona/pbm", "describe-restore", "2026-03-26T08:14:16.764090759Z", "--config", "/etc/pbm/pbm_config.yaml", "--out", "json"], "pod": "some-name-rs0-0"} 2026-03-26T08:14:23.846Z DEBUG PBM restore status {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "4400f92b-a8fa-4ec5-b1a3-675990dcffec", "status": {"type":"physical","opid":"","name":"2026-03-26T08:14:16.764090759Z","replsets":[{"name":"rs0","start_ts":0,"status":"","size":0,"size_uncompressed":0,"last_transition_ts":0,"first_write_ts":{"T":0,"I":0},"last_write_ts":{"T":0,"I":0},"node":"","conditions":null}],"compression":"","store":{"type":""},"size":0,"size_uncompressed":0,"mongodb_version":"","fcv":"","start_ts":0,"last_transition_ts":0,"first_write_ts":{"T":0,"I":0},"last_write_ts":{"T":0,"I":0},"hb":{"T":0,"I":0},"status":"","conditions":null,"n":null,"pbm_version":"","balancer":""}} 2026-03-26T08:14:28.847Z DEBUG checking if restore is allowed {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "5ad555e6-d7d6-440f-be2f-be44b0030357", "cluster": "some-name", "namespace": "demand-backup-incremental-gcp-s3-9166"} 2026-03-26T08:14:28.848Z DEBUG Pod is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "5ad555e6-d7d6-440f-be2f-be44b0030357", "pod": "some-name-rs0-1"} 2026-03-26T08:14:28.848Z DEBUG Pod is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "5ad555e6-d7d6-440f-be2f-be44b0030357", "pod": "some-name-rs0-2"} 2026-03-26T08:14:28.848Z DEBUG Pod is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "5ad555e6-d7d6-440f-be2f-be44b0030357", "pod": "some-name-rs0-0"} 2026-03-26T08:14:28.848Z DEBUG Statefulset is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "5ad555e6-d7d6-440f-be2f-be44b0030357", "sts": "some-name-rs0", "replset": "rs0"} 2026-03-26T08:14:28.848Z DEBUG Check restore status {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "5ad555e6-d7d6-440f-be2f-be44b0030357", "command": ["/opt/percona/pbm", "describe-restore", "2026-03-26T08:14:16.764090759Z", "--config", "/etc/pbm/pbm_config.yaml", "--out", "json"], "pod": "some-name-rs0-0"} 2026-03-26T08:14:29.793Z DEBUG PBM restore status {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "5ad555e6-d7d6-440f-be2f-be44b0030357", "status": {"type":"physical","opid":"","name":"2026-03-26T08:14:16.764090759Z","replsets":[{"name":"rs0","start_ts":0,"status":"starting","size":0,"size_uncompressed":0,"last_transition_ts":1774512864,"first_write_ts":{"T":0,"I":0},"last_write_ts":{"T":0,"I":0},"node":"","conditions":null}],"compression":"","store":{"type":""},"size":0,"size_uncompressed":0,"mongodb_version":"","fcv":"","start_ts":0,"last_transition_ts":0,"first_write_ts":{"T":0,"I":0},"last_write_ts":{"T":0,"I":0},"hb":{"T":0,"I":0},"status":"","conditions":null,"n":null,"pbm_version":"","balancer":""}} 2026-03-26T08:14:34.794Z DEBUG checking if restore is allowed {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "5ab014c7-726d-4bb5-87b7-e7b75a9a4eee", "cluster": "some-name", "namespace": "demand-backup-incremental-gcp-s3-9166"} 2026-03-26T08:14:34.795Z DEBUG Pod is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "5ab014c7-726d-4bb5-87b7-e7b75a9a4eee", "pod": "some-name-rs0-0"} 2026-03-26T08:14:34.795Z DEBUG Pod is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "5ab014c7-726d-4bb5-87b7-e7b75a9a4eee", "pod": "some-name-rs0-1"} 2026-03-26T08:14:34.795Z DEBUG Pod is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "5ab014c7-726d-4bb5-87b7-e7b75a9a4eee", "pod": "some-name-rs0-2"} 2026-03-26T08:14:34.795Z DEBUG Statefulset is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "5ab014c7-726d-4bb5-87b7-e7b75a9a4eee", "sts": "some-name-rs0", "replset": "rs0"} 2026-03-26T08:14:34.796Z DEBUG Check restore status {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "5ab014c7-726d-4bb5-87b7-e7b75a9a4eee", "command": ["/opt/percona/pbm", "describe-restore", "2026-03-26T08:14:16.764090759Z", "--config", "/etc/pbm/pbm_config.yaml", "--out", "json"], "pod": "some-name-rs0-0"} 2026-03-26T08:14:36.017Z DEBUG PBM restore status {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "5ab014c7-726d-4bb5-87b7-e7b75a9a4eee", "status": {"type":"physical","opid":"","name":"2026-03-26T08:14:16.764090759Z","replsets":[{"name":"rs0","start_ts":0,"status":"starting","size":0,"size_uncompressed":0,"last_transition_ts":1774512864,"first_write_ts":{"T":0,"I":0},"last_write_ts":{"T":0,"I":0},"node":"","conditions":null}],"compression":"","store":{"type":""},"size":0,"size_uncompressed":0,"mongodb_version":"","fcv":"","start_ts":0,"last_transition_ts":1774512869,"first_write_ts":{"T":0,"I":0},"last_write_ts":{"T":0,"I":0},"hb":{"T":0,"I":0},"status":"starting","conditions":null,"n":null,"pbm_version":"","balancer":""}} 2026-03-26T08:14:41.018Z DEBUG checking if restore is allowed {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "aac8017f-6859-4b48-a254-7d0449a8b902", "cluster": "some-name", "namespace": "demand-backup-incremental-gcp-s3-9166"} 2026-03-26T08:14:41.019Z DEBUG Pod is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "aac8017f-6859-4b48-a254-7d0449a8b902", "pod": "some-name-rs0-1"} 2026-03-26T08:14:41.019Z DEBUG Pod is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "aac8017f-6859-4b48-a254-7d0449a8b902", "pod": "some-name-rs0-2"} 2026-03-26T08:14:41.019Z DEBUG Pod is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "aac8017f-6859-4b48-a254-7d0449a8b902", "pod": "some-name-rs0-0"} 2026-03-26T08:14:41.019Z DEBUG Statefulset is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "aac8017f-6859-4b48-a254-7d0449a8b902", "sts": "some-name-rs0", "replset": "rs0"} 2026-03-26T08:14:41.019Z DEBUG Check restore status {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "aac8017f-6859-4b48-a254-7d0449a8b902", "command": ["/opt/percona/pbm", "describe-restore", "2026-03-26T08:14:16.764090759Z", "--config", "/etc/pbm/pbm_config.yaml", "--out", "json"], "pod": "some-name-rs0-0"} 2026-03-26T08:14:43.212Z DEBUG PBM restore status {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "aac8017f-6859-4b48-a254-7d0449a8b902", "status": {"type":"physical","opid":"","name":"2026-03-26T08:14:16.764090759Z","replsets":[{"name":"rs0","start_ts":0,"status":"running","size":0,"size_uncompressed":0,"last_transition_ts":1774512880,"first_write_ts":{"T":0,"I":0},"last_write_ts":{"T":0,"I":0},"node":"","conditions":null}],"compression":"","store":{"type":""},"size":0,"size_uncompressed":0,"mongodb_version":"","fcv":"","start_ts":0,"last_transition_ts":1774512869,"first_write_ts":{"T":0,"I":0},"last_write_ts":{"T":0,"I":0},"hb":{"T":0,"I":0},"status":"starting","conditions":null,"n":null,"pbm_version":"","balancer":""}} 2026-03-26T08:14:43.212Z INFO Restore state changed {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "aac8017f-6859-4b48-a254-7d0449a8b902", "previous": "requested", "current": "running"} 2026-03-26T08:14:43.788Z DEBUG checking if restore is allowed {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "58fb79a7-6bd6-4361-8c52-88f1e52e0509", "cluster": "some-name", "namespace": "demand-backup-incremental-gcp-s3-9166"} 2026-03-26T08:14:43.788Z DEBUG Pod is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "58fb79a7-6bd6-4361-8c52-88f1e52e0509", "pod": "some-name-rs0-0"} 2026-03-26T08:14:43.788Z DEBUG Pod is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "58fb79a7-6bd6-4361-8c52-88f1e52e0509", "pod": "some-name-rs0-1"} 2026-03-26T08:14:43.788Z DEBUG Pod is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "58fb79a7-6bd6-4361-8c52-88f1e52e0509", "pod": "some-name-rs0-2"} 2026-03-26T08:14:43.788Z DEBUG Statefulset is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "58fb79a7-6bd6-4361-8c52-88f1e52e0509", "sts": "some-name-rs0", "replset": "rs0"} 2026-03-26T08:14:43.788Z DEBUG Check restore status {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "58fb79a7-6bd6-4361-8c52-88f1e52e0509", "command": ["/opt/percona/pbm", "describe-restore", "2026-03-26T08:14:16.764090759Z", "--config", "/etc/pbm/pbm_config.yaml", "--out", "json"], "pod": "some-name-rs0-0"} 2026-03-26T08:14:44.936Z DEBUG PBM restore status {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "58fb79a7-6bd6-4361-8c52-88f1e52e0509", "status": {"type":"physical","opid":"","name":"2026-03-26T08:14:16.764090759Z","replsets":[{"name":"rs0","start_ts":0,"status":"running","size":0,"size_uncompressed":0,"last_transition_ts":1774512880,"first_write_ts":{"T":0,"I":0},"last_write_ts":{"T":0,"I":0},"node":"","conditions":null}],"compression":"","store":{"type":""},"size":0,"size_uncompressed":0,"mongodb_version":"","fcv":"","start_ts":0,"last_transition_ts":1774512869,"first_write_ts":{"T":0,"I":0},"last_write_ts":{"T":0,"I":0},"hb":{"T":0,"I":0},"status":"starting","conditions":null,"n":null,"pbm_version":"","balancer":""}} 2026-03-26T08:14:49.938Z DEBUG checking if restore is allowed {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "a228742f-42d5-4044-8f48-257e7a1b49c0", "cluster": "some-name", "namespace": "demand-backup-incremental-gcp-s3-9166"} 2026-03-26T08:14:49.939Z DEBUG Pod is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "a228742f-42d5-4044-8f48-257e7a1b49c0", "pod": "some-name-rs0-0"} 2026-03-26T08:14:49.939Z DEBUG Pod is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "a228742f-42d5-4044-8f48-257e7a1b49c0", "pod": "some-name-rs0-1"} 2026-03-26T08:14:49.939Z DEBUG Pod is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "a228742f-42d5-4044-8f48-257e7a1b49c0", "pod": "some-name-rs0-2"} 2026-03-26T08:14:49.939Z DEBUG Statefulset is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "a228742f-42d5-4044-8f48-257e7a1b49c0", "sts": "some-name-rs0", "replset": "rs0"} 2026-03-26T08:14:49.939Z DEBUG Check restore status {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "a228742f-42d5-4044-8f48-257e7a1b49c0", "command": ["/opt/percona/pbm", "describe-restore", "2026-03-26T08:14:16.764090759Z", "--config", "/etc/pbm/pbm_config.yaml", "--out", "json"], "pod": "some-name-rs0-0"} 2026-03-26T08:14:51.296Z DEBUG PBM restore status {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "a228742f-42d5-4044-8f48-257e7a1b49c0", "status": {"type":"physical","opid":"","name":"2026-03-26T08:14:16.764090759Z","replsets":[{"name":"rs0","start_ts":0,"status":"running","size":0,"size_uncompressed":0,"last_transition_ts":1774512880,"first_write_ts":{"T":0,"I":0},"last_write_ts":{"T":0,"I":0},"node":"","conditions":null}],"compression":"","store":{"type":""},"size":0,"size_uncompressed":0,"mongodb_version":"","fcv":"","start_ts":0,"last_transition_ts":1774512886,"first_write_ts":{"T":0,"I":0},"last_write_ts":{"T":0,"I":0},"hb":{"T":0,"I":0},"status":"running","conditions":null,"n":null,"pbm_version":"","balancer":""}} 2026-03-26T08:14:56.296Z DEBUG checking if restore is allowed {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "df5a9ff9-520a-4fef-84de-b37e4bdb4a56", "cluster": "some-name", "namespace": "demand-backup-incremental-gcp-s3-9166"} 2026-03-26T08:14:56.297Z DEBUG Pod is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "df5a9ff9-520a-4fef-84de-b37e4bdb4a56", "pod": "some-name-rs0-2"} 2026-03-26T08:14:56.297Z DEBUG Pod is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "df5a9ff9-520a-4fef-84de-b37e4bdb4a56", "pod": "some-name-rs0-0"} 2026-03-26T08:14:56.297Z DEBUG Pod is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "df5a9ff9-520a-4fef-84de-b37e4bdb4a56", "pod": "some-name-rs0-1"} 2026-03-26T08:14:56.297Z DEBUG Statefulset is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "df5a9ff9-520a-4fef-84de-b37e4bdb4a56", "sts": "some-name-rs0", "replset": "rs0"} 2026-03-26T08:14:56.297Z DEBUG Check restore status {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "df5a9ff9-520a-4fef-84de-b37e4bdb4a56", "command": ["/opt/percona/pbm", "describe-restore", "2026-03-26T08:14:16.764090759Z", "--config", "/etc/pbm/pbm_config.yaml", "--out", "json"], "pod": "some-name-rs0-0"} 2026-03-26T08:14:58.299Z DEBUG PBM restore status {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "df5a9ff9-520a-4fef-84de-b37e4bdb4a56", "status": {"type":"physical","opid":"","name":"2026-03-26T08:14:16.764090759Z","replsets":[{"name":"rs0","start_ts":0,"status":"running","size":0,"size_uncompressed":0,"last_transition_ts":1774512880,"first_write_ts":{"T":0,"I":0},"last_write_ts":{"T":0,"I":0},"node":"","conditions":null}],"compression":"","store":{"type":""},"size":0,"size_uncompressed":0,"mongodb_version":"","fcv":"","start_ts":0,"last_transition_ts":1774512886,"first_write_ts":{"T":0,"I":0},"last_write_ts":{"T":0,"I":0},"hb":{"T":0,"I":0},"status":"running","conditions":null,"n":null,"pbm_version":"","balancer":""}} 2026-03-26T08:15:03.300Z DEBUG checking if restore is allowed {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "b2cb493e-e710-4343-b33a-767455350358", "cluster": "some-name", "namespace": "demand-backup-incremental-gcp-s3-9166"} 2026-03-26T08:15:03.300Z DEBUG Pod is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "b2cb493e-e710-4343-b33a-767455350358", "pod": "some-name-rs0-2"} 2026-03-26T08:15:03.300Z DEBUG Pod is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "b2cb493e-e710-4343-b33a-767455350358", "pod": "some-name-rs0-0"} 2026-03-26T08:15:03.300Z DEBUG Pod is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "b2cb493e-e710-4343-b33a-767455350358", "pod": "some-name-rs0-1"} 2026-03-26T08:15:03.300Z DEBUG Statefulset is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "b2cb493e-e710-4343-b33a-767455350358", "sts": "some-name-rs0", "replset": "rs0"} 2026-03-26T08:15:03.301Z DEBUG Check restore status {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "b2cb493e-e710-4343-b33a-767455350358", "command": ["/opt/percona/pbm", "describe-restore", "2026-03-26T08:14:16.764090759Z", "--config", "/etc/pbm/pbm_config.yaml", "--out", "json"], "pod": "some-name-rs0-0"} 2026-03-26T08:15:04.903Z DEBUG PBM restore status {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "b2cb493e-e710-4343-b33a-767455350358", "status": {"type":"physical","opid":"","name":"2026-03-26T08:14:16.764090759Z","replsets":[{"name":"rs0","start_ts":0,"status":"running","size":0,"size_uncompressed":0,"last_transition_ts":1774512880,"first_write_ts":{"T":0,"I":0},"last_write_ts":{"T":0,"I":0},"node":"","conditions":null}],"compression":"","store":{"type":""},"size":0,"size_uncompressed":0,"mongodb_version":"","fcv":"","start_ts":0,"last_transition_ts":1774512886,"first_write_ts":{"T":0,"I":0},"last_write_ts":{"T":0,"I":0},"hb":{"T":0,"I":0},"status":"running","conditions":null,"n":null,"pbm_version":"","balancer":""}} 2026-03-26T08:15:09.904Z DEBUG checking if restore is allowed {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "d836f243-ece9-497e-83ca-8ff8a60fc400", "cluster": "some-name", "namespace": "demand-backup-incremental-gcp-s3-9166"} 2026-03-26T08:15:09.905Z DEBUG Pod is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "d836f243-ece9-497e-83ca-8ff8a60fc400", "pod": "some-name-rs0-0"} 2026-03-26T08:15:09.905Z DEBUG Pod is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "d836f243-ece9-497e-83ca-8ff8a60fc400", "pod": "some-name-rs0-1"} 2026-03-26T08:15:09.905Z DEBUG Pod is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "d836f243-ece9-497e-83ca-8ff8a60fc400", "pod": "some-name-rs0-2"} 2026-03-26T08:15:09.905Z DEBUG Statefulset is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "d836f243-ece9-497e-83ca-8ff8a60fc400", "sts": "some-name-rs0", "replset": "rs0"} 2026-03-26T08:15:09.905Z DEBUG Check restore status {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "d836f243-ece9-497e-83ca-8ff8a60fc400", "command": ["/opt/percona/pbm", "describe-restore", "2026-03-26T08:14:16.764090759Z", "--config", "/etc/pbm/pbm_config.yaml", "--out", "json"], "pod": "some-name-rs0-0"} 2026-03-26T08:15:11.765Z DEBUG PBM restore status {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "d836f243-ece9-497e-83ca-8ff8a60fc400", "status": {"type":"physical","opid":"","name":"2026-03-26T08:14:16.764090759Z","replsets":[{"name":"rs0","start_ts":0,"status":"running","size":0,"size_uncompressed":0,"last_transition_ts":1774512880,"first_write_ts":{"T":0,"I":0},"last_write_ts":{"T":0,"I":0},"node":"","conditions":null}],"compression":"","store":{"type":""},"size":0,"size_uncompressed":0,"mongodb_version":"","fcv":"","start_ts":0,"last_transition_ts":1774512886,"first_write_ts":{"T":0,"I":0},"last_write_ts":{"T":0,"I":0},"hb":{"T":0,"I":0},"status":"running","conditions":null,"n":null,"pbm_version":"","balancer":""}} 2026-03-26T08:15:16.766Z DEBUG checking if restore is allowed {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "7e876468-08cf-4b4e-b1bb-46eee7e32437", "cluster": "some-name", "namespace": "demand-backup-incremental-gcp-s3-9166"} 2026-03-26T08:15:16.767Z DEBUG Pod is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "7e876468-08cf-4b4e-b1bb-46eee7e32437", "pod": "some-name-rs0-0"} 2026-03-26T08:15:16.767Z DEBUG Pod is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "7e876468-08cf-4b4e-b1bb-46eee7e32437", "pod": "some-name-rs0-1"} 2026-03-26T08:15:16.767Z DEBUG Pod is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "7e876468-08cf-4b4e-b1bb-46eee7e32437", "pod": "some-name-rs0-2"} 2026-03-26T08:15:16.767Z DEBUG Statefulset is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "7e876468-08cf-4b4e-b1bb-46eee7e32437", "sts": "some-name-rs0", "replset": "rs0"} 2026-03-26T08:15:16.767Z DEBUG Check restore status {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "7e876468-08cf-4b4e-b1bb-46eee7e32437", "command": ["/opt/percona/pbm", "describe-restore", "2026-03-26T08:14:16.764090759Z", "--config", "/etc/pbm/pbm_config.yaml", "--out", "json"], "pod": "some-name-rs0-0"} 2026-03-26T08:15:18.481Z DEBUG PBM restore status {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "7e876468-08cf-4b4e-b1bb-46eee7e32437", "status": {"type":"physical","opid":"","name":"2026-03-26T08:14:16.764090759Z","replsets":[{"name":"rs0","start_ts":0,"status":"running","size":0,"size_uncompressed":0,"last_transition_ts":1774512880,"first_write_ts":{"T":0,"I":0},"last_write_ts":{"T":0,"I":0},"node":"","conditions":null}],"compression":"","store":{"type":""},"size":0,"size_uncompressed":0,"mongodb_version":"","fcv":"","start_ts":0,"last_transition_ts":1774512886,"first_write_ts":{"T":0,"I":0},"last_write_ts":{"T":0,"I":0},"hb":{"T":0,"I":0},"status":"running","conditions":null,"n":null,"pbm_version":"","balancer":""}} 2026-03-26T08:15:23.482Z DEBUG checking if restore is allowed {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "add92e89-2444-4c62-b0e7-38ca3d286255", "cluster": "some-name", "namespace": "demand-backup-incremental-gcp-s3-9166"} 2026-03-26T08:15:23.483Z DEBUG Pod is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "add92e89-2444-4c62-b0e7-38ca3d286255", "pod": "some-name-rs0-1"} 2026-03-26T08:15:23.483Z DEBUG Pod is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "add92e89-2444-4c62-b0e7-38ca3d286255", "pod": "some-name-rs0-2"} 2026-03-26T08:15:23.483Z DEBUG Pod is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "add92e89-2444-4c62-b0e7-38ca3d286255", "pod": "some-name-rs0-0"} 2026-03-26T08:15:23.483Z DEBUG Statefulset is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "add92e89-2444-4c62-b0e7-38ca3d286255", "sts": "some-name-rs0", "replset": "rs0"} 2026-03-26T08:15:23.483Z DEBUG Check restore status {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "add92e89-2444-4c62-b0e7-38ca3d286255", "command": ["/opt/percona/pbm", "describe-restore", "2026-03-26T08:14:16.764090759Z", "--config", "/etc/pbm/pbm_config.yaml", "--out", "json"], "pod": "some-name-rs0-0"} 2026-03-26T08:15:24.863Z DEBUG PBM restore status {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "add92e89-2444-4c62-b0e7-38ca3d286255", "status": {"type":"physical","opid":"","name":"2026-03-26T08:14:16.764090759Z","replsets":[{"name":"rs0","start_ts":0,"status":"running","size":0,"size_uncompressed":0,"last_transition_ts":1774512880,"first_write_ts":{"T":0,"I":0},"last_write_ts":{"T":0,"I":0},"node":"","conditions":null}],"compression":"","store":{"type":""},"size":0,"size_uncompressed":0,"mongodb_version":"","fcv":"","start_ts":0,"last_transition_ts":1774512886,"first_write_ts":{"T":0,"I":0},"last_write_ts":{"T":0,"I":0},"hb":{"T":0,"I":0},"status":"running","conditions":null,"n":null,"pbm_version":"","balancer":""}} 2026-03-26T08:15:29.864Z DEBUG checking if restore is allowed {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "2a35b5c3-52a1-4d78-b8bb-0b0fa770cf10", "cluster": "some-name", "namespace": "demand-backup-incremental-gcp-s3-9166"} 2026-03-26T08:15:29.865Z DEBUG Pod is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "2a35b5c3-52a1-4d78-b8bb-0b0fa770cf10", "pod": "some-name-rs0-0"} 2026-03-26T08:15:29.865Z DEBUG Pod is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "2a35b5c3-52a1-4d78-b8bb-0b0fa770cf10", "pod": "some-name-rs0-1"} 2026-03-26T08:15:29.865Z DEBUG Pod is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "2a35b5c3-52a1-4d78-b8bb-0b0fa770cf10", "pod": "some-name-rs0-2"} 2026-03-26T08:15:29.865Z DEBUG Statefulset is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "2a35b5c3-52a1-4d78-b8bb-0b0fa770cf10", "sts": "some-name-rs0", "replset": "rs0"} 2026-03-26T08:15:29.865Z DEBUG Check restore status {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "2a35b5c3-52a1-4d78-b8bb-0b0fa770cf10", "command": ["/opt/percona/pbm", "describe-restore", "2026-03-26T08:14:16.764090759Z", "--config", "/etc/pbm/pbm_config.yaml", "--out", "json"], "pod": "some-name-rs0-0"} 2026-03-26T08:15:31.335Z DEBUG PBM restore status {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "2a35b5c3-52a1-4d78-b8bb-0b0fa770cf10", "status": {"type":"physical","opid":"","name":"2026-03-26T08:14:16.764090759Z","replsets":[{"name":"rs0","start_ts":0,"status":"down","size":0,"size_uncompressed":0,"last_transition_ts":1774512929,"first_write_ts":{"T":0,"I":0},"last_write_ts":{"T":0,"I":0},"node":"","conditions":null}],"compression":"","store":{"type":""},"size":0,"size_uncompressed":0,"mongodb_version":"","fcv":"","start_ts":0,"last_transition_ts":1774512886,"first_write_ts":{"T":0,"I":0},"last_write_ts":{"T":0,"I":0},"hb":{"T":0,"I":0},"status":"running","conditions":null,"n":null,"pbm_version":"","balancer":""}} 2026-03-26T08:15:36.336Z DEBUG checking if restore is allowed {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "3ad26d1f-a19c-49cf-bba7-d7b6e1d6580e", "cluster": "some-name", "namespace": "demand-backup-incremental-gcp-s3-9166"} 2026-03-26T08:15:36.337Z DEBUG Pod is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "3ad26d1f-a19c-49cf-bba7-d7b6e1d6580e", "pod": "some-name-rs0-0"} 2026-03-26T08:15:36.337Z DEBUG Pod is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "3ad26d1f-a19c-49cf-bba7-d7b6e1d6580e", "pod": "some-name-rs0-1"} 2026-03-26T08:15:36.337Z DEBUG Pod is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "3ad26d1f-a19c-49cf-bba7-d7b6e1d6580e", "pod": "some-name-rs0-2"} 2026-03-26T08:15:36.337Z DEBUG Statefulset is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "3ad26d1f-a19c-49cf-bba7-d7b6e1d6580e", "sts": "some-name-rs0", "replset": "rs0"} 2026-03-26T08:15:36.337Z DEBUG Check restore status {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "3ad26d1f-a19c-49cf-bba7-d7b6e1d6580e", "command": ["/opt/percona/pbm", "describe-restore", "2026-03-26T08:14:16.764090759Z", "--config", "/etc/pbm/pbm_config.yaml", "--out", "json"], "pod": "some-name-rs0-0"} 2026-03-26T08:15:37.818Z DEBUG PBM restore status {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "3ad26d1f-a19c-49cf-bba7-d7b6e1d6580e", "status": {"type":"physical","opid":"","name":"2026-03-26T08:14:16.764090759Z","replsets":[{"name":"rs0","start_ts":0,"status":"down","size":0,"size_uncompressed":0,"last_transition_ts":1774512929,"first_write_ts":{"T":0,"I":0},"last_write_ts":{"T":0,"I":0},"node":"","conditions":null}],"compression":"","store":{"type":""},"size":0,"size_uncompressed":0,"mongodb_version":"","fcv":"","start_ts":0,"last_transition_ts":1774512886,"first_write_ts":{"T":0,"I":0},"last_write_ts":{"T":0,"I":0},"hb":{"T":0,"I":0},"status":"running","conditions":null,"n":null,"pbm_version":"","balancer":""}} 2026-03-26T08:15:42.819Z DEBUG checking if restore is allowed {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "ab4ad6cb-2d68-4ef4-ab12-47569a1a14ac", "cluster": "some-name", "namespace": "demand-backup-incremental-gcp-s3-9166"} 2026-03-26T08:15:42.820Z DEBUG Check restore status {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "ab4ad6cb-2d68-4ef4-ab12-47569a1a14ac", "command": ["/opt/percona/pbm", "describe-restore", "2026-03-26T08:14:16.764090759Z", "--config", "/etc/pbm/pbm_config.yaml", "--out", "json"], "pod": "some-name-rs0-0"} 2026-03-26T08:15:44.792Z DEBUG PBM restore status {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "ab4ad6cb-2d68-4ef4-ab12-47569a1a14ac", "status": {"type":"physical","opid":"","name":"2026-03-26T08:14:16.764090759Z","replsets":[{"name":"rs0","start_ts":0,"status":"down","size":0,"size_uncompressed":0,"last_transition_ts":1774512929,"first_write_ts":{"T":0,"I":0},"last_write_ts":{"T":0,"I":0},"node":"","conditions":null}],"compression":"","store":{"type":""},"size":0,"size_uncompressed":0,"mongodb_version":"","fcv":"","start_ts":0,"last_transition_ts":1774512886,"first_write_ts":{"T":0,"I":0},"last_write_ts":{"T":0,"I":0},"hb":{"T":0,"I":0},"status":"running","conditions":null,"n":null,"pbm_version":"","balancer":""}} 2026-03-26T08:15:49.793Z DEBUG checking if restore is allowed {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "a8b43023-9101-4499-9049-f75021011acf", "cluster": "some-name", "namespace": "demand-backup-incremental-gcp-s3-9166"} 2026-03-26T08:15:49.794Z DEBUG Check restore status {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "a8b43023-9101-4499-9049-f75021011acf", "command": ["/opt/percona/pbm", "describe-restore", "2026-03-26T08:14:16.764090759Z", "--config", "/etc/pbm/pbm_config.yaml", "--out", "json"], "pod": "some-name-rs0-0"} 2026-03-26T08:15:51.468Z DEBUG PBM restore status {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "a8b43023-9101-4499-9049-f75021011acf", "status": {"type":"physical","opid":"","name":"2026-03-26T08:14:16.764090759Z","replsets":[{"name":"rs0","start_ts":0,"status":"down","size":0,"size_uncompressed":0,"last_transition_ts":1774512929,"first_write_ts":{"T":0,"I":0},"last_write_ts":{"T":0,"I":0},"node":"","conditions":null}],"compression":"","store":{"type":""},"size":0,"size_uncompressed":0,"mongodb_version":"","fcv":"","start_ts":0,"last_transition_ts":1774512886,"first_write_ts":{"T":0,"I":0},"last_write_ts":{"T":0,"I":0},"hb":{"T":0,"I":0},"status":"running","conditions":null,"n":null,"pbm_version":"","balancer":""}} 2026-03-26T08:15:56.468Z DEBUG checking if restore is allowed {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "3f222fe7-3086-4c4b-abb5-f0461bbc7a19", "cluster": "some-name", "namespace": "demand-backup-incremental-gcp-s3-9166"} 2026-03-26T08:15:56.469Z DEBUG Check restore status {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "3f222fe7-3086-4c4b-abb5-f0461bbc7a19", "command": ["/opt/percona/pbm", "describe-restore", "2026-03-26T08:14:16.764090759Z", "--config", "/etc/pbm/pbm_config.yaml", "--out", "json"], "pod": "some-name-rs0-0"} 2026-03-26T08:16:21.675Z ERROR failed to make restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "3f222fe7-3086-4c4b-abb5-f0461bbc7a19", "restore": "restore-backup-gcp-cs-s3", "backup": "backup-gcp-cs-s3", "error": "describe restore stderr: Error: get restore meta: get file .pbm.restore/2026-03-26T08:14:16.764090759Z.json: list with parts for mw file stat op: fetching pbm file parts base for .pbm.restore/2026-03-26T08:14:16.764090759Z.json: get object: Get \"http://operator-testing.storage.googleapis.com/?location\": dial tcp [2607:f8b0:4001:c05::cf]:80: connect: cannot assign requested address\n stdout: : command terminated with exit code 1", "errorVerbose": "command terminated with exit code 1\ndescribe restore stderr: Error: get restore meta: get file .pbm.restore/2026-03-26T08:14:16.764090759Z.json: list with parts for mw file stat op: fetching pbm file parts base for .pbm.restore/2026-03-26T08:14:16.764090759Z.json: get object: Get \"http://operator-testing.storage.googleapis.com/?location\": dial tcp [2607:f8b0:4001:c05::cf]:80: connect: cannot assign requested address\n stdout: \ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).reconcilePhysicalRestore.func6\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/physical.go:231\nk8s.io/client-go/util/retry.OnError.func1\n\t/go/pkg/mod/k8s.io/client-go@v0.35.2/util/retry/util.go:51\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection\n\t/go/pkg/mod/k8s.io/apimachinery@v0.35.2/pkg/util/wait/wait.go:150\nk8s.io/apimachinery/pkg/util/wait.ExponentialBackoff\n\t/go/pkg/mod/k8s.io/apimachinery@v0.35.2/pkg/util/wait/backoff.go:477\nk8s.io/client-go/util/retry.OnError\n\t/go/pkg/mod/k8s.io/client-go@v0.35.2/util/retry/util.go:50\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).reconcilePhysicalRestore\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/physical.go:208\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/psmdb_restore_controller.go:260\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:222\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:479\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:313\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1693"} 2026-03-26T08:16:21.676Z INFO Restore state changed {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "3f222fe7-3086-4c4b-abb5-f0461bbc7a19", "previous": "running", "current": "error"} 2026-03-26T08:16:22.264Z INFO Warning: Reconciler returned both a result with either RequeueAfter or Requeue set and a non-nil error. RequeueAfter and Requeue will always be ignored if the error is non-nil. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "3f222fe7-3086-4c4b-abb5-f0461bbc7a19"} 2026-03-26T08:16:22.264Z ERROR Reconciler error {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"}, "namespace": "demand-backup-incremental-gcp-s3-9166", "name": "restore-backup-gcp-cs-s3", "reconcileID": "3f222fe7-3086-4c4b-abb5-f0461bbc7a19", "error": "reconcile physical restore: describe restore stderr: Error: get restore meta: get file .pbm.restore/2026-03-26T08:14:16.764090759Z.json: list with parts for mw file stat op: fetching pbm file parts base for .pbm.restore/2026-03-26T08:14:16.764090759Z.json: get object: Get \"http://operator-testing.storage.googleapis.com/?location\": dial tcp [2607:f8b0:4001:c05::cf]:80: connect: cannot assign requested address\n stdout: : command terminated with exit code 1", "errorVerbose": "command terminated with exit code 1\ndescribe restore stderr: Error: get restore meta: get file .pbm.restore/2026-03-26T08:14:16.764090759Z.json: list with parts for mw file stat op: fetching pbm file parts base for .pbm.restore/2026-03-26T08:14:16.764090759Z.json: get object: Get \"http://operator-testing.storage.googleapis.com/?location\": dial tcp [2607:f8b0:4001:c05::cf]:80: connect: cannot assign requested address\n stdout: \ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).reconcilePhysicalRestore.func6\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/physical.go:231\nk8s.io/client-go/util/retry.OnError.func1\n\t/go/pkg/mod/k8s.io/client-go@v0.35.2/util/retry/util.go:51\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection\n\t/go/pkg/mod/k8s.io/apimachinery@v0.35.2/pkg/util/wait/wait.go:150\nk8s.io/apimachinery/pkg/util/wait.ExponentialBackoff\n\t/go/pkg/mod/k8s.io/apimachinery@v0.35.2/pkg/util/wait/backoff.go:477\nk8s.io/client-go/util/retry.OnError\n\t/go/pkg/mod/k8s.io/client-go@v0.35.2/util/retry/util.go:50\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).reconcilePhysicalRestore\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/physical.go:208\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/psmdb_restore_controller.go:260\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:222\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:479\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:313\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1693\nreconcile physical restore\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/psmdb_restore_controller.go:262\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:222\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:479\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:313\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1693"} apiVersion: psmdb.percona.com/v1 kind: PerconaServerMongoDBRestore metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"psmdb.percona.com/v1","kind":"PerconaServerMongoDBRestore","metadata":{"annotations":{},"name":"restore-backup-gcp-cs-s3","namespace":"demand-backup-incremental-gcp-s3-9166"},"spec":{"backupName":"backup-gcp-cs-s3","clusterName":"some-name"}} creationTimestamp: "2026-03-26T08:12:13Z" generation: 1 name: restore-backup-gcp-cs-s3 namespace: demand-backup-incremental-gcp-s3-9166 resourceVersion: "1774512981695551016" uid: 5880bea0-58c8-4515-be5f-e4018f264694 spec: backupName: backup-gcp-cs-s3 clusterName: some-name status: error: |- describe restore stderr: Error: get restore meta: get file .pbm.restore/2026-03-26T08:14:16.764090759Z.json: list with parts for mw file stat op: fetching pbm file parts base for .pbm.restore/2026-03-26T08:14:16.764090759Z.json: get object: Get "http://operator-testing.storage.googleapis.com/?location": dial tcp [2607:f8b0:4001:c05::cf]:80: connect: cannot assign requested address stdout: : command terminated with exit code 1 pbmName: "2026-03-26T08:14:16.764090759Z" state: error Name: restore-backup-gcp-cs-s3 Namespace: demand-backup-incremental-gcp-s3-9166 Labels: Annotations: API Version: psmdb.percona.com/v1 Kind: PerconaServerMongoDBRestore Metadata: Creation Timestamp: 2026-03-26T08:12:13Z Generation: 1 Resource Version: 1774512981695551016 UID: 5880bea0-58c8-4515-be5f-e4018f264694 Spec: Backup Name: backup-gcp-cs-s3 Cluster Name: some-name Status: Error: describe restore stderr: Error: get restore meta: get file .pbm.restore/2026-03-26T08:14:16.764090759Z.json: list with parts for mw file stat op: fetching pbm file parts base for .pbm.restore/2026-03-26T08:14:16.764090759Z.json: get object: Get "http://operator-testing.storage.googleapis.com/?location": dial tcp [2607:f8b0:4001:c05::cf]:80: connect: cannot assign requested address stdout: : command terminated with exit code 1 Pbm Name: 2026-03-26T08:14:16.764090759Z State: error Events: Restore object restore-backup-gcp-cs-s3 is in error state after 2 minutes. something went wrong with operator or kubernetes cluster