Log: /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2230/e2e-tests/logs/demand-backup-fs.log Warning: version difference between client (1.35) and server (1.32) exceeds the supported minor version skew of +/-1 Warning: version difference between client (1.35) and server (1.32) exceeds the supported minor version skew of +/-1 Warning: version difference between client (1.35) and server (1.32) exceeds the supported minor version skew of +/-1 ----------------------------------------------------------------------------------- get and delete old CRDs and RBAC ----------------------------------------------------------------------------------- grep: warning: stray \ before - grep: warning: stray \ before - error: the server doesn't have a resource type "perconaservermongodbbackups" + kubectl patch perconaservermongodbbackups.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbbackups" error: the server doesn't have a resource type "perconaservermongodbrestores" + kubectl patch perconaservermongodbrestores.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbrestores" error: the server doesn't have a resource type "perconaservermongodbs" + kubectl patch perconaservermongodbs.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbs" Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found ----------------------------------------------------------------------------------- destroy chaos-mesh ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces psmdb-operator ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- create namespace psmdb-operator ----------------------------------------------------------------------------------- namespace/psmdb-operator created Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-2230-4109adf5-2-cluster12" modified. ----------------------------------------------------------------------------------- start PSMDB operator: perconalab/percona-server-mongodb-operator:PR-2230-4109adf5 ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io/perconaservermongodbbackups.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbrestores.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbs.psmdb.percona.com serverside-applied clusterrole.rbac.authorization.k8s.io/percona-server-mongodb-operator created serviceaccount/percona-server-mongodb-operator created clusterrolebinding.rbac.authorization.k8s.io/service-account-percona-server-mongodb-operator created deployment.apps/percona-server-mongodb-operator created waiting for pod/percona-server-mongodb-operator-59bd4bd74d-rwmh7 to be ready.OK Print operator info from log 2026-02-09T09:04:14.257Z INFO setup Manager starting up {"gitCommit": "4109adf5d85ff0c9b937397f53fb65e2720b87be", "gitBranch": "PR-2230-4109adf5", "buildTime": "", "goVersion": "go1.25.7", "os": "linux", "arch": "amd64"} ----------------------------------------------------------------------------------- destroy chaos-mesh ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces demand-backup-fs-3633 ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- create namespace demand-backup-fs-3633 ----------------------------------------------------------------------------------- namespace/demand-backup-fs-3633 created Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-2230-4109adf5-2-cluster12" modified. Error from server (NotFound): namespaces "storage" not found Error from server (NotFound): namespaces "storage" not found Error from server (NotFound): namespaces "storage" not found Error from server (NotFound): namespaces "storage" not found [2026-02-09T09:05:02+0000] deploying NFS server namespace/storage created serviceaccount/nfs-server created rolebinding.rbac.authorization.k8s.io/system:openshift:scc:privileged created persistentvolumeclaim/nfs-pvc created deployment.apps/nfs-server created service/nfs-service created [2026-02-09T09:05:14+0000] Waiting for nfs-server-7654469b4d-6vgdn to start Running [2026-02-09T09:05:16+0000] Waiting for nfs-server-7654469b4d-6vgdn to start Running [2026-02-09T09:05:18+0000] Waiting for nfs-server-7654469b4d-6vgdn to start Running [2026-02-09T09:05:20+0000] Waiting for nfs-server-7654469b4d-6vgdn to start Running [2026-02-09T09:05:22+0000] Waiting for nfs-server-7654469b4d-6vgdn to start Running [2026-02-09T09:05:24+0000] Waiting for nfs-server-7654469b4d-6vgdn to start Running [2026-02-09T09:05:26+0000] Waiting for nfs-server-7654469b4d-6vgdn to start Running [2026-02-09T09:05:31+0000] creating secrets and start client secret/some-users created deployment.apps/psmdb-client created [2026-02-09T09:05:33+0000] creating PSMDB cluster some-name perconaservermongodb.psmdb.percona.com/some-name created [2026-02-09T09:05:36+0000] wait for all 3 pods to start waiting for pod/some-name-rs0-0 to be ready.................OK waiting for pod/some-name-rs0-1 to be ready..............OK waiting for pod/some-name-rs0-2 to be ready...................OK Waiting for cluster readyness......... [2026-02-09T09:08:13+0000] checking if statefulset created with expected config [2026-02-09T09:08:15+0000] compare_kubectl: statefulset/some-name-rs0 OK [2026-02-09T09:08:15+0000] creating user Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://some-name-rs0-2.some-name-rs0.demand-backup-fs-3633.svc.cluster.local:27017,some-name-rs0-0.some-name-rs0.demand-backup-fs-3633.svc.cluster.local:27017,some-name-rs0-1.some-name-rs0.demand-backup-fs-3633.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb&replicaSet=rs0&ssl=false Implicit session: session { "id" : UUID("1c60459e-e1de-45e7-90ea-82a4a2f26850") } Percona Server for MongoDB server version: v8.0.17-6 WARNING: shell and server versions do not match Successfully added user: { "user" : "myApp", "roles" : [ { "db" : "myApp", "role" : "readWrite" } ] } bye [2026-02-09T09:08:19+0000] write initial data Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://some-name-rs0-0.some-name-rs0.demand-backup-fs-3633.svc.cluster.local:27017,some-name-rs0-1.some-name-rs0.demand-backup-fs-3633.svc.cluster.local:27017,some-name-rs0-2.some-name-rs0.demand-backup-fs-3633.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb&replicaSet=rs0&ssl=false Implicit session: session { "id" : UUID("ffdd4942-19b6-4cea-8097-cef624dba901") } Percona Server for MongoDB server version: v8.0.17-6 WARNING: shell and server versions do not match switched to db myApp WriteResult({ "nInserted" : 1 }) bye [2026-02-09T09:08:22+0000] running db.test.find() in myApp waiting for pbm-agent to be ready in some-name-rs0-0...2026-02-09T09:07:35.000+0000 I listening for the commands waiting for pbm-agent to be ready in some-name-rs0-1...2026-02-09T09:08:11.000+0000 I listening for the commands waiting for pbm-agent to be ready in some-name-rs0-2...2026-02-09T09:08:12.000+0000 I listening for the commands ----------------------------------------------------------------------------------- CASE 1: Logical backup and restore ----------------------------------------------------------------------------------- [2026-02-09T09:08:29+0000] running backup backup-nfs-logical perconaservermongodbbackup.psmdb.percona.com/backup-nfs-logical created waiting for backup-nfs-logical to reach ready state.......OK Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://some-name-rs0-2.some-name-rs0.demand-backup-fs-3633.svc.cluster.local:27017,some-name-rs0-0.some-name-rs0.demand-backup-fs-3633.svc.cluster.local:27017,some-name-rs0-1.some-name-rs0.demand-backup-fs-3633.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb&replicaSet=rs0&ssl=false Implicit session: session { "id" : UUID("427666d8-5a3d-4496-b4ff-4b5ba62ab132") } Percona Server for MongoDB server version: v8.0.17-6 WARNING: shell and server versions do not match switched to db myApp WriteResult({ "nInserted" : 1 }) bye [2026-02-09T09:08:46+0000] running db.test.find() in myApp [2026-02-09T09:08:49+0000] running restore restore-backup-nfs-logical perconaservermongodbrestore.psmdb.percona.com/restore-backup-nfs-logical created Waiting for the psmdb-restore/restore-backup-nfs-logical object to be created.OK Waiting psmdb-restore/restore-backup-nfs-logical to reach state "ready" .OK after 0 minutes waiting for cluster readyness.OK [2026-02-09T09:09:11+0000] running db.test.find() in myApp ----------------------------------------------------------------------------------- CASE 2: Logical backup and PiTR ----------------------------------------------------------------------------------- [2026-02-09T09:09:14+0000] running backup backup-nfs-logical-pitr perconaservermongodbbackup.psmdb.percona.com/backup-nfs-logical-pitr created waiting for backup-nfs-logical-pitr to reach ready state.......OK Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://some-name-rs0-2.some-name-rs0.demand-backup-fs-3633.svc.cluster.local:27017,some-name-rs0-0.some-name-rs0.demand-backup-fs-3633.svc.cluster.local:27017,some-name-rs0-1.some-name-rs0.demand-backup-fs-3633.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb&replicaSet=rs0&ssl=false Implicit session: session { "id" : UUID("ad6e59c5-41ed-4d1a-9b27-b286f6207309") } Percona Server for MongoDB server version: v8.0.17-6 WARNING: shell and server versions do not match switched to db myApp WriteResult({ "nInserted" : 1 }) bye [2026-02-09T09:09:31+0000] running db.test.find() in myApp [2026-02-09T09:09:39+0000] Waiting for last oplog chunk (2026-02-09 09:09:17) to be greater than last write (2026-02-09 09:09:21) [2026-02-09T09:09:51+0000] Waiting for last oplog chunk (2026-02-09 09:09:17) to be greater than last write (2026-02-09 09:09:21) [2026-02-09T09:10:03+0000] Waiting for last oplog chunk (2026-02-09 09:09:21) to be greater than last write (2026-02-09 09:09:21) [2026-02-09T09:10:14+0000] Waiting for last oplog chunk (2026-02-09 09:09:21) to be greater than last write (2026-02-09 09:09:21) [2026-02-09T09:10:26+0000] Waiting for last oplog chunk (2026-02-09 09:10:22) to be greater than last write (2026-02-09 09:09:21) [2026-02-09T09:10:38+0000] dropping test collection Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://some-name-rs0-2.some-name-rs0.demand-backup-fs-3633.svc.cluster.local:27017,some-name-rs0-0.some-name-rs0.demand-backup-fs-3633.svc.cluster.local:27017,some-name-rs0-1.some-name-rs0.demand-backup-fs-3633.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb&replicaSet=rs0&ssl=false Implicit session: session { "id" : UUID("ee738374-5c65-4cc6-9070-ef6cb079966d") } Percona Server for MongoDB server version: v8.0.17-6 WARNING: shell and server versions do not match switched to db myApp true bye [2026-02-09T09:10:40+0000] checking pitr... backup: backup-nfs-logical-pitr target: 2026-02-09 09:10:22 perconaservermongodbrestore.psmdb.percona.com/restore-backup-nfs-logical-pitr created Waiting for the psmdb-restore/restore-backup-nfs-logical-pitr object to be created.OK Waiting psmdb-restore/restore-backup-nfs-logical-pitr to reach state "ready" .OK after 0 minutes waiting for cluster readyness.OK [2026-02-09T09:11:17+0000] running db.test.find() in myApp ----------------------------------------------------------------------------------- CASE 3: Physical backup and restore ----------------------------------------------------------------------------------- [2026-02-09T09:11:19+0000] running backup backup-nfs-physical perconaservermongodbbackup.psmdb.percona.com/backup-nfs-physical created waiting for backup-nfs-physical to reach ready state.......OK Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://some-name-rs0-2.some-name-rs0.demand-backup-fs-3633.svc.cluster.local:27017,some-name-rs0-0.some-name-rs0.demand-backup-fs-3633.svc.cluster.local:27017,some-name-rs0-1.some-name-rs0.demand-backup-fs-3633.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb&replicaSet=rs0&ssl=false Implicit session: session { "id" : UUID("1e091d0d-9523-4fdd-af90-1f1e5969276d") } Percona Server for MongoDB server version: v8.0.17-6 WARNING: shell and server versions do not match switched to db myApp WriteResult({ "nInserted" : 1 }) bye [2026-02-09T09:11:36+0000] running db.test.find() in myApp [2026-02-09T09:11:38+0000] running restore restore-backup-nfs-physical perconaservermongodbrestore.psmdb.percona.com/restore-backup-nfs-physical created Waiting for the psmdb-restore/restore-backup-nfs-physical object to be created.OK Waiting psmdb-restore/restore-backup-nfs-physical to reach state "ready" ... ----------------------------------------------------------------------------------- pbm-agent logs from some-name-rs0-0 ----------------------------------------------------------------------------------- Defaulted container "mongod" out of: mongod, mongo-init (init), pbm-init (init) error: Internal error occurred: error sending request: Post "https://10.219.0.16:10250/exec/demand-backup-fs-3633/some-name-rs0-0/mongod?command=cat&command=%2Ftmp%2Fpbm-agent.log&error=1&output=1": dial tcp 10.219.0.16:10250: i/o timeout Defaulted container "mongod" out of: mongod, mongo-init (init), pbm-init (init) error: Internal error occurred: error sending request: Post "https://10.219.0.16:10250/exec/demand-backup-fs-3633/some-name-rs0-0/mongod?command=cat&command=%2Ftmp%2Fpbm-agent.log&error=1&output=1": dial tcp 10.219.0.16:10250: i/o timeout Defaulted container "mongod" out of: mongod, mongo-init (init), pbm-init (init) error: Internal error occurred: error sending request: Post "https://10.219.0.16:10250/exec/demand-backup-fs-3633/some-name-rs0-0/mongod?command=cat&command=%2Ftmp%2Fpbm-agent.log&error=1&output=1": dial tcp 10.219.0.16:10250: i/o timeout Defaulted container "mongod" out of: mongod, mongo-init (init), pbm-init (init) error: Internal error occurred: error sending request: Post "https://10.219.0.16:10250/exec/demand-backup-fs-3633/some-name-rs0-0/mongod?command=cat&command=%2Ftmp%2Fpbm-agent.log&error=1&output=1": dial tcp 10.219.0.16:10250: i/o timeout failed to get /tmp/pbm-agent.log from some-name-rs0-0 ----------------------------------------------------------------------------------- pbm-agent logs from some-name-rs0-1 ----------------------------------------------------------------------------------- 2026/02/09 09:12:50 [entrypoint] starting `pbm-agent` 2026-02-09T09:12:55.000+0000 I % _____ %%% | __ \ ###%%%%%%%%%%%%* | |__) |__ _ __ ___ ___ _ __ __ _ ### ##%% %%%% | ___/ _ \ '__/ __/ _ \| '_ \ / _` | #### ##% %%%% | | | __/ | | (_| (_) | | | | (_| | ### #### %%% |_| \___|_| \___\___/|_| |_|\__,_| ,((### ### %%% _____ _ (((( (### #### %%%% / ____| | | ((( ((# ###### | (___ __ _ _ _ __ _ __| | (((( (((# #### \___ \ / _` | | | |/ _` |/ _` | /(( ,((( *### ____) | (_| | |_| | (_| | (_| | //// ((( #### |_____/ \__, |\__,_|\__,_|\__,_| /// (((( #### | | /////////////(((((((((((((((((######## |_| Join @ squad.percona.com/mongodb ** Join Percona Squad! ** Participate in monthly SWAG raffles, get early access to new product features, invite-only ”ask me anything” sessions with database performance experts. Interested? Fill in the form at squad.percona.com/mongodb 2026-02-09T09:12:55.000+0000 I log options: log-path=/dev/stderr, log-level:D, log-json:false 2026-02-09T09:12:55.000+0000 I starting PITR routine 2026-02-09T09:12:55.000+0000 I pbm-agent: Version: 2.12.0 Platform: linux/amd64 GitCommit: 2f7100a2a1da2bb7074e15d757693999cac7574e GitBranch: release-2.12.0 BuildTime: 2025-10-31_09:21_UTC GoVersion: go1.25.3 2026-02-09T09:12:55.000+0000 I node: rs0/some-name-rs0-1.some-name-rs0.demand-backup-fs-3633.svc.cluster.local:27017 2026-02-09T09:12:55.000+0000 I conn level ReadConcern: majority; WriteConcern: majority 2026-02-09T09:12:55.000+0000 I listening for the commands 2026-02-09T09:13:44.000+0000 I got command restore [name: 2026-02-09T09:13:43.960765373Z, snapshot: 2026-02-09T09:11:21Z] , opid: 6989a547a31321bde45ac460 2026-02-09T09:13:44.000+0000 I got epoch {1770628301 1} 2026-02-09T09:13:44.000+0000 I [restore/2026-02-09T09:13:43.960765373Z] oplog slicer disabled 2026-02-09T09:13:44.000+0000 I [restore/2026-02-09T09:13:43.960765373Z] backup: 2026-02-09T09:11:21Z 2026-02-09T09:13:44.000+0000 I [restore/2026-02-09T09:13:43.960765373Z] recovery started 2026-02-09T09:13:44.000+0000 D [restore/2026-02-09T09:13:43.960765373Z] port: 27816 2026-02-09T09:13:45.000+0000 D [restore/2026-02-09T09:13:43.960765373Z] mongod binary: mongod, version: v8.0.17-6 2026-02-09T09:13:45.000+0000 D [restore/2026-02-09T09:13:43.960765373Z] restore opts: fallbackEnabled: false; allowPartlyDone: true 2026-02-09T09:13:45.000+0000 I [restore/2026-02-09T09:13:43.960765373Z] moving to state starting 2026-02-09T09:13:45.000+0000 I [restore/2026-02-09T09:13:43.960765373Z] waiting for `starting` status in rs map[.pbm.restore/2026-02-09T09:13:43.960765373Z/rs.rs0/node.some-name-rs0-0.some-name-rs0.demand-backup-fs-3633.svc.cluster.local:27017:{} .pbm.restore/2026-02-09T09:13:43.960765373Z/rs.rs0/node.some-name-rs0-1.some-name-rs0.demand-backup-fs-3633.svc.cluster.local:27017:{} .pbm.restore/2026-02-09T09:13:43.960765373Z/rs.rs0/node.some-name-rs0-2.some-name-rs0.demand-backup-fs-3633.svc.cluster.local:27017:{}] 2026-02-09T09:13:50.000+0000 I [restore/2026-02-09T09:13:43.960765373Z] waiting for shards map[.pbm.restore/2026-02-09T09:13:43.960765373Z/rs.rs0/rs:{}] 2026-02-09T09:13:55.000+0000 I [restore/2026-02-09T09:13:43.960765373Z] waiting for cluster 2026-02-09T09:14:00.000+0000 D [restore/2026-02-09T09:13:43.960765373Z] converged to state starting 2026-02-09T09:14:00.000+0000 D [restore/2026-02-09T09:13:43.960765373Z] starting 2026-02-09T09:14:00.000+0000 I [restore/2026-02-09T09:13:43.960765373Z] moving to state running 2026-02-09T09:14:00.000+0000 I [restore/2026-02-09T09:13:43.960765373Z] waiting for `running` status in rs map[.pbm.restore/2026-02-09T09:13:43.960765373Z/rs.rs0/node.some-name-rs0-0.some-name-rs0.demand-backup-fs-3633.svc.cluster.local:27017:{} .pbm.restore/2026-02-09T09:13:43.960765373Z/rs.rs0/node.some-name-rs0-1.some-name-rs0.demand-backup-fs-3633.svc.cluster.local:27017:{} .pbm.restore/2026-02-09T09:13:43.960765373Z/rs.rs0/node.some-name-rs0-2.some-name-rs0.demand-backup-fs-3633.svc.cluster.local:27017:{}] Defaulted container "mongod" out of: mongod, mongo-init (init), pbm-init (init) ----------------------------------------------------------------------------------- pbm-agent logs from some-name-rs0-2 ----------------------------------------------------------------------------------- 2026/02/09 09:12:05 [entrypoint] starting `pbm-agent` 2026-02-09T09:12:11.000+0000 I % _____ %%% | __ \ ###%%%%%%%%%%%%* | |__) |__ _ __ ___ ___ _ __ __ _ ### ##%% %%%% | ___/ _ \ '__/ __/ _ \| '_ \ / _` | #### ##% %%%% | | | __/ | | (_| (_) | | | | (_| | ### #### %%% |_| \___|_| \___\___/|_| |_|\__,_| ,((### ### %%% _____ _ (((( (### #### %%%% / ____| | | ((( ((# ###### | (___ __ _ _ _ __ _ __| | (((( (((# #### \___ \ / _` | | | |/ _` |/ _` | /(( ,((( *### ____) | (_| | |_| | (_| | (_| | //// ((( #### |_____/ \__, |\__,_|\__,_|\__,_| /// (((( #### | | /////////////(((((((((((((((((######## |_| Join @ squad.percona.com/mongodb ** Join Percona Squad! ** Participate in monthly SWAG raffles, get early access to new product features, invite-only ”ask me anything” sessions with database performance experts. Interested? Fill in the form at squad.percona.com/mongodb 2026-02-09T09:12:11.000+0000 I log options: log-path=/dev/stderr, log-level:D, log-json:false 2026-02-09T09:12:11.000+0000 I pbm-agent: Version: 2.12.0 Platform: linux/amd64 GitCommit: 2f7100a2a1da2bb7074e15d757693999cac7574e GitBranch: release-2.12.0 BuildTime: 2025-10-31_09:21_UTC GoVersion: go1.25.3 2026-02-09T09:12:11.000+0000 I starting PITR routine 2026-02-09T09:12:11.000+0000 I node: rs0/some-name-rs0-2.some-name-rs0.demand-backup-fs-3633.svc.cluster.local:27017 2026-02-09T09:12:11.000+0000 I conn level ReadConcern: majority; WriteConcern: majority 2026-02-09T09:12:11.000+0000 I listening for the commands 2026-02-09T09:13:44.000+0000 I got command restore [name: 2026-02-09T09:13:43.960765373Z, snapshot: 2026-02-09T09:11:21Z] , opid: 6989a547a31321bde45ac460 2026-02-09T09:13:44.000+0000 I got epoch {1770628301 1} 2026-02-09T09:13:44.000+0000 I [restore/2026-02-09T09:13:43.960765373Z] backup: 2026-02-09T09:11:21Z 2026-02-09T09:13:44.000+0000 I [restore/2026-02-09T09:13:43.960765373Z] recovery started 2026-02-09T09:13:44.000+0000 D [restore/2026-02-09T09:13:43.960765373Z] port: 28016 2026-02-09T09:13:44.000+0000 D [restore/2026-02-09T09:13:43.960765373Z] mongod binary: mongod, version: v8.0.17-6 2026-02-09T09:13:44.000+0000 D [restore/2026-02-09T09:13:43.960765373Z] restore opts: fallbackEnabled: false; allowPartlyDone: true 2026-02-09T09:13:44.000+0000 I [restore/2026-02-09T09:13:43.960765373Z] moving to state starting 2026-02-09T09:13:44.000+0000 I [restore/2026-02-09T09:13:43.960765373Z] waiting for cluster Defaulted container "mongod" out of: mongod, mongo-init (init), pbm-init (init) Error from server: Get "https://10.219.0.18:10250/containerLogs/psmdb-operator/percona-server-mongodb-operator-59bd4bd74d-d5w6h/percona-server-mongodb-operator": dial timeout, backstop 2026-02-09T09:14:30.794Z DEBUG checking if restore is allowed {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-nfs-physical","namespace":"demand-backup-fs-3633"}, "namespace": "demand-backup-fs-3633", "name": "restore-backup-nfs-physical", "reconcileID": "3ed67c93-4159-4a08-a806-cfd6a49e70e2", "cluster": "some-name", "namespace": "demand-backup-fs-3633"} 2026-02-09T09:14:30.895Z DEBUG Pod is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-nfs-physical","namespace":"demand-backup-fs-3633"}, "namespace": "demand-backup-fs-3633", "name": "restore-backup-nfs-physical", "reconcileID": "3ed67c93-4159-4a08-a806-cfd6a49e70e2", "pod": "some-name-rs0-2"} 2026-02-09T09:14:30.895Z DEBUG Pod is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-nfs-physical","namespace":"demand-backup-fs-3633"}, "namespace": "demand-backup-fs-3633", "name": "restore-backup-nfs-physical", "reconcileID": "3ed67c93-4159-4a08-a806-cfd6a49e70e2", "pod": "some-name-rs0-0"} 2026-02-09T09:14:30.895Z DEBUG Pod is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-nfs-physical","namespace":"demand-backup-fs-3633"}, "namespace": "demand-backup-fs-3633", "name": "restore-backup-nfs-physical", "reconcileID": "3ed67c93-4159-4a08-a806-cfd6a49e70e2", "pod": "some-name-rs0-1"} 2026-02-09T09:14:30.895Z DEBUG Statefulset is ready for physical restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-nfs-physical","namespace":"demand-backup-fs-3633"}, "namespace": "demand-backup-fs-3633", "name": "restore-backup-nfs-physical", "reconcileID": "3ed67c93-4159-4a08-a806-cfd6a49e70e2", "sts": "some-name-rs0", "replset": "rs0"} 2026-02-09T09:14:30.895Z DEBUG Check restore status {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-nfs-physical","namespace":"demand-backup-fs-3633"}, "namespace": "demand-backup-fs-3633", "name": "restore-backup-nfs-physical", "reconcileID": "3ed67c93-4159-4a08-a806-cfd6a49e70e2", "command": ["/opt/percona/pbm", "describe-restore", "2026-02-09T09:13:43.960765373Z", "--config", "/etc/pbm/pbm_config.yaml", "--out", "json"], "pod": "some-name-rs0-0"} 2026-02-09T09:14:30.963Z ERROR failed to make restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-nfs-physical","namespace":"demand-backup-fs-3633"}, "namespace": "demand-backup-fs-3633", "name": "restore-backup-nfs-physical", "reconcileID": "3ed67c93-4159-4a08-a806-cfd6a49e70e2", "restore": "restore-backup-nfs-physical", "backup": "backup-nfs-physical", "error": "describe restore stderr: stdout: : Internal error occurred: error executing command in container: failed to exec in container: failed to start exec \"07091df695f70e670f37428a7ca22cea6853de30a146fffb81b4a968db1870d3\": OCI runtime exec failed: exec failed: unable to start container process: error executing setns process: exit status 1: unknown", "errorVerbose": "Internal error occurred: error executing command in container: failed to exec in container: failed to start exec \"07091df695f70e670f37428a7ca22cea6853de30a146fffb81b4a968db1870d3\": OCI runtime exec failed: exec failed: unable to start container process: error executing setns process: exit status 1: unknown\ndescribe restore stderr: stdout: \ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).reconcilePhysicalRestore.func6\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/physical.go:232\nk8s.io/client-go/util/retry.OnError.func1\n\t/go/pkg/mod/k8s.io/client-go@v0.35.0/util/retry/util.go:51\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection\n\t/go/pkg/mod/k8s.io/apimachinery@v0.35.0/pkg/util/wait/wait.go:150\nk8s.io/apimachinery/pkg/util/wait.ExponentialBackoff\n\t/go/pkg/mod/k8s.io/apimachinery@v0.35.0/pkg/util/wait/backoff.go:477\nk8s.io/client-go/util/retry.OnError\n\t/go/pkg/mod/k8s.io/client-go@v0.35.0/util/retry/util.go:50\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).reconcilePhysicalRestore\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/physical.go:209\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/psmdb_restore_controller.go:260\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1693"} 2026-02-09T09:14:30.963Z INFO Restore state changed {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-nfs-physical","namespace":"demand-backup-fs-3633"}, "namespace": "demand-backup-fs-3633", "name": "restore-backup-nfs-physical", "reconcileID": "3ed67c93-4159-4a08-a806-cfd6a49e70e2", "previous": "requested", "current": "error"} 2026-02-09T09:14:31.592Z INFO Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-nfs-physical","namespace":"demand-backup-fs-3633"}, "namespace": "demand-backup-fs-3633", "name": "restore-backup-nfs-physical", "reconcileID": "3ed67c93-4159-4a08-a806-cfd6a49e70e2"} 2026-02-09T09:14:31.592Z ERROR Reconciler error {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-nfs-physical","namespace":"demand-backup-fs-3633"}, "namespace": "demand-backup-fs-3633", "name": "restore-backup-nfs-physical", "reconcileID": "3ed67c93-4159-4a08-a806-cfd6a49e70e2", "error": "reconcile physical restore: describe restore stderr: stdout: : Internal error occurred: error executing command in container: failed to exec in container: failed to start exec \"07091df695f70e670f37428a7ca22cea6853de30a146fffb81b4a968db1870d3\": OCI runtime exec failed: exec failed: unable to start container process: error executing setns process: exit status 1: unknown", "errorVerbose": "Internal error occurred: error executing command in container: failed to exec in container: failed to start exec \"07091df695f70e670f37428a7ca22cea6853de30a146fffb81b4a968db1870d3\": OCI runtime exec failed: exec failed: unable to start container process: error executing setns process: exit status 1: unknown\ndescribe restore stderr: stdout: \ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).reconcilePhysicalRestore.func6\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/physical.go:232\nk8s.io/client-go/util/retry.OnError.func1\n\t/go/pkg/mod/k8s.io/client-go@v0.35.0/util/retry/util.go:51\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection\n\t/go/pkg/mod/k8s.io/apimachinery@v0.35.0/pkg/util/wait/wait.go:150\nk8s.io/apimachinery/pkg/util/wait.ExponentialBackoff\n\t/go/pkg/mod/k8s.io/apimachinery@v0.35.0/pkg/util/wait/backoff.go:477\nk8s.io/client-go/util/retry.OnError\n\t/go/pkg/mod/k8s.io/client-go@v0.35.0/util/retry/util.go:50\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).reconcilePhysicalRestore\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/physical.go:209\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/psmdb_restore_controller.go:260\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1693\nreconcile physical restore\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/psmdb_restore_controller.go:262\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1693"} apiVersion: psmdb.percona.com/v1 kind: PerconaServerMongoDBRestore metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"psmdb.percona.com/v1","kind":"PerconaServerMongoDBRestore","metadata":{"annotations":{},"name":"restore-backup-nfs-physical","namespace":"demand-backup-fs-3633"},"spec":{"backupName":"backup-nfs-physical","clusterName":"some-name"}} creationTimestamp: "2026-02-09T09:11:41Z" generation: 1 name: restore-backup-nfs-physical namespace: demand-backup-fs-3633 resourceVersion: "1770628470980559005" uid: 88e21410-852c-4464-9e93-840da21b2bbd spec: backupName: backup-nfs-physical clusterName: some-name status: error: 'describe restore stderr: stdout: : Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "07091df695f70e670f37428a7ca22cea6853de30a146fffb81b4a968db1870d3": OCI runtime exec failed: exec failed: unable to start container process: error executing setns process: exit status 1: unknown' pbmName: "2026-02-09T09:13:43.960765373Z" state: error Name: restore-backup-nfs-physical Namespace: demand-backup-fs-3633 Labels: Annotations: API Version: psmdb.percona.com/v1 Kind: PerconaServerMongoDBRestore Metadata: Creation Timestamp: 2026-02-09T09:11:41Z Generation: 1 Resource Version: 1770628470980559005 UID: 88e21410-852c-4464-9e93-840da21b2bbd Spec: Backup Name: backup-nfs-physical Cluster Name: some-name Status: Error: describe restore stderr: stdout: : Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "07091df695f70e670f37428a7ca22cea6853de30a146fffb81b4a968db1870d3": OCI runtime exec failed: exec failed: unable to start container process: error executing setns process: exit status 1: unknown Pbm Name: 2026-02-09T09:13:43.960765373Z State: error Events: Restore object restore-backup-nfs-physical is in error state after 3 minutes. something went wrong with operator or kubernetes cluster