Log: /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2285/e2e-tests/logs/demand-backup-fs.log Warning: version difference between client (1.35) and server (1.32) exceeds the supported minor version skew of +/-1 Warning: version difference between client (1.35) and server (1.32) exceeds the supported minor version skew of +/-1 Warning: version difference between client (1.35) and server (1.32) exceeds the supported minor version skew of +/-1 ----------------------------------------------------------------------------------- get and delete old CRDs and RBAC ----------------------------------------------------------------------------------- grep: warning: stray \ before - grep: warning: stray \ before - error: the server doesn't have a resource type "perconaservermongodbbackups" + kubectl patch perconaservermongodbbackups.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbbackups" error: the server doesn't have a resource type "perconaservermongodbrestores" + kubectl patch perconaservermongodbrestores.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbrestores" error: the server doesn't have a resource type "perconaservermongodbs" + kubectl patch perconaservermongodbs.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbs" Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found ----------------------------------------------------------------------------------- destroy chaos-mesh ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces psmdb-operator ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- create namespace psmdb-operator ----------------------------------------------------------------------------------- namespace/psmdb-operator created Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-2285-ace12b60-1-cluster12" modified. ----------------------------------------------------------------------------------- start PSMDB operator: docker.io/perconalab/percona-server-mongodb-operator:PR-2285-ace12b60 ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io/perconaservermongodbbackups.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbrestores.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbs.psmdb.percona.com serverside-applied clusterrole.rbac.authorization.k8s.io/percona-server-mongodb-operator created serviceaccount/percona-server-mongodb-operator created clusterrolebinding.rbac.authorization.k8s.io/service-account-percona-server-mongodb-operator created deployment.apps/percona-server-mongodb-operator created waiting for pod/percona-server-mongodb-operator-c46d8d6c4-727lp to be ready.OK Print operator info from log 2026-03-15T16:43:49.037Z INFO setup Manager starting up {"gitCommit": "ace12b605f36447907325d565c954115594804c6", "gitBranch": "PR-2285-ace12b60", "buildTime": "", "goVersion": "go1.25.8", "os": "linux", "arch": "amd64"} ----------------------------------------------------------------------------------- destroy chaos-mesh ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces demand-backup-fs-28716 ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- create namespace demand-backup-fs-28716 ----------------------------------------------------------------------------------- namespace/demand-backup-fs-28716 created Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-2285-ace12b60-1-cluster12" modified. Error from server (NotFound): namespaces "storage" not found Error from server (NotFound): namespaces "storage" not found Error from server (NotFound): namespaces "storage" not found Error from server (NotFound): namespaces "storage" not found [2026-03-15T16:44:32+0000] deploying NFS server namespace/storage created serviceaccount/nfs-server created rolebinding.rbac.authorization.k8s.io/system:openshift:scc:privileged created persistentvolumeclaim/nfs-pvc created deployment.apps/nfs-server created service/nfs-service created [2026-03-15T16:44:44+0000] Waiting for nfs-server-7654469b4d-xn2qj to start Running [2026-03-15T16:44:46+0000] Waiting for nfs-server-7654469b4d-xn2qj to start Running [2026-03-15T16:44:48+0000] Waiting for nfs-server-7654469b4d-xn2qj to start Running [2026-03-15T16:44:50+0000] Waiting for nfs-server-7654469b4d-xn2qj to start Running [2026-03-15T16:44:52+0000] Waiting for nfs-server-7654469b4d-xn2qj to start Running [2026-03-15T16:44:54+0000] Waiting for nfs-server-7654469b4d-xn2qj to start Running [2026-03-15T16:44:56+0000] Waiting for nfs-server-7654469b4d-xn2qj to start Running [2026-03-15T16:44:58+0000] Waiting for nfs-server-7654469b4d-xn2qj to start Running [2026-03-15T16:45:03+0000] creating secrets and start client secret/some-users created deployment.apps/psmdb-client created [2026-03-15T16:45:05+0000] creating PSMDB cluster some-name perconaservermongodb.psmdb.percona.com/some-name created [2026-03-15T16:45:08+0000] wait for all 3 pods to start waiting for pod/some-name-rs0-0 to be ready........................OK waiting for pod/some-name-rs0-1 to be ready.........................OK waiting for pod/some-name-rs0-2 to be ready........................OK Waiting for cluster readyness...... [2026-03-15T16:48:09+0000] checking if statefulset created with expected config [2026-03-15T16:48:10+0000] compare_kubectl: statefulset/some-name-rs0 OK [2026-03-15T16:48:10+0000] creating user Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://some-name-rs0-1.some-name-rs0.demand-backup-fs-28716.svc.cluster.local:27017,some-name-rs0-0.some-name-rs0.demand-backup-fs-28716.svc.cluster.local:27017,some-name-rs0-2.some-name-rs0.demand-backup-fs-28716.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb&replicaSet=rs0&ssl=false Implicit session: session { "id" : UUID("d14a1c68-47c2-409f-87b8-272e12d792f0") } Percona Server for MongoDB server version: v8.0.19-7 WARNING: shell and server versions do not match Successfully added user: { "user" : "myApp", "roles" : [ { "db" : "myApp", "role" : "readWrite" } ] } bye [2026-03-15T16:48:15+0000] write initial data Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://some-name-rs0-1.some-name-rs0.demand-backup-fs-28716.svc.cluster.local:27017,some-name-rs0-0.some-name-rs0.demand-backup-fs-28716.svc.cluster.local:27017,some-name-rs0-2.some-name-rs0.demand-backup-fs-28716.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb&replicaSet=rs0&ssl=false Implicit session: session { "id" : UUID("98cf5769-036f-4a98-9f2e-4fce2eda5967") } Percona Server for MongoDB server version: v8.0.19-7 WARNING: shell and server versions do not match switched to db myApp WriteResult({ "nInserted" : 1 }) bye [2026-03-15T16:48:18+0000] running db.test.find() in myApp waiting for pbm-agent to be ready in some-name-rs0-0...2026-03-15T16:47:33.000+0000 I listening for the commands waiting for pbm-agent to be ready in some-name-rs0-1...2026-03-15T16:48:02.000+0000 I listening for the commands waiting for pbm-agent to be ready in some-name-rs0-2...2026-03-15T16:48:09.000+0000 I listening for the commands ----------------------------------------------------------------------------------- CASE 1: Logical backup and restore ----------------------------------------------------------------------------------- [2026-03-15T16:48:25+0000] running backup backup-nfs-logical perconaservermongodbbackup.psmdb.percona.com/backup-nfs-logical created waiting for backup-nfs-logical to reach ready state.......OK Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://some-name-rs0-0.some-name-rs0.demand-backup-fs-28716.svc.cluster.local:27017,some-name-rs0-2.some-name-rs0.demand-backup-fs-28716.svc.cluster.local:27017,some-name-rs0-1.some-name-rs0.demand-backup-fs-28716.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb&replicaSet=rs0&ssl=false Implicit session: session { "id" : UUID("938f522d-ec25-4afa-90b0-efb661ac5177") } Percona Server for MongoDB server version: v8.0.19-7 WARNING: shell and server versions do not match switched to db myApp WriteResult({ "nInserted" : 1 }) bye [2026-03-15T16:48:43+0000] running db.test.find() in myApp [2026-03-15T16:48:45+0000] running restore restore-backup-nfs-logical perconaservermongodbrestore.psmdb.percona.com/restore-backup-nfs-logical created Waiting for the psmdb-restore/restore-backup-nfs-logical object to be created.OK Waiting psmdb-restore/restore-backup-nfs-logical to reach state "ready" .OK after 0 minutes waiting for cluster readyness.OK [2026-03-15T16:49:08+0000] running db.test.find() in myApp ----------------------------------------------------------------------------------- CASE 2: Logical backup and PiTR ----------------------------------------------------------------------------------- [2026-03-15T16:49:10+0000] running backup backup-nfs-logical-pitr perconaservermongodbbackup.psmdb.percona.com/backup-nfs-logical-pitr created waiting for backup-nfs-logical-pitr to reach ready state.......OK Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://some-name-rs0-1.some-name-rs0.demand-backup-fs-28716.svc.cluster.local:27017,some-name-rs0-0.some-name-rs0.demand-backup-fs-28716.svc.cluster.local:27017,some-name-rs0-2.some-name-rs0.demand-backup-fs-28716.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb&replicaSet=rs0&ssl=false Implicit session: session { "id" : UUID("7a58c526-732b-4452-a198-fb48d8da894e") } Percona Server for MongoDB server version: v8.0.19-7 WARNING: shell and server versions do not match switched to db myApp WriteResult({ "nInserted" : 1 }) bye [2026-03-15T16:49:28+0000] running db.test.find() in myApp date: invalid date ‘@null’ [2026-03-15T16:49:36+0000] Waiting for last oplog chunk () to be greater than last write (2026-03-15 16:49:19) date: invalid date ‘@null’ [2026-03-15T16:49:47+0000] Waiting for last oplog chunk () to be greater than last write (2026-03-15 16:49:19) date: invalid date ‘@null’ [2026-03-15T16:49:59+0000] Waiting for last oplog chunk () to be greater than last write (2026-03-15 16:49:19) date: invalid date ‘@null’ [2026-03-15T16:50:11+0000] Waiting for last oplog chunk () to be greater than last write (2026-03-15 16:49:19) date: invalid date ‘@null’ [2026-03-15T16:50:23+0000] Waiting for last oplog chunk () to be greater than last write (2026-03-15 16:49:19) date: invalid date ‘@null’ [2026-03-15T16:50:35+0000] Waiting for last oplog chunk () to be greater than last write (2026-03-15 16:49:19) date: invalid date ‘@null’ [2026-03-15T16:50:47+0000] Waiting for last oplog chunk () to be greater than last write (2026-03-15 16:49:19) [2026-03-15T16:50:59+0000] Waiting for last oplog chunk (2026-03-15 16:50:50) to be greater than last write (2026-03-15 16:49:19) [2026-03-15T16:51:11+0000] dropping test collection Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://some-name-rs0-1.some-name-rs0.demand-backup-fs-28716.svc.cluster.local:27017,some-name-rs0-0.some-name-rs0.demand-backup-fs-28716.svc.cluster.local:27017,some-name-rs0-2.some-name-rs0.demand-backup-fs-28716.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb&replicaSet=rs0&ssl=false Implicit session: session { "id" : UUID("efbc0c92-dce2-4530-a0a8-130795a40e72") } Percona Server for MongoDB server version: v8.0.19-7 WARNING: shell and server versions do not match switched to db myApp true bye [2026-03-15T16:51:13+0000] checking pitr... backup: backup-nfs-logical-pitr target: 2026-03-15 16:50:50 perconaservermongodbrestore.psmdb.percona.com/restore-backup-nfs-logical-pitr created Waiting for the psmdb-restore/restore-backup-nfs-logical-pitr object to be created.OK Waiting psmdb-restore/restore-backup-nfs-logical-pitr to reach state "ready" .OK after 0 minutes waiting for cluster readyness.OK [2026-03-15T16:51:45+0000] running db.test.find() in myApp ----------------------------------------------------------------------------------- CASE 3: Physical backup and restore ----------------------------------------------------------------------------------- [2026-03-15T16:51:48+0000] running backup backup-nfs-physical perconaservermongodbbackup.psmdb.percona.com/backup-nfs-physical created waiting for backup-nfs-physical to reach ready state.......OK Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://some-name-rs0-1.some-name-rs0.demand-backup-fs-28716.svc.cluster.local:27017,some-name-rs0-0.some-name-rs0.demand-backup-fs-28716.svc.cluster.local:27017,some-name-rs0-2.some-name-rs0.demand-backup-fs-28716.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb&replicaSet=rs0&ssl=false Implicit session: session { "id" : UUID("32178f81-5854-4251-998a-8b46bd99851e") } Percona Server for MongoDB server version: v8.0.19-7 WARNING: shell and server versions do not match switched to db myApp WriteResult({ "nInserted" : 1 }) bye [2026-03-15T16:52:04+0000] running db.test.find() in myApp [2026-03-15T16:52:07+0000] running restore restore-backup-nfs-physical perconaservermongodbrestore.psmdb.percona.com/restore-backup-nfs-physical created Waiting for the psmdb-restore/restore-backup-nfs-physical object to be created.OK Waiting psmdb-restore/restore-backup-nfs-physical to reach state "ready" .....OK after 4 minutes waiting for cluster readyness..........OK [2026-03-15T16:58:23+0000] running db.test.find() in myApp ----------------------------------------------------------------------------------- CASE 4: Physical backup and PiTR ----------------------------------------------------------------------------------- [2026-03-15T16:58:26+0000] running backup backup-nfs-physical-pitr perconaservermongodbbackup.psmdb.percona.com/backup-nfs-physical-pitr created waiting for backup-nfs-physical-pitr to reach ready state.......OK Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://some-name-rs0-1.some-name-rs0.demand-backup-fs-28716.svc.cluster.local:27017,some-name-rs0-0.some-name-rs0.demand-backup-fs-28716.svc.cluster.local:27017,some-name-rs0-2.some-name-rs0.demand-backup-fs-28716.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb&replicaSet=rs0&ssl=false Implicit session: session { "id" : UUID("ec4a2900-abfb-4094-9cc9-380e7468a3a7") } Percona Server for MongoDB server version: v8.0.19-7 WARNING: shell and server versions do not match switched to db myApp WriteResult({ "nInserted" : 1 }) bye [2026-03-15T16:58:42+0000] running db.test.find() in myApp command terminated with exit code 137 error: cannot exec into a container in a completed pod; current phase is Failed error: cannot exec into a container in a completed pod; current phase is Failed error: cannot exec into a container in a completed pod; current phase is Failed --- /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2285/e2e-tests/demand-backup-fs/compare/find-5th.json 2026-03-15 16:25:25.573162803 +0000 +++ /tmp/tmp.W1AGEEzB3c/find-5th 2026-03-15 16:58:42.936812246 +0000 @@ -1,5 +0,0 @@ -switched to db myApp -{ "_id" : , "x" : 100500 } -{ "_id" : , "x" : 100502 } -{ "_id" : , "x" : 100503 } -bye