Log: /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2296/e2e-tests/logs/replset-remapping-sharded.log Warning: version difference between client (1.35) and server (1.32) exceeds the supported minor version skew of +/-1 Warning: version difference between client (1.35) and server (1.32) exceeds the supported minor version skew of +/-1 Warning: version difference between client (1.35) and server (1.32) exceeds the supported minor version skew of +/-1 ----------------------------------------------------------------------------------- get and delete old CRDs and RBAC ----------------------------------------------------------------------------------- grep: warning: stray \ before - grep: warning: stray \ before - error: the server doesn't have a resource type "perconaservermongodbbackups" + kubectl patch perconaservermongodbbackups.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbbackups" error: the server doesn't have a resource type "perconaservermongodbrestores" + kubectl patch perconaservermongodbrestores.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbrestores" error: the server doesn't have a resource type "perconaservermongodbs" + kubectl patch perconaservermongodbs.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbs" Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found ----------------------------------------------------------------------------------- destroy chaos-mesh ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces psmdb-operator ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- create namespace psmdb-operator ----------------------------------------------------------------------------------- namespace/psmdb-operator created Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-2296-796466d8-1-cluster11" modified. ----------------------------------------------------------------------------------- start PSMDB operator: docker.io/perconalab/percona-server-mongodb-operator:PR-2296-796466d8 ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io/perconaservermongodbbackups.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbrestores.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbs.psmdb.percona.com serverside-applied clusterrole.rbac.authorization.k8s.io/percona-server-mongodb-operator created serviceaccount/percona-server-mongodb-operator created clusterrolebinding.rbac.authorization.k8s.io/service-account-percona-server-mongodb-operator created deployment.apps/percona-server-mongodb-operator created waiting for pod/percona-server-mongodb-operator-5cdbcd8964-lk6pq to be ready.OK Print operator info from log 2026-03-30T02:44:52.950Z INFO setup Manager starting up {"gitCommit": "796466d835269084ebd982866588fda2e529c48d", "gitBranch": "PR-2296-796466d8", "buildTime": "", "goVersion": "go1.25.8", "os": "linux", "arch": "amd64"} ----------------------------------------------------------------------------------- destroy chaos-mesh ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces replset-remapping-sharded-1920 ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- create namespace replset-remapping-sharded-1920 ----------------------------------------------------------------------------------- namespace/replset-remapping-sharded-1920 created Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-2296-796466d8-1-cluster11" modified. deployment.apps/psmdb-client created secret/some-users created secret/some-name-ssl created secret/some-name-ssl-internal created secret/minio-secret created ----------------------------------------------------------------------------------- install MinIO: minio-service ----------------------------------------------------------------------------------- "minio" has been removed from your repositories "minio" has been added to your repositories NAME: minio-service LAST DEPLOYED: Mon Mar 30 02:45:34 2026 NAMESPACE: replset-remapping-sharded-1920 STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: MinIO can be accessed via port 9000 on the following DNS name from within your cluster: minio-service.replset-remapping-sharded-1920.cluster.local To access MinIO from localhost, run the below commands: 1. export POD_NAME=$(kubectl get pods --namespace replset-remapping-sharded-1920 -l "release=minio-service" -o jsonpath="{.items[0].metadata.name}") 2. kubectl port-forward $POD_NAME 9000 --namespace replset-remapping-sharded-1920 Read more about port forwarding here: http://kubernetes.io/docs/user-guide/kubectl/kubectl_port-forward/ You can now access MinIO server on http://localhost:9000. Follow the below steps to connect to MinIO server with mc client: 1. Download the MinIO mc client - https://min.io/docs/minio/linux/reference/minio-mc.html#quickstart 2. export MC_HOST_minio-service-local=http://$(kubectl get secret --namespace replset-remapping-sharded-1920 minio-service -o jsonpath="{.data.rootUser}" | base64 --decode):$(kubectl get secret --namespace replset-remapping-sharded-1920 minio-service -o jsonpath="{.data.rootPassword}" | base64 --decode)@localhost:9000 3. mc ls minio-service-local waiting for pod/minio-service-6d5f646cdc-z4zmb to be ready.OK service/minio-service created make_bucket: operator-testing pod "aws-cli" deleted from replset-remapping-sharded-1920 namespace ----------------------------------------------------------------------------------- setting up source cluster: some-name-source ----------------------------------------------------------------------------------- [2026-03-30T02:46:10+0000] creating PSMDB cluster: some-name-source perconaservermongodb.psmdb.percona.com/some-name-source created waiting for pod/some-name-source-cfg-0 to be ready....OK waiting for pod/some-name-source-cfg-1 to be ready.......OK waiting for pod/some-name-source-cfg-2 to be ready.....OK Waiting for cluster readyness.................................................................... waiting for pod/some-name-source-rs0-0 to be ready.OK waiting for pod/some-name-source-rs0-1 to be ready.OK waiting for pod/some-name-source-rs0-2 to be ready.OK Waiting for cluster readyness waiting for pod/some-name-source-rs1-0 to be ready.OK waiting for pod/some-name-source-rs1-1 to be ready.OK waiting for pod/some-name-source-rs1-2 to be ready.OK Waiting for cluster readyness waiting for pod/some-name-source-mongos-0 to be ready.OK waiting for pod/some-name-source-mongos-1 to be ready.OK waiting for pod/some-name-source-mongos-2 to be ready........................................................OK Waiting for cluster readyness.. waiting for cluster readyness.OK [2026-03-30T02:52:55+0000] enable sharding Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://some-name-source-mongos.replset-remapping-sharded-1920.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("f1744ebd-0d76-43ce-ad06-354be03e38cd") } Percona Server for MongoDB server version: v8.0.19-7 WARNING: shell and server versions do not match { "ok" : 1, "$clusterTime" : { "clusterTime" : Timestamp(1774839177, 10), "signature" : { "hash" : BinData(0,"wwgEUc1lCEXQ7irjMn7OChrEeyQ="), "keyId" : NumberLong("7622874812125282324") } }, "operationTime" : Timestamp(1774839177, 7) } bye [2026-03-30T02:53:00+0000] write some data Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://some-name-source-mongos.replset-remapping-sharded-1920.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("02759c42-1b74-43b5-bec8-fa1f336e53c2") } Percona Server for MongoDB server version: v8.0.19-7 WARNING: shell and server versions do not match switched to db myApp WriteResult({ "nInserted" : 1 }) bye [2026-03-30T02:53:02+0000] shard collection Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://some-name-source-mongos.replset-remapping-sharded-1920.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("24f411cc-da56-4d4f-b74f-9e0377522ed7") } Percona Server for MongoDB server version: v8.0.19-7 WARNING: shell and server versions do not match { "collectionsharded" : "myApp.test", "ok" : 1, "$clusterTime" : { "clusterTime" : Timestamp(1774839185, 41), "signature" : { "hash" : BinData(0,"TTUY81IPinsIv9njEyXh4/b61vw="), "keyId" : NumberLong("7622874812125282324") } }, "operationTime" : Timestamp(1774839185, 41) } bye [2026-03-30T02:53:06+0000] running db.test.command() in myApp [2026-03-30T02:53:08+0000] running backup backup-minio-logical perconaservermongodbbackup.psmdb.percona.com/backup-minio-logical created waiting for backup-minio-logical to reach ready state.................OK [2026-03-30T02:53:43+0000] running backup backup-minio-physical perconaservermongodbbackup.psmdb.percona.com/backup-minio-physical created waiting for backup-minio-physical to reach ready state.................OK [2026-03-30T02:54:19+0000] deleting PSMDB cluster: some-name-source perconaservermongodb.psmdb.percona.com "some-name-source" deleted from replset-remapping-sharded-1920 namespace waiting for psmdb/some-name-source to be deletedError from server (NotFound): perconaservermongodbs.psmdb.percona.com "some-name-source" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "some-name-source" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "some-name-source" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "some-name-source" not found waiting for pod/some-name-source-rs0-0 to be deletedError from server (NotFound): pods "some-name-source-rs0-0" not found Error from server (NotFound): pods "some-name-source-rs0-0" not found Error from server (NotFound): pods "some-name-source-rs0-0" not found Error from server (NotFound): pods "some-name-source-rs0-0" not found ----------------------------------------------------------------------------------- setting up target cluster: some-name-target ----------------------------------------------------------------------------------- [2026-03-30T02:54:59+0000] creating PSMDB cluster: some-name-target secret/some-users configured secret/some-name-ssl unchanged secret/some-name-ssl-internal unchanged Warning: resource secrets/some-users is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically. perconaservermongodb.psmdb.percona.com/some-name-target created waiting for pod/some-name-target-cfg-0 to be ready......OK waiting for pod/some-name-target-cfg-1 to be ready......OK waiting for pod/some-name-target-cfg-2 to be ready......OK Waiting for cluster readyness.............................................................. waiting for pod/some-name-target-replset0-0 to be ready.OK waiting for pod/some-name-target-replset0-1 to be ready.OK waiting for pod/some-name-target-replset0-2 to be ready.OK Waiting for cluster readyness waiting for pod/some-name-target-replset1-0 to be ready.OK waiting for pod/some-name-target-replset1-1 to be ready.OK waiting for pod/some-name-target-replset1-2 to be ready.OK Waiting for cluster readyness waiting for pod/some-name-target-mongos-0 to be ready.OK waiting for pod/some-name-target-mongos-1 to be ready.OK waiting for pod/some-name-target-mongos-2 to be ready.OK Waiting for cluster readyness waiting for cluster readyness.OK ----------------------------------------------------------------------------------- CASE 1: Logical restore with replset remapping ----------------------------------------------------------------------------------- [2026-03-30T02:59:37+0000] running restore restore-backup-minio-logical perconaservermongodbrestore.psmdb.percona.com/restore-backup-minio-logical created Waiting for the psmdb-restore/restore-backup-minio-logical object to be created.OK Waiting psmdb-restore/restore-backup-minio-logical to reach state "ready" .OK after 0 minutes waiting for cluster readyness......OK waiting for cluster readyness.OK [2026-03-30T03:01:23+0000] running db.test.command() in myApp ----------------------------------------------------------------------------------- CASE 1: PASSED ----------------------------------------------------------------------------------- [2026-03-30T03:01:26+0000] dropping test collection Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://some-name-target-mongos.replset-remapping-sharded-1920.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("3fdff98a-070b-4e3d-be11-9d2e33f3d96f") } Percona Server for MongoDB server version: v8.0.19-7 WARNING: shell and server versions do not match switched to db myApp true bye ----------------------------------------------------------------------------------- CASE 2: Physical restore with replset remapping ----------------------------------------------------------------------------------- [2026-03-30T03:01:29+0000] running restore restore-backup-minio-physical perconaservermongodbrestore.psmdb.percona.com/restore-backup-minio-physical created Waiting for the psmdb-restore/restore-backup-minio-physical object to be created.OK Waiting psmdb-restore/restore-backup-minio-physical to reach state "ready" .Error from server: Get "https://10.218.0.91:10250/containerLogs/psmdb-operator/percona-server-mongodb-operator-5cdbcd8964-6z4hk/percona-server-mongodb-operator": dial timeout, backstop 2026-03-30T03:01:57.902Z DEBUG checking if restore is allowed {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-minio-physical","namespace":"replset-remapping-sharded-1920"}, "namespace": "replset-remapping-sharded-1920", "name": "restore-backup-minio-physical", "reconcileID": "19cd7c77-7d84-4037-843f-93554d91de0b", "cluster": "some-name-target", "namespace": "replset-remapping-sharded-1920"} 2026-03-30T03:02:01.526Z ERROR failed to make restore {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-minio-physical","namespace":"replset-remapping-sharded-1920"}, "namespace": "replset-remapping-sharded-1920", "name": "restore-backup-minio-physical", "reconcileID": "19cd7c77-7d84-4037-843f-93554d91de0b", "restore": "restore-backup-minio-physical", "backup": "backup-minio-physical", "error": "failed to validate backup: validate backup in storage: get backup metadata from storage: get from store: list with parts for mw source reader: fetching pbm file parts base for 2026-03-30T02:53:46Z.pbm.json: get S3 object header: operation error S3: HeadObject, exceeded maximum number of attempts, 3, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head \"http://minio-service:9000/operator-testing/2026-03-30T02%3A53%3A46Z.pbm.json\": dial tcp 34.118.230.58:9000: connect: connection refused", "errorVerbose": "operation error S3: HeadObject, exceeded maximum number of attempts, 3, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head \"http://minio-service:9000/operator-testing/2026-03-30T02%3A53%3A46Z.pbm.json\": dial tcp 34.118.230.58:9000: connect: connection refused\nget S3 object header\nfetching pbm file parts base for 2026-03-30T02:53:46Z.pbm.json\nlist with parts for mw source reader\nget from store\nget backup metadata from storage\ngithub.com/percona/percona-server-mongodb-operator/pkg/psmdb/backup.(*pbmC).ValidateBackupInStorage\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/psmdb/backup/pbm.go:792\ngithub.com/percona/percona-server-mongodb-operator/pkg/psmdb/backup.(*pbmC).ValidateBackup\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/psmdb/backup/pbm.go:762\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).validate\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/validate.go:78\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/psmdb_restore_controller.go:191\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:222\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:479\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:313\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1693\nvalidate backup in storage\ngithub.com/percona/percona-server-mongodb-operator/pkg/psmdb/backup.(*pbmC).ValidateBackup\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/psmdb/backup/pbm.go:763\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).validate\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/validate.go:78\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/psmdb_restore_controller.go:191\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:222\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:479\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:313\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1693\nfailed to validate backup\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).validate\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/validate.go:79\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/psmdb_restore_controller.go:191\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:222\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:479\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:313\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1693"} 2026-03-30T03:02:01.526Z INFO Restore state changed {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-minio-physical","namespace":"replset-remapping-sharded-1920"}, "namespace": "replset-remapping-sharded-1920", "name": "restore-backup-minio-physical", "reconcileID": "19cd7c77-7d84-4037-843f-93554d91de0b", "previous": "", "current": "error"} 2026-03-30T03:02:02.110Z ERROR Reconciler error {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"restore-backup-minio-physical","namespace":"replset-remapping-sharded-1920"}, "namespace": "replset-remapping-sharded-1920", "name": "restore-backup-minio-physical", "reconcileID": "19cd7c77-7d84-4037-843f-93554d91de0b", "error": "failed to validate restore: failed to validate backup: validate backup in storage: get backup metadata from storage: get from store: list with parts for mw source reader: fetching pbm file parts base for 2026-03-30T02:53:46Z.pbm.json: get S3 object header: operation error S3: HeadObject, exceeded maximum number of attempts, 3, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head \"http://minio-service:9000/operator-testing/2026-03-30T02%3A53%3A46Z.pbm.json\": dial tcp 34.118.230.58:9000: connect: connection refused", "errorVerbose": "operation error S3: HeadObject, exceeded maximum number of attempts, 3, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head \"http://minio-service:9000/operator-testing/2026-03-30T02%3A53%3A46Z.pbm.json\": dial tcp 34.118.230.58:9000: connect: connection refused\nget S3 object header\nfetching pbm file parts base for 2026-03-30T02:53:46Z.pbm.json\nlist with parts for mw source reader\nget from store\nget backup metadata from storage\ngithub.com/percona/percona-server-mongodb-operator/pkg/psmdb/backup.(*pbmC).ValidateBackupInStorage\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/psmdb/backup/pbm.go:792\ngithub.com/percona/percona-server-mongodb-operator/pkg/psmdb/backup.(*pbmC).ValidateBackup\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/psmdb/backup/pbm.go:762\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).validate\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/validate.go:78\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/psmdb_restore_controller.go:191\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:222\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:479\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:313\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1693\nvalidate backup in storage\ngithub.com/percona/percona-server-mongodb-operator/pkg/psmdb/backup.(*pbmC).ValidateBackup\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/psmdb/backup/pbm.go:763\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).validate\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/validate.go:78\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/psmdb_restore_controller.go:191\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:222\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:479\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:313\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1693\nfailed to validate backup\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).validate\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/validate.go:79\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/psmdb_restore_controller.go:191\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:222\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:479\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:313\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1693\nfailed to validate restore\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/psmdb_restore_controller.go:222\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:222\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:479\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:313\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1693"} apiVersion: psmdb.percona.com/v1 kind: PerconaServerMongoDBRestore metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"psmdb.percona.com/v1","kind":"PerconaServerMongoDBRestore","metadata":{"annotations":{},"name":"restore-backup-minio-physical","namespace":"replset-remapping-sharded-1920"},"spec":{"backupName":"backup-minio-physical","clusterName":"some-name-target","replsetRemapping":{"cfg":"csReplSet","rs0":"replset0","rs1":"replset1"}}} creationTimestamp: "2026-03-30T03:01:32Z" generation: 1 name: restore-backup-minio-physical namespace: replset-remapping-sharded-1920 resourceVersion: "1774839721543999016" uid: afb3a23d-c097-4108-8978-fded6f94f4fd spec: backupName: backup-minio-physical clusterName: some-name-target replsetRemapping: cfg: csReplSet rs0: replset0 rs1: replset1 status: error: 'failed to validate backup: validate backup in storage: get backup metadata from storage: get from store: list with parts for mw source reader: fetching pbm file parts base for 2026-03-30T02:53:46Z.pbm.json: get S3 object header: operation error S3: HeadObject, exceeded maximum number of attempts, 3, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "http://minio-service:9000/operator-testing/2026-03-30T02%3A53%3A46Z.pbm.json": dial tcp 34.118.230.58:9000: connect: connection refused' state: error Name: restore-backup-minio-physical Namespace: replset-remapping-sharded-1920 Labels: Annotations: API Version: psmdb.percona.com/v1 Kind: PerconaServerMongoDBRestore Metadata: Creation Timestamp: 2026-03-30T03:01:32Z Generation: 1 Resource Version: 1774839721543999016 UID: afb3a23d-c097-4108-8978-fded6f94f4fd Spec: Backup Name: backup-minio-physical Cluster Name: some-name-target Replset Remapping: Cfg: csReplSet rs0: replset0 rs1: replset1 Status: Error: failed to validate backup: validate backup in storage: get backup metadata from storage: get from store: list with parts for mw source reader: fetching pbm file parts base for 2026-03-30T02:53:46Z.pbm.json: get S3 object header: operation error S3: HeadObject, exceeded maximum number of attempts, 3, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "http://minio-service:9000/operator-testing/2026-03-30T02%3A53%3A46Z.pbm.json": dial tcp 34.118.230.58:9000: connect: connection refused State: error Events: Restore object restore-backup-minio-physical is in error state after 1 minutes. something went wrong with operator or kubernetes cluster