Log: /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2277/e2e-tests/logs/demand-backup-if-unhealthy.log Warning: version difference between client (1.35) and server (1.32) exceeds the supported minor version skew of +/-1 Warning: version difference between client (1.35) and server (1.32) exceeds the supported minor version skew of +/-1 Warning: version difference between client (1.35) and server (1.32) exceeds the supported minor version skew of +/-1 ----------------------------------------------------------------------------------- get and delete old CRDs and RBAC ----------------------------------------------------------------------------------- grep: warning: stray \ before - grep: warning: stray \ before - error: the server doesn't have a resource type "perconaservermongodbbackups" + kubectl patch perconaservermongodbbackups.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbbackups" error: the server doesn't have a resource type "perconaservermongodbrestores" + kubectl patch perconaservermongodbrestores.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbrestores" error: the server doesn't have a resource type "perconaservermongodbs" + kubectl patch perconaservermongodbs.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbs" Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found ----------------------------------------------------------------------------------- destroy chaos-mesh ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces psmdb-operator ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- create namespace psmdb-operator ----------------------------------------------------------------------------------- namespace/psmdb-operator created Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-2277-0a1aa642f-2-cluster13" modified. ----------------------------------------------------------------------------------- start PSMDB operator: docker.io/perconalab/percona-server-mongodb-operator:PR-2277-0a1aa642f ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io/perconaservermongodbbackups.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbrestores.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbs.psmdb.percona.com serverside-applied clusterrole.rbac.authorization.k8s.io/percona-server-mongodb-operator created serviceaccount/percona-server-mongodb-operator created clusterrolebinding.rbac.authorization.k8s.io/service-account-percona-server-mongodb-operator created deployment.apps/percona-server-mongodb-operator created waiting for pod/percona-server-mongodb-operator-5b5f6fcd94-hvm5q to be ready.OK Print operator info from log 2026-03-09T08:58:44.446Z INFO setup Manager starting up {"gitCommit": "0a1aa642fab677b3a3eb9b1dee5f81fc42e34249", "gitBranch": "PR-2277-0a1aa642f", "buildTime": "", "goVersion": "go1.25.8", "os": "linux", "arch": "amd64"} ----------------------------------------------------------------------------------- destroy chaos-mesh ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces demand-backup-if-unhealthy-21026 ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- create namespace demand-backup-if-unhealthy-21026 ----------------------------------------------------------------------------------- namespace/demand-backup-if-unhealthy-21026 created Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-2277-0a1aa642f-2-cluster13" modified. ----------------------------------------------------------------------------------- install MinIO: minio-service ----------------------------------------------------------------------------------- "minio" has been removed from your repositories "minio" has been added to your repositories NAME: minio-service LAST DEPLOYED: Mon Mar 9 08:59:17 2026 NAMESPACE: demand-backup-if-unhealthy-21026 STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: MinIO can be accessed via port 9000 on the following DNS name from within your cluster: minio-service.demand-backup-if-unhealthy-21026.cluster.local To access MinIO from localhost, run the below commands: 1. export POD_NAME=$(kubectl get pods --namespace demand-backup-if-unhealthy-21026 -l "release=minio-service" -o jsonpath="{.items[0].metadata.name}") 2. kubectl port-forward $POD_NAME 9000 --namespace demand-backup-if-unhealthy-21026 Read more about port forwarding here: http://kubernetes.io/docs/user-guide/kubectl/kubectl_port-forward/ You can now access MinIO server on http://localhost:9000. Follow the below steps to connect to MinIO server with mc client: 1. Download the MinIO mc client - https://min.io/docs/minio/linux/reference/minio-mc.html#quickstart 2. export MC_HOST_minio-service-local=http://$(kubectl get secret --namespace demand-backup-if-unhealthy-21026 minio-service -o jsonpath="{.data.rootUser}" | base64 --decode):$(kubectl get secret --namespace demand-backup-if-unhealthy-21026 minio-service -o jsonpath="{.data.rootPassword}" | base64 --decode)@localhost:9000 3. mc ls minio-service-local waiting for pod/minio-service-6d5f646cdc-nxfk5 to be ready.OK service/minio-service created pod "aws-cli" deleted from demand-backup-if-unhealthy-21026 namespace All commands and output from this session will be recorded in container logs, including credentials and sensitive information passed through the command prompt. If you don't see a command prompt, try pressing enter. Creating secrets and start client secret/some-users created deployment.apps/psmdb-client created ----------------------------------------------------------------------------------- create secrets for cloud storages ----------------------------------------------------------------------------------- secret/minio-secret created secret/aws-s3-secret created secret/gcp-cs-secret created secret/azure-secret created secret/gcp-cs-sa-key-secret created ----------------------------------------------------------------------------------- create PSMDB cluster some-name ----------------------------------------------------------------------------------- perconaservermongodb.psmdb.percona.com/some-name created Waiting for psmdb/some-name to reach ready state.................................................................................................................................................................... [2026-03-09T09:05:33+0000] psmdb/some-name is ready: OK ----------------------------------------------------------------------------------- Adding new user ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://some-name-rs0-2.some-name-rs0.demand-backup-if-unhealthy-21026.svc.cluster.local:27017,some-name-rs0-1.some-name-rs0.demand-backup-if-unhealthy-21026.svc.cluster.local:27017,some-name-rs0-0.some-name-rs0.demand-backup-if-unhealthy-21026.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb&replicaSet=rs0&ssl=false Implicit session: session { "id" : UUID("dc40e4f3-f1bc-432b-8f8d-603f2da9cc27") } Percona Server for MongoDB server version: v8.0.19-7 WARNING: shell and server versions do not match Successfully added user: { "user" : "myApp", "roles" : [ { "db" : "myApp", "role" : "readWrite" } ] } bye Verifying user creation... ----------------------------------------------------------------------------------- Testing logical backup with an unhealthy container ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://some-name-rs0-1.some-name-rs0.demand-backup-if-unhealthy-21026.svc.cluster.local:27017,some-name-rs0-0.some-name-rs0.demand-backup-if-unhealthy-21026.svc.cluster.local:27017,some-name-rs0-2.some-name-rs0.demand-backup-if-unhealthy-21026.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb&replicaSet=rs0&ssl=false Implicit session: session { "id" : UUID("34eaa18d-b9fe-466c-a065-dccb65a60828") } Percona Server for MongoDB server version: v8.0.19-7 WARNING: shell and server versions do not match switched to db myApp WriteResult({ "nInserted" : 1 }) bye [2026-03-09T09:05:42+0000] running db.test.find() in myApp Updating cluster with invalid image... perconaservermongodb.psmdb.percona.com/some-name configured Wait for statefulset some-name-rs0 to become not ready........Waiting for psmdb/some-name to reach initializing|error state [2026-03-09T09:06:20+0000] psmdb/some-name is initializing|error: OK [2026-03-09T09:06:20+0000] running backup backup-minio-logical perconaservermongodbbackup.psmdb.percona.com/backup-minio-logical created Waiting for psmdb/some-name to reach initializing|error state [2026-03-09T09:06:26+0000] psmdb/some-name is initializing|error: OK ----------------------------------------------------------------------------------- Verifying backup-minio-logical and update cluster back to ready state ----------------------------------------------------------------------------------- waiting for backup-minio-logical to reach ready state..............2026-03-09T09:06:03.334Z INFO setup Manager starting up {"gitCommit": "0a1aa642fab677b3a3eb9b1dee5f81fc42e34249", "gitBranch": "PR-2277-0a1aa642f", "buildTime": "", "goVersion": "go1.25.8", "os": "linux", "arch": "amd64"} 2026-03-09T09:06:03.367Z INFO setup Multi-cluster services (MCS) are not available: ServiceExport resource not found in cluster 2026-03-09T09:06:03.383Z INFO server version {"platform": "kubernetes", "version": "v1.32.12-gke.1127000"} 2026-03-09T09:06:03.414Z INFO controller-runtime.metrics Starting metrics server 2026-03-09T09:06:03.414Z INFO starting server {"name": "health probe", "addr": "[::]:8081"} I0309 09:06:03.414370 1 leaderelection.go:258] "Attempting to acquire leader lease..." lock="psmdb-operator/08db0feb.percona.com" 2026-03-09T09:06:03.414Z INFO controller-runtime.metrics Serving metrics server {"bindAddress": ":8080", "secure": false} I0309 09:06:22.603445 1 leaderelection.go:272] "Successfully acquired lease" lock="psmdb-operator/08db0feb.percona.com" 2026-03-09T09:06:22.603Z DEBUG events percona-server-mongodb-operator-5b5f6fcd94-6v8bn_430acbcb-e5f0-43a5-91be-87a5b4db3dc2 became leader {"type": "Normal", "object": {"kind":"Lease","namespace":"psmdb-operator","name":"08db0feb.percona.com","uid":"bb23c399-6102-4c0d-be14-e4fcfbf60af0","apiVersion":"coordination.k8s.io/v1","resourceVersion":"1773047182595503015"}, "reason": "LeaderElection"} 2026-03-09T09:06:22.603Z INFO Starting EventSource {"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "source": "kind source: *v1.PerconaServerMongoDB"} 2026-03-09T09:06:22.604Z INFO Starting EventSource {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "source": "kind source: *v1.Pod"} 2026-03-09T09:06:22.604Z INFO Starting EventSource {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "source": "kind source: *v1.PerconaServerMongoDBRestore"} 2026-03-09T09:06:22.604Z INFO Starting EventSource {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "source": "kind source: *v1.Pod"} 2026-03-09T09:06:22.604Z INFO Starting EventSource {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "source": "kind source: *v1.PerconaServerMongoDBBackup"} 2026-03-09T09:06:22.823Z INFO Starting Controller {"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB"} 2026-03-09T09:06:22.823Z INFO Starting workers {"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "worker count": 1} 2026-03-09T09:06:22.823Z INFO Starting Controller {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore"} 2026-03-09T09:06:22.823Z INFO Starting workers {"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "worker count": 1} 2026-03-09T09:06:22.825Z INFO Starting Controller {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup"} 2026-03-09T09:06:22.825Z INFO Starting workers {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "worker count": 1} 2026-03-09T09:06:22.933Z DEBUG Reconciling {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio-logical","namespace":"demand-backup-if-unhealthy-21026"}, "namespace": "demand-backup-if-unhealthy-21026", "name": "backup-minio-logical", "reconcileID": "7bafb8c1-3f75-469c-a11d-b805d0a3c778"} 2026-03-09T09:06:22.978Z DEBUG Got object from API server {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio-logical","namespace":"demand-backup-if-unhealthy-21026"}, "namespace": "demand-backup-if-unhealthy-21026", "name": "backup-minio-logical", "reconcileID": "7bafb8c1-3f75-469c-a11d-b805d0a3c778", "state": ""} 2026-03-09T09:06:23.393Z DEBUG checking if backup is allowed {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio-logical","namespace":"demand-backup-if-unhealthy-21026"}, "namespace": "demand-backup-if-unhealthy-21026", "name": "backup-minio-logical", "reconcileID": "7bafb8c1-3f75-469c-a11d-b805d0a3c778", "cluster": "some-name", "namespace": "demand-backup-if-unhealthy-21026"} 2026-03-09T09:06:23.393Z DEBUG Checking for active jobs {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio-logical","namespace":"demand-backup-if-unhealthy-21026"}, "namespace": "demand-backup-if-unhealthy-21026", "name": "backup-minio-logical", "reconcileID": "7bafb8c1-3f75-469c-a11d-b805d0a3c778", "currentJob": {"Name":"backup-minio-logical","Type":0}} 2026-03-09T09:06:23.523Z INFO Acquiring the backup lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio-logical","namespace":"demand-backup-if-unhealthy-21026"}, "namespace": "demand-backup-if-unhealthy-21026", "name": "backup-minio-logical", "reconcileID": "7bafb8c1-3f75-469c-a11d-b805d0a3c778"} 2026-03-09T09:06:23.697Z DEBUG Added finalizer {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio-logical","namespace":"demand-backup-if-unhealthy-21026"}, "namespace": "demand-backup-if-unhealthy-21026", "name": "backup-minio-logical", "reconcileID": "7bafb8c1-3f75-469c-a11d-b805d0a3c778", "finalizer": "internal.percona.com/release-lock"} 2026-03-09T09:06:23.697Z INFO Starting backup {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio-logical","namespace":"demand-backup-if-unhealthy-21026"}, "namespace": "demand-backup-if-unhealthy-21026", "name": "backup-minio-logical", "reconcileID": "7bafb8c1-3f75-469c-a11d-b805d0a3c778", "backup": "backup-minio-logical", "storage": "minio"} 2026-03-09T09:06:23.697Z INFO Sending backup command {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio-logical","namespace":"demand-backup-if-unhealthy-21026"}, "namespace": "demand-backup-if-unhealthy-21026", "name": "backup-minio-logical", "reconcileID": "7bafb8c1-3f75-469c-a11d-b805d0a3c778", "backup": "backup-minio-logical", "storage": "minio", "backupCmd": "backup [name: 2026-03-09T09:06:23Z, compression: gzip (level: default)] ", "profile": ""} 2026-03-09T09:06:25.278Z INFO SmartUpdate StatefulSet is changed, starting smart update {"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"some-name","namespace":"demand-backup-if-unhealthy-21026"}, "namespace": "demand-backup-if-unhealthy-21026", "name": "some-name", "reconcileID": "58d36be6-4635-45c2-a38c-bc13dab3bb95", "statefulset": "some-name-rs0", "replset": "rs0"} 2026-03-09T09:06:25.281Z INFO SmartUpdate can't start/continue 'SmartUpdate': waiting for all replicas are ready {"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"some-name","namespace":"demand-backup-if-unhealthy-21026"}, "namespace": "demand-backup-if-unhealthy-21026", "name": "some-name", "reconcileID": "58d36be6-4635-45c2-a38c-bc13dab3bb95", "statefulset": "some-name-rs0", "replset": "rs0"} 2026-03-09T09:06:35.595Z ERROR failed to reconcile cluster {"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"some-name","namespace":"demand-backup-if-unhealthy-21026"}, "namespace": "demand-backup-if-unhealthy-21026", "name": "some-name", "reconcileID": "58d36be6-4635-45c2-a38c-bc13dab3bb95", "replset": "rs0", "error": "dial: ping mongo: server selection error: context deadline exceeded, current topology: { Type: ReplicaSetNoPrimary, Servers: [{ Addr: some-name-rs0-0.some-name-rs0.demand-backup-if-unhealthy-21026.svc.cluster.local:27017, Type: RSSecondary, Tag sets: serviceName=some-name,podName=some-name-rs0-0,region=us-central1,zone=us-central1-a,nodeName=gke-jen-psmdb-2277-0a1aa-default-pool-24a6e21c-vbzl, Average RTT: 95226457 }, { Addr: some-name-rs0-1.some-name-rs0.demand-backup-if-unhealthy-21026.svc.cluster.local:27017, Type: Unknown, Last error: dial tcp 10.218.65.11:27017: i/o timeout }, { Addr: some-name-rs0-2.some-name-rs0.demand-backup-if-unhealthy-21026.svc.cluster.local:27017, Type: Unknown, Last error: dial tcp: lookup some-name-rs0-2.some-name-rs0.demand-backup-if-unhealthy-21026.svc.cluster.local on 34.118.224.10:53: no such host }, ] }", "errorVerbose": "server selection error: context deadline exceeded, current topology: { Type: ReplicaSetNoPrimary, Servers: [{ Addr: some-name-rs0-0.some-name-rs0.demand-backup-if-unhealthy-21026.svc.cluster.local:27017, Type: RSSecondary, Tag sets: serviceName=some-name,podName=some-name-rs0-0,region=us-central1,zone=us-central1-a,nodeName=gke-jen-psmdb-2277-0a1aa-default-pool-24a6e21c-vbzl, Average RTT: 95226457 }, { Addr: some-name-rs0-1.some-name-rs0.demand-backup-if-unhealthy-21026.svc.cluster.local:27017, Type: Unknown, Last error: dial tcp 10.218.65.11:27017: i/o timeout }, { Addr: some-name-rs0-2.some-name-rs0.demand-backup-if-unhealthy-21026.svc.cluster.local:27017, Type: Unknown, Last error: dial tcp: lookup some-name-rs0-2.some-name-rs0.demand-backup-if-unhealthy-21026.svc.cluster.local on 34.118.224.10:53: no such host }, ] }\nping mongo\ngithub.com/percona/percona-server-mongodb-operator/pkg/psmdb/mongo.Dial\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/psmdb/mongo/mongo.go:126\ngithub.com/percona/percona-server-mongodb-operator/pkg/psmdb.MongoClient\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/psmdb/client.go:62\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*mongoClientProvider).Mongo\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/connections.go:38\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).mongoClientWithRole\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/connections.go:60\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileCluster\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/mgo.go:94\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileReplsets\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:590\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:448\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:222\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:479\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:313\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1693\ndial\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileCluster\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/mgo.go:111\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileReplsets\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:590\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:448\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:222\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:479\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:313\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1693"} github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileReplsets /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:592 github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:448 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:313 2026-03-09T09:06:35.595Z DEBUG Replset status is not healthy {"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"some-name","namespace":"demand-backup-if-unhealthy-21026"}, "namespace": "demand-backup-if-unhealthy-21026", "name": "some-name", "reconcileID": "58d36be6-4635-45c2-a38c-bc13dab3bb95", "replset": "rs0", "status": "error"} 2026-03-09T09:06:35.661Z ERROR Reconciler error {"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"some-name","namespace":"demand-backup-if-unhealthy-21026"}, "namespace": "demand-backup-if-unhealthy-21026", "name": "some-name", "reconcileID": "58d36be6-4635-45c2-a38c-bc13dab3bb95", "error": "reconcile statefulsets: dial: ping mongo: server selection error: context deadline exceeded, current topology: { Type: ReplicaSetNoPrimary, Servers: [{ Addr: some-name-rs0-0.some-name-rs0.demand-backup-if-unhealthy-21026.svc.cluster.local:27017, Type: RSSecondary, Tag sets: serviceName=some-name,podName=some-name-rs0-0,region=us-central1,zone=us-central1-a,nodeName=gke-jen-psmdb-2277-0a1aa-default-pool-24a6e21c-vbzl, Average RTT: 95226457 }, { Addr: some-name-rs0-1.some-name-rs0.demand-backup-if-unhealthy-21026.svc.cluster.local:27017, Type: Unknown, Last error: dial tcp 10.218.65.11:27017: i/o timeout }, { Addr: some-name-rs0-2.some-name-rs0.demand-backup-if-unhealthy-21026.svc.cluster.local:27017, Type: Unknown, Last error: dial tcp: lookup some-name-rs0-2.some-name-rs0.demand-backup-if-unhealthy-21026.svc.cluster.local on 34.118.224.10:53: no such host }, ] }", "errorVerbose": "dial: ping mongo: server selection error: context deadline exceeded, current topology: { Type: ReplicaSetNoPrimary, Servers: [{ Addr: some-name-rs0-0.some-name-rs0.demand-backup-if-unhealthy-21026.svc.cluster.local:27017, Type: RSSecondary, Tag sets: serviceName=some-name,podName=some-name-rs0-0,region=us-central1,zone=us-central1-a,nodeName=gke-jen-psmdb-2277-0a1aa-default-pool-24a6e21c-vbzl, Average RTT: 95226457 }, { Addr: some-name-rs0-1.some-name-rs0.demand-backup-if-unhealthy-21026.svc.cluster.local:27017, Type: Unknown, Last error: dial tcp 10.218.65.11:27017: i/o timeout }, { Addr: some-name-rs0-2.some-name-rs0.demand-backup-if-unhealthy-21026.svc.cluster.local:27017, Type: Unknown, Last error: dial tcp: lookup some-name-rs0-2.some-name-rs0.demand-backup-if-unhealthy-21026.svc.cluster.local on 34.118.224.10:53: no such host }, ] }\nreconcile statefulsets\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:450\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:222\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:479\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:313\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1693"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:313 2026-03-09T09:06:36.631Z INFO SmartUpdate StatefulSet is changed, starting smart update {"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"some-name","namespace":"demand-backup-if-unhealthy-21026"}, "namespace": "demand-backup-if-unhealthy-21026", "name": "some-name", "reconcileID": "4fd8a430-4b15-4264-b254-d8e36b9dfc67", "statefulset": "some-name-rs0", "replset": "rs0"} 2026-03-09T09:06:36.631Z INFO SmartUpdate can't start/continue 'SmartUpdate': waiting for all replicas are ready {"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"some-name","namespace":"demand-backup-if-unhealthy-21026"}, "namespace": "demand-backup-if-unhealthy-21026", "name": "some-name", "reconcileID": "4fd8a430-4b15-4264-b254-d8e36b9dfc67", "statefulset": "some-name-rs0", "replset": "rs0"} 2026-03-09T09:06:47.137Z ERROR FULL CLUSTER CRASH {"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"some-name","namespace":"demand-backup-if-unhealthy-21026"}, "namespace": "demand-backup-if-unhealthy-21026", "name": "some-name", "reconcileID": "4fd8a430-4b15-4264-b254-d8e36b9dfc67", "error": "ping mongo: server selection error: server selection timeout, current topology: { Type: ReplicaSetNoPrimary, Servers: [{ Addr: some-name-rs0-0.some-name-rs0.demand-backup-if-unhealthy-21026.svc.cluster.local:27017, Type: Unknown, Last error: dial tcp: lookup some-name-rs0-0.some-name-rs0.demand-backup-if-unhealthy-21026.svc.cluster.local: i/o timeout }, { Addr: some-name-rs0-1.some-name-rs0.demand-backup-if-unhealthy-21026.svc.cluster.local:27017, Type: Unknown, Last error: dial tcp: lookup some-name-rs0-1.some-name-rs0.demand-backup-if-unhealthy-21026.svc.cluster.local: i/o timeout }, { Addr: some-name-rs0-2.some-name-rs0.demand-backup-if-unhealthy-21026.svc.cluster.local:27017, Type: Unknown, Last error: dial tcp: lookup some-name-rs0-2.some-name-rs0.demand-backup-if-unhealthy-21026.svc.cluster.local: i/o timeout }, ] }", "errorVerbose": "server selection error: server selection timeout, current topology: { Type: ReplicaSetNoPrimary, Servers: [{ Addr: some-name-rs0-0.some-name-rs0.demand-backup-if-unhealthy-21026.svc.cluster.local:27017, Type: Unknown, Last error: dial tcp: lookup some-name-rs0-0.some-name-rs0.demand-backup-if-unhealthy-21026.svc.cluster.local: i/o timeout }, { Addr: some-name-rs0-1.some-name-rs0.demand-backup-if-unhealthy-21026.svc.cluster.local:27017, Type: Unknown, Last error: dial tcp: lookup some-name-rs0-1.some-name-rs0.demand-backup-if-unhealthy-21026.svc.cluster.local: i/o timeout }, { Addr: some-name-rs0-2.some-name-rs0.demand-backup-if-unhealthy-21026.svc.cluster.local:27017, Type: Unknown, Last error: dial tcp: lookup some-name-rs0-2.some-name-rs0.demand-backup-if-unhealthy-21026.svc.cluster.local: i/o timeout }, ] }\nping mongo\ngithub.com/percona/percona-server-mongodb-operator/pkg/psmdb/mongo.Dial\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/psmdb/mongo/mongo.go:126\ngithub.com/percona/percona-server-mongodb-operator/pkg/psmdb.MongoClient\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/psmdb/client.go:62\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*mongoClientProvider).Mongo\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/connections.go:38\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).mongoClientWithRole\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/connections.go:60\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileCluster\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/mgo.go:94\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileReplsets\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:590\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:448\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:222\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:479\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:313\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1693"} github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileCluster /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/mgo.go:101 github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileReplsets /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:590 github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:448 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:313 2026-03-09T09:06:47.137Z INFO handleReplicaSetNoPrimary Connecting to pod {"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"some-name","namespace":"demand-backup-if-unhealthy-21026"}, "namespace": "demand-backup-if-unhealthy-21026", "name": "some-name", "reconcileID": "4fd8a430-4b15-4264-b254-d8e36b9dfc67", "pod": "some-name-rs0-0", "user": "clusterAdmin"} 2026-03-09T09:06:53.892Z ERROR failed to make backup {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio-logical","namespace":"demand-backup-if-unhealthy-21026"}, "namespace": "demand-backup-if-unhealthy-21026", "name": "backup-minio-logical", "reconcileID": "7bafb8c1-3f75-469c-a11d-b805d0a3c778", "backup": "backup-minio-logical", "error": "write exception: write concern error: (InterruptedDueToReplStateChange) operation was interrupted"} github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).Reconcile.func1 /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/psmdb_backup_controller.go:147 github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).Reconcile /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/psmdb_backup_controller.go:228 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:313 2026-03-09T09:06:53.892Z INFO Backup state changed {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio-logical","namespace":"demand-backup-if-unhealthy-21026"}, "namespace": "demand-backup-if-unhealthy-21026", "name": "backup-minio-logical", "reconcileID": "7bafb8c1-3f75-469c-a11d-b805d0a3c778", "previous": "", "current": "error"} 2026-03-09T09:06:54.120Z INFO Releasing backup lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio-logical","namespace":"demand-backup-if-unhealthy-21026"}, "namespace": "demand-backup-if-unhealthy-21026", "name": "backup-minio-logical", "reconcileID": "7bafb8c1-3f75-469c-a11d-b805d0a3c778", "lease": "psmdb-some-name-backup-lock"} 2026-03-09T09:06:54.302Z DEBUG Reconcile finished {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio-logical","namespace":"demand-backup-if-unhealthy-21026"}, "namespace": "demand-backup-if-unhealthy-21026", "name": "backup-minio-logical", "reconcileID": "7bafb8c1-3f75-469c-a11d-b805d0a3c778"} 2026-03-09T09:06:54.303Z INFO Warning: Reconciler returned both a result with either RequeueAfter or Requeue set and a non-nil error. RequeueAfter and Requeue will always be ignored if the error is non-nil. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio-logical","namespace":"demand-backup-if-unhealthy-21026"}, "namespace": "demand-backup-if-unhealthy-21026", "name": "backup-minio-logical", "reconcileID": "7bafb8c1-3f75-469c-a11d-b805d0a3c778"} 2026-03-09T09:06:54.303Z ERROR Reconciler error {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio-logical","namespace":"demand-backup-if-unhealthy-21026"}, "namespace": "demand-backup-if-unhealthy-21026", "name": "backup-minio-logical", "reconcileID": "7bafb8c1-3f75-469c-a11d-b805d0a3c778", "error": "reconcile backup: write exception: write concern error: (InterruptedDueToReplStateChange) operation was interrupted", "errorVerbose": "write exception: write concern error: (InterruptedDueToReplStateChange) operation was interrupted\nreconcile backup\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/psmdb_backup_controller.go:224\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:222\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:479\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:313\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1693"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:313 2026-03-09T09:06:54.303Z DEBUG Reconciling {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio-logical","namespace":"demand-backup-if-unhealthy-21026"}, "namespace": "demand-backup-if-unhealthy-21026", "name": "backup-minio-logical", "reconcileID": "60722744-14f7-4644-9c8e-640805125de1"} 2026-03-09T09:06:54.445Z DEBUG Got object from API server {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio-logical","namespace":"demand-backup-if-unhealthy-21026"}, "namespace": "demand-backup-if-unhealthy-21026", "name": "backup-minio-logical", "reconcileID": "60722744-14f7-4644-9c8e-640805125de1", "state": "error"} 2026-03-09T09:06:54.445Z DEBUG Reconcile finished {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"backup-minio-logical","namespace":"demand-backup-if-unhealthy-21026"}, "namespace": "demand-backup-if-unhealthy-21026", "name": "backup-minio-logical", "reconcileID": "60722744-14f7-4644-9c8e-640805125de1"} NAME CLUSTER STORAGE DESTINATION TYPE SIZE STATUS COMPLETED AGE backup-minio-logical some-name minio error 35s Name: backup-minio-logical Namespace: demand-backup-if-unhealthy-21026 Labels: Annotations: API Version: psmdb.percona.com/v1 Kind: PerconaServerMongoDBBackup Metadata: Creation Timestamp: 2026-03-09T09:06:22Z Finalizers: percona.com/delete-backup internal.percona.com/release-lock Generation: 1 Resource Version: 1773047214051487004 UID: c7e7b79a-821f-41d7-bc73-15c421c48af8 Spec: Cluster Name: some-name Starting Deadline Seconds: 120 Storage Name: minio Type: logical Status: Error: write exception: write concern error: (InterruptedDueToReplStateChange) operation was interrupted State: error Events: Backup object psmdb-backup/backup-minio-logical is in error state.