Log: /mnt/jenkins/workspace/cloud-psmdb-operator_PR-2285/e2e-tests/logs/cross-site-sharded.log Warning: version difference between client (1.35) and server (1.32) exceeds the supported minor version skew of +/-1 Warning: version difference between client (1.35) and server (1.32) exceeds the supported minor version skew of +/-1 Warning: version difference between client (1.35) and server (1.32) exceeds the supported minor version skew of +/-1 Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. ----------------------------------------------------------------------------------- create main cluster ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- get and delete old CRDs and RBAC ----------------------------------------------------------------------------------- grep: warning: stray \ before - grep: warning: stray \ before - error: the server doesn't have a resource type "perconaservermongodbbackups" + kubectl patch perconaservermongodbbackups.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbbackups" error: the server doesn't have a resource type "perconaservermongodbrestores" + kubectl patch perconaservermongodbrestores.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbrestores" error: the server doesn't have a resource type "perconaservermongodbs" + kubectl patch perconaservermongodbs.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbs" Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found ----------------------------------------------------------------------------------- destroy chaos-mesh ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces cross-site-sharded-7884 ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- create namespace cross-site-sharded-7884 ----------------------------------------------------------------------------------- namespace/cross-site-sharded-7884 created Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-2285-ace12b60-1-cluster3" modified. ----------------------------------------------------------------------------------- start PSMDB operator: docker.io/perconalab/percona-server-mongodb-operator:PR-2285-ace12b60 ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io/perconaservermongodbbackups.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbrestores.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbs.psmdb.percona.com serverside-applied role.rbac.authorization.k8s.io/percona-server-mongodb-operator created serviceaccount/percona-server-mongodb-operator created rolebinding.rbac.authorization.k8s.io/service-account-percona-server-mongodb-operator created deployment.apps/percona-server-mongodb-operator created waiting for pod/percona-server-mongodb-operator-677dfb6b6f-8k7ll to be ready.OK Print operator info from log 2026-03-15T16:44:42.802Z INFO setup Manager starting up {"gitCommit": "ace12b605f36447907325d565c954115594804c6", "gitBranch": "PR-2285-ace12b60", "buildTime": "", "goVersion": "go1.25.8", "os": "linux", "arch": "amd64"} ----------------------------------------------------------------------------------- create secrets and start client ----------------------------------------------------------------------------------- deployment.apps/psmdb-client created secret/cross-site-sharded-main-secrets created secret/cross-site-sharded-main-ssl created secret/cross-site-sharded-main-ssl-internal created ----------------------------------------------------------------------------------- create main PSMDB cluster cross-site-sharded-main. ----------------------------------------------------------------------------------- perconaservermongodb.psmdb.percona.com/cross-site-sharded-main created ----------------------------------------------------------------------------------- check if all 3 Pods started ----------------------------------------------------------------------------------- waiting for pod/cross-site-sharded-main-rs0-0 to be ready...........................OK waiting for pod/cross-site-sharded-main-rs0-1 to be ready..............OK waiting for pod/cross-site-sharded-main-rs0-2 to be ready.............OK Waiting for cluster readyness.................................................. waiting for pod/cross-site-sharded-main-cfg-0 to be ready.OK waiting for pod/cross-site-sharded-main-cfg-1 to be ready.OK waiting for pod/cross-site-sharded-main-cfg-2 to be ready.OK ----------------------------------------------------------------------------------- create user ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://cross-site-sharded-main-mongos.cross-site-sharded-7884.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("5d97bc1e-0a9f-49db-818a-8c92e95d8e2d") } Percona Server for MongoDB server version: v8.0.19-7 WARNING: shell and server versions do not match Successfully added user: { "user" : "user", "roles" : [ { "db" : "app", "role" : "readWrite" } ] } bye ----------------------------------------------------------------------------------- set chunk size to 2 MB ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://cross-site-sharded-main-mongos.cross-site-sharded-7884.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("9a395a1b-516b-4a66-8e57-cba9827e943d") } Percona Server for MongoDB server version: v8.0.19-7 WARNING: shell and server versions do not match switched to db config WriteResult({ "nMatched" : 0, "nUpserted" : 1, "nModified" : 0, "_id" : "chunksize" }) bye ----------------------------------------------------------------------------------- write data ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://cross-site-sharded-main-mongos.cross-site-sharded-7884.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("01eb6106-7ddf-473e-b74f-b425ef973bcc") } Percona Server for MongoDB server version: v8.0.19-7 WARNING: shell and server versions do not match ----------------------------------------------------------------------------------- shard collection ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://cross-site-sharded-main-mongos.cross-site-sharded-7884.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("d2440746-3338-4ff1-bb48-e46f5c7c798b") } Percona Server for MongoDB server version: v8.0.19-7 WARNING: shell and server versions do not match { "ok" : 1, "$clusterTime" : { "clusterTime" : Timestamp(1773593407, 2), "signature" : { "hash" : BinData(0,"dCbfwzad4yeqg/pR2dtoYzpmIYs="), "keyId" : NumberLong("7617524764638183427") } }, "operationTime" : Timestamp(1773593407, 2) } bye Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://cross-site-sharded-main-mongos.cross-site-sharded-7884.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("3596af04-4265-4ef9-9654-af4266328c18") } Percona Server for MongoDB server version: v8.0.19-7 WARNING: shell and server versions do not match { "collectionsharded" : "app.city", "ok" : 1, "$clusterTime" : { "clusterTime" : Timestamp(1773593414, 24), "signature" : { "hash" : BinData(0,"OEPhgYKoVYUukHemy1H91kXk7/M="), "keyId" : NumberLong("7617524764638183427") } }, "operationTime" : Timestamp(1773593414, 24) } bye ----------------------------------------------------------------------------------- Check chunks ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- rs0 has 1 chunks ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- rs1 has 3 chunks ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- create replica cluster ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces cross-site-sharded-replica-4263 ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- create namespace cross-site-sharded-replica-4263 ----------------------------------------------------------------------------------- namespace/cross-site-sharded-replica-4263 created Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-2285-ace12b60-1-cluster3" modified. ----------------------------------------------------------------------------------- start PSMDB operator: docker.io/perconalab/percona-server-mongodb-operator:PR-2285-ace12b60 ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io/perconaservermongodbbackups.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbrestores.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbs.psmdb.percona.com serverside-applied role.rbac.authorization.k8s.io/percona-server-mongodb-operator created serviceaccount/percona-server-mongodb-operator created rolebinding.rbac.authorization.k8s.io/service-account-percona-server-mongodb-operator created deployment.apps/percona-server-mongodb-operator created waiting for pod/percona-server-mongodb-operator-677dfb6b6f-sgzdg to be ready.OK Print operator info from log 2026-03-15T16:52:39.609Z INFO setup Manager starting up {"gitCommit": "ace12b605f36447907325d565c954115594804c6", "gitBranch": "PR-2285-ace12b60", "buildTime": "", "goVersion": "go1.25.8", "os": "linux", "arch": "amd64"} ----------------------------------------------------------------------------------- start client ----------------------------------------------------------------------------------- deployment.apps/psmdb-client created ----------------------------------------------------------------------------------- copy secrets from main to replica namespace and create all of them ----------------------------------------------------------------------------------- secret/cross-site-sharded-replica-secrets created secret/cross-site-sharded-replica-ssl-internal created secret/cross-site-sharded-replica-ssl created ----------------------------------------------------------------------------------- create replica PSMDB cluster ----------------------------------------------------------------------------------- perconaservermongodb.psmdb.percona.com/cross-site-sharded-replica created waiting for pod/cross-site-sharded-replica-rs0-0 to be ready...............OK waiting for pod/cross-site-sharded-replica-rs0-1 to be ready..............OK waiting for pod/cross-site-sharded-replica-rs0-2 to be ready.............OK waiting for pod/cross-site-sharded-replica-rs1-0 to be ready.OK waiting for pod/cross-site-sharded-replica-rs1-1 to be ready.OK waiting for pod/cross-site-sharded-replica-rs1-2 to be ready.OK waiting for pod/cross-site-sharded-replica-cfg-0 to be ready.OK waiting for pod/cross-site-sharded-replica-cfg-1 to be ready.OK waiting for pod/cross-site-sharded-replica-cfg-2 to be ready.OK Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-2285-ace12b60-1-cluster3" modified. perconaservermongodb.psmdb.percona.com/cross-site-sharded-main patched command terminated with exit code 1 command terminated with exit code 1 waiting for all members to be configured in cfg. waiting for all members to be configured in cfg. waiting for all members to be configured in cfg. waiting for all members to be configured in cfg. waiting for all members to be configured in cfg. waiting for all members to be configured in cfg. waiting for all members to be configured in cfg. waiting for all members to be configured in cfg. waiting for all members to be configured in cfg. command terminated with exit code 1 command terminated with exit code 1 error: cannot exec into a container in a completed pod; current phase is Failed error: cannot exec into a container in a completed pod; current phase is Failed waiting for all members to be configured in cfg. Error from server (BadRequest): pod psmdb-client-bb8b97679-4vr2t does not have a host assigned Error from server (BadRequest): pod psmdb-client-bb8b97679-4vr2t does not have a host assigned Error from server (BadRequest): pod psmdb-client-bb8b97679-4vr2t does not have a host assigned Error from server (BadRequest): pod psmdb-client-bb8b97679-4vr2t does not have a host assigned waiting for all members to be configured in cfg. Error from server (BadRequest): pod psmdb-client-bb8b97679-4vr2t does not have a host assigned Error from server (BadRequest): pod psmdb-client-bb8b97679-4vr2t does not have a host assigned Error from server (BadRequest): pod psmdb-client-bb8b97679-4vr2t does not have a host assigned Error from server (BadRequest): pod psmdb-client-bb8b97679-4vr2t does not have a host assigned waiting for all members to be configured in cfg. Error from server (BadRequest): pod psmdb-client-bb8b97679-4vr2t does not have a host assigned Error from server (BadRequest): pod psmdb-client-bb8b97679-4vr2t does not have a host assigned Error from server (BadRequest): pod psmdb-client-bb8b97679-4vr2t does not have a host assigned Error from server (BadRequest): pod psmdb-client-bb8b97679-4vr2t does not have a host assigned waiting for all members to be configured in cfg. error: Internal error occurred: unable to upgrade connection: container not found ("psmdb-client") error: Internal error occurred: unable to upgrade connection: container not found ("psmdb-client") error: Internal error occurred: unable to upgrade connection: container not found ("psmdb-client") error: Internal error occurred: unable to upgrade connection: container not found ("psmdb-client") waiting for all members to be configured in cfg. command terminated with exit code 1 command terminated with exit code 1 command terminated with exit code 1 command terminated with exit code 1 waiting for all members to be configured in cfgMax retry count 15 reached. something went wrong with mongo cluster. Config for endpoint 34.118.239.153 has Error: couldn't connect to server 34.118.239.153:27017, connection attempt failed: SocketException: Error connecting to 34.118.239.153:27017 :: caused by :: Connection refused : connect@src/mongo/shell/mongo.js:374:17 @(connect):2:6 exception: connect failed exiting with code 1 Error: couldn't connect to server 34.118.239.153:27017, connection attempt failed: SocketException: Error connecting to 34.118.239.153:27017 :: caused by :: Connection refused : connect@src/mongo/shell/mongo.js:374:17 @(connect):2:6 exception: connect failed exiting with code 1 Error: couldn't connect to server 34.118.239.153:27017, connection attempt failed: SocketException: Error connecting to 34.118.239.153:27017 :: caused by :: Connection refused : connect@src/mongo/shell/mongo.js:374:17 @(connect):2:6 exception: connect failed exiting with code 1 Error: couldn't connect to server 34.118.239.153:27017, connection attempt failed: SocketException: Error connecting to 34.118.239.153:27017 :: caused by :: Connection refused : connect@src/mongo/shell/mongo.js:374:17 @(connect):2:6 exception: connect failed exiting with code 1 but expected 6.