Log: /mnt/jenkins/workspace/cloud-psmdb-operator_PR-1545/e2e-tests/logs/cross-site-sharded.log E0507 17:46:38.301312 5240 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:46:38.531022 5240 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:46:38.639281 5240 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:46:38.749637 5240 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request WARNING: version difference between client (1.30) and server (1.26) exceeds the supported minor version skew of +/-1 WARNING: version difference between client (1.30) and server (1.26) exceeds the supported minor version skew of +/-1 WARNING: version difference between client (1.30) and server (1.26) exceeds the supported minor version skew of +/-1 E0507 17:46:43.548872 5909 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:46:43.771106 5909 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:46:45.277577 6110 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:46:45.590152 6110 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:46:45.696029 6110 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:00.157313 7587 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:00.359081 7587 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:00.465420 7587 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:03.070929 7875 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:03.378283 7875 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:03.483000 7875 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. ----------------------------------------------------------------------------------- create main cluster ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- get and delete old CRDs and RBAC ----------------------------------------------------------------------------------- E0507 17:47:05.060524 8067 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:05.273693 8067 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:07.327514 8243 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:07.543728 8243 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:07.649772 8243 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:07.766921 8243 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:08.359841 8243 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:08.554427 8243 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:08.664270 8243 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:08.770871 8243 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:08.877111 8243 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request error: the server doesn't have a resource type "perconaservermongodbbackups" + kubectl patch perconaservermongodbbackups.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' E0507 17:47:10.206759 8603 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:10.434112 8603 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:10.540759 8603 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:10.650373 8603 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:10.984238 8603 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:11.207985 8603 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:11.321485 8603 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:11.427593 8603 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:11.536261 8603 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request error: the server doesn't have a resource type "perconaservermongodbbackups" E0507 17:47:12.768080 8872 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:12.989244 8872 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:13.096138 8872 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:13.202674 8872 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:14.508373 9047 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:14.726280 9047 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:14.834071 9047 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:14.941986 9047 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:15.268122 9047 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:15.492949 9047 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:15.604986 9047 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:15.714065 9047 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:15.822388 9047 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request error: the server doesn't have a resource type "perconaservermongodbrestores" + kubectl patch perconaservermongodbrestores.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' E0507 17:47:16.999725 9209 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:17.225689 9209 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:17.332442 9209 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:17.439049 9209 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:17.772822 9209 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:17.986533 9209 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:18.095808 9209 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:18.202089 9209 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:18.309211 9209 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request error: the server doesn't have a resource type "perconaservermongodbrestores" E0507 17:47:19.568217 9408 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:19.682587 9408 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:19.788990 9408 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:19.895259 9408 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:21.266649 9515 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:21.479425 9515 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:21.586248 9515 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:21.693032 9515 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:22.021751 9515 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:22.224853 9515 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:22.334256 9515 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:22.441209 9515 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:22.548481 9515 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request error: the server doesn't have a resource type "perconaservermongodbs" + kubectl patch perconaservermongodbs.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' E0507 17:47:23.740360 9716 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:24.050518 9716 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:24.156202 9716 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:24.262072 9716 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:24.684132 9716 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:24.794835 9716 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:24.902761 9716 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:25.008700 9716 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 17:47:25.114812 9716 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request error: the server doesn't have a resource type "perconaservermongodbs" Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found ----------------------------------------------------------------------------------- destroy chaos-mesh ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces cross-site-sharded-29846 ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- create namespace cross-site-sharded-29846 ----------------------------------------------------------------------------------- namespace/cross-site-sharded-29846 created Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-1545-63b8c179-7-cluster4" modified. ----------------------------------------------------------------------------------- start PSMDB operator ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io/perconaservermongodbbackups.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbrestores.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbs.psmdb.percona.com serverside-applied role.rbac.authorization.k8s.io/percona-server-mongodb-operator created serviceaccount/percona-server-mongodb-operator created rolebinding.rbac.authorization.k8s.io/service-account-percona-server-mongodb-operator created deployment.apps/percona-server-mongodb-operator created waiting for pod/percona-server-mongodb-operator-6964d95648-7x2mk to be ready..OK ----------------------------------------------------------------------------------- create secrets and start client ----------------------------------------------------------------------------------- deployment.apps/psmdb-client created secret/cross-site-sharded-main-secrets created secret/cross-site-sharded-main-ssl created secret/cross-site-sharded-main-ssl-internal created ----------------------------------------------------------------------------------- create main PSMDB cluster cross-site-sharded-main. ----------------------------------------------------------------------------------- perconaservermongodb.psmdb.percona.com/cross-site-sharded-main created ----------------------------------------------------------------------------------- check if all 3 Pods started ----------------------------------------------------------------------------------- waiting for pod/cross-site-sharded-main-rs0-0 to be ready......................OK waiting for pod/cross-site-sharded-main-rs0-1 to be ready............OK waiting for pod/cross-site-sharded-main-rs0-2 to be ready............OK Waiting for cluster readyness....................................... waiting for pod/cross-site-sharded-main-cfg-0 to be ready.OK waiting for pod/cross-site-sharded-main-cfg-1 to be ready.OK waiting for pod/cross-site-sharded-main-cfg-2 to be ready.OK ----------------------------------------------------------------------------------- create user ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://cross-site-sharded-main-mongos.cross-site-sharded-29846.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("5964b2b5-4c45-4512-a0cb-2b754ec36a97") } Percona Server for MongoDB server version: v7.0.8-5 WARNING: shell and server versions do not match Successfully added user: { "user" : "user", "roles" : [ { "db" : "app", "role" : "readWrite" } ] } bye ----------------------------------------------------------------------------------- set chunk size to 2 MB ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://cross-site-sharded-main-mongos.cross-site-sharded-29846.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("f485868c-9a46-42a8-9ec8-be7bba7146c8") } Percona Server for MongoDB server version: v7.0.8-5 WARNING: shell and server versions do not match switched to db config WriteResult({ "nMatched" : 0, "nUpserted" : 1, "nModified" : 0, "_id" : "chunksize" }) bye ----------------------------------------------------------------------------------- write data ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://cross-site-sharded-main-mongos.cross-site-sharded-29846.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("fbd4cc66-335b-4888-9fde-bf046568b7e0") } Percona Server for MongoDB server version: v7.0.8-5 WARNING: shell and server versions do not match ----------------------------------------------------------------------------------- shard collection ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://cross-site-sharded-main-mongos.cross-site-sharded-29846.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("10bb3c0a-c2d5-4cc8-bd8e-7e5fbca23d76") } Percona Server for MongoDB server version: v7.0.8-5 WARNING: shell and server versions do not match { "ok" : 1, "$clusterTime" : { "clusterTime" : Timestamp(1715104388, 1), "signature" : { "hash" : BinData(0,"BcLXYuAeTowMPMGFtVIKefY4sJs="), "keyId" : NumberLong("7366316418167472136") } }, "operationTime" : Timestamp(1715104388, 1) } bye Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://cross-site-sharded-main-mongos.cross-site-sharded-29846.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("841d136a-03bb-4a9f-af52-0837a23f6bd0") } Percona Server for MongoDB server version: v7.0.8-5 WARNING: shell and server versions do not match { "collectionsharded" : "app.city", "ok" : 1, "$clusterTime" : { "clusterTime" : Timestamp(1715104394, 35), "signature" : { "hash" : BinData(0,"/jp/pTdEsvBOsVQ/J235Kl0VCjA="), "keyId" : NumberLong("7366316418167472136") } }, "operationTime" : Timestamp(1715104394, 35) } bye ----------------------------------------------------------------------------------- Check chunks ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- rs0 has 3 chunks ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- rs1 has 1 chunks ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- create replica cluster ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces cross-site-sharded-replica-14346 ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- create namespace cross-site-sharded-replica-14346 ----------------------------------------------------------------------------------- namespace/cross-site-sharded-replica-14346 created Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-1545-63b8c179-7-cluster4" modified. ----------------------------------------------------------------------------------- start PSMDB operator ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io/perconaservermongodbbackups.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbrestores.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbs.psmdb.percona.com serverside-applied role.rbac.authorization.k8s.io/percona-server-mongodb-operator created serviceaccount/percona-server-mongodb-operator created rolebinding.rbac.authorization.k8s.io/service-account-percona-server-mongodb-operator created deployment.apps/percona-server-mongodb-operator created waiting for pod/percona-server-mongodb-operator-6964d95648-psfxb to be ready.OK ----------------------------------------------------------------------------------- start client ----------------------------------------------------------------------------------- deployment.apps/psmdb-client created ----------------------------------------------------------------------------------- copy secrets from main to replica namespace and create all of them ----------------------------------------------------------------------------------- secret/cross-site-sharded-replica-secrets created secret/cross-site-sharded-replica-ssl-internal created secret/cross-site-sharded-replica-ssl created ----------------------------------------------------------------------------------- create replica PSMDB cluster ----------------------------------------------------------------------------------- perconaservermongodb.psmdb.percona.com/cross-site-sharded-replica created waiting for pod/cross-site-sharded-replica-rs0-0 to be ready..............OK waiting for pod/cross-site-sharded-replica-rs0-1 to be ready..........OK waiting for pod/cross-site-sharded-replica-rs0-2 to be ready............OK waiting for pod/cross-site-sharded-replica-rs1-0 to be ready.OK waiting for pod/cross-site-sharded-replica-rs1-1 to be ready.OK waiting for pod/cross-site-sharded-replica-rs1-2 to be ready.OK waiting for pod/cross-site-sharded-replica-cfg-0 to be ready.OK waiting for pod/cross-site-sharded-replica-cfg-1 to be ready.OK waiting for pod/cross-site-sharded-replica-cfg-2 to be ready.OK Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-1545-63b8c179-7-cluster4" modified. perconaservermongodb.psmdb.percona.com/cross-site-sharded-main patched command terminated with exit code 1 waiting for all members to be configured in cfg .waiting for all members to be configured in cfg .waiting for all members to be configured in rs0 .waiting for all members to be configured in rs1 .Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-1545-63b8c179-7-cluster4" modified. ----------------------------------------------------------------------------------- check if all 3 Pods started ----------------------------------------------------------------------------------- waiting for pod/cross-site-sharded-replica-rs0-0 to be ready.OK waiting for pod/cross-site-sharded-replica-rs0-1 to be ready.OK waiting for pod/cross-site-sharded-replica-rs0-2 to be ready.OK Waiting for cluster readyness waiting for pod/cross-site-sharded-replica-cfg-0 to be ready.OK waiting for pod/cross-site-sharded-replica-cfg-1 to be ready.OK waiting for pod/cross-site-sharded-replica-cfg-2 to be ready.OK ----------------------------------------------------------------------------------- create user ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://cross-site-sharded-main-mongos.cross-site-sharded-29846.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("4a516bf4-1f9b-411c-adff-ab3f8a8be85b") } Percona Server for MongoDB server version: v7.0.8-5 WARNING: shell and server versions do not match Successfully added user: { "user" : "myApp", "roles" : [ { "db" : "myApp", "role" : "readWrite" } ] } bye ----------------------------------------------------------------------------------- write data, read from all ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://cross-site-sharded-main-mongos.cross-site-sharded-29846.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("1f1a473d-48e8-4a78-bfd9-871ec02d30d6") } Percona Server for MongoDB server version: v7.0.8-5 WARNING: shell and server versions do not match switched to db myApp WriteResult({ "nInserted" : 1 }) bye ----------------------------------------------------------------------------------- Compare data ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- test failover ----------------------------------------------------------------------------------- Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-1545-63b8c179-7-cluster4" modified. perconaservermongodb.psmdb.percona.com "cross-site-sharded-main" deleted ----------------------------------------------------------------------------------- run disaster recovery script for replset: cfg ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://10.219.63.18:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("d89f6fec-56dc-45d2-a741-8e2b3e44d3d3") } Percona Server for MongoDB server version: v7.0.8-5 WARNING: shell and server versions do not match ----------------------------------------------------------------------------------- run disaster recovery script for replset: rs0 ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://10.219.58.107:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("7ff56d74-c9d4-45ee-8931-0c009fa2669e") } Percona Server for MongoDB server version: v7.0.8-5 WARNING: shell and server versions do not match ----------------------------------------------------------------------------------- run disaster recovery script for replset: rs1 ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://10.219.53.47:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("ecbd3d3a-1b41-4275-afe4-91dae43f665f") } Percona Server for MongoDB server version: v7.0.8-5 WARNING: shell and server versions do not match ----------------------------------------------------------------------------------- make replica cluster managed ----------------------------------------------------------------------------------- Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-1545-63b8c179-7-cluster4" modified. perconaservermongodb.psmdb.percona.com/cross-site-sharded-replica patched waiting for pod/cross-site-sharded-replica-rs0-0 to be ready.OK waiting for pod/cross-site-sharded-replica-rs0-1 to be ready.OK waiting for pod/cross-site-sharded-replica-rs0-2 to be ready.OK Waiting for cluster readyness waiting for pod/cross-site-sharded-replica-cfg-0 to be ready.OK waiting for pod/cross-site-sharded-replica-cfg-1 to be ready.OK waiting for pod/cross-site-sharded-replica-cfg-2 to be ready.OK Waiting for cluster readyness ----------------------------------------------------------------------------------- check failover status ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- Failover check finished successfully ----------------------------------------------------------------------------------- waiting for cluster readyness ----------------------------------------------------------------------------------- destroy cluster/operator and all other resources ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- get and delete old CRDs and RBAC ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io "perconaservermongodbbackups.psmdb.percona.com" deleted customresourcedefinition.apiextensions.k8s.io "perconaservermongodbrestores.psmdb.percona.com" deleted customresourcedefinition.apiextensions.k8s.io "perconaservermongodbs.psmdb.percona.com" deleted error: the server doesn't have a resource type "perconaservermongodbbackups" + kubectl patch perconaservermongodbbackups.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbbackups" error: the server doesn't have a resource type "perconaservermongodbrestores" + kubectl patch perconaservermongodbrestores.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbrestores" error: the server doesn't have a resource type "perconaservermongodbs" + kubectl patch perconaservermongodbs.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbs" role.rbac.authorization.k8s.io "percona-server-mongodb-operator" deleted serviceaccount "percona-server-mongodb-operator" deleted rolebinding.rbac.authorization.k8s.io "service-account-percona-server-mongodb-operator" deleted ----------------------------------------------------------------------------------- destroy cluster/operator and all other resources ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- get and delete old CRDs and RBAC ----------------------------------------------------------------------------------- error: the server doesn't have a resource type "perconaservermongodbbackups" + kubectl patch perconaservermongodbbackups.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbbackups" error: the server doesn't have a resource type "perconaservermongodbrestores" + kubectl patch perconaservermongodbrestores.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbrestores" error: the server doesn't have a resource type "perconaservermongodbs" + kubectl patch perconaservermongodbs.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbs" namespace "cross-site-sharded-29846" force deleted Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. ----------------------------------------------------------------------------------- test passed -----------------------------------------------------------------------------------