Log: /mnt/jenkins/workspace/cloud-psmdb-operator_PR-1539/e2e-tests/logs/cross-site-sharded.log E0501 02:28:13.583587 15603 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:13.806317 15603 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:13.916734 15603 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:14.025302 15603 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request WARNING: version difference between client (1.30) and server (1.26) exceeds the supported minor version skew of +/-1 WARNING: version difference between client (1.30) and server (1.26) exceeds the supported minor version skew of +/-1 WARNING: version difference between client (1.30) and server (1.26) exceeds the supported minor version skew of +/-1 E0501 02:28:18.437474 16070 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:18.760879 16070 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:20.346691 16306 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:20.455376 16306 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:20.563360 16306 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:33.510639 17619 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:33.725247 17619 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:33.831280 17619 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:36.591171 17944 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:36.699041 17944 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:36.807332 17944 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. ----------------------------------------------------------------------------------- create main cluster ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- get and delete old CRDs and RBAC ----------------------------------------------------------------------------------- E0501 02:28:38.676015 18261 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:38.988325 18261 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:40.607403 18488 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:40.919942 18488 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:41.026742 18488 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:41.134105 18488 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:41.456607 18488 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:41.668800 18488 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:41.779511 18488 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:41.886697 18488 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:41.995234 18488 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request error: the server doesn't have a resource type "perconaservermongodbbackups" + kubectl patch perconaservermongodbbackups.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' E0501 02:28:43.155344 18767 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:43.380422 18767 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:43.489578 18767 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:43.598263 18767 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:44.163598 18767 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:44.459477 18767 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:44.593809 18767 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:44.730881 18767 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:44.840226 18767 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request error: the server doesn't have a resource type "perconaservermongodbbackups" E0501 02:28:46.128254 19071 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:46.449345 19071 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:46.557914 19071 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:46.668081 19071 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:48.000715 19359 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:48.233843 19359 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:48.341607 19359 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:48.448676 19359 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:48.771251 19359 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:48.986377 19359 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:49.096912 19359 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:49.204158 19359 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:49.311275 19359 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request error: the server doesn't have a resource type "perconaservermongodbrestores" + kubectl patch perconaservermongodbrestores.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' E0501 02:28:50.683021 19697 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:50.887416 19697 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:50.993325 19697 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:51.100041 19697 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:51.424395 19697 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:51.638288 19697 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:51.744566 19697 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:51.851237 19697 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:51.956997 19697 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request error: the server doesn't have a resource type "perconaservermongodbrestores" E0501 02:28:53.239548 20007 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:53.552661 20007 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:53.660222 20007 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:53.766998 20007 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:55.322942 20273 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:55.646099 20273 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:55.754478 20273 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:55.863124 20273 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:56.193192 20273 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:56.408280 20273 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:56.520860 20273 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:56.632275 20273 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:56.741374 20273 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request error: the server doesn't have a resource type "perconaservermongodbs" + kubectl patch perconaservermongodbs.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' E0501 02:28:58.445854 20655 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:58.664655 20655 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:58.775790 20655 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:58.934296 20655 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:59.374484 20655 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:59.655124 20655 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:59.765003 20655 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:59.871484 20655 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:28:59.977947 20655 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request error: the server doesn't have a resource type "perconaservermongodbs" E0501 02:29:01.284042 21099 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:29:01.604223 21099 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:29:01.712389 21099 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0501 02:29:01.820394 21099 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found ----------------------------------------------------------------------------------- destroy chaos-mesh ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces cross-site-sharded-2930 ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- create namespace cross-site-sharded-2930 ----------------------------------------------------------------------------------- namespace/cross-site-sharded-2930 created Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-1539-b60f2a71-1-cluster4" modified. ----------------------------------------------------------------------------------- start PSMDB operator ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io/perconaservermongodbbackups.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbrestores.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbs.psmdb.percona.com serverside-applied role.rbac.authorization.k8s.io/percona-server-mongodb-operator created serviceaccount/percona-server-mongodb-operator created rolebinding.rbac.authorization.k8s.io/service-account-percona-server-mongodb-operator created deployment.apps/percona-server-mongodb-operator created waiting for pod/percona-server-mongodb-operator-5b88b4fc5c-kfcq4 to be ready..OK ----------------------------------------------------------------------------------- create secrets and start client ----------------------------------------------------------------------------------- deployment.apps/psmdb-client created secret/cross-site-sharded-main-secrets created secret/cross-site-sharded-main-ssl created secret/cross-site-sharded-main-ssl-internal created ----------------------------------------------------------------------------------- create main PSMDB cluster cross-site-sharded-main. ----------------------------------------------------------------------------------- perconaservermongodb.psmdb.percona.com/cross-site-sharded-main created ----------------------------------------------------------------------------------- check if all 3 Pods started ----------------------------------------------------------------------------------- waiting for pod/cross-site-sharded-main-rs0-0 to be ready.................OK waiting for pod/cross-site-sharded-main-rs0-1 to be ready.............OK waiting for pod/cross-site-sharded-main-rs0-2 to be ready..............OK Waiting for cluster readyness......................................... waiting for pod/cross-site-sharded-main-cfg-0 to be ready.OK waiting for pod/cross-site-sharded-main-cfg-1 to be ready.OK waiting for pod/cross-site-sharded-main-cfg-2 to be ready.OK ----------------------------------------------------------------------------------- create user ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://cross-site-sharded-main-mongos.cross-site-sharded-2930.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("eb2fd80e-c128-491c-9cf1-2213c1ed544d") } Percona Server for MongoDB server version: v7.0.8-5 WARNING: shell and server versions do not match Successfully added user: { "user" : "user", "roles" : [ { "db" : "app", "role" : "readWrite" } ] } bye ----------------------------------------------------------------------------------- set chunk size to 2 MB ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://cross-site-sharded-main-mongos.cross-site-sharded-2930.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("af67ffb7-3ad5-4ae1-8365-af463af34758") } Percona Server for MongoDB server version: v7.0.8-5 WARNING: shell and server versions do not match switched to db config WriteResult({ "nMatched" : 0, "nUpserted" : 1, "nModified" : 0, "_id" : "chunksize" }) bye ----------------------------------------------------------------------------------- write data ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://cross-site-sharded-main-mongos.cross-site-sharded-2930.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("635c8429-7d6c-40ef-af2b-3e3b257426b2") } Percona Server for MongoDB server version: v7.0.8-5 WARNING: shell and server versions do not match ----------------------------------------------------------------------------------- shard collection ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://cross-site-sharded-main-mongos.cross-site-sharded-2930.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("598e9d2e-0bd4-4c60-bbd3-1d86c1aa8e60") } Percona Server for MongoDB server version: v7.0.8-5 WARNING: shell and server versions do not match { "ok" : 1, "$clusterTime" : { "clusterTime" : Timestamp(1714530882, 1), "signature" : { "hash" : BinData(0,"tgvrBswUby8zqbkkQGu4bouYt2g="), "keyId" : NumberLong("7363853237243346963") } }, "operationTime" : Timestamp(1714530882, 1) } bye Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://cross-site-sharded-main-mongos.cross-site-sharded-2930.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("1c06894e-9448-4070-9682-59489d4ce302") } Percona Server for MongoDB server version: v7.0.8-5 WARNING: shell and server versions do not match { "collectionsharded" : "app.city", "ok" : 1, "$clusterTime" : { "clusterTime" : Timestamp(1714530888, 35), "signature" : { "hash" : BinData(0,"4354AOAr08FwFrLejSMjJtrMZvI="), "keyId" : NumberLong("7363853237243346963") } }, "operationTime" : Timestamp(1714530888, 35) } bye ----------------------------------------------------------------------------------- Check chunks ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- rs0 has 3 chunks ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- rs1 has 1 chunks ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- create replica cluster ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces cross-site-sharded-replica-3340 ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- create namespace cross-site-sharded-replica-3340 ----------------------------------------------------------------------------------- namespace/cross-site-sharded-replica-3340 created Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-1539-b60f2a71-1-cluster4" modified. ----------------------------------------------------------------------------------- start PSMDB operator ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io/perconaservermongodbbackups.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbrestores.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbs.psmdb.percona.com serverside-applied role.rbac.authorization.k8s.io/percona-server-mongodb-operator created serviceaccount/percona-server-mongodb-operator created rolebinding.rbac.authorization.k8s.io/service-account-percona-server-mongodb-operator created deployment.apps/percona-server-mongodb-operator created waiting for pod/percona-server-mongodb-operator-5b88b4fc5c-j6spb to be ready.OK ----------------------------------------------------------------------------------- start client ----------------------------------------------------------------------------------- deployment.apps/psmdb-client created ----------------------------------------------------------------------------------- copy secrets from main to replica namespace and create all of them ----------------------------------------------------------------------------------- secret/cross-site-sharded-replica-secrets created secret/cross-site-sharded-replica-ssl-internal created secret/cross-site-sharded-replica-ssl created ----------------------------------------------------------------------------------- create replica PSMDB cluster ----------------------------------------------------------------------------------- perconaservermongodb.psmdb.percona.com/cross-site-sharded-replica created waiting for pod/cross-site-sharded-replica-rs0-0 to be ready.............OK waiting for pod/cross-site-sharded-replica-rs0-1 to be ready.............OK waiting for pod/cross-site-sharded-replica-rs0-2 to be ready.............OK waiting for pod/cross-site-sharded-replica-rs1-0 to be ready.OK waiting for pod/cross-site-sharded-replica-rs1-1 to be ready.OK waiting for pod/cross-site-sharded-replica-rs1-2 to be ready.OK waiting for pod/cross-site-sharded-replica-cfg-0 to be ready.OK waiting for pod/cross-site-sharded-replica-cfg-1 to be ready.OK waiting for pod/cross-site-sharded-replica-cfg-2 to be ready.OK Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-1539-b60f2a71-1-cluster4" modified. perconaservermongodb.psmdb.percona.com/cross-site-sharded-main patched waiting for all members to be configured in cfg .waiting for all members to be configured in cfg .waiting for all members to be configured in rs0 .waiting for all members to be configured in rs1 .Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-1539-b60f2a71-1-cluster4" modified. ----------------------------------------------------------------------------------- check if all 3 Pods started ----------------------------------------------------------------------------------- waiting for pod/cross-site-sharded-replica-rs0-0 to be ready.OK waiting for pod/cross-site-sharded-replica-rs0-1 to be ready.OK waiting for pod/cross-site-sharded-replica-rs0-2 to be ready.OK Waiting for cluster readyness waiting for pod/cross-site-sharded-replica-cfg-0 to be ready.OK waiting for pod/cross-site-sharded-replica-cfg-1 to be ready.OK waiting for pod/cross-site-sharded-replica-cfg-2 to be ready.OK ----------------------------------------------------------------------------------- create user ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://cross-site-sharded-main-mongos.cross-site-sharded-2930.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("79ef0432-5a0a-40c0-94f9-7adaeb7b444c") } Percona Server for MongoDB server version: v7.0.8-5 WARNING: shell and server versions do not match Successfully added user: { "user" : "myApp", "roles" : [ { "db" : "myApp", "role" : "readWrite" } ] } bye ----------------------------------------------------------------------------------- write data, read from all ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://cross-site-sharded-main-mongos.cross-site-sharded-2930.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("473e87e6-6a36-46d5-b9fe-96b37052e53a") } Percona Server for MongoDB server version: v7.0.8-5 WARNING: shell and server versions do not match switched to db myApp WriteResult({ "nInserted" : 1 }) bye ----------------------------------------------------------------------------------- Compare data ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- test failover ----------------------------------------------------------------------------------- Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-1539-b60f2a71-1-cluster4" modified. perconaservermongodb.psmdb.percona.com "cross-site-sharded-main" deleted ----------------------------------------------------------------------------------- run disaster recovery script for replset: cfg ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://10.92.244.12:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("f9e9b928-ae71-4dc2-9cae-1637d85d1274") } Percona Server for MongoDB server version: v7.0.8-5 WARNING: shell and server versions do not match ----------------------------------------------------------------------------------- run disaster recovery script for replset: rs0 ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://10.92.249.242:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("afca8696-91ee-411b-8671-6d6b9e69d33f") } Percona Server for MongoDB server version: v7.0.8-5 WARNING: shell and server versions do not match ----------------------------------------------------------------------------------- run disaster recovery script for replset: rs1 ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://10.92.249.197:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("adf3cd51-7b42-4a71-a5b6-f50d39d0fd9b") } Percona Server for MongoDB server version: v7.0.8-5 WARNING: shell and server versions do not match ----------------------------------------------------------------------------------- make replica cluster managed ----------------------------------------------------------------------------------- Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-1539-b60f2a71-1-cluster4" modified. perconaservermongodb.psmdb.percona.com/cross-site-sharded-replica patched waiting for pod/cross-site-sharded-replica-rs0-0 to be ready.OK waiting for pod/cross-site-sharded-replica-rs0-1 to be ready.OK waiting for pod/cross-site-sharded-replica-rs0-2 to be ready.OK Waiting for cluster readyness waiting for pod/cross-site-sharded-replica-cfg-0 to be ready.OK waiting for pod/cross-site-sharded-replica-cfg-1 to be ready.OK waiting for pod/cross-site-sharded-replica-cfg-2 to be ready.OK Waiting for cluster readyness ----------------------------------------------------------------------------------- check failover status ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- Failover check finished successfully ----------------------------------------------------------------------------------- waiting for cluster readyness ----------------------------------------------------------------------------------- destroy cluster/operator and all other resources ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- get and delete old CRDs and RBAC ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io "perconaservermongodbbackups.psmdb.percona.com" deleted customresourcedefinition.apiextensions.k8s.io "perconaservermongodbrestores.psmdb.percona.com" deleted customresourcedefinition.apiextensions.k8s.io "perconaservermongodbs.psmdb.percona.com" deleted error: the server doesn't have a resource type "perconaservermongodbbackups" + kubectl patch perconaservermongodbbackups.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbbackups" E0501 02:43:46.802266 5063 memcache.go:287] couldn't get resource list for psmdb.percona.com/v1: the server could not find the requested resource E0501 02:43:46.803251 5063 memcache.go:287] couldn't get resource list for psmdb.percona.com/v1-10-0: the server could not find the requested resource E0501 02:43:46.804327 5063 memcache.go:287] couldn't get resource list for psmdb.percona.com/v1-11-0: the server could not find the requested resource E0501 02:43:46.807675 5063 memcache.go:287] couldn't get resource list for psmdb.percona.com/v1-12-0: the server could not find the requested resource error: the server doesn't have a resource type "perconaservermongodbrestores" + kubectl patch perconaservermongodbrestores.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbrestores" error: the server doesn't have a resource type "perconaservermongodbs" + kubectl patch perconaservermongodbs.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbs" role.rbac.authorization.k8s.io "percona-server-mongodb-operator" deleted serviceaccount "percona-server-mongodb-operator" deleted rolebinding.rbac.authorization.k8s.io "service-account-percona-server-mongodb-operator" deleted ----------------------------------------------------------------------------------- destroy cluster/operator and all other resources ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- get and delete old CRDs and RBAC ----------------------------------------------------------------------------------- error: the server doesn't have a resource type "perconaservermongodbbackups" + kubectl patch perconaservermongodbbackups.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbbackups" error: the server doesn't have a resource type "perconaservermongodbrestores" + kubectl patch perconaservermongodbrestores.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbrestores" error: the server doesn't have a resource type "perconaservermongodbs" + kubectl patch perconaservermongodbs.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbs" namespace "cross-site-sharded-2930" force deleted Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. ----------------------------------------------------------------------------------- test passed -----------------------------------------------------------------------------------