Log: /mnt/jenkins/workspace/cloud-psmdb-operator_PR-1543/e2e-tests/logs/cross-site-sharded.log E0507 10:57:48.076619 30352 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:57:48.316904 30352 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:57:48.438708 30352 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:57:48.558787 30352 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request WARNING: version difference between client (1.30) and server (1.26) exceeds the supported minor version skew of +/-1 WARNING: version difference between client (1.30) and server (1.26) exceeds the supported minor version skew of +/-1 WARNING: version difference between client (1.30) and server (1.26) exceeds the supported minor version skew of +/-1 E0507 10:57:52.637900 30973 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:57:52.967846 30973 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:57:54.365971 31232 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:57:54.600957 31232 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:57:54.713591 31232 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:07.254314 32444 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:07.367074 32444 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:07.478675 32444 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:10.146383 32759 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:10.258653 32759 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:10.371514 32759 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. ----------------------------------------------------------------------------------- create main cluster ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- get and delete old CRDs and RBAC ----------------------------------------------------------------------------------- E0507 10:58:12.327994 528 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:12.655686 528 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:14.759721 884 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:15.094652 884 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:15.206877 884 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:15.319009 884 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:15.661891 884 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:15.886561 884 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:16.003549 884 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:16.116543 884 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:16.228683 884 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request error: the server doesn't have a resource type "perconaservermongodbbackups" + kubectl patch perconaservermongodbbackups.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' E0507 10:58:17.445201 1169 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:17.770253 1169 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:17.881332 1169 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:18.278818 1169 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:18.802575 1169 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:19.078666 1169 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:19.259621 1169 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:19.370055 1169 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:19.480910 1169 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request error: the server doesn't have a resource type "perconaservermongodbbackups" E0507 10:58:20.722416 1505 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:20.834116 1505 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:20.944785 1505 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:21.055515 1505 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:22.604514 1790 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:22.928819 1790 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:23.040151 1790 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:23.151003 1790 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:23.603311 1790 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:23.718664 1790 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:23.832234 1790 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:23.943129 1790 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:24.055036 1790 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request error: the server doesn't have a resource type "perconaservermongodbrestores" + kubectl patch perconaservermongodbrestores.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' E0507 10:58:25.233930 2226 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:25.563482 2226 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:25.675525 2226 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:25.786891 2226 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:26.131452 2226 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:26.345868 2226 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:26.460213 2226 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:26.571659 2226 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:26.683366 2226 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request error: the server doesn't have a resource type "perconaservermongodbrestores" E0507 10:58:27.792662 2486 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:28.123101 2486 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:28.237129 2486 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:28.349243 2486 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:29.720038 2619 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:29.963024 2619 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:30.076605 2619 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:30.190053 2619 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:30.549217 2619 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:30.760515 2619 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:30.875820 2619 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:30.989212 2619 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:31.102677 2619 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request error: the server doesn't have a resource type "perconaservermongodbs" + kubectl patch perconaservermongodbs.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' E0507 10:58:32.184244 2822 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:32.516626 2822 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:32.630885 2822 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:32.743742 2822 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:33.328590 2822 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:33.578328 2822 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:33.696534 2822 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:33.831021 2822 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:33.943087 2822 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request error: the server doesn't have a resource type "perconaservermongodbs" E0507 10:58:35.377575 3175 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:35.708598 3175 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:35.821300 3175 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:35.933638 3175 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:37.329794 3408 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0507 10:58:37.560761 3408 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found ----------------------------------------------------------------------------------- destroy chaos-mesh ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces cross-site-sharded-30626 ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- create namespace cross-site-sharded-30626 ----------------------------------------------------------------------------------- namespace/cross-site-sharded-30626 created Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-1543-c8a64a78-9-cluster4" modified. ----------------------------------------------------------------------------------- start PSMDB operator ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io/perconaservermongodbbackups.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbrestores.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbs.psmdb.percona.com serverside-applied role.rbac.authorization.k8s.io/percona-server-mongodb-operator created serviceaccount/percona-server-mongodb-operator created rolebinding.rbac.authorization.k8s.io/service-account-percona-server-mongodb-operator created deployment.apps/percona-server-mongodb-operator created waiting for pod/percona-server-mongodb-operator-55f6d87c8f-mcnnb to be ready.OK ----------------------------------------------------------------------------------- create secrets and start client ----------------------------------------------------------------------------------- deployment.apps/psmdb-client created secret/cross-site-sharded-main-secrets created secret/cross-site-sharded-main-ssl created secret/cross-site-sharded-main-ssl-internal created ----------------------------------------------------------------------------------- create main PSMDB cluster cross-site-sharded-main. ----------------------------------------------------------------------------------- perconaservermongodb.psmdb.percona.com/cross-site-sharded-main created ----------------------------------------------------------------------------------- check if all 3 Pods started ----------------------------------------------------------------------------------- waiting for pod/cross-site-sharded-main-rs0-0 to be ready...................OK waiting for pod/cross-site-sharded-main-rs0-1 to be ready.............OK waiting for pod/cross-site-sharded-main-rs0-2 to be ready.............OK Waiting for cluster readyness......................................... waiting for pod/cross-site-sharded-main-cfg-0 to be ready.OK waiting for pod/cross-site-sharded-main-cfg-1 to be ready.OK waiting for pod/cross-site-sharded-main-cfg-2 to be ready.OK ----------------------------------------------------------------------------------- create user ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://cross-site-sharded-main-mongos.cross-site-sharded-30626.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("b62b142d-ba41-47e8-9d12-bf295a16b1a7") } Percona Server for MongoDB server version: v7.0.8-5 WARNING: shell and server versions do not match Successfully added user: { "user" : "user", "roles" : [ { "db" : "app", "role" : "readWrite" } ] } bye ----------------------------------------------------------------------------------- set chunk size to 2 MB ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://cross-site-sharded-main-mongos.cross-site-sharded-30626.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("e8818412-d302-46d8-9362-a9b0278e8595") } Percona Server for MongoDB server version: v7.0.8-5 WARNING: shell and server versions do not match switched to db config WriteResult({ "nMatched" : 0, "nUpserted" : 1, "nModified" : 0, "_id" : "chunksize" }) bye ----------------------------------------------------------------------------------- write data ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://cross-site-sharded-main-mongos.cross-site-sharded-30626.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("d63ffc6d-679e-4a54-891f-1a768a30a58b") } Percona Server for MongoDB server version: v7.0.8-5 WARNING: shell and server versions do not match ----------------------------------------------------------------------------------- shard collection ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://cross-site-sharded-main-mongos.cross-site-sharded-30626.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("9b55ce85-c346-4673-8d29-965829338f32") } Percona Server for MongoDB server version: v7.0.8-5 WARNING: shell and server versions do not match { "ok" : 1, "$clusterTime" : { "clusterTime" : Timestamp(1715079849, 1), "signature" : { "hash" : BinData(0,"hNz/hLqCIsnMYOhtXt2kCuX6mkI="), "keyId" : NumberLong("7366211079799570441") } }, "operationTime" : Timestamp(1715079849, 1) } bye Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://cross-site-sharded-main-mongos.cross-site-sharded-30626.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("d9ae36cd-af62-45d7-a6e3-7cfccfe087c5") } Percona Server for MongoDB server version: v7.0.8-5 WARNING: shell and server versions do not match { "collectionsharded" : "app.city", "ok" : 1, "$clusterTime" : { "clusterTime" : Timestamp(1715079854, 35), "signature" : { "hash" : BinData(0,"DvNSsm1JdgbpIM8mopKhJ18tlyY="), "keyId" : NumberLong("7366211079799570441") } }, "operationTime" : Timestamp(1715079854, 35) } bye ----------------------------------------------------------------------------------- Check chunks ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- rs0 has 1 chunks ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- rs1 has 3 chunks ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- create replica cluster ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces cross-site-sharded-replica-19235 ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- create namespace cross-site-sharded-replica-19235 ----------------------------------------------------------------------------------- namespace/cross-site-sharded-replica-19235 created Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-1543-c8a64a78-9-cluster4" modified. ----------------------------------------------------------------------------------- start PSMDB operator ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io/perconaservermongodbbackups.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbrestores.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbs.psmdb.percona.com serverside-applied role.rbac.authorization.k8s.io/percona-server-mongodb-operator created serviceaccount/percona-server-mongodb-operator created rolebinding.rbac.authorization.k8s.io/service-account-percona-server-mongodb-operator created deployment.apps/percona-server-mongodb-operator created waiting for pod/percona-server-mongodb-operator-55f6d87c8f-2jnhw to be ready.OK ----------------------------------------------------------------------------------- start client ----------------------------------------------------------------------------------- deployment.apps/psmdb-client created ----------------------------------------------------------------------------------- copy secrets from main to replica namespace and create all of them ----------------------------------------------------------------------------------- secret/cross-site-sharded-replica-secrets created secret/cross-site-sharded-replica-ssl-internal created secret/cross-site-sharded-replica-ssl created ----------------------------------------------------------------------------------- create replica PSMDB cluster ----------------------------------------------------------------------------------- perconaservermongodb.psmdb.percona.com/cross-site-sharded-replica created waiting for pod/cross-site-sharded-replica-rs0-0 to be ready..............OK waiting for pod/cross-site-sharded-replica-rs0-1 to be ready..............OK waiting for pod/cross-site-sharded-replica-rs0-2 to be ready.............OK waiting for pod/cross-site-sharded-replica-rs1-0 to be ready.OK waiting for pod/cross-site-sharded-replica-rs1-1 to be ready.OK waiting for pod/cross-site-sharded-replica-rs1-2 to be ready.OK waiting for pod/cross-site-sharded-replica-cfg-0 to be ready.OK waiting for pod/cross-site-sharded-replica-cfg-1 to be ready.OK waiting for pod/cross-site-sharded-replica-cfg-2 to be ready.OK Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-1543-c8a64a78-9-cluster4" modified. perconaservermongodb.psmdb.percona.com/cross-site-sharded-main patched waiting for all members to be configured in cfg .waiting for all members to be configured in cfg .waiting for all members to be configured in rs0 .waiting for all members to be configured in rs1 .Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-1543-c8a64a78-9-cluster4" modified. ----------------------------------------------------------------------------------- check if all 3 Pods started ----------------------------------------------------------------------------------- waiting for pod/cross-site-sharded-replica-rs0-0 to be ready.OK waiting for pod/cross-site-sharded-replica-rs0-1 to be ready.OK waiting for pod/cross-site-sharded-replica-rs0-2 to be ready.OK Waiting for cluster readyness waiting for pod/cross-site-sharded-replica-cfg-0 to be ready.OK waiting for pod/cross-site-sharded-replica-cfg-1 to be ready.OK waiting for pod/cross-site-sharded-replica-cfg-2 to be ready.OK ----------------------------------------------------------------------------------- create user ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://cross-site-sharded-main-mongos.cross-site-sharded-30626.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("29c0149a-2f21-4257-9f83-929e6f3e2b00") } Percona Server for MongoDB server version: v7.0.8-5 WARNING: shell and server versions do not match Successfully added user: { "user" : "myApp", "roles" : [ { "db" : "myApp", "role" : "readWrite" } ] } bye ----------------------------------------------------------------------------------- write data, read from all ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://cross-site-sharded-main-mongos.cross-site-sharded-30626.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("35eb9542-a3a7-4086-b2e0-125cd9eb2b17") } Percona Server for MongoDB server version: v7.0.8-5 WARNING: shell and server versions do not match switched to db myApp WriteResult({ "nInserted" : 1 }) bye ----------------------------------------------------------------------------------- Compare data ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- test failover ----------------------------------------------------------------------------------- Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-1543-c8a64a78-9-cluster4" modified. perconaservermongodb.psmdb.percona.com "cross-site-sharded-main" deleted ----------------------------------------------------------------------------------- run disaster recovery script for replset: cfg ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://10.216.185.14:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("1bf0f266-6a0a-475a-a629-bc40c5cd656d") } Percona Server for MongoDB server version: v7.0.8-5 WARNING: shell and server versions do not match ----------------------------------------------------------------------------------- run disaster recovery script for replset: rs0 ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://10.216.182.181:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("6d68dc16-9374-4c55-b87b-41fc42fa871f") } Percona Server for MongoDB server version: v7.0.8-5 WARNING: shell and server versions do not match ----------------------------------------------------------------------------------- run disaster recovery script for replset: rs1 ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://10.216.191.69:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("d0cf2285-e3bf-4d99-a3f3-d993da677010") } Percona Server for MongoDB server version: v7.0.8-5 WARNING: shell and server versions do not match ----------------------------------------------------------------------------------- make replica cluster managed ----------------------------------------------------------------------------------- Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-1543-c8a64a78-9-cluster4" modified. perconaservermongodb.psmdb.percona.com/cross-site-sharded-replica patched waiting for pod/cross-site-sharded-replica-rs0-0 to be ready.OK waiting for pod/cross-site-sharded-replica-rs0-1 to be ready.OK waiting for pod/cross-site-sharded-replica-rs0-2 to be ready.OK Waiting for cluster readyness waiting for pod/cross-site-sharded-replica-cfg-0 to be ready.OK waiting for pod/cross-site-sharded-replica-cfg-1 to be ready.OK waiting for pod/cross-site-sharded-replica-cfg-2 to be ready.OK Waiting for cluster readyness ----------------------------------------------------------------------------------- check failover status ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- Failover check finished successfully ----------------------------------------------------------------------------------- waiting for cluster readyness ----------------------------------------------------------------------------------- destroy cluster/operator and all other resources ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- get and delete old CRDs and RBAC ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io "perconaservermongodbbackups.psmdb.percona.com" deleted customresourcedefinition.apiextensions.k8s.io "perconaservermongodbrestores.psmdb.percona.com" deleted customresourcedefinition.apiextensions.k8s.io "perconaservermongodbs.psmdb.percona.com" deleted error: the server doesn't have a resource type "perconaservermongodbbackups" + kubectl patch perconaservermongodbbackups.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbbackups" error: the server doesn't have a resource type "perconaservermongodbrestores" + kubectl patch perconaservermongodbrestores.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbrestores" error: the server doesn't have a resource type "perconaservermongodbs" + kubectl patch perconaservermongodbs.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbs" role.rbac.authorization.k8s.io "percona-server-mongodb-operator" deleted serviceaccount "percona-server-mongodb-operator" deleted rolebinding.rbac.authorization.k8s.io "service-account-percona-server-mongodb-operator" deleted ----------------------------------------------------------------------------------- destroy cluster/operator and all other resources ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- get and delete old CRDs and RBAC ----------------------------------------------------------------------------------- error: the server doesn't have a resource type "perconaservermongodbbackups" + kubectl patch perconaservermongodbbackups.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbbackups" error: the server doesn't have a resource type "perconaservermongodbrestores" + kubectl patch perconaservermongodbrestores.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbrestores" error: the server doesn't have a resource type "perconaservermongodbs" + kubectl patch perconaservermongodbs.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbs" namespace "cross-site-sharded-30626" force deleted Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. ----------------------------------------------------------------------------------- test passed -----------------------------------------------------------------------------------