Log: /mnt/jenkins/workspace/cloud-psmdb-operator_PR-1377/e2e-tests/logs/cross-site-sharded.log E0127 16:10:31.943868 25199 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:10:32.058984 25199 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:10:32.174669 25199 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:10:32.283825 25199 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request WARNING: version difference between client (1.29) and server (1.25) exceeds the supported minor version skew of +/-1 WARNING: version difference between client (1.29) and server (1.25) exceeds the supported minor version skew of +/-1 WARNING: version difference between client (1.29) and server (1.25) exceeds the supported minor version skew of +/-1 E0127 16:10:36.562225 25538 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:10:36.739006 25538 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:10:38.360968 25691 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:10:38.583250 25691 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:10:38.693111 25691 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:10:48.636890 26680 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:10:48.970396 26680 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:10:49.082081 26680 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:10:51.584917 27035 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:10:52.114568 27035 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:10:52.243231 27035 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. ----------------------------------------------------------------------------------- create main cluster ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- get and delete old CRDs and RBAC ----------------------------------------------------------------------------------- E0127 16:10:53.899939 27225 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:10:54.222027 27225 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:10:56.069869 27398 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:10:56.397418 27398 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:10:56.507721 27398 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:10:56.617942 27398 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:10:56.967608 27398 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:10:57.173889 27398 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:10:57.291997 27398 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:10:57.404067 27398 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:10:57.513786 27398 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request error: the server doesn't have a resource type "perconaservermongodbbackups" + kubectl patch perconaservermongodbbackups.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' E0127 16:10:59.062147 27616 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:10:59.291884 27616 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:10:59.401859 27616 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:10:59.511489 27616 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:10:59.850323 27616 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:00.064254 27616 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:00.178890 27616 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:00.288998 27616 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:00.399121 27616 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request error: the server doesn't have a resource type "perconaservermongodbbackups" E0127 16:11:01.813083 27930 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:02.045312 27930 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:02.157074 27930 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:02.270069 27930 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:04.096689 28153 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:04.331628 28153 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:04.441641 28153 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:04.553034 28153 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:04.900939 28153 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:05.101878 28153 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:05.216096 28153 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:05.326641 28153 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:05.436675 28153 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request error: the server doesn't have a resource type "perconaservermongodbrestores" + kubectl patch perconaservermongodbrestores.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' E0127 16:11:07.143239 28476 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:07.471186 28476 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:07.583092 28476 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:07.694404 28476 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:08.038507 28476 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:08.256582 28476 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:08.371256 28476 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:08.482965 28476 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:08.598698 28476 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request error: the server doesn't have a resource type "perconaservermongodbrestores" E0127 16:11:10.517247 28844 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:10.779990 28844 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:10.890799 28844 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:11.001195 28844 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:12.711183 29028 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:12.950458 29028 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:13.062055 29028 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:13.173441 29028 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:13.517756 29028 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:13.731727 29028 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:13.846224 29028 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:13.959934 29028 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:14.071497 29028 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request error: the server doesn't have a resource type "perconaservermongodbs" + kubectl patch perconaservermongodbs.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' E0127 16:11:16.018254 29402 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:16.139157 29402 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:16.250200 29402 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:16.361583 29402 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:16.703688 29402 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:16.920728 29402 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:17.034734 29402 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:17.145688 29402 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:17.256619 29402 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request error: the server doesn't have a resource type "perconaservermongodbs" E0127 16:11:19.047222 29757 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:19.272383 29757 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:19.383016 29757 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:19.493392 29757 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:21.430596 30079 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:21.790796 30079 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:23.965768 30408 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:24.193440 30408 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0127 16:11:24.303945 30408 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found ----------------------------------------------------------------------------------- destroy chaos-mesh ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces cross-site-sharded-13421 ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- create namespace cross-site-sharded-13421 ----------------------------------------------------------------------------------- namespace/cross-site-sharded-13421 created Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-1377-eeba1539-6-cluster4" modified. ----------------------------------------------------------------------------------- start PSMDB operator ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io/perconaservermongodbbackups.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbrestores.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbs.psmdb.percona.com serverside-applied role.rbac.authorization.k8s.io/percona-server-mongodb-operator created serviceaccount/percona-server-mongodb-operator created rolebinding.rbac.authorization.k8s.io/service-account-percona-server-mongodb-operator created deployment.apps/percona-server-mongodb-operator created waiting for pod/percona-server-mongodb-operator-66b4f875cc-h9lkc to be ready..OK ----------------------------------------------------------------------------------- create secrets and start client ----------------------------------------------------------------------------------- deployment.apps/psmdb-client created secret/cross-site-sharded-main-secrets created secret/cross-site-sharded-main-ssl created secret/cross-site-sharded-main-ssl-internal created ----------------------------------------------------------------------------------- create main PSMDB cluster cross-site-sharded-main. ----------------------------------------------------------------------------------- perconaservermongodb.psmdb.percona.com/cross-site-sharded-main created ----------------------------------------------------------------------------------- check if all 3 Pods started ----------------------------------------------------------------------------------- waiting for pod/cross-site-sharded-main-rs0-0 to be ready..................OK waiting for pod/cross-site-sharded-main-rs0-1 to be ready...........OK waiting for pod/cross-site-sharded-main-rs0-2 to be ready..........OK Waiting for cluster readyness................ waiting for pod/cross-site-sharded-main-cfg-0 to be ready.OK waiting for pod/cross-site-sharded-main-cfg-1 to be ready.OK waiting for pod/cross-site-sharded-main-cfg-2 to be ready.OK ----------------------------------------------------------------------------------- create user ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.27-26 connecting to: mongodb://cross-site-sharded-main-mongos.cross-site-sharded-13421.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("de6eab7a-4c3e-4aae-9237-617e476b935f") } Percona Server for MongoDB server version: v5.0.23-20 WARNING: shell and server versions do not match Successfully added user: { "user" : "user", "roles" : [ { "db" : "app", "role" : "readWrite" } ] } bye ----------------------------------------------------------------------------------- set chunk size to 2 MB ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.27-26 connecting to: mongodb://cross-site-sharded-main-mongos.cross-site-sharded-13421.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("ee21f502-386d-4023-a1cb-414bcfb71ac5") } Percona Server for MongoDB server version: v5.0.23-20 WARNING: shell and server versions do not match switched to db config WriteResult({ "nMatched" : 0, "nUpserted" : 1, "nModified" : 0, "_id" : "chunksize" }) bye ----------------------------------------------------------------------------------- write data ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.27-26 connecting to: mongodb://cross-site-sharded-main-mongos.cross-site-sharded-13421.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("e5701a1e-a1d2-4719-9a60-af196f1c914d") } Percona Server for MongoDB server version: v5.0.23-20 WARNING: shell and server versions do not match ----------------------------------------------------------------------------------- shard collection ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.27-26 connecting to: mongodb://cross-site-sharded-main-mongos.cross-site-sharded-13421.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("054f5cdd-faba-4953-aec4-de206faf7ecf") } Percona Server for MongoDB server version: v5.0.23-20 WARNING: shell and server versions do not match { "ok" : 1, "$clusterTime" : { "clusterTime" : Timestamp(1706372175, 1), "signature" : { "hash" : BinData(0,"d9EQnJAPgE1s8FIiQFYhGYM/qCc="), "keyId" : NumberLong("7328811999234621464") } }, "operationTime" : Timestamp(1706372175, 1) } bye Percona Server for MongoDB shell version v4.4.27-26 connecting to: mongodb://cross-site-sharded-main-mongos.cross-site-sharded-13421.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("db381b77-4749-47e0-a5a7-68ffb7bb420e") } Percona Server for MongoDB server version: v5.0.23-20 WARNING: shell and server versions do not match { "collectionsharded" : "app.city", "ok" : 1, "$clusterTime" : { "clusterTime" : Timestamp(1706372181, 35), "signature" : { "hash" : BinData(0,"VW+Y7kktdGfy1qp79k8AXOqpK8c="), "keyId" : NumberLong("7328811999234621464") } }, "operationTime" : Timestamp(1706372181, 31) } bye ----------------------------------------------------------------------------------- Check chunks ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- rs0 has 4 chunks ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- rs1 has 5 chunks ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- create replica cluster ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces cross-site-sharded-replica-5427 ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- create namespace cross-site-sharded-replica-5427 ----------------------------------------------------------------------------------- namespace/cross-site-sharded-replica-5427 created Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-1377-eeba1539-6-cluster4" modified. ----------------------------------------------------------------------------------- start PSMDB operator ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io/perconaservermongodbbackups.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbrestores.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbs.psmdb.percona.com serverside-applied role.rbac.authorization.k8s.io/percona-server-mongodb-operator created serviceaccount/percona-server-mongodb-operator created rolebinding.rbac.authorization.k8s.io/service-account-percona-server-mongodb-operator created deployment.apps/percona-server-mongodb-operator created waiting for pod/percona-server-mongodb-operator-66b4f875cc-xgcrb to be ready.OK ----------------------------------------------------------------------------------- start client ----------------------------------------------------------------------------------- deployment.apps/psmdb-client created ----------------------------------------------------------------------------------- copy secrets from main to replica namespace and create all of them ----------------------------------------------------------------------------------- secret/cross-site-sharded-replica-secrets created secret/cross-site-sharded-replica-ssl-internal created secret/cross-site-sharded-replica-ssl created ----------------------------------------------------------------------------------- create replica PSMDB cluster ----------------------------------------------------------------------------------- perconaservermongodb.psmdb.percona.com/cross-site-sharded-replica created Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-1377-eeba1539-6-cluster4" modified. perconaservermongodb.psmdb.percona.com/cross-site-sharded-main patched Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-1377-eeba1539-6-cluster4" modified. ----------------------------------------------------------------------------------- check if all 3 Pods started ----------------------------------------------------------------------------------- waiting for pod/cross-site-sharded-replica-rs0-0 to be ready.OK waiting for pod/cross-site-sharded-replica-rs0-1 to be ready.OK waiting for pod/cross-site-sharded-replica-rs0-2 to be ready.OK Waiting for cluster readyness waiting for pod/cross-site-sharded-replica-cfg-0 to be ready.OK waiting for pod/cross-site-sharded-replica-cfg-1 to be ready.OK waiting for pod/cross-site-sharded-replica-cfg-2 to be ready.OK ----------------------------------------------------------------------------------- create user ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.27-26 connecting to: mongodb://cross-site-sharded-main-mongos.cross-site-sharded-13421.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("e85ec264-7d7a-48d7-a4c1-5fd01826ae9d") } Percona Server for MongoDB server version: v5.0.23-20 WARNING: shell and server versions do not match Successfully added user: { "user" : "myApp", "roles" : [ { "db" : "myApp", "role" : "readWrite" } ] } bye ----------------------------------------------------------------------------------- write data, read from all ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.27-26 connecting to: mongodb://cross-site-sharded-main-mongos.cross-site-sharded-13421.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("ccc74fa4-9f3a-4445-8aaf-625cecf9f544") } Percona Server for MongoDB server version: v5.0.23-20 WARNING: shell and server versions do not match switched to db myApp WriteResult({ "nInserted" : 1 }) bye ----------------------------------------------------------------------------------- Compare data ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- test failover ----------------------------------------------------------------------------------- Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-1377-eeba1539-6-cluster4" modified. perconaservermongodb.psmdb.percona.com "cross-site-sharded-main" deleted ----------------------------------------------------------------------------------- run disaster recovery script for replset: cfg ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.27-26 connecting to: mongodb://10.67.188.208:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("d5e5c17b-9082-474e-970f-111ae3a9cbb8") } Percona Server for MongoDB server version: v5.0.23-20 WARNING: shell and server versions do not match ----------------------------------------------------------------------------------- run disaster recovery script for replset: rs0 ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.27-26 connecting to: mongodb://10.67.184.22:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("e0fd44a1-d23f-4739-95ef-97ec04589047") } Percona Server for MongoDB server version: v5.0.23-20 WARNING: shell and server versions do not match ----------------------------------------------------------------------------------- run disaster recovery script for replset: rs1 ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.27-26 connecting to: mongodb://10.67.187.143:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("fa04d26e-ef45-4913-b4ce-6acb3c35969a") } Percona Server for MongoDB server version: v5.0.23-20 WARNING: shell and server versions do not match ----------------------------------------------------------------------------------- make replica cluster managed ----------------------------------------------------------------------------------- Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-1377-eeba1539-6-cluster4" modified. perconaservermongodb.psmdb.percona.com/cross-site-sharded-replica patched ----------------------------------------------------------------------------------- check failover status ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- Failover check finished successfully ----------------------------------------------------------------------------------- waiting for cluster readyness ----------------------------------------------------------------------------------- destroy cluster/operator and all other resources ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- get and delete old CRDs and RBAC ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io "perconaservermongodbbackups.psmdb.percona.com" deleted customresourcedefinition.apiextensions.k8s.io "perconaservermongodbrestores.psmdb.percona.com" deleted customresourcedefinition.apiextensions.k8s.io "perconaservermongodbs.psmdb.percona.com" deleted error: the server doesn't have a resource type "perconaservermongodbbackups" + kubectl patch perconaservermongodbbackups.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbbackups" error: the server doesn't have a resource type "perconaservermongodbrestores" + kubectl patch perconaservermongodbrestores.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbrestores" error: the server doesn't have a resource type "perconaservermongodbs" + kubectl patch perconaservermongodbs.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbs" role.rbac.authorization.k8s.io "percona-server-mongodb-operator" deleted serviceaccount "percona-server-mongodb-operator" deleted rolebinding.rbac.authorization.k8s.io "service-account-percona-server-mongodb-operator" deleted ----------------------------------------------------------------------------------- destroy cluster/operator and all other resources ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- get and delete old CRDs and RBAC ----------------------------------------------------------------------------------- error: the server doesn't have a resource type "perconaservermongodbbackups" + kubectl patch perconaservermongodbbackups.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbbackups" error: the server doesn't have a resource type "perconaservermongodbrestores" + kubectl patch perconaservermongodbrestores.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbrestores" error: the server doesn't have a resource type "perconaservermongodbs" + kubectl patch perconaservermongodbs.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbs" namespace "cross-site-sharded-13421" force deleted Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. ----------------------------------------------------------------------------------- test passed -----------------------------------------------------------------------------------