Log: /mnt/jenkins/workspace/cloud-psmdb-operator_PR-1556/e2e-tests/logs/cross-site-sharded.log E0524 14:53:05.096967 4936 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:05.205161 4936 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:05.315241 4936 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:05.421756 4936 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request WARNING: version difference between client (1.30) and server (1.26) exceeds the supported minor version skew of +/-1 WARNING: version difference between client (1.30) and server (1.26) exceeds the supported minor version skew of +/-1 WARNING: version difference between client (1.30) and server (1.26) exceeds the supported minor version skew of +/-1 E0524 14:53:08.853762 5283 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:09.098956 5283 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:10.476176 5405 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:10.639055 5405 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:10.766022 5405 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:24.234845 6749 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:24.556513 6749 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:24.663744 6749 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:27.497365 7167 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:27.607877 7167 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:27.715844 7167 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. ----------------------------------------------------------------------------------- create main cluster ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- get and delete old CRDs and RBAC ----------------------------------------------------------------------------------- E0524 14:53:29.397804 7360 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:29.504001 7360 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:31.283421 7666 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:31.391921 7666 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:31.500527 7666 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:31.609414 7666 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:31.941366 7666 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:32.164110 7666 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:32.282234 7666 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:32.390582 7666 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:32.499217 7666 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request error: the server doesn't have a resource type "perconaservermongodbbackups" + kubectl patch perconaservermongodbbackups.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' E0524 14:53:34.037816 7965 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:34.269977 7965 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:34.377347 7965 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:34.485164 7965 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:34.809674 7965 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:35.029780 7965 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:35.140738 7965 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:35.249038 7965 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:35.362708 7965 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request error: the server doesn't have a resource type "perconaservermongodbbackups" E0524 14:53:36.502204 8281 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:36.726091 8281 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:36.834231 8281 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:36.942318 8281 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:38.182714 8570 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:38.398656 8570 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:38.507298 8570 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:38.615015 8570 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:38.945634 8570 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:39.157736 8570 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:39.268910 8570 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:39.376935 8570 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:39.484828 8570 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request error: the server doesn't have a resource type "perconaservermongodbrestores" + kubectl patch perconaservermongodbrestores.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' E0524 14:53:41.484578 8914 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:41.795709 8914 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:41.902394 8914 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:42.009498 8914 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:42.345385 8914 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:42.552727 8914 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:42.662350 8914 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:42.771417 8914 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:42.878120 8914 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request error: the server doesn't have a resource type "perconaservermongodbrestores" E0524 14:53:44.293938 9288 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:44.538581 9288 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:44.646540 9288 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0524 14:53:44.754069 9288 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request error: the server doesn't have a resource type "perconaservermongodbs" + kubectl patch perconaservermongodbs.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbs" Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found ----------------------------------------------------------------------------------- destroy chaos-mesh ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces cross-site-sharded-16775 ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- create namespace cross-site-sharded-16775 ----------------------------------------------------------------------------------- namespace/cross-site-sharded-16775 created Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-1556-43640f06-2-cluster4" modified. ----------------------------------------------------------------------------------- start PSMDB operator ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io/perconaservermongodbbackups.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbrestores.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbs.psmdb.percona.com serverside-applied role.rbac.authorization.k8s.io/percona-server-mongodb-operator created serviceaccount/percona-server-mongodb-operator created rolebinding.rbac.authorization.k8s.io/service-account-percona-server-mongodb-operator created deployment.apps/percona-server-mongodb-operator created waiting for pod/percona-server-mongodb-operator-8df647d98-plpz6 to be ready.OK ----------------------------------------------------------------------------------- create secrets and start client ----------------------------------------------------------------------------------- deployment.apps/psmdb-client created secret/cross-site-sharded-main-secrets created secret/cross-site-sharded-main-ssl created secret/cross-site-sharded-main-ssl-internal created ----------------------------------------------------------------------------------- create main PSMDB cluster cross-site-sharded-main. ----------------------------------------------------------------------------------- perconaservermongodb.psmdb.percona.com/cross-site-sharded-main created ----------------------------------------------------------------------------------- check if all 3 Pods started ----------------------------------------------------------------------------------- waiting for pod/cross-site-sharded-main-rs0-0 to be ready...................OK waiting for pod/cross-site-sharded-main-rs0-1 to be ready..............OK waiting for pod/cross-site-sharded-main-rs0-2 to be ready..............OK Waiting for cluster readyness...................................... waiting for pod/cross-site-sharded-main-cfg-0 to be ready.OK waiting for pod/cross-site-sharded-main-cfg-1 to be ready.OK waiting for pod/cross-site-sharded-main-cfg-2 to be ready.OK ----------------------------------------------------------------------------------- create user ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://cross-site-sharded-main-mongos.cross-site-sharded-16775.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("e6636ada-a2cf-4195-8f8a-519b21b52e35") } Percona Server for MongoDB server version: v7.0.8-5 WARNING: shell and server versions do not match Successfully added user: { "user" : "user", "roles" : [ { "db" : "app", "role" : "readWrite" } ] } bye ----------------------------------------------------------------------------------- set chunk size to 2 MB ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://cross-site-sharded-main-mongos.cross-site-sharded-16775.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("de90c64b-0bde-4957-9756-579b49154ec3") } Percona Server for MongoDB server version: v7.0.8-5 WARNING: shell and server versions do not match switched to db config WriteResult({ "nMatched" : 0, "nUpserted" : 1, "nModified" : 0, "_id" : "chunksize" }) bye ----------------------------------------------------------------------------------- write data ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://cross-site-sharded-main-mongos.cross-site-sharded-16775.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("d1ba0d97-2786-4eb1-91d2-99eae9dd35de") } Percona Server for MongoDB server version: v7.0.8-5 WARNING: shell and server versions do not match ----------------------------------------------------------------------------------- shard collection ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://cross-site-sharded-main-mongos.cross-site-sharded-16775.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("300de3a6-085d-4c8e-92a4-d8f3a975b1af") } Percona Server for MongoDB server version: v7.0.8-5 WARNING: shell and server versions do not match { "ok" : 1, "$clusterTime" : { "clusterTime" : Timestamp(1716562765, 1), "signature" : { "hash" : BinData(0,"SIp7Z3Mxg0YsrLdZpsMEa42oJeo="), "keyId" : NumberLong("7372580112572612630") } }, "operationTime" : Timestamp(1716562765, 1) } bye Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://cross-site-sharded-main-mongos.cross-site-sharded-16775.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("fdd2bb32-c993-4b09-ab71-e143716f5b98") } Percona Server for MongoDB server version: v7.0.8-5 WARNING: shell and server versions do not match { "collectionsharded" : "app.city", "ok" : 1, "$clusterTime" : { "clusterTime" : Timestamp(1716562772, 21), "signature" : { "hash" : BinData(0,"LMcX/Mex1+3Myi0pZpSNnBpM08U="), "keyId" : NumberLong("7372580112572612630") } }, "operationTime" : Timestamp(1716562772, 21) } bye ----------------------------------------------------------------------------------- Check chunks ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- rs0 has 1 chunks ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- rs1 has 3 chunks ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- create replica cluster ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces cross-site-sharded-replica-14829 ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- create namespace cross-site-sharded-replica-14829 ----------------------------------------------------------------------------------- namespace/cross-site-sharded-replica-14829 created Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-1556-43640f06-2-cluster4" modified. ----------------------------------------------------------------------------------- start PSMDB operator ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io/perconaservermongodbbackups.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbrestores.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbs.psmdb.percona.com serverside-applied role.rbac.authorization.k8s.io/percona-server-mongodb-operator created serviceaccount/percona-server-mongodb-operator created rolebinding.rbac.authorization.k8s.io/service-account-percona-server-mongodb-operator created deployment.apps/percona-server-mongodb-operator created waiting for pod/percona-server-mongodb-operator-8df647d98-xqfjn to be ready.OK ----------------------------------------------------------------------------------- start client ----------------------------------------------------------------------------------- deployment.apps/psmdb-client created ----------------------------------------------------------------------------------- copy secrets from main to replica namespace and create all of them ----------------------------------------------------------------------------------- secret/cross-site-sharded-replica-secrets created secret/cross-site-sharded-replica-ssl-internal created secret/cross-site-sharded-replica-ssl created ----------------------------------------------------------------------------------- create replica PSMDB cluster ----------------------------------------------------------------------------------- perconaservermongodb.psmdb.percona.com/cross-site-sharded-replica created waiting for pod/cross-site-sharded-replica-rs0-0 to be ready.............OK waiting for pod/cross-site-sharded-replica-rs0-1 to be ready...............OK waiting for pod/cross-site-sharded-replica-rs0-2 to be ready...............OK waiting for pod/cross-site-sharded-replica-rs1-0 to be ready.OK waiting for pod/cross-site-sharded-replica-rs1-1 to be ready.OK waiting for pod/cross-site-sharded-replica-rs1-2 to be ready.OK waiting for pod/cross-site-sharded-replica-cfg-0 to be ready.OK waiting for pod/cross-site-sharded-replica-cfg-1 to be ready.OK waiting for pod/cross-site-sharded-replica-cfg-2 to be ready.OK Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-1556-43640f06-2-cluster4" modified. perconaservermongodb.psmdb.percona.com/cross-site-sharded-main patched error: cannot exec into a container in a completed pod; current phase is Failed error: cannot exec into a container in a completed pod; current phase is Failed error: cannot exec into a container in a completed pod; current phase is Failed error: cannot exec into a container in a completed pod; current phase is Failed waiting for all members to be configured in cfg .error: cannot exec into a container in a completed pod; current phase is Failed error: cannot exec into a container in a completed pod; current phase is Failed error: cannot exec into a container in a completed pod; current phase is Failed error: cannot exec into a container in a completed pod; current phase is Failed waiting for all members to be configured in cfg .waiting for all members to be configured in cfg .waiting for all members to be configured in cfg .waiting for all members to be configured in cfg .waiting for all members to be configured in cfg .waiting for all members to be configured in cfg .waiting for all members to be configured in rs0 .waiting for all members to be configured in rs1 .Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-1556-43640f06-2-cluster4" modified. ----------------------------------------------------------------------------------- check if all 3 Pods started ----------------------------------------------------------------------------------- waiting for pod/cross-site-sharded-replica-rs0-0 to be ready.OK waiting for pod/cross-site-sharded-replica-rs0-1 to be ready.OK waiting for pod/cross-site-sharded-replica-rs0-2 to be ready.OK Waiting for cluster readyness waiting for pod/cross-site-sharded-replica-cfg-0 to be ready.OK waiting for pod/cross-site-sharded-replica-cfg-1 to be ready.OK waiting for pod/cross-site-sharded-replica-cfg-2 to be ready.OK ----------------------------------------------------------------------------------- create user ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://cross-site-sharded-main-mongos.cross-site-sharded-16775.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("c11bb339-656b-48c7-8918-a2e2b59ed734") } Percona Server for MongoDB server version: v7.0.8-5 WARNING: shell and server versions do not match Successfully added user: { "user" : "myApp", "roles" : [ { "db" : "myApp", "role" : "readWrite" } ] } bye ----------------------------------------------------------------------------------- write data, read from all ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://cross-site-sharded-main-mongos.cross-site-sharded-16775.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("d0321d50-409b-4280-bef9-c69c84c7464e") } Percona Server for MongoDB server version: v7.0.8-5 WARNING: shell and server versions do not match switched to db myApp WriteResult({ "nInserted" : 1 }) bye ----------------------------------------------------------------------------------- Compare data ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- test failover ----------------------------------------------------------------------------------- Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-1556-43640f06-2-cluster4" modified. perconaservermongodb.psmdb.percona.com "cross-site-sharded-main" deleted ----------------------------------------------------------------------------------- run disaster recovery script for replset: cfg ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://10.184.84.142:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("442df7dd-7be8-4508-9958-a5bc13e70715") } Percona Server for MongoDB server version: v7.0.8-5 WARNING: shell and server versions do not match ----------------------------------------------------------------------------------- run disaster recovery script for replset: rs0 ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://10.184.93.225:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("52d591c1-e9d8-4e86-80b0-4d94b00ecaee") } Percona Server for MongoDB server version: v7.0.8-5 WARNING: shell and server versions do not match ----------------------------------------------------------------------------------- run disaster recovery script for replset: rs1 ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://10.184.85.35:27017/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("2274d539-0caf-45e1-9213-f0a30c0fe65c") } Percona Server for MongoDB server version: v7.0.8-5 WARNING: shell and server versions do not match ----------------------------------------------------------------------------------- make replica cluster managed ----------------------------------------------------------------------------------- Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-1556-43640f06-2-cluster4" modified. perconaservermongodb.psmdb.percona.com/cross-site-sharded-replica patched waiting for pod/cross-site-sharded-replica-rs0-0 to be ready.OK waiting for pod/cross-site-sharded-replica-rs0-1 to be ready.OK waiting for pod/cross-site-sharded-replica-rs0-2 to be ready.OK Waiting for cluster readyness waiting for pod/cross-site-sharded-replica-cfg-0 to be ready.OK waiting for pod/cross-site-sharded-replica-cfg-1 to be ready.OK waiting for pod/cross-site-sharded-replica-cfg-2 to be ready.OK Waiting for cluster readyness ----------------------------------------------------------------------------------- check failover status ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- Failover check finished successfully ----------------------------------------------------------------------------------- waiting for cluster readyness ----------------------------------------------------------------------------------- destroy cluster/operator and all other resources ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- get and delete old CRDs and RBAC ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io "perconaservermongodbbackups.psmdb.percona.com" deleted customresourcedefinition.apiextensions.k8s.io "perconaservermongodbrestores.psmdb.percona.com" deleted customresourcedefinition.apiextensions.k8s.io "perconaservermongodbs.psmdb.percona.com" deleted error: the server doesn't have a resource type "perconaservermongodbbackups" + kubectl patch perconaservermongodbbackups.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbbackups" error: the server doesn't have a resource type "perconaservermongodbrestores" + kubectl patch perconaservermongodbrestores.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbrestores" error: the server doesn't have a resource type "perconaservermongodbs" + kubectl patch perconaservermongodbs.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbs" role.rbac.authorization.k8s.io "percona-server-mongodb-operator" deleted serviceaccount "percona-server-mongodb-operator" deleted rolebinding.rbac.authorization.k8s.io "service-account-percona-server-mongodb-operator" deleted ----------------------------------------------------------------------------------- destroy cluster/operator and all other resources ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- get and delete old CRDs and RBAC ----------------------------------------------------------------------------------- error: the server doesn't have a resource type "perconaservermongodbbackups" + kubectl patch perconaservermongodbbackups.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbbackups" error: the server doesn't have a resource type "perconaservermongodbrestores" + kubectl patch perconaservermongodbrestores.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbrestores" error: the server doesn't have a resource type "perconaservermongodbs" + kubectl patch perconaservermongodbs.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbs" namespace "cross-site-sharded-16775" force deleted Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. ----------------------------------------------------------------------------------- test passed -----------------------------------------------------------------------------------