Log: /mnt/jenkins/workspace/cloud-pxc-operator_PR-2467/e2e-tests/logs/proxy-switch-8-0.log Warning: version difference between client (1.36) and server (1.33) exceeds the supported minor version skew of +/-1 Warning: version difference between client (1.36) and server (1.33) exceeds the supported minor version skew of +/-1 No resources found + kubectl patch pxc -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: resource(s) were provided, but no name was specified No resources found No resources found No resources found error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces pxc-operator ----------------------------------------------------------------------------------- Error from server (NotFound): namespaces "pxc-operator" not found waiting for namespace/pxc-operator to be deletederror: resource(s) were provided, but no name was specified Error from server (NotFound): namespaces "pxc-operator" not found ----------------------------------------------------------------------------------- create namespace pxc-operator ----------------------------------------------------------------------------------- namespace/pxc-operator created Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2467-3dc7f023-16-cluster12" modified. ----------------------------------------------------------------------------------- start PXC operator ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterbackups.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterrestores.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusters.pxc.percona.com serverside-applied clusterrole.rbac.authorization.k8s.io/percona-xtradb-cluster-operator unchanged serviceaccount/percona-xtradb-cluster-operator created clusterrolebinding.rbac.authorization.k8s.io/service-account-percona-xtradb-cluster-operator unchanged deployment.apps/percona-xtradb-cluster-operator created service/percona-xtradb-cluster-operator created pod/percona-xtradb-cluster-operator-9d9fbdb5-mrrqh condition met E0516 19:55:28.060985 9121 reflector.go:227] "Failed to watch" err="Get \"https://34.121.115.52/api/v1/namespaces/pxc-operator/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpercona-xtradb-cluster-operator-9d9fbdb5-mrrqh&resourceVersion=1778961327702232000&timeoutSeconds=452&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" pod/percona-xtradb-cluster-operator-9d9fbdb5-mrrqh condition met E0516 19:55:32.896573 9784 reflector.go:227] "Failed to watch" err="Get \"https://34.121.115.52/api/v1/namespaces/pxc-operator/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpercona-xtradb-cluster-operator-9d9fbdb5-mrrqh&resourceVersion=1778961330918178000&timeoutSeconds=540&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/percona-xtradb-cluster-operator-9d9fbdb5-mrrqh to become Ready.Ok error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces proxy-switch-11779 ----------------------------------------------------------------------------------- Error from server (NotFound): namespaces "proxy-switch-11779" not found waiting for namespace/proxy-switch-11779 to be deletederror: resource(s) were provided, but no name was specified Error from server (NotFound): namespaces "proxy-switch-11779" not found ----------------------------------------------------------------------------------- create namespace proxy-switch-11779 ----------------------------------------------------------------------------------- namespace/proxy-switch-11779 created Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2467-3dc7f023-16-cluster12" modified. ----------------------------------------------------------------------------------- create secrets for cloud storages ----------------------------------------------------------------------------------- secret/minio-secret created secret/aws-s3-secret created secret/do-spaces-secret created secret/gcp-cs-secret created secret/azure-secret created ----------------------------------------------------------------------------------- create PXC cluster with HAProxy ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- create first PXC cluster ----------------------------------------------------------------------------------- secret/my-cluster-secrets created secret/some-name-ssl created secret/some-name-ssl-internal created deployment.apps/pxc-client created perconaxtradbcluster.pxc.percona.com/proxy-switch created ----------------------------------------------------------------------------------- check if all 3 Pods started ----------------------------------------------------------------------------------- error: no matching resources found ----------------------------------------------------------------------------------- wait for running cluster ----------------------------------------------------------------------------------- pod/proxy-switch-haproxy-0 condition met waiting for pod/proxy-switch-haproxy-0 to become Ready.Ok ----------------------------------------------------------------------------------- wait for running cluster ----------------------------------------------------------------------------------- pod/proxy-switch-pxc-0 condition met E0516 19:57:51.629860 27743 reflector.go:227] "Failed to watch" err="Get \"https://34.121.115.52/api/v1/namespaces/proxy-switch-11779/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-pxc-0&resourceVersion=1778961468931055007&timeoutSeconds=569&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-pxc-0 to become Ready.Ok pod/proxy-switch-pxc-1 condition met waiting for pod/proxy-switch-pxc-1 to become Ready.Ok pod/proxy-switch-pxc-2 condition met waiting for pod/proxy-switch-pxc-2 to become Ready.Ok ----------------------------------------------------------------------------------- write data ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-dhfg7 condition met E0516 20:00:36.827401 13117 reflector.go:227] "Failed to watch" err="Get \"https://34.121.115.52/api/v1/namespaces/proxy-switch-11779/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-dhfg7&resourceVersion=1778961636028655000&timeoutSeconds=567&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-dhfg7 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-dhfg7 condition met E0516 20:00:44.681769 13999 reflector.go:227] "Failed to watch" err="Get \"https://34.121.115.52/api/v1/namespaces/proxy-switch-11779/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-dhfg7&resourceVersion=1778961642862812000&timeoutSeconds=541&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-dhfg7 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-dhfg7 condition met E0516 20:01:22.389816 17638 reflector.go:227] "Failed to watch" err="Get \"https://34.121.115.52/api/v1/namespaces/proxy-switch-11779/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-dhfg7&resourceVersion=1778961681384696000&timeoutSeconds=536&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-dhfg7 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-dhfg7 condition met E0516 20:01:29.290004 18484 reflector.go:227] "Failed to watch" err="Get \"https://34.121.115.52/api/v1/namespaces/proxy-switch-11779/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-dhfg7&resourceVersion=1778961688444537000&timeoutSeconds=303&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-dhfg7 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-dhfg7 condition met E0516 20:01:38.809009 19602 reflector.go:227] "Failed to watch" err="Get \"https://34.121.115.52/api/v1/namespaces/proxy-switch-11779/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-dhfg7&resourceVersion=1778961696830271000&timeoutSeconds=558&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-dhfg7 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- check cluster is ready with HAProxy ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- wait for running cluster ----------------------------------------------------------------------------------- pod/proxy-switch-pxc-0 condition met E0516 20:01:49.934390 21141 reflector.go:227] "Failed to watch" err="Get \"https://34.121.115.52/api/v1/namespaces/proxy-switch-11779/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-pxc-0&resourceVersion=1778961706830949000&timeoutSeconds=362&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-pxc-0 to become Ready.Ok pod/proxy-switch-pxc-1 condition met E0516 20:01:56.671277 22162 reflector.go:227] "Failed to watch" err="Get \"https://34.121.115.52/api/v1/namespaces/proxy-switch-11779/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-pxc-1&resourceVersion=1778961711830991000&timeoutSeconds=304&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-pxc-1 to become Ready.Ok pod/proxy-switch-pxc-2 condition met E0516 20:02:04.740240 23150 reflector.go:227] "Failed to watch" err="Get \"https://34.121.115.52/api/v1/namespaces/proxy-switch-11779/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-pxc-2&resourceVersion=1778961721832237000&timeoutSeconds=559&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-pxc-2 to become Ready.Ok ----------------------------------------------------------------------------------- wait for running cluster ----------------------------------------------------------------------------------- pod/proxy-switch-haproxy-0 condition met E0516 20:02:13.111664 24272 reflector.go:227] "Failed to watch" err="Get \"https://34.121.115.52/api/v1/namespaces/proxy-switch-11779/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-haproxy-0&resourceVersion=1778961731832912000&timeoutSeconds=461&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-haproxy-0 to become Ready.Ok pod/proxy-switch-haproxy-1 condition met E0516 20:02:20.022687 25302 reflector.go:227] "Failed to watch" err="Get \"https://34.121.115.52/api/v1/namespaces/proxy-switch-11779/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-haproxy-1&resourceVersion=1778961736833398000&timeoutSeconds=548&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-haproxy-1 to become Ready.Ok pod/proxy-switch-haproxy-2 condition met E0516 20:02:26.550920 26102 reflector.go:227] "Failed to watch" err="Get \"https://34.121.115.52/api/v1/namespaces/proxy-switch-11779/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-haproxy-2&resourceVersion=1778961741833412000&timeoutSeconds=480&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-haproxy-2 to become Ready.Ok ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/proxy-switch to be ready ----------------------------------------------------------------------------------- write data and check connectivity through HAProxy ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-dhfg7 condition met E0516 20:02:45.960000 28913 reflector.go:227] "Failed to watch" err="Get \"https://34.121.115.52/api/v1/namespaces/proxy-switch-11779/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-dhfg7&resourceVersion=1778961765100088000&timeoutSeconds=311&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-dhfg7 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-dhfg7 condition met E0516 20:02:54.341358 29944 reflector.go:227] "Failed to watch" err="Get \"https://34.121.115.52/api/v1/namespaces/proxy-switch-11779/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-dhfg7&resourceVersion=1778961772602711000&timeoutSeconds=586&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-dhfg7 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ERROR 1062 (23000) at line 1: Duplicate entry '100500' for key 'myApp.PRIMARY' command terminated with exit code 1 pod/pxc-client-67fc4995bb-dhfg7 condition met E0516 20:03:12.130050 32071 reflector.go:227] "Failed to watch" err="Get \"https://34.121.115.52/api/v1/namespaces/proxy-switch-11779/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-dhfg7&resourceVersion=1778961790378558000&timeoutSeconds=418&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-dhfg7 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- verify data exists on all PXC nodes ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-dhfg7 condition met E0516 20:03:20.148613 715 reflector.go:227] "Failed to watch" err="Get \"https://34.121.115.52/api/v1/namespaces/proxy-switch-11779/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-dhfg7&resourceVersion=1778961798556035000&timeoutSeconds=473&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-dhfg7 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-dhfg7 condition met E0516 20:03:32.116954 2431 reflector.go:227] "Failed to watch" err="Get \"https://34.121.115.52/api/v1/namespaces/proxy-switch-11779/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-dhfg7&resourceVersion=1778961809706027000&timeoutSeconds=436&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-dhfg7 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-dhfg7 condition met E0516 20:03:43.244332 4192 reflector.go:227] "Failed to watch" err="Get \"https://34.121.115.52/api/v1/namespaces/proxy-switch-11779/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-dhfg7&resourceVersion=1778961821382424000&timeoutSeconds=370&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-dhfg7 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- switch from HAProxy to ProxySQL ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/proxy-switch patched ----------------------------------------------------------------------------------- wait for ProxySQL to be ready and HAProxy to be removed ----------------------------------------------------------------------------------- waiting for sts/proxy-switch-haproxy to be deletedError from server (NotFound): statefulsets.apps "proxy-switch-haproxy" not found ----------------------------------------------------------------------------------- wait for running cluster ----------------------------------------------------------------------------------- pod/proxy-switch-proxysql-0 condition met waiting for pod/proxy-switch-proxysql-0 to become Ready.Ok pod/proxy-switch-proxysql-1 condition met waiting for pod/proxy-switch-proxysql-1 to become Ready.Ok pod/proxy-switch-proxysql-2 condition met waiting for pod/proxy-switch-proxysql-2 to become Ready.Ok ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/proxy-switch to be ready ----------------------------------------------------------------------------------- verify HAProxy pods are deleted ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- verify ProxySQL service exists ----------------------------------------------------------------------------------- NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE proxy-switch-proxysql ClusterIP 34.118.229.226 3306/TCP,33062/TCP,6070/TCP 75s ----------------------------------------------------------------------------------- check connectivity through ProxySQL after switch ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-dhfg7 condition met E0516 20:05:29.143411 18364 reflector.go:227] "Failed to watch" err="Get \"https://34.121.115.52/api/v1/namespaces/proxy-switch-11779/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-dhfg7&resourceVersion=1778961928273646000&timeoutSeconds=594&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-dhfg7 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- write new data through ProxySQL ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-dhfg7 condition met E0516 20:05:41.644883 19928 reflector.go:227] "Failed to watch" err="Get \"https://34.121.115.52/api/v1/namespaces/proxy-switch-11779/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-dhfg7&resourceVersion=1778961938712722000&timeoutSeconds=374&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-dhfg7 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-dhfg7 condition met E0516 20:06:06.418752 23494 reflector.go:227] "Failed to watch" err="Get \"https://34.121.115.52/api/v1/namespaces/proxy-switch-11779/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-dhfg7&resourceVersion=1778961963888678000&timeoutSeconds=497&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-dhfg7 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- verify new data exists on all PXC nodes ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-dhfg7 condition met E0516 20:06:24.665525 26077 reflector.go:227] "Failed to watch" err="Get \"https://34.121.115.52/api/v1/namespaces/proxy-switch-11779/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-dhfg7&resourceVersion=1778961981809226000&timeoutSeconds=327&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-dhfg7 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-dhfg7 condition met E0516 20:06:39.317907 28389 reflector.go:227] "Failed to watch" err="Get \"https://34.121.115.52/api/v1/namespaces/proxy-switch-11779/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-dhfg7&resourceVersion=1778961996876884000&timeoutSeconds=463&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-dhfg7 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-dhfg7 condition met E0516 20:06:53.678167 30385 reflector.go:227] "Failed to watch" err="Get \"https://34.121.115.52/api/v1/namespaces/proxy-switch-11779/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-dhfg7&resourceVersion=1778962011677989000&timeoutSeconds=542&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-dhfg7 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- verify ProxySQL is routing to primary ----------------------------------------------------------------------------------- ProxySQL is routing to primary: proxy-switch-pxc-0 ----------------------------------------------------------------------------------- switch back to HAProxy from ProxySQL ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/proxy-switch patched ----------------------------------------------------------------------------------- wait for HAProxy to be ready and ProxySQL to be removed ----------------------------------------------------------------------------------- waiting for sts/proxy-switch-proxysql to be deletedError from server (NotFound): statefulsets.apps "proxy-switch-proxysql" not found ----------------------------------------------------------------------------------- wait for running cluster ----------------------------------------------------------------------------------- pod/proxy-switch-haproxy-0 condition met waiting for pod/proxy-switch-haproxy-0 to become Ready.Ok pod/proxy-switch-haproxy-1 condition met waiting for pod/proxy-switch-haproxy-1 to become Ready.Ok pod/proxy-switch-haproxy-2 condition met waiting for pod/proxy-switch-haproxy-2 to become Ready.Ok ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/proxy-switch to be ready ----------------------------------------------------------------------------------- verify ProxySQL pods are deleted ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- check cluster is ready with HAProxy ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- wait for running cluster ----------------------------------------------------------------------------------- pod/proxy-switch-pxc-0 condition met E0516 20:08:36.724141 12304 reflector.go:227] "Failed to watch" err="Get \"https://34.121.115.52/api/v1/namespaces/proxy-switch-11779/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-pxc-0&resourceVersion=1778962114178686000&timeoutSeconds=400&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-pxc-0 to become Ready.Ok pod/proxy-switch-pxc-1 condition met E0516 20:08:44.199882 13372 reflector.go:227] "Failed to watch" err="Get \"https://34.121.115.52/api/v1/namespaces/proxy-switch-11779/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-pxc-1&resourceVersion=1778962119178725000&timeoutSeconds=486&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-pxc-1 to become Ready.Ok pod/proxy-switch-pxc-2 condition met E0516 20:08:50.651326 14380 reflector.go:227] "Failed to watch" err="Get \"https://34.121.115.52/api/v1/namespaces/proxy-switch-11779/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-pxc-2&resourceVersion=1778962129179263000&timeoutSeconds=410&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-pxc-2 to become Ready.Ok ----------------------------------------------------------------------------------- wait for running cluster ----------------------------------------------------------------------------------- pod/proxy-switch-haproxy-0 condition met E0516 20:08:57.629281 15365 reflector.go:227] "Failed to watch" err="Get \"https://34.121.115.52/api/v1/namespaces/proxy-switch-11779/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-haproxy-0&resourceVersion=1778962134179889000&timeoutSeconds=541&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-haproxy-0 to become Ready.Ok pod/proxy-switch-haproxy-1 condition met E0516 20:09:03.311364 16184 reflector.go:227] "Failed to watch" err="Get \"https://34.121.115.52/api/v1/namespaces/proxy-switch-11779/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-haproxy-1&resourceVersion=1778962139180185000&timeoutSeconds=497&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-haproxy-1 to become Ready.Ok pod/proxy-switch-haproxy-2 condition met E0516 20:09:09.836724 17135 reflector.go:227] "Failed to watch" err="Get \"https://34.121.115.52/api/v1/namespaces/proxy-switch-11779/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-haproxy-2&resourceVersion=1778962149180261000&timeoutSeconds=303&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-haproxy-2 to become Ready.Ok ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/proxy-switch to be ready ----------------------------------------------------------------------------------- verify HAProxy service exists ----------------------------------------------------------------------------------- NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE proxy-switch-haproxy ClusterIP 34.118.229.52 3306/TCP,3309/TCP,33062/TCP,33060/TCP,8404/TCP 2m24s ----------------------------------------------------------------------------------- write data and check connectivity through HAProxy ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-dhfg7 condition met E0516 20:09:30.984468 20095 reflector.go:227] "Failed to watch" err="Get \"https://34.121.115.52/api/v1/namespaces/proxy-switch-11779/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-dhfg7&resourceVersion=1778962168716289000&timeoutSeconds=531&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-dhfg7 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-dhfg7 condition met E0516 20:09:42.511393 21742 reflector.go:227] "Failed to watch" err="Get \"https://34.121.115.52/api/v1/namespaces/proxy-switch-11779/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-dhfg7&resourceVersion=1778962180548327000&timeoutSeconds=515&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-dhfg7 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-dhfg7 condition met E0516 20:10:01.194420 24342 reflector.go:227] "Failed to watch" err="Get \"https://34.121.115.52/api/v1/namespaces/proxy-switch-11779/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-dhfg7&resourceVersion=1778962199377403000&timeoutSeconds=567&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-dhfg7 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- verify data exists on all PXC nodes ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-dhfg7 condition met E0516 20:10:12.372583 25892 reflector.go:227] "Failed to watch" err="Get \"https://34.121.115.52/api/v1/namespaces/proxy-switch-11779/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-dhfg7&resourceVersion=1778962210302559000&timeoutSeconds=466&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-dhfg7 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-dhfg7 condition met E0516 20:10:18.011824 26713 reflector.go:227] "Failed to watch" err="Get \"https://34.121.115.52/api/v1/namespaces/proxy-switch-11779/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-dhfg7&resourceVersion=1778962217232139000&timeoutSeconds=577&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-dhfg7 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-dhfg7 condition met E0516 20:10:25.490227 27557 reflector.go:227] "Failed to watch" err="Get \"https://34.121.115.52/api/v1/namespaces/proxy-switch-11779/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-dhfg7&resourceVersion=1778962223474290000&timeoutSeconds=568&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-dhfg7 to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- clean up ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- destroy cluster/operator and all other resources ----------------------------------------------------------------------------------- + kubectl patch pxc -n proxy-switch-11779 proxy-switch --type=merge -p '{"metadata":{"finalizers":[]}}' perconaxtradbcluster.pxc.percona.com/proxy-switch patched perconaxtradbcluster.pxc.percona.com "proxy-switch" deleted from proxy-switch-11779 namespace No resources found No resources found validatingwebhookconfiguration.admissionregistration.k8s.io "percona-xtradbcluster-webhook" deleted ----------------------------------------------------------------------------------- test passed -----------------------------------------------------------------------------------