Log: /mnt/jenkins/workspace/cloud-pxc-operator_PR-2525/e2e-tests/logs/proxy-switch-8-0.log Warning: version difference between client (1.36) and server (1.33) exceeds the supported minor version skew of +/-1 Warning: version difference between client (1.36) and server (1.33) exceeds the supported minor version skew of +/-1 No resources found No resources found No resources found error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces pxc-operator ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified namespace "pxc-operator" deleted waiting for namespace/pxc-operator to be deletedError from server (NotFound): namespaces "pxc-operator" not found ----------------------------------------------------------------------------------- create namespace pxc-operator ----------------------------------------------------------------------------------- namespace/pxc-operator created Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2525-c81e47f0-4-cluster10" modified. ----------------------------------------------------------------------------------- start PXC operator ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterbackups.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterrestores.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusters.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusters.pxc.percona.com condition met E0806 02:59:09.785858 750353 reflector.go:227] "Failed to watch" err="Get \"https://34.121.204.218/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dperconaxtradbclusters.pxc.percona.com&resourceVersion=1785985145451952000&timeoutSeconds=324&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterbackups.pxc.percona.com condition met E0806 02:59:12.921667 750739 reflector.go:227] "Failed to watch" err="Get \"https://34.121.204.218/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dperconaxtradbclusterbackups.pxc.percona.com&resourceVersion=1785985150452067000&timeoutSeconds=412&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterrestores.pxc.percona.com condition met E0806 02:59:17.154117 751099 reflector.go:227] "Failed to watch" err="Get \"https://34.121.204.218/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dperconaxtradbclusterrestores.pxc.percona.com&resourceVersion=1785985155452191000&timeoutSeconds=525&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" clusterrole.rbac.authorization.k8s.io/percona-xtradb-cluster-operator unchanged serviceaccount/percona-xtradb-cluster-operator created clusterrolebinding.rbac.authorization.k8s.io/service-account-percona-xtradb-cluster-operator unchanged deployment.apps/percona-xtradb-cluster-operator created service/percona-xtradb-cluster-operator created pod/percona-xtradb-cluster-operator-6699755d74-f5xrw condition met E0806 02:59:46.063481 754814 reflector.go:227] "Failed to watch" err="Get \"https://34.121.204.218/api/v1/namespaces/pxc-operator/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpercona-xtradb-cluster-operator-6699755d74-f5xrw&resourceVersion=1785985185704073000&timeoutSeconds=379&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" pod/percona-xtradb-cluster-operator-6699755d74-f5xrw condition met E0806 02:59:56.036223 756105 reflector.go:227] "Failed to watch" err="Get \"https://34.121.204.218/api/v1/namespaces/pxc-operator/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpercona-xtradb-cluster-operator-6699755d74-f5xrw&resourceVersion=1785985192256101000&timeoutSeconds=469&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/percona-xtradb-cluster-operator-6699755d74-f5xrw to become Ready.Ok error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces proxy-switch-4064 ----------------------------------------------------------------------------------- Error from server (NotFound): namespaces "proxy-switch-4064" not found waiting for namespace/proxy-switch-4064 to be deletederror: resource(s) were provided, but no name was specified Error from server (NotFound): namespaces "proxy-switch-4064" not found ----------------------------------------------------------------------------------- create namespace proxy-switch-4064 ----------------------------------------------------------------------------------- namespace/proxy-switch-4064 created Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2525-c81e47f0-4-cluster10" modified. ----------------------------------------------------------------------------------- create secrets for cloud storages ----------------------------------------------------------------------------------- secret/minio-secret created secret/aws-s3-secret created secret/do-spaces-secret created secret/gcp-cs-secret created secret/azure-secret created ----------------------------------------------------------------------------------- create PXC cluster with HAProxy ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- create first PXC cluster ----------------------------------------------------------------------------------- secret/my-cluster-secrets created secret/some-name-ssl created secret/some-name-ssl-internal created deployment.apps/pxc-client created perconaxtradbcluster.pxc.percona.com/proxy-switch created ----------------------------------------------------------------------------------- waiting pods to be running for proxy-switch ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- Waiting for 3 running pods for proxy-switch-pxc ----------------------------------------------------------------------------------- pod/proxy-switch-pxc-0 condition met waiting for pod/proxy-switch-pxc-0 to become Ready.Ok pod/proxy-switch-pxc-1 condition met waiting for pod/proxy-switch-pxc-1 to become Ready.Ok pod/proxy-switch-pxc-2 condition met waiting for pod/proxy-switch-pxc-2 to become Ready.Ok ----------------------------------------------------------------------------------- Waiting for 3 running pods for proxy-switch-haproxy ----------------------------------------------------------------------------------- pod/proxy-switch-haproxy-0 condition met E0806 03:05:12.419230 796684 reflector.go:227] "Failed to watch" err="Get \"https://34.121.204.218/api/v1/namespaces/proxy-switch-4064/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-haproxy-0&resourceVersion=1785985511490186000&timeoutSeconds=581&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-haproxy-0 to become Ready.Ok pod/proxy-switch-haproxy-1 condition met E0806 03:05:22.206104 797853 reflector.go:227] "Failed to watch" err="Get \"https://34.121.204.218/api/v1/namespaces/proxy-switch-4064/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-haproxy-1&resourceVersion=1785985521490337000&timeoutSeconds=510&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-haproxy-1 to become Ready.Ok pod/proxy-switch-haproxy-2 condition met E0806 03:05:36.717999 799648 reflector.go:227] "Failed to watch" err="Get \"https://34.121.204.218/api/v1/namespaces/proxy-switch-4064/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-haproxy-2&resourceVersion=1785985536490553000&timeoutSeconds=535&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-haproxy-2 to become Ready.Ok ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/proxy-switch to be ready ----------------------------------------------------------------------------------- write data ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-q4ttw condition met E0806 03:06:24.101019 805882 reflector.go:227] "Failed to watch" err="Get \"https://34.121.204.218/api/v1/namespaces/proxy-switch-4064/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-q4ttw&resourceVersion=1785985581338737000&timeoutSeconds=303&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-q4ttw to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-q4ttw condition met E0806 03:06:39.412685 808201 reflector.go:227] "Failed to watch" err="Get \"https://34.121.204.218/api/v1/namespaces/proxy-switch-4064/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-q4ttw&resourceVersion=1785985598374410000&timeoutSeconds=456&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-q4ttw to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- compare initial data ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-q4ttw condition met E0806 03:07:31.141154 814616 reflector.go:227] "Failed to watch" err="Get \"https://34.121.204.218/api/v1/namespaces/proxy-switch-4064/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-q4ttw&resourceVersion=1785985648714510000&timeoutSeconds=506&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-q4ttw to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-q4ttw condition met E0806 03:07:45.115486 816536 reflector.go:227] "Failed to watch" err="Get \"https://34.121.204.218/api/v1/namespaces/proxy-switch-4064/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-q4ttw&resourceVersion=1785985662862172000&timeoutSeconds=548&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-q4ttw to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-q4ttw condition met E0806 03:07:59.573082 818300 reflector.go:227] "Failed to watch" err="Get \"https://34.121.204.218/api/v1/namespaces/proxy-switch-4064/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-q4ttw&resourceVersion=1785985676721777000&timeoutSeconds=517&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-q4ttw to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- check cluster is ready with HAProxy ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- Waiting for 3 running pods for proxy-switch-pxc ----------------------------------------------------------------------------------- pod/proxy-switch-pxc-0 condition met E0806 03:08:17.989923 820747 reflector.go:227] "Failed to watch" err="Get \"https://34.121.204.218/api/v1/namespaces/proxy-switch-4064/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-pxc-0&resourceVersion=1785985696721849000&timeoutSeconds=385&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-pxc-0 to become Ready.Ok pod/proxy-switch-pxc-1 condition met E0806 03:08:26.066116 821905 reflector.go:227] "Failed to watch" err="Get \"https://34.121.204.218/api/v1/namespaces/proxy-switch-4064/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-pxc-1&resourceVersion=1785985701721893000&timeoutSeconds=466&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-pxc-1 to become Ready.Ok pod/proxy-switch-pxc-2 condition met E0806 03:08:33.912153 822915 reflector.go:227] "Failed to watch" err="Get \"https://34.121.204.218/api/v1/namespaces/proxy-switch-4064/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-pxc-2&resourceVersion=1785985711722021000&timeoutSeconds=546&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-pxc-2 to become Ready.Ok ----------------------------------------------------------------------------------- Waiting for 3 running pods for proxy-switch-haproxy ----------------------------------------------------------------------------------- pod/proxy-switch-haproxy-0 condition met E0806 03:08:43.119467 824227 reflector.go:227] "Failed to watch" err="Get \"https://34.121.204.218/api/v1/namespaces/proxy-switch-4064/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-haproxy-0&resourceVersion=1785985721722084000&timeoutSeconds=448&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-haproxy-0 to become Ready.Ok pod/proxy-switch-haproxy-1 condition met E0806 03:08:49.947419 825148 reflector.go:227] "Failed to watch" err="Get \"https://34.121.204.218/api/v1/namespaces/proxy-switch-4064/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-haproxy-1&resourceVersion=1785985726722093000&timeoutSeconds=307&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-haproxy-1 to become Ready.Ok pod/proxy-switch-haproxy-2 condition met E0806 03:08:58.127813 826216 reflector.go:227] "Failed to watch" err="Get \"https://34.121.204.218/api/v1/namespaces/proxy-switch-4064/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-haproxy-2&resourceVersion=1785985736722161000&timeoutSeconds=372&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-haproxy-2 to become Ready.Ok ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/proxy-switch to be ready ----------------------------------------------------------------------------------- write data and check connectivity through HAProxy ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-q4ttw condition met E0806 03:09:14.057288 828095 reflector.go:227] "Failed to watch" err="Get \"https://34.121.204.218/api/v1/namespaces/proxy-switch-4064/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-q4ttw&resourceVersion=1785985752398508000&timeoutSeconds=377&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-q4ttw to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-q4ttw condition met E0806 03:09:23.661535 829278 reflector.go:227] "Failed to watch" err="Get \"https://34.121.204.218/api/v1/namespaces/proxy-switch-4064/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-q4ttw&resourceVersion=1785985761712819000&timeoutSeconds=365&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-q4ttw to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ERROR 1062 (23000) at line 1: Duplicate entry '100500' for key 'myApp.PRIMARY' command terminated with exit code 1 pod/pxc-client-67fc4995bb-q4ttw condition met E0806 03:09:44.188025 831494 reflector.go:227] "Failed to watch" err="Get \"https://34.121.204.218/api/v1/namespaces/proxy-switch-4064/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-q4ttw&resourceVersion=1785985781792502000&timeoutSeconds=509&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-q4ttw to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- verify data exists on all PXC nodes ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-q4ttw condition met E0806 03:09:56.302838 833136 reflector.go:227] "Failed to watch" err="Get \"https://34.121.204.218/api/v1/namespaces/proxy-switch-4064/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-q4ttw&resourceVersion=1785985794086750000&timeoutSeconds=340&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-q4ttw to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-q4ttw condition met E0806 03:10:11.861511 835056 reflector.go:227] "Failed to watch" err="Get \"https://34.121.204.218/api/v1/namespaces/proxy-switch-4064/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-q4ttw&resourceVersion=1785985808758555000&timeoutSeconds=374&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-q4ttw to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-q4ttw condition met E0806 03:10:29.722125 837342 reflector.go:227] "Failed to watch" err="Get \"https://34.121.204.218/api/v1/namespaces/proxy-switch-4064/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-q4ttw&resourceVersion=1785985826207694000&timeoutSeconds=306&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-q4ttw to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- switch from HAProxy to ProxySQL ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/proxy-switch patched ----------------------------------------------------------------------------------- wait for ProxySQL to be ready and HAProxy to be removed ----------------------------------------------------------------------------------- waiting for sts/proxy-switch-haproxy to be deletedError from server (NotFound): statefulsets.apps "proxy-switch-haproxy" not found ----------------------------------------------------------------------------------- Waiting for 3 running pods for proxy-switch-proxysql ----------------------------------------------------------------------------------- pod/proxy-switch-proxysql-0 condition met waiting for pod/proxy-switch-proxysql-0 to become Ready.Ok pod/proxy-switch-proxysql-1 condition met waiting for pod/proxy-switch-proxysql-1 to become Ready.Ok pod/proxy-switch-proxysql-2 condition met waiting for pod/proxy-switch-proxysql-2 to become Ready.Ok ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/proxy-switch to be ready ----------------------------------------------------------------------------------- verify HAProxy pods are deleted ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- verify ProxySQL service exists ----------------------------------------------------------------------------------- NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE proxy-switch-proxysql ClusterIP 34.118.227.130 3306/TCP,33062/TCP,6070/TCP 84s ----------------------------------------------------------------------------------- check connectivity through ProxySQL after switch ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-q4ttw condition met E0806 03:12:33.457822 853652 reflector.go:227] "Failed to watch" err="Get \"https://34.121.204.218/api/v1/namespaces/proxy-switch-4064/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-q4ttw&resourceVersion=1785985950608292000&timeoutSeconds=541&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-q4ttw to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- write new data through ProxySQL ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-q4ttw condition met E0806 03:12:49.167946 855773 reflector.go:227] "Failed to watch" err="Get \"https://34.121.204.218/api/v1/namespaces/proxy-switch-4064/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-q4ttw&resourceVersion=1785985966448903000&timeoutSeconds=524&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-q4ttw to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-q4ttw condition met E0806 03:13:13.217736 858967 reflector.go:227] "Failed to watch" err="Get \"https://34.121.204.218/api/v1/namespaces/proxy-switch-4064/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-q4ttw&resourceVersion=1785985991327786000&timeoutSeconds=542&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-q4ttw to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- verify new data exists on all PXC nodes ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-q4ttw condition met E0806 03:13:24.248974 860363 reflector.go:227] "Failed to watch" err="Get \"https://34.121.204.218/api/v1/namespaces/proxy-switch-4064/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-q4ttw&resourceVersion=1785986002538036000&timeoutSeconds=497&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-q4ttw to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-q4ttw condition met E0806 03:13:31.296241 861375 reflector.go:227] "Failed to watch" err="Get \"https://34.121.204.218/api/v1/namespaces/proxy-switch-4064/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-q4ttw&resourceVersion=1785986010495662000&timeoutSeconds=498&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-q4ttw to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-q4ttw condition met E0806 03:13:40.099844 862379 reflector.go:227] "Failed to watch" err="Get \"https://34.121.204.218/api/v1/namespaces/proxy-switch-4064/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-q4ttw&resourceVersion=1785986018516981000&timeoutSeconds=322&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-q4ttw to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- verify ProxySQL is routing to primary ----------------------------------------------------------------------------------- ProxySQL is routing to primary: proxy-switch-pxc-0 ----------------------------------------------------------------------------------- switch back to HAProxy from ProxySQL ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/proxy-switch patched ----------------------------------------------------------------------------------- wait for HAProxy to be ready and ProxySQL to be removed ----------------------------------------------------------------------------------- waiting for sts/proxy-switch-proxysql to be deletedError from server (NotFound): statefulsets.apps "proxy-switch-proxysql" not found ----------------------------------------------------------------------------------- Waiting for 3 running pods for proxy-switch-haproxy ----------------------------------------------------------------------------------- pod/proxy-switch-haproxy-0 condition met waiting for pod/proxy-switch-haproxy-0 to become Ready.Ok pod/proxy-switch-haproxy-1 condition met waiting for pod/proxy-switch-haproxy-1 to become Ready.Ok pod/proxy-switch-haproxy-2 condition met waiting for pod/proxy-switch-haproxy-2 to become Ready.Ok ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/proxy-switch to be ready ----------------------------------------------------------------------------------- verify ProxySQL pods are deleted ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- check cluster is ready with HAProxy ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- Waiting for 3 running pods for proxy-switch-pxc ----------------------------------------------------------------------------------- pod/proxy-switch-pxc-0 condition met E0806 03:15:32.617433 875871 reflector.go:227] "Failed to watch" err="Get \"https://34.121.204.218/api/v1/namespaces/proxy-switch-4064/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-pxc-0&resourceVersion=1785986128708968000&timeoutSeconds=462&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-pxc-0 to become Ready.Ok pod/proxy-switch-pxc-1 condition met E0806 03:15:43.854602 877331 reflector.go:227] "Failed to watch" err="Get \"https://34.121.204.218/api/v1/namespaces/proxy-switch-4064/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-pxc-1&resourceVersion=1785986143709213000&timeoutSeconds=384&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-pxc-1 to become Ready.Ok pod/proxy-switch-pxc-2 condition met E0806 03:15:53.931065 878751 reflector.go:227] "Failed to watch" err="Get \"https://34.121.204.218/api/v1/namespaces/proxy-switch-4064/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-pxc-2&resourceVersion=1785986153709302000&timeoutSeconds=480&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-pxc-2 to become Ready.Ok ----------------------------------------------------------------------------------- Waiting for 3 running pods for proxy-switch-haproxy ----------------------------------------------------------------------------------- pod/proxy-switch-haproxy-0 condition met E0806 03:16:02.059856 879795 reflector.go:227] "Failed to watch" err="Get \"https://34.121.204.218/api/v1/namespaces/proxy-switch-4064/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-haproxy-0&resourceVersion=1785986158709362000&timeoutSeconds=320&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-haproxy-0 to become Ready.Ok pod/proxy-switch-haproxy-1 condition met E0806 03:16:14.837093 881088 reflector.go:227] "Failed to watch" err="Get \"https://34.121.204.218/api/v1/namespaces/proxy-switch-4064/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-haproxy-1&resourceVersion=1785986173709450000&timeoutSeconds=525&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-haproxy-1 to become Ready.Ok pod/proxy-switch-haproxy-2 condition met E0806 03:16:44.331813 883765 reflector.go:227] "Failed to watch" err="Get \"https://34.121.204.218/api/v1/namespaces/proxy-switch-4064/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dproxy-switch-haproxy-2&resourceVersion=1785986203710091000&timeoutSeconds=491&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/proxy-switch-haproxy-2 to become Ready.Ok ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/proxy-switch to be ready ----------------------------------------------------------------------------------- verify HAProxy service exists ----------------------------------------------------------------------------------- NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE proxy-switch-haproxy ClusterIP 34.118.225.112 3306/TCP,3309/TCP,33062/TCP,33060/TCP,8404/TCP 3m21s ----------------------------------------------------------------------------------- write data and check connectivity through HAProxy ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-q4ttw condition met E0806 03:17:15.401820 888025 reflector.go:227] "Failed to watch" err="Get \"https://34.121.204.218/api/v1/namespaces/proxy-switch-4064/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-q4ttw&resourceVersion=1785986233582119000&timeoutSeconds=414&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-q4ttw to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-q4ttw condition met E0806 03:17:25.892174 889547 reflector.go:227] "Failed to watch" err="Get \"https://34.121.204.218/api/v1/namespaces/proxy-switch-4064/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-q4ttw&resourceVersion=1785986244826930000&timeoutSeconds=310&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-q4ttw to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-q4ttw condition met E0806 03:17:49.835664 892414 reflector.go:227] "Failed to watch" err="Get \"https://34.121.204.218/api/v1/namespaces/proxy-switch-4064/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-q4ttw&resourceVersion=1785986266932025000&timeoutSeconds=463&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-q4ttw to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- verify data exists on all PXC nodes ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-q4ttw condition met E0806 03:18:08.673686 894651 reflector.go:227] "Failed to watch" err="Get \"https://34.121.204.218/api/v1/namespaces/proxy-switch-4064/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-q4ttw&resourceVersion=1785986284515535000&timeoutSeconds=580&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-q4ttw to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-q4ttw condition met E0806 03:18:28.063688 897289 reflector.go:227] "Failed to watch" err="Get \"https://34.121.204.218/api/v1/namespaces/proxy-switch-4064/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-q4ttw&resourceVersion=1785986304282144000&timeoutSeconds=425&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-q4ttw to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-q4ttw condition met E0806 03:18:46.147690 899708 reflector.go:227] "Failed to watch" err="Get \"https://34.121.204.218/api/v1/namespaces/proxy-switch-4064/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-q4ttw&resourceVersion=1785986324086594000&timeoutSeconds=450&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-q4ttw to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- clean up ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- destroy cluster/operator and all other resources ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/proxy-switch patched perconaxtradbcluster.pxc.percona.com "proxy-switch" deleted from proxy-switch-4064 namespace No resources found validatingwebhookconfiguration.admissionregistration.k8s.io "percona-xtradbcluster-webhook" deleted namespace "proxy-switch-4064" force deleted Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. ----------------------------------------------------------------------------------- test passed -----------------------------------------------------------------------------------