Log: /mnt/jenkins/workspace/cloud-pxc-operator_PR-2525/e2e-tests/logs/proxysql-scheduler-8-0.log Warning: version difference between client (1.36) and server (1.33) exceeds the supported minor version skew of +/-1 Warning: version difference between client (1.36) and server (1.33) exceeds the supported minor version skew of +/-1 No resources found No resources found No resources found error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces pxc-operator ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified namespace "pxc-operator" deleted waiting for namespace/pxc-operator to be deletedError from server (NotFound): namespaces "pxc-operator" not found ----------------------------------------------------------------------------------- create namespace pxc-operator ----------------------------------------------------------------------------------- namespace/pxc-operator created Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2525-c81e47f0-4-cluster4" modified. ----------------------------------------------------------------------------------- start PXC operator ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterbackups.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterrestores.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusters.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusters.pxc.percona.com condition met E0806 02:58:21.195114 744009 reflector.go:227] "Failed to watch" err="Get \"https://34.170.55.135/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dperconaxtradbclusters.pxc.percona.com&resourceVersion=1785985099864436000&timeoutSeconds=321&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterbackups.pxc.percona.com condition met E0806 02:58:24.711358 744457 reflector.go:227] "Failed to watch" err="Get \"https://34.170.55.135/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dperconaxtradbclusterbackups.pxc.percona.com&resourceVersion=1785985099864436000&timeoutSeconds=490&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterrestores.pxc.percona.com condition met E0806 02:58:28.433422 744858 reflector.go:227] "Failed to watch" err="Get \"https://34.170.55.135/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dperconaxtradbclusterrestores.pxc.percona.com&resourceVersion=1785985104864444000&timeoutSeconds=311&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" clusterrole.rbac.authorization.k8s.io/percona-xtradb-cluster-operator unchanged serviceaccount/percona-xtradb-cluster-operator created clusterrolebinding.rbac.authorization.k8s.io/service-account-percona-xtradb-cluster-operator unchanged deployment.apps/percona-xtradb-cluster-operator created service/percona-xtradb-cluster-operator created pod/percona-xtradb-cluster-operator-6699755d74-z8qns condition met E0806 02:58:52.512242 748060 reflector.go:227] "Failed to watch" err="Get \"https://34.170.55.135/api/v1/namespaces/pxc-operator/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpercona-xtradb-cluster-operator-6699755d74-z8qns&resourceVersion=1785985132152700000&timeoutSeconds=572&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" pod/percona-xtradb-cluster-operator-6699755d74-z8qns condition met E0806 02:58:58.773539 749131 reflector.go:227] "Failed to watch" err="Get \"https://34.170.55.135/api/v1/namespaces/pxc-operator/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpercona-xtradb-cluster-operator-6699755d74-z8qns&resourceVersion=1785985137754868000&timeoutSeconds=379&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/percona-xtradb-cluster-operator-6699755d74-z8qns to become Ready.Ok error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces proxysql-scheduler-15034 ----------------------------------------------------------------------------------- Error from server (NotFound): namespaces "proxysql-scheduler-15034" not found waiting for namespace/proxysql-scheduler-15034 to be deletederror: resource(s) were provided, but no name was specified Error from server (NotFound): namespaces "proxysql-scheduler-15034" not found ----------------------------------------------------------------------------------- create namespace proxysql-scheduler-15034 ----------------------------------------------------------------------------------- namespace/proxysql-scheduler-15034 created Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2525-c81e47f0-4-cluster4" modified. ----------------------------------------------------------------------------------- create secrets for cloud storages ----------------------------------------------------------------------------------- secret/minio-secret created secret/aws-s3-secret created secret/do-spaces-secret created secret/gcp-cs-secret created secret/azure-secret created ----------------------------------------------------------------------------------- create PXC cluster: some-name ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- create first PXC cluster ----------------------------------------------------------------------------------- secret/my-cluster-secrets created secret/some-name-ssl created secret/some-name-ssl-internal created deployment.apps/pxc-client created perconaxtradbcluster.pxc.percona.com/some-name created ----------------------------------------------------------------------------------- waiting pods to be running for some-name ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- Waiting for 3 running pods for some-name-pxc ----------------------------------------------------------------------------------- pod/some-name-pxc-0 condition met waiting for pod/some-name-pxc-0 to become Ready.Ok pod/some-name-pxc-1 condition met waiting for pod/some-name-pxc-1 to become Ready.Ok pod/some-name-pxc-2 condition met waiting for pod/some-name-pxc-2 to become Ready.Ok ----------------------------------------------------------------------------------- Waiting for 2 running pods for some-name-proxysql ----------------------------------------------------------------------------------- pod/some-name-proxysql-0 condition met E0806 03:04:23.884363 790148 reflector.go:227] "Failed to watch" err="Get \"https://34.170.55.135/api/v1/namespaces/proxysql-scheduler-15034/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-proxysql-0&resourceVersion=1785985459525074000&timeoutSeconds=441&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/some-name-proxysql-0 to become Ready.Ok pod/some-name-proxysql-1 condition met E0806 03:04:36.944350 791836 reflector.go:227] "Failed to watch" err="Get \"https://34.170.55.135/api/v1/namespaces/proxysql-scheduler-15034/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-proxysql-1&resourceVersion=1785985474525217000&timeoutSeconds=512&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/some-name-proxysql-1 to become Ready.Ok ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/some-name to be ready ----------------------------------------------------------------------------------- write data ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-npqqf condition met E0806 03:05:31.510456 799048 reflector.go:227] "Failed to watch" err="Get \"https://34.170.55.135/api/v1/namespaces/proxysql-scheduler-15034/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-npqqf&resourceVersion=1785985528199377000&timeoutSeconds=396&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-npqqf to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-npqqf condition met E0806 03:06:03.598983 803060 reflector.go:227] "Failed to watch" err="Get \"https://34.170.55.135/api/v1/namespaces/proxysql-scheduler-15034/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-npqqf&resourceVersion=1785985559437950000&timeoutSeconds=334&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-npqqf to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- compare initial data ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-npqqf condition met E0806 03:06:55.019460 810069 reflector.go:227] "Failed to watch" err="Get \"https://34.170.55.135/api/v1/namespaces/proxysql-scheduler-15034/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-npqqf&resourceVersion=1785985612420639000&timeoutSeconds=556&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-npqqf to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-npqqf condition met E0806 03:07:13.588517 812350 reflector.go:227] "Failed to watch" err="Get \"https://34.170.55.135/api/v1/namespaces/proxysql-scheduler-15034/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-npqqf&resourceVersion=1785985630075083000&timeoutSeconds=363&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-npqqf to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-npqqf condition met E0806 03:07:29.940486 814376 reflector.go:227] "Failed to watch" err="Get \"https://34.170.55.135/api/v1/namespaces/proxysql-scheduler-15034/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-npqqf&resourceVersion=1785985646766953000&timeoutSeconds=510&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-npqqf to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/some-name to be ready ----------------------------------------------------------------------------------- check proxysql custom config ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- check if service and statefulset created with expected config ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- compare statefulset/some-name-pxc- ----------------------------------------------------------------------------------- [2026-08-06T03:08:06+0000] compare_kubectl: statefulset/some-name-pxc OK ----------------------------------------------------------------------------------- compare statefulset/some-name-proxysql- ----------------------------------------------------------------------------------- [2026-08-06T03:08:10+0000] compare_kubectl: statefulset/some-name-proxysql OK ----------------------------------------------------------------------------------- compare service/some-name-pxc- ----------------------------------------------------------------------------------- [2026-08-06T03:08:12+0000] compare_kubectl: service/some-name-pxc OK ----------------------------------------------------------------------------------- compare service/some-name-proxysql- ----------------------------------------------------------------------------------- [2026-08-06T03:08:15+0000] compare_kubectl: service/some-name-proxysql OK ----------------------------------------------------------------------------------- compare service/some-name-proxysql-unready- ----------------------------------------------------------------------------------- [2026-08-06T03:08:17+0000] compare_kubectl: service/some-name-proxysql-unready OK ----------------------------------------------------------------------------------- check if scheduler is enabled in all ProxySQL servers ----------------------------------------------------------------------------------- [2026-08-06T03:10:21+0000] scheduler is enabled in some-name-proxysql-0: OK [2026-08-06T03:10:24+0000] scheduler is enabled in some-name-proxysql-1: OK ----------------------------------------------------------------------------------- check if scheduler is doing its job in all ProxySQL servers ----------------------------------------------------------------------------------- check runtime_mysql_servers in some-name-proxysql-0 [2026-08-06T03:10:28+0000] mysql_servers are configured in some-name-proxysql-0: OK check runtime_mysql_servers in some-name-proxysql-1 [2026-08-06T03:10:31+0000] mysql_servers are configured in some-name-proxysql-1: OK ----------------------------------------------------------------------------------- check disabling scheduler ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/some-name patched [2026-08-06T03:10:34+0000] sleeping for 90 seconds ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/some-name to be ready [2026-08-06T03:12:46+0000] scheduler is disabled in some-name-proxysql-0: OK [2026-08-06T03:13:12+0000] scheduler is disabled in some-name-proxysql-1: OK NAME READY STATUS RESTARTS AGE pxc-client-67fc4995bb-npqqf 2/2 Running 0 13m some-name-proxysql-0 3/3 Running 0 2m7s some-name-proxysql-1 3/3 Running 0 2m39s some-name-pxc-0 1/1 Running 0 13m some-name-pxc-1 1/1 Running 0 11m some-name-pxc-2 1/1 Running 0 10m NAME ENDPOINT STATUS PXC PROXYSQL HAPROXY AGE some-name some-name-proxysql.proxysql-scheduler-15034 ready 3 2 13m ----------------------------------------------------------------------------------- check enabling scheduler ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/some-name patched [2026-08-06T03:13:17+0000] sleeping for 90 seconds ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/some-name to be ready [2026-08-06T03:15:08+0000] scheduler is enabled in some-name-proxysql-0: OK [2026-08-06T03:15:12+0000] scheduler is enabled in some-name-proxysql-1: OK ----------------------------------------------------------------------------------- check PXC pod 1 is promoted to writer when pod-0 is down ----------------------------------------------------------------------------------- LOOP_PID=873780 pod "some-name-pxc-0" deleted from proxysql-scheduler-15034 namespace [2026-08-06T03:15:22+0000] waiting for pod0 to be removed from proxysql check runtime_mysql_servers in some-name-proxysql-0 [2026-08-06T03:15:25+0000] mysql_servers are configured in some-name-proxysql-0 when pxc-0 is down: OK check runtime_mysql_servers in some-name-proxysql-1 [2026-08-06T03:15:28+0000] mysql_servers are configured in some-name-proxysql-1 when pxc-0 is down: OK ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/some-name to be ready... ----------------------------------------------------------------------------------- check scaling PXC up to 5 replicas ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/some-name patched pod/some-name-pxc-3 condition met waiting for pod/some-name-pxc-3 to become Ready.Ok pod/some-name-pxc-4 condition met waiting for pod/some-name-pxc-4 to become Ready.Ok check runtime_mysql_servers in some-name-proxysql-0 [2026-08-06T03:21:53+0000] new mysql_servers are configured in some-name-proxysql-0: OK check runtime_mysql_servers in some-name-proxysql-1 [2026-08-06T03:21:56+0000] new mysql_servers are configured in some-name-proxysql-1: OK ----------------------------------------------------------------------------------- check scaling ProxySQL up ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/some-name patched pod/some-name-proxysql-2 condition met waiting for pod/some-name-proxysql-2 to become Ready.Ok [2026-08-06T03:24:30+0000] scheduler is enabled in some-name-proxysql-2: OK check runtime_mysql_servers in some-name-proxysql-2 [2026-08-06T03:24:33+0000] mysql_servers are configured in some-name-proxysql-2: OK ----------------------------------------------------------------------------------- check writerIsAlsoReader = false ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/some-name patched ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/some-name to be ready.. check runtime_mysql_servers in some-name-proxysql-0 [2026-08-06T03:26:53+0000] new mysql_servers are configured in some-name-proxysql-0: OK check runtime_mysql_servers in some-name-proxysql-1 [2026-08-06T03:26:55+0000] new mysql_servers are configured in some-name-proxysql-1: OK check runtime_mysql_servers in some-name-proxysql-2 [2026-08-06T03:26:57+0000] mysql_servers are configured in some-name-proxysql-2: OK ----------------------------------------------------------------------------------- check scaling PXC down to 3 replicas ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/some-name patched ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/some-name to be ready.. check runtime_mysql_servers in some-name-proxysql-0 [2026-08-06T03:28:46+0000] mysql_servers are configured in some-name-proxysql-0: OK check runtime_mysql_servers in some-name-proxysql-1 [2026-08-06T03:28:49+0000] mysql_servers are configured in some-name-proxysql-1: OK check runtime_mysql_servers in some-name-proxysql-2 [2026-08-06T03:28:51+0000] mysql_servers are configured in some-name-proxysql-2: OK ----------------------------------------------------------------------------------- check PXC pod 0 is reader and writer when pods 1 and 2 are down ----------------------------------------------------------------------------------- [2026-08-06T03:28:51+0000] scaling PXC down to 1 replica statefulset.apps/some-name-pxc scaled check runtime_mysql_servers in some-name-proxysql-0.... [2026-08-06T03:30:11+0000] mysql_servers are configured in some-name-proxysql-0 when 2 pods are down: OK check runtime_mysql_servers in some-name-proxysql-1 [2026-08-06T03:30:14+0000] mysql_servers are configured in some-name-proxysql-1 when 2 pods are down: OK check runtime_mysql_servers in some-name-proxysql-2 [2026-08-06T03:30:17+0000] mysql_servers are configured in some-name-proxysql-2 when 2 pods are down: OK ----------------------------------------------------------------------------------- test passed ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- destroy cluster/operator and all other resources ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/some-name patched perconaxtradbcluster.pxc.percona.com "some-name" deleted from proxysql-scheduler-15034 namespace No resources found validatingwebhookconfiguration.admissionregistration.k8s.io "percona-xtradbcluster-webhook" deleted namespace "proxysql-scheduler-15034" force deleted Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.