Log: /mnt/jenkins/workspace/cloud-pxc-operator_PR-2539/e2e-tests/logs/proxysql-scheduler-8-0.log Warning: version difference between client (1.36) and server (1.33) exceeds the supported minor version skew of +/-1 Warning: version difference between client (1.36) and server (1.33) exceeds the supported minor version skew of +/-1 No resources found No resources found No resources found error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces pxc-operator ----------------------------------------------------------------------------------- Error from server (NotFound): namespaces "pxc-operator" not found waiting for namespace/pxc-operator to be deletedError from server (NotFound): namespaces "pxc-operator" not found ----------------------------------------------------------------------------------- create namespace pxc-operator ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified namespace/pxc-operator created Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2539-dcdf1e8a-1-cluster14" modified. ----------------------------------------------------------------------------------- start PXC operator ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterbackups.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterrestores.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusters.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusters.pxc.percona.com condition met E0702 02:59:52.945264 30130 reflector.go:227] "Failed to watch" err="Get \"https://34.61.66.163/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dperconaxtradbclusters.pxc.percona.com&resourceVersion=1782961188622966000&timeoutSeconds=348&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterbackups.pxc.percona.com condition met E0702 02:59:55.878025 30529 reflector.go:227] "Failed to watch" err="Get \"https://34.61.66.163/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dperconaxtradbclusterbackups.pxc.percona.com&resourceVersion=1782961193623249000&timeoutSeconds=482&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterrestores.pxc.percona.com condition met E0702 02:59:58.417265 30953 reflector.go:227] "Failed to watch" err="Get \"https://34.61.66.163/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dperconaxtradbclusterrestores.pxc.percona.com&resourceVersion=1782961193623249000&timeoutSeconds=552&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" clusterrole.rbac.authorization.k8s.io/percona-xtradb-cluster-operator unchanged serviceaccount/percona-xtradb-cluster-operator created clusterrolebinding.rbac.authorization.k8s.io/service-account-percona-xtradb-cluster-operator unchanged deployment.apps/percona-xtradb-cluster-operator created service/percona-xtradb-cluster-operator created pod/percona-xtradb-cluster-operator-65b5d85575-zsmtp condition met E0702 03:00:22.760087 1935 reflector.go:227] "Failed to watch" err="Get \"https://34.61.66.163/api/v1/namespaces/pxc-operator/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpercona-xtradb-cluster-operator-65b5d85575-zsmtp&resourceVersion=1782961222401330000&timeoutSeconds=375&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" pod/percona-xtradb-cluster-operator-65b5d85575-zsmtp condition met E0702 03:00:31.084069 3264 reflector.go:227] "Failed to watch" err="Get \"https://34.61.66.163/api/v1/namespaces/pxc-operator/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpercona-xtradb-cluster-operator-65b5d85575-zsmtp&resourceVersion=1782961228236865000&timeoutSeconds=390&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/percona-xtradb-cluster-operator-65b5d85575-zsmtp to become Ready.Ok error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces proxysql-scheduler-3168 ----------------------------------------------------------------------------------- Error from server (NotFound): namespaces "proxysql-scheduler-3168" not found waiting for namespace/proxysql-scheduler-3168 to be deletederror: resource(s) were provided, but no name was specified Error from server (NotFound): namespaces "proxysql-scheduler-3168" not found ----------------------------------------------------------------------------------- create namespace proxysql-scheduler-3168 ----------------------------------------------------------------------------------- namespace/proxysql-scheduler-3168 created Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2539-dcdf1e8a-1-cluster14" modified. ----------------------------------------------------------------------------------- create secrets for cloud storages ----------------------------------------------------------------------------------- secret/minio-secret created secret/aws-s3-secret created secret/do-spaces-secret created secret/gcp-cs-secret created secret/azure-secret created ----------------------------------------------------------------------------------- create PXC cluster: some-name ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- create first PXC cluster ----------------------------------------------------------------------------------- secret/my-cluster-secrets created secret/some-name-ssl created secret/some-name-ssl-internal created deployment.apps/pxc-client created perconaxtradbcluster.pxc.percona.com/some-name created ----------------------------------------------------------------------------------- waiting pods to be running for some-name ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- Waiting for 3 running pods for some-name-pxc ----------------------------------------------------------------------------------- pod/some-name-pxc-0 condition met waiting for pod/some-name-pxc-0 to become Ready.Ok pod/some-name-pxc-1 condition met waiting for pod/some-name-pxc-1 to become Ready.Ok pod/some-name-pxc-2 condition met waiting for pod/some-name-pxc-2 to become Ready.Ok ----------------------------------------------------------------------------------- Waiting for 2 running pods for some-name-proxysql ----------------------------------------------------------------------------------- pod/some-name-proxysql-0 condition met E0702 03:05:01.203482 11460 reflector.go:227] "Failed to watch" err="Get \"https://34.61.66.163/api/v1/namespaces/proxysql-scheduler-3168/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-proxysql-0&resourceVersion=1782961497355011000&timeoutSeconds=318&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/some-name-proxysql-0 to become Ready.Ok pod/some-name-proxysql-1 condition met E0702 03:05:10.659220 12897 reflector.go:227] "Failed to watch" err="Get \"https://34.61.66.163/api/v1/namespaces/proxysql-scheduler-3168/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-proxysql-1&resourceVersion=1782961507355086000&timeoutSeconds=594&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/some-name-proxysql-1 to become Ready.Ok ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/some-name to be ready ----------------------------------------------------------------------------------- write data ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-7gwpj condition met E0702 03:05:53.962317 19346 reflector.go:227] "Failed to watch" err="Get \"https://34.61.66.163/api/v1/namespaces/proxysql-scheduler-3168/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-7gwpj&resourceVersion=1782961550774775000&timeoutSeconds=562&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-7gwpj to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-7gwpj condition met E0702 03:06:13.826370 22460 reflector.go:227] "Failed to watch" err="Get \"https://34.61.66.163/api/v1/namespaces/proxysql-scheduler-3168/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-7gwpj&resourceVersion=1782961571425516000&timeoutSeconds=587&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-7gwpj to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- compare initial data ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-7gwpj condition met E0702 03:06:59.384393 29100 reflector.go:227] "Failed to watch" err="Get \"https://34.61.66.163/api/v1/namespaces/proxysql-scheduler-3168/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-7gwpj&resourceVersion=1782961617609931000&timeoutSeconds=350&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-7gwpj to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-7gwpj condition met E0702 03:07:11.981728 30840 reflector.go:227] "Failed to watch" err="Get \"https://34.61.66.163/api/v1/namespaces/proxysql-scheduler-3168/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-7gwpj&resourceVersion=1782961628238497000&timeoutSeconds=420&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-7gwpj to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-7gwpj condition met E0702 03:07:22.689731 32534 reflector.go:227] "Failed to watch" err="Get \"https://34.61.66.163/api/v1/namespaces/proxysql-scheduler-3168/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-7gwpj&resourceVersion=1782961639594847000&timeoutSeconds=591&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-7gwpj to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/some-name to be ready ----------------------------------------------------------------------------------- check proxysql custom config ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- check if service and statefulset created with expected config ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- compare statefulset/some-name-pxc- ----------------------------------------------------------------------------------- [2026-07-02T03:07:58+0000] compare_kubectl: statefulset/some-name-pxc OK ----------------------------------------------------------------------------------- compare statefulset/some-name-proxysql- ----------------------------------------------------------------------------------- [2026-07-02T03:08:00+0000] compare_kubectl: statefulset/some-name-proxysql OK ----------------------------------------------------------------------------------- compare service/some-name-pxc- ----------------------------------------------------------------------------------- [2026-07-02T03:08:02+0000] compare_kubectl: service/some-name-pxc OK ----------------------------------------------------------------------------------- compare service/some-name-proxysql- ----------------------------------------------------------------------------------- [2026-07-02T03:08:03+0000] compare_kubectl: service/some-name-proxysql OK ----------------------------------------------------------------------------------- compare service/some-name-proxysql-unready- ----------------------------------------------------------------------------------- [2026-07-02T03:08:05+0000] compare_kubectl: service/some-name-proxysql-unready OK ----------------------------------------------------------------------------------- check if scheduler is enabled in all ProxySQL servers ----------------------------------------------------------------------------------- --- /mnt/jenkins/workspace/cloud-pxc-operator_PR-2539/e2e-tests/proxysql-scheduler/compare/scheduler-0.sql 2026-07-02 01:10:10.562883882 +0000 +++ /tmp/tmp.D0DPnk0HPY/scheduler-0.sql 2026-07-02 03:10:37.774980469 +0000 @@ -1 +1 @@ -1 1 2000 /usr/bin/pxc_scheduler_handler --configfile=scheduler-config.toml --configpath=/tmp NULL NULL NULL { hgW:11, hgR:10 } +error: Internal error occurred: error sending request: Post "https://10.221.0.97:10250/exec/proxysql-scheduler-3168/some-name-proxysql-0/proxysql?command=bash&command=-c&command=printf+%27%25s%5Cn%27+%22SELECT+%2A+FROM+runtime_scheduler%3B%22+%7C+mysql+-sN+-h127.0.0.1+-P6032+-uproxyadmin+-padmin_password&error=1&output=1": dial timeout, backstop