Log: /mnt/jenkins/workspace/cloud-pxc-operator_PR-2538/e2e-tests/logs/proxysql-scheduler-8-0.log Warning: version difference between client (1.36) and server (1.33) exceeds the supported minor version skew of +/-1 Warning: version difference between client (1.36) and server (1.33) exceeds the supported minor version skew of +/-1 No resources found No resources found No resources found error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces pxc-operator ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified namespace "pxc-operator" deleted waiting for namespace/pxc-operator to be deletedError from server (NotFound): namespaces "pxc-operator" not found ----------------------------------------------------------------------------------- create namespace pxc-operator ----------------------------------------------------------------------------------- namespace/pxc-operator created Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2538-70c04858-1-cluster8" modified. ----------------------------------------------------------------------------------- start PXC operator ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterbackups.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterrestores.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusters.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusters.pxc.percona.com condition met E0702 02:58:05.656740 3370 reflector.go:227] "Failed to watch" err="Get \"https://34.9.252.246/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dperconaxtradbclusters.pxc.percona.com&resourceVersion=1782961082553613000&timeoutSeconds=320&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterbackups.pxc.percona.com condition met E0702 02:58:08.547908 3850 reflector.go:227] "Failed to watch" err="Get \"https://34.9.252.246/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dperconaxtradbclusterbackups.pxc.percona.com&resourceVersion=1782961087553894000&timeoutSeconds=515&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterrestores.pxc.percona.com condition met E0702 02:58:11.123423 4307 reflector.go:227] "Failed to watch" err="Get \"https://34.9.252.246/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dperconaxtradbclusterrestores.pxc.percona.com&resourceVersion=1782961087553894000&timeoutSeconds=359&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" clusterrole.rbac.authorization.k8s.io/percona-xtradb-cluster-operator unchanged serviceaccount/percona-xtradb-cluster-operator created clusterrolebinding.rbac.authorization.k8s.io/service-account-percona-xtradb-cluster-operator unchanged deployment.apps/percona-xtradb-cluster-operator created service/percona-xtradb-cluster-operator created pod/percona-xtradb-cluster-operator-59848dbdcc-dlgpl condition met E0702 02:58:33.780852 7798 reflector.go:227] "Failed to watch" err="Get \"https://34.9.252.246/api/v1/namespaces/pxc-operator/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpercona-xtradb-cluster-operator-59848dbdcc-dlgpl&resourceVersion=1782961113412560000&timeoutSeconds=539&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" pod/percona-xtradb-cluster-operator-59848dbdcc-dlgpl condition met E0702 02:58:44.777694 9355 reflector.go:227] "Failed to watch" err="Get \"https://34.9.252.246/api/v1/namespaces/pxc-operator/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpercona-xtradb-cluster-operator-59848dbdcc-dlgpl&resourceVersion=1782961120967752000&timeoutSeconds=372&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/percona-xtradb-cluster-operator-59848dbdcc-dlgpl to become Ready.Ok error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces proxysql-scheduler-7201 ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified Error from server (NotFound): namespaces "proxysql-scheduler-7201" not found waiting for namespace/proxysql-scheduler-7201 to be deletedError from server (NotFound): namespaces "proxysql-scheduler-7201" not found ----------------------------------------------------------------------------------- create namespace proxysql-scheduler-7201 ----------------------------------------------------------------------------------- namespace/proxysql-scheduler-7201 created Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2538-70c04858-1-cluster8" modified. ----------------------------------------------------------------------------------- create secrets for cloud storages ----------------------------------------------------------------------------------- secret/minio-secret created secret/aws-s3-secret created secret/do-spaces-secret created secret/gcp-cs-secret created secret/azure-secret created ----------------------------------------------------------------------------------- create PXC cluster: some-name ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- create first PXC cluster ----------------------------------------------------------------------------------- secret/my-cluster-secrets created secret/some-name-ssl created secret/some-name-ssl-internal created deployment.apps/pxc-client created perconaxtradbcluster.pxc.percona.com/some-name created ----------------------------------------------------------------------------------- waiting pods to be running for some-name ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- Waiting for 3 running pods for some-name-pxc ----------------------------------------------------------------------------------- pod/some-name-pxc-0 condition met waiting for pod/some-name-pxc-0 to become Ready.Ok pod/some-name-pxc-1 condition met waiting for pod/some-name-pxc-1 to become Ready.Ok pod/some-name-pxc-2 condition met waiting for pod/some-name-pxc-2 to become Ready.Ok ----------------------------------------------------------------------------------- Waiting for 2 running pods for some-name-proxysql ----------------------------------------------------------------------------------- pod/some-name-proxysql-0 condition met E0702 03:03:24.923954 18246 reflector.go:227] "Failed to watch" err="Get \"https://34.9.252.246/api/v1/namespaces/proxysql-scheduler-7201/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-proxysql-0&resourceVersion=1782961403620379000&timeoutSeconds=528&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/some-name-proxysql-0 to become Ready.Ok pod/some-name-proxysql-1 condition met E0702 03:03:38.433056 20227 reflector.go:227] "Failed to watch" err="Get \"https://34.9.252.246/api/v1/namespaces/proxysql-scheduler-7201/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-proxysql-1&resourceVersion=1782961413620415000&timeoutSeconds=447&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/some-name-proxysql-1 to become Ready.Ok ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/some-name to be ready ----------------------------------------------------------------------------------- write data ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-vttxk condition met E0702 03:04:39.933608 29523 reflector.go:227] "Failed to watch" err="Get \"https://34.9.252.246/api/v1/namespaces/proxysql-scheduler-7201/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-vttxk&resourceVersion=1782961477208458000&timeoutSeconds=557&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-vttxk to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-vttxk condition met E0702 03:05:09.568270 1429 reflector.go:227] "Failed to watch" err="Get \"https://34.9.252.246/api/v1/namespaces/proxysql-scheduler-7201/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-vttxk&resourceVersion=1782961505823093000&timeoutSeconds=429&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-vttxk to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- compare initial data ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-vttxk condition met E0702 03:06:02.091001 9619 reflector.go:227] "Failed to watch" err="Get \"https://34.9.252.246/api/v1/namespaces/proxysql-scheduler-7201/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-vttxk&resourceVersion=1782961558964872000&timeoutSeconds=354&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-vttxk to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-vttxk condition met E0702 03:06:22.067066 12493 reflector.go:227] "Failed to watch" err="Get \"https://34.9.252.246/api/v1/namespaces/proxysql-scheduler-7201/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-vttxk&resourceVersion=1782961579062346000&timeoutSeconds=528&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-vttxk to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-vttxk condition met E0702 03:06:38.576795 14931 reflector.go:227] "Failed to watch" err="Get \"https://34.9.252.246/api/v1/namespaces/proxysql-scheduler-7201/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-vttxk&resourceVersion=1782961596136340000&timeoutSeconds=398&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-vttxk to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/some-name to be ready ----------------------------------------------------------------------------------- check proxysql custom config ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- check if service and statefulset created with expected config ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- compare statefulset/some-name-pxc- ----------------------------------------------------------------------------------- [2026-07-02T03:07:25+0000] compare_kubectl: statefulset/some-name-pxc OK ----------------------------------------------------------------------------------- compare statefulset/some-name-proxysql- ----------------------------------------------------------------------------------- [2026-07-02T03:07:29+0000] compare_kubectl: statefulset/some-name-proxysql OK ----------------------------------------------------------------------------------- compare service/some-name-pxc- ----------------------------------------------------------------------------------- [2026-07-02T03:07:32+0000] compare_kubectl: service/some-name-pxc OK ----------------------------------------------------------------------------------- compare service/some-name-proxysql- ----------------------------------------------------------------------------------- [2026-07-02T03:07:35+0000] compare_kubectl: service/some-name-proxysql OK ----------------------------------------------------------------------------------- compare service/some-name-proxysql-unready- ----------------------------------------------------------------------------------- [2026-07-02T03:07:38+0000] compare_kubectl: service/some-name-proxysql-unready OK ----------------------------------------------------------------------------------- check if scheduler is enabled in all ProxySQL servers ----------------------------------------------------------------------------------- [2026-07-02T03:09:42+0000] scheduler is enabled in some-name-proxysql-0: OK [2026-07-02T03:09:46+0000] scheduler is enabled in some-name-proxysql-1: OK ----------------------------------------------------------------------------------- check if scheduler is doing its job in all ProxySQL servers ----------------------------------------------------------------------------------- check runtime_mysql_servers in some-name-proxysql-0 [2026-07-02T03:09:49+0000] mysql_servers are configured in some-name-proxysql-0: OK check runtime_mysql_servers in some-name-proxysql-1 [2026-07-02T03:09:53+0000] mysql_servers are configured in some-name-proxysql-1: OK ----------------------------------------------------------------------------------- check disabling scheduler ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/some-name patched [2026-07-02T03:09:56+0000] sleeping for 90 seconds ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/some-name to be ready [2026-07-02T03:12:13+0000] scheduler is disabled in some-name-proxysql-0: OK [2026-07-02T03:12:39+0000] scheduler is disabled in some-name-proxysql-1: OK NAME READY STATUS RESTARTS AGE pxc-client-67fc4995bb-vttxk 2/2 Running 0 13m some-name-proxysql-0 3/3 Running 0 2m26s some-name-proxysql-1 3/3 Running 0 2m45s some-name-pxc-0 1/1 Running 0 12m some-name-pxc-1 1/1 Running 0 12m some-name-pxc-2 1/1 Running 0 10m NAME ENDPOINT STATUS PXC PROXYSQL HAPROXY AGE some-name some-name-proxysql.proxysql-scheduler-7201 ready 3 2 12m ----------------------------------------------------------------------------------- check enabling scheduler ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/some-name patched [2026-07-02T03:12:46+0000] sleeping for 90 seconds ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/some-name to be ready [2026-07-02T03:14:42+0000] scheduler is enabled in some-name-proxysql-0: OK [2026-07-02T03:14:45+0000] scheduler is enabled in some-name-proxysql-1: OK ----------------------------------------------------------------------------------- check PXC pod 1 is promoted to writer when pod-0 is down ----------------------------------------------------------------------------------- LOOP_PID=24425 pod "some-name-pxc-0" deleted from proxysql-scheduler-7201 namespace [2026-07-02T03:14:55+0000] waiting for pod0 to be removed from proxysql check runtime_mysql_servers in some-name-proxysql-0 [2026-07-02T03:14:58+0000] mysql_servers are configured in some-name-proxysql-0 when pxc-0 is down: OK check runtime_mysql_servers in some-name-proxysql-1 [2026-07-02T03:15:00+0000] mysql_servers are configured in some-name-proxysql-1 when pxc-0 is down: OK ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/some-name to be ready.. ----------------------------------------------------------------------------------- check scaling PXC up to 5 replicas ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/some-name patched pod/some-name-pxc-3 condition met waiting for pod/some-name-pxc-3 to become Ready.Ok pod/some-name-pxc-4 condition met waiting for pod/some-name-pxc-4 to become Ready.Ok check runtime_mysql_servers in some-name-proxysql-0 [2026-07-02T03:21:02+0000] new mysql_servers are configured in some-name-proxysql-0: OK check runtime_mysql_servers in some-name-proxysql-1 [2026-07-02T03:21:07+0000] new mysql_servers are configured in some-name-proxysql-1: OK ----------------------------------------------------------------------------------- check scaling ProxySQL up ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/some-name patched pod/some-name-proxysql-2 condition met waiting for pod/some-name-proxysql-2 to become Ready.Ok [2026-07-02T03:23:45+0000] scheduler is enabled in some-name-proxysql-2: OK check runtime_mysql_servers in some-name-proxysql-2 [2026-07-02T03:23:50+0000] mysql_servers are configured in some-name-proxysql-2: OK ----------------------------------------------------------------------------------- check writerIsAlsoReader = false ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/some-name patched ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/some-name to be ready. check runtime_mysql_servers in some-name-proxysql-0 [2026-07-02T03:25:50+0000] new mysql_servers are configured in some-name-proxysql-0: OK check runtime_mysql_servers in some-name-proxysql-1 [2026-07-02T03:25:53+0000] new mysql_servers are configured in some-name-proxysql-1: OK check runtime_mysql_servers in some-name-proxysql-2 [2026-07-02T03:25:56+0000] mysql_servers are configured in some-name-proxysql-2: OK ----------------------------------------------------------------------------------- check scaling PXC down to 3 replicas ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/some-name patched ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/some-name to be ready.. check runtime_mysql_servers in some-name-proxysql-0 [2026-07-02T03:28:02+0000] mysql_servers are configured in some-name-proxysql-0: OK check runtime_mysql_servers in some-name-proxysql-1 [2026-07-02T03:28:07+0000] mysql_servers are configured in some-name-proxysql-1: OK check runtime_mysql_servers in some-name-proxysql-2 [2026-07-02T03:28:11+0000] mysql_servers are configured in some-name-proxysql-2: OK ----------------------------------------------------------------------------------- check PXC pod 0 is reader and writer when pods 1 and 2 are down ----------------------------------------------------------------------------------- [2026-07-02T03:28:11+0000] scaling PXC down to 1 replica statefulset.apps/some-name-pxc scaled check runtime_mysql_servers in some-name-proxysql-0... [2026-07-02T03:29:22+0000] mysql_servers are configured in some-name-proxysql-0 when 2 pods are down: OK check runtime_mysql_servers in some-name-proxysql-1 [2026-07-02T03:29:26+0000] mysql_servers are configured in some-name-proxysql-1 when 2 pods are down: OK check runtime_mysql_servers in some-name-proxysql-2 [2026-07-02T03:29:30+0000] mysql_servers are configured in some-name-proxysql-2 when 2 pods are down: OK ----------------------------------------------------------------------------------- test passed ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- destroy cluster/operator and all other resources ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/some-name patched perconaxtradbcluster.pxc.percona.com "some-name" deleted from proxysql-scheduler-7201 namespace No resources found validatingwebhookconfiguration.admissionregistration.k8s.io "percona-xtradbcluster-webhook" deleted namespace "pxc-operator" force deleted Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. namespace "proxysql-scheduler-7201" force deleted Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.