Log: /mnt/jenkins/workspace/cloud-pxc-operator_PR-2538/e2e-tests/logs/cross-site-proxysql-8-4.log Warning: version difference between client (1.36) and server (1.33) exceeds the supported minor version skew of +/-1 Warning: version difference between client (1.36) and server (1.33) exceeds the supported minor version skew of +/-1 ----------------------------------------------------------------------------------- Create source cluster ----------------------------------------------------------------------------------- error: the server doesn't have a resource type "pxc" error: the server doesn't have a resource type "pxc-backup" error: the server doesn't have a resource type "pxc-restore" error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces pxc-operator ----------------------------------------------------------------------------------- Error from server (NotFound): namespaces "pxc-operator" not found waiting for namespace/pxc-operator to be deletederror: resource(s) were provided, but no name was specified Error from server (NotFound): namespaces "pxc-operator" not found ----------------------------------------------------------------------------------- create namespace pxc-operator ----------------------------------------------------------------------------------- namespace/pxc-operator created Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2538-70c04858-1-cluster7" modified. ----------------------------------------------------------------------------------- start PXC operator ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterbackups.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterrestores.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusters.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusters.pxc.percona.com condition met customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterbackups.pxc.percona.com condition met E0702 01:29:51.289152 24579 reflector.go:227] "Failed to watch" err="Get \"https://136.65.98.190/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dperconaxtradbclusterbackups.pxc.percona.com&resourceVersion=1782955789112735007&timeoutSeconds=549&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterrestores.pxc.percona.com condition met E0702 01:29:53.431648 24794 reflector.go:227] "Failed to watch" err="Get \"https://136.65.98.190/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dperconaxtradbclusterrestores.pxc.percona.com&resourceVersion=1782955789112735007&timeoutSeconds=559&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" clusterrole.rbac.authorization.k8s.io/percona-xtradb-cluster-operator created serviceaccount/percona-xtradb-cluster-operator created clusterrolebinding.rbac.authorization.k8s.io/service-account-percona-xtradb-cluster-operator created deployment.apps/percona-xtradb-cluster-operator created service/percona-xtradb-cluster-operator created pod/percona-xtradb-cluster-operator-59848dbdcc-snktk condition met E0702 01:30:21.297166 28643 reflector.go:227] "Failed to watch" err="Get \"https://136.65.98.190/api/v1/namespaces/pxc-operator/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpercona-xtradb-cluster-operator-59848dbdcc-snktk&resourceVersion=1782955820936054000&timeoutSeconds=385&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" pod/percona-xtradb-cluster-operator-59848dbdcc-snktk condition met E0702 01:30:32.224925 30220 reflector.go:227] "Failed to watch" err="Get \"https://136.65.98.190/api/v1/namespaces/pxc-operator/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpercona-xtradb-cluster-operator-59848dbdcc-snktk&resourceVersion=1782955828602479009&timeoutSeconds=509&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/percona-xtradb-cluster-operator-59848dbdcc-snktk to become Ready.Ok error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces cross-site-proxysql-28589 ----------------------------------------------------------------------------------- Error from server (NotFound): namespaces "cross-site-proxysql-28589" not found waiting for namespace/cross-site-proxysql-28589 to be deletederror: resource(s) were provided, but no name was specified Error from server (NotFound): namespaces "cross-site-proxysql-28589" not found ----------------------------------------------------------------------------------- create namespace cross-site-proxysql-28589 ----------------------------------------------------------------------------------- namespace/cross-site-proxysql-28589 created Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2538-70c04858-1-cluster7" modified. ----------------------------------------------------------------------------------- create secrets for cloud storages ----------------------------------------------------------------------------------- secret/minio-secret created secret/aws-s3-secret created secret/do-spaces-secret created secret/gcp-cs-secret created secret/azure-secret created "hashicorp" already exists with the same configuration, skipping "minio" already exists with the same configuration, skipping Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "minio" chart repository ...Successfully got an update from the "hashicorp" chart repository Update Complete. ⎈Happy Helming!⎈ ----------------------------------------------------------------------------------- install Minio ----------------------------------------------------------------------------------- Error: uninstall: Release not loaded: minio-service: release: not found NAME: minio-service LAST DEPLOYED: Thu Jul 2 01:31:31 2026 NAMESPACE: cross-site-proxysql-28589 STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: MinIO can be accessed via port 9000 on the following DNS name from within your cluster: minio-service.cross-site-proxysql-28589.cluster.local To access MinIO from localhost, run the below commands: 1. export POD_NAME=$(kubectl get pods --namespace cross-site-proxysql-28589 -l "release=minio-service" -o jsonpath="{.items[0].metadata.name}") 2. kubectl port-forward $POD_NAME 9000 --namespace cross-site-proxysql-28589 Read more about port forwarding here: http://kubernetes.io/docs/user-guide/kubectl/kubectl_port-forward/ You can now access MinIO server on http://localhost:9000. Follow the below steps to connect to MinIO server with mc client: 1. Download the MinIO mc client - https://min.io/docs/minio/linux/reference/minio-mc.html#quickstart 2. export MC_HOST_minio-service-local=http://$(kubectl get secret --namespace cross-site-proxysql-28589 minio-service -o jsonpath="{.data.rootUser}" | base64 --decode):$(kubectl get secret --namespace cross-site-proxysql-28589 minio-service -o jsonpath="{.data.rootPassword}" | base64 --decode)@localhost:9000 3. mc ls minio-service-local pod/minio-service-5fd5489bdc-qq79l condition met E0702 01:35:11.592900 29246 reflector.go:227] "Failed to watch" err="Get \"https://136.65.98.190/api/v1/namespaces/cross-site-proxysql-28589/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dminio-service-5fd5489bdc-qq79l&resourceVersion=1782956110841352000&timeoutSeconds=595&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/minio-service-5fd5489bdc-qq79l to become Ready.Ok make_bucket: operator-testing pod "aws-cli" deleted from cross-site-proxysql-28589 namespace All commands and output from this session will be recorded in container logs, including credentials and sensitive information passed through the command prompt. If you don't see a command prompt, try pressing enter. ----------------------------------------------------------------------------------- create secrets for cloud storages ----------------------------------------------------------------------------------- secret/minio-secret unchanged secret/aws-s3-secret unchanged secret/do-spaces-secret unchanged secret/gcp-cs-secret unchanged secret/azure-secret unchanged ----------------------------------------------------------------------------------- create first PXC cluster ----------------------------------------------------------------------------------- secret/my-cluster-secrets created secret/some-name-ssl created secret/some-name-ssl-internal created deployment.apps/pxc-client created perconaxtradbcluster.pxc.percona.com/cross-site-source created ----------------------------------------------------------------------------------- waiting pods to be running for cross-site-source ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- Waiting for 3 running pods for cross-site-source-pxc ----------------------------------------------------------------------------------- pod/cross-site-source-pxc-0 condition met waiting for pod/cross-site-source-pxc-0 to become Ready.Ok pod/cross-site-source-pxc-1 condition met waiting for pod/cross-site-source-pxc-1 to become Ready.Ok pod/cross-site-source-pxc-2 condition met waiting for pod/cross-site-source-pxc-2 to become Ready.Ok ----------------------------------------------------------------------------------- Waiting for 3 running pods for cross-site-source-proxysql ----------------------------------------------------------------------------------- pod/cross-site-source-proxysql-0 condition met E0702 01:40:02.755226 604 reflector.go:227] "Failed to watch" err="Get \"https://136.65.98.190/api/v1/namespaces/cross-site-proxysql-28589/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dcross-site-source-proxysql-0&resourceVersion=1782956399425323000&timeoutSeconds=518&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/cross-site-source-proxysql-0 to become Ready.Ok pod/cross-site-source-proxysql-1 condition met E0702 01:40:20.077067 3548 reflector.go:227] "Failed to watch" err="Get \"https://136.65.98.190/api/v1/namespaces/cross-site-proxysql-28589/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dcross-site-source-proxysql-1&resourceVersion=1782956419425530000&timeoutSeconds=346&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/cross-site-source-proxysql-1 to become Ready.Ok pod/cross-site-source-proxysql-2 condition met E0702 01:40:33.433031 5636 reflector.go:227] "Failed to watch" err="Get \"https://136.65.98.190/api/v1/namespaces/cross-site-proxysql-28589/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dcross-site-source-proxysql-2&resourceVersion=1782956429425593000&timeoutSeconds=335&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/cross-site-source-proxysql-2 to become Ready.Ok ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/cross-site-source to be ready ----------------------------------------------------------------------------------- write data ----------------------------------------------------------------------------------- pod/pxc-client-56fd5498cd-vqznf condition met E0702 01:41:38.787801 15100 reflector.go:227] "Failed to watch" err="Get \"https://136.65.98.190/api/v1/namespaces/cross-site-proxysql-28589/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-vqznf&resourceVersion=1782956494763254000&timeoutSeconds=303&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-vqznf to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-56fd5498cd-vqznf condition met E0702 01:42:15.335106 20561 reflector.go:227] "Failed to watch" err="Get \"https://136.65.98.190/api/v1/namespaces/cross-site-proxysql-28589/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-vqznf&resourceVersion=1782956531937111000&timeoutSeconds=327&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-vqznf to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- compare initial data ----------------------------------------------------------------------------------- pod/pxc-client-56fd5498cd-vqznf condition met E0702 01:43:21.177575 30318 reflector.go:227] "Failed to watch" err="Get \"https://136.65.98.190/api/v1/namespaces/cross-site-proxysql-28589/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-vqznf&resourceVersion=1782956597606064000&timeoutSeconds=318&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-vqznf to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-56fd5498cd-vqznf condition met E0702 01:43:37.221950 311 reflector.go:227] "Failed to watch" err="Get \"https://136.65.98.190/api/v1/namespaces/cross-site-proxysql-28589/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-vqznf&resourceVersion=1782956613980281000&timeoutSeconds=330&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-vqznf to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-56fd5498cd-vqznf condition met E0702 01:43:57.311586 3262 reflector.go:227] "Failed to watch" err="Get \"https://136.65.98.190/api/v1/namespaces/cross-site-proxysql-28589/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-vqznf&resourceVersion=1782956634058498000&timeoutSeconds=499&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-vqznf to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- get main cluster services endpoints ----------------------------------------------------------------------------------- pod/pxc-client-56fd5498cd-vqznf condition met E0702 01:45:09.480706 13391 reflector.go:227] "Failed to watch" err="Get \"https://136.65.98.190/api/v1/namespaces/cross-site-proxysql-28589/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-vqznf&resourceVersion=1782956704973569000&timeoutSeconds=523&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-vqznf to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- patch source cluster with replicationChannels settings ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/cross-site-source patched ----------------------------------------------------------------------------------- patch main cluster secrets with replication user ----------------------------------------------------------------------------------- secret/my-cluster-secrets patched ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/cross-site-source to be ready ----------------------------------------------------------------------------------- write data to source cluster ----------------------------------------------------------------------------------- pod/pxc-client-56fd5498cd-vqznf condition met E0702 01:46:27.955519 24935 reflector.go:227] "Failed to watch" err="Get \"https://136.65.98.190/api/v1/namespaces/cross-site-proxysql-28589/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-vqznf&resourceVersion=1782956783744407000&timeoutSeconds=567&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-vqznf to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-56fd5498cd-vqznf condition met E0702 01:46:48.414340 28404 reflector.go:227] "Failed to watch" err="Get \"https://136.65.98.190/api/v1/namespaces/cross-site-proxysql-28589/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-vqznf&resourceVersion=1782956807365281000&timeoutSeconds=504&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-vqznf to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- take backup of source cluster ----------------------------------------------------------------------------------- [2026-07-02T01:47:05+0000] run pxc-backup/backup-minio-source perconaxtradbclusterbackup.pxc.percona.com/backup-minio-source created waiting for pxc-backup/backup-minio-source to reach Succeeded state.......Succeeded ----------------------------------------------------------------------------------- create replica cluster ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces cross-site-proxysql-replica-29606 ----------------------------------------------------------------------------------- Error from server (NotFound): namespaces "cross-site-proxysql-replica-29606" not found waiting for namespace/cross-site-proxysql-replica-29606 to be deletedError from server (NotFound): namespaces "cross-site-proxysql-replica-29606" not found ----------------------------------------------------------------------------------- create namespace cross-site-proxysql-replica-29606 ----------------------------------------------------------------------------------- namespace/cross-site-proxysql-replica-29606 created Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2538-70c04858-1-cluster7" modified. ----------------------------------------------------------------------------------- start PXC operator ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterbackups.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterrestores.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusters.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusters.pxc.percona.com condition met E0702 01:48:06.222379 6679 reflector.go:227] "Failed to watch" err="Get \"https://136.65.98.190/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dperconaxtradbclusters.pxc.percona.com&resourceVersion=1782956885837054000&timeoutSeconds=421&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterbackups.pxc.percona.com condition met E0702 01:48:10.873440 7433 reflector.go:227] "Failed to watch" err="Get \"https://136.65.98.190/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dperconaxtradbclusterbackups.pxc.percona.com&resourceVersion=1782956885837054000&timeoutSeconds=498&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterrestores.pxc.percona.com condition met E0702 01:48:13.722468 8098 reflector.go:227] "Failed to watch" err="Get \"https://136.65.98.190/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dperconaxtradbclusterrestores.pxc.percona.com&resourceVersion=1782956890837104000&timeoutSeconds=329&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" clusterrole.rbac.authorization.k8s.io/percona-xtradb-cluster-operator unchanged serviceaccount/percona-xtradb-cluster-operator created clusterrolebinding.rbac.authorization.k8s.io/service-account-percona-xtradb-cluster-operator unchanged deployment.apps/percona-xtradb-cluster-operator created service/percona-xtradb-cluster-operator created pod/percona-xtradb-cluster-operator-59848dbdcc-7z52l condition met pod/percona-xtradb-cluster-operator-59848dbdcc-snktk condition met E0702 01:48:58.729876 14672 reflector.go:227] "Failed to watch" err="Get \"https://136.65.98.190/api/v1/namespaces/pxc-operator/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpercona-xtradb-cluster-operator-59848dbdcc-snktk&resourceVersion=1782956935628091000&timeoutSeconds=419&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/percona-xtradb-cluster-operator-59848dbdcc-snktk to become Ready.Ok secret/cross-site-replica-ssl-internal created ----------------------------------------------------------------------------------- create secrets for cloud storages ----------------------------------------------------------------------------------- secret/minio-secret created secret/aws-s3-secret created secret/do-spaces-secret created secret/gcp-cs-secret created secret/azure-secret created ----------------------------------------------------------------------------------- create first PXC cluster ----------------------------------------------------------------------------------- secret/my-cluster-secrets created secret/some-name-ssl created secret/some-name-ssl-internal created deployment.apps/pxc-client created perconaxtradbcluster.pxc.percona.com/cross-site-replica created ----------------------------------------------------------------------------------- waiting pods to be running for cross-site-replica ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- Waiting for 3 running pods for cross-site-replica-pxc ----------------------------------------------------------------------------------- pod/cross-site-replica-pxc-0 condition met waiting for pod/cross-site-replica-pxc-0 to become Ready.Ok pod/cross-site-replica-pxc-1 condition met waiting for pod/cross-site-replica-pxc-1 to become Ready.Ok pod/cross-site-replica-pxc-2 condition met waiting for pod/cross-site-replica-pxc-2 to become Ready.Ok ----------------------------------------------------------------------------------- Waiting for 3 running pods for cross-site-replica-proxysql ----------------------------------------------------------------------------------- pod/cross-site-replica-proxysql-0 condition met E0702 01:53:53.451363 25526 reflector.go:227] "Failed to watch" err="Get \"https://136.65.98.190/api/v1/namespaces/cross-site-proxysql-replica-29606/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dcross-site-replica-proxysql-0&resourceVersion=1782957231890398000&timeoutSeconds=427&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/cross-site-replica-proxysql-0 to become Ready.Ok pod/cross-site-replica-proxysql-1 condition met E0702 01:54:04.346246 27434 reflector.go:227] "Failed to watch" err="Get \"https://136.65.98.190/api/v1/namespaces/cross-site-proxysql-replica-29606/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dcross-site-replica-proxysql-1&resourceVersion=1782957241890538000&timeoutSeconds=459&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/cross-site-replica-proxysql-1 to become Ready.Ok pod/cross-site-replica-proxysql-2 condition met E0702 01:54:17.822536 29022 reflector.go:227] "Failed to watch" err="Get \"https://136.65.98.190/api/v1/namespaces/cross-site-proxysql-replica-29606/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dcross-site-replica-proxysql-2&resourceVersion=1782957256890812000&timeoutSeconds=356&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/cross-site-replica-proxysql-2 to become Ready.Ok ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/cross-site-replica to be ready ----------------------------------------------------------------------------------- write data ----------------------------------------------------------------------------------- pod/pxc-client-56fd5498cd-zk74k condition met E0702 01:54:58.297601 2657 reflector.go:227] "Failed to watch" err="Get \"https://136.65.98.190/api/v1/namespaces/cross-site-proxysql-replica-29606/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-zk74k&resourceVersion=1782957296273458000&timeoutSeconds=575&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-zk74k to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-56fd5498cd-zk74k condition met E0702 01:55:13.736958 4845 reflector.go:227] "Failed to watch" err="Get \"https://136.65.98.190/api/v1/namespaces/cross-site-proxysql-replica-29606/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-zk74k&resourceVersion=1782957311379383000&timeoutSeconds=387&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-zk74k to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- compare initial data ----------------------------------------------------------------------------------- pod/pxc-client-56fd5498cd-zk74k condition met E0702 01:56:02.431627 12378 reflector.go:227] "Failed to watch" err="Get \"https://136.65.98.190/api/v1/namespaces/cross-site-proxysql-replica-29606/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-zk74k&resourceVersion=1782957361328374000&timeoutSeconds=339&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-zk74k to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-56fd5498cd-zk74k condition met E0702 01:56:14.715653 13924 reflector.go:227] "Failed to watch" err="Get \"https://136.65.98.190/api/v1/namespaces/cross-site-proxysql-replica-29606/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-zk74k&resourceVersion=1782957371671892000&timeoutSeconds=443&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-zk74k to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-56fd5498cd-zk74k condition met E0702 01:56:28.736346 16093 reflector.go:227] "Failed to watch" err="Get \"https://136.65.98.190/api/v1/namespaces/cross-site-proxysql-replica-29606/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-zk74k&resourceVersion=1782957386286627000&timeoutSeconds=561&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-zk74k to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- restore backup from source cluster ----------------------------------------------------------------------------------- perconaxtradbclusterrestore.pxc.percona.com/backup-minio created ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/cross-site-replica to be ready..................... ----------------------------------------------------------------------------------- get replica cluster services endpoints ----------------------------------------------------------------------------------- pod/pxc-client-56fd5498cd-zk74k condition met E0702 02:03:13.511689 12190 reflector.go:227] "Failed to watch" err="Get \"https://136.65.98.190/api/v1/namespaces/cross-site-proxysql-replica-29606/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-zk74k&resourceVersion=1782957791590093000&timeoutSeconds=478&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-zk74k to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-56fd5498cd-zk74k condition met E0702 02:03:26.448003 14366 reflector.go:227] "Failed to watch" err="Get \"https://136.65.98.190/api/v1/namespaces/cross-site-proxysql-replica-29606/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-zk74k&resourceVersion=1782957805620167000&timeoutSeconds=461&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-zk74k to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- patch replica cluster with replicationChannels settings ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/cross-site-replica patched ----------------------------------------------------------------------------------- patch replica cluster secrets with replication user ----------------------------------------------------------------------------------- secret/my-cluster-secrets patched ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/cross-site-replica to be ready.. ----------------------------------------------------------------------------------- Check replication works between source -> replica ----------------------------------------------------------------------------------- pod/pxc-client-56fd5498cd-zk74k condition met E0702 02:05:20.464843 31155 reflector.go:227] "Failed to watch" err="Get \"https://136.65.98.190/api/v1/namespaces/cross-site-proxysql-replica-29606/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-zk74k&resourceVersion=1782957916604507000&timeoutSeconds=357&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-zk74k to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-56fd5498cd-zk74k condition met E0702 02:05:41.641346 1823 reflector.go:227] "Failed to watch" err="Get \"https://136.65.98.190/api/v1/namespaces/cross-site-proxysql-replica-29606/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-zk74k&resourceVersion=1782957936910266000&timeoutSeconds=341&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-zk74k to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-56fd5498cd-zk74k condition met E0702 02:05:57.260117 4481 reflector.go:227] "Failed to watch" err="Get \"https://136.65.98.190/api/v1/namespaces/cross-site-proxysql-replica-29606/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-zk74k&resourceVersion=1782957955228325000&timeoutSeconds=378&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-zk74k to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- check that ProxySQL on replica cluster does not have writer hostgroup (11) ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- verify cluster stability and no ProxySQL pod restarts over 60 seconds ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/cross-site-replica to be ready ----------------------------------------------------------------------------------- check read access via ProxySQL on replica cluster ----------------------------------------------------------------------------------- pod/pxc-client-56fd5498cd-zk74k condition met E0702 02:07:52.905831 21173 reflector.go:227] "Failed to watch" err="Get \"https://136.65.98.190/api/v1/namespaces/cross-site-proxysql-replica-29606/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-zk74k&resourceVersion=1782958068867532000&timeoutSeconds=521&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-zk74k to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok [2026-07-02T02:08:08+0000] run pxc-backup/backup-minio-replica perconaxtradbclusterbackup.pxc.percona.com/backup-minio-replica created waiting for pxc-backup/backup-minio-replica to reach Succeeded state..........Succeeded ----------------------------------------------------------------------------------- Switch clusters over ----------------------------------------------------------------------------------- Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2538-70c04858-1-cluster7" modified. ----------------------------------------------------------------------------------- rebuild source cluster ----------------------------------------------------------------------------------- perconaxtradbclusterrestore.pxc.percona.com/backup-minio created ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/cross-site-source to be ready................... pod/pxc-client-56fd5498cd-vqznf condition met E0702 02:15:43.942190 25509 reflector.go:227] "Failed to watch" err="Get \"https://136.65.98.190/api/v1/namespaces/cross-site-proxysql-28589/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-vqznf&resourceVersion=1782958540929245000&timeoutSeconds=579&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-vqznf to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-56fd5498cd-vqznf condition met E0702 02:16:03.498963 28396 reflector.go:227] "Failed to watch" err="Get \"https://136.65.98.190/api/v1/namespaces/cross-site-proxysql-28589/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-vqznf&resourceVersion=1782958560337511000&timeoutSeconds=349&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-vqznf to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- configure old replica as source ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/cross-site-replica patched perconaxtradbcluster.pxc.percona.com/cross-site-replica patched ----------------------------------------------------------------------------------- wait for promoted cluster to be ready ----------------------------------------------------------------------------------- Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2538-70c04858-1-cluster7" modified. ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/cross-site-replica to be ready.... ----------------------------------------------------------------------------------- check that ProxySQL on promoted cluster has writer (11) and reader (10) hostgroups ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- write data via ProxySQL on promoted cluster and verify ----------------------------------------------------------------------------------- pod/pxc-client-56fd5498cd-zk74k condition met E0702 02:19:55.957794 2651 reflector.go:227] "Failed to watch" err="Get \"https://136.65.98.190/api/v1/namespaces/cross-site-proxysql-replica-29606/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-zk74k&resourceVersion=1782958791804092000&timeoutSeconds=411&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-zk74k to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-56fd5498cd-zk74k condition met E0702 02:20:18.887410 6559 reflector.go:227] "Failed to watch" err="Get \"https://136.65.98.190/api/v1/namespaces/cross-site-proxysql-replica-29606/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-zk74k&resourceVersion=1782958814141666000&timeoutSeconds=417&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-zk74k to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-56fd5498cd-zk74k condition met E0702 02:20:39.331242 10337 reflector.go:227] "Failed to watch" err="Get \"https://136.65.98.190/api/v1/namespaces/cross-site-proxysql-replica-29606/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-zk74k&resourceVersion=1782958834497166000&timeoutSeconds=504&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-zk74k to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2538-70c04858-1-cluster7" modified. ----------------------------------------------------------------------------------- configure old source as replica ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/cross-site-source patched perconaxtradbcluster.pxc.percona.com/cross-site-source patched Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2538-70c04858-1-cluster7" modified. ----------------------------------------------------------------------------------- Write data to replica cluster ----------------------------------------------------------------------------------- pod/pxc-client-56fd5498cd-zk74k condition met E0702 02:21:14.466798 16133 reflector.go:227] "Failed to watch" err="Get \"https://136.65.98.190/api/v1/namespaces/cross-site-proxysql-replica-29606/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-zk74k&resourceVersion=1782958874206127022&timeoutSeconds=510&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-zk74k to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-56fd5498cd-zk74k condition met E0702 02:21:36.839116 19739 reflector.go:227] "Failed to watch" err="Get \"https://136.65.98.190/api/v1/namespaces/cross-site-proxysql-replica-29606/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-zk74k&resourceVersion=1782958896271983001&timeoutSeconds=477&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-zk74k to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- Check replication works between replica -> source ----------------------------------------------------------------------------------- Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2538-70c04858-1-cluster7" modified. pod/pxc-client-56fd5498cd-vqznf condition met E0702 02:22:13.864770 26092 reflector.go:227] "Failed to watch" err="Get \"https://136.65.98.190/api/v1/namespaces/cross-site-proxysql-28589/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-vqznf&resourceVersion=1782958929290362000&timeoutSeconds=425&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-vqznf to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-56fd5498cd-vqznf condition met E0702 02:22:35.290627 29735 reflector.go:227] "Failed to watch" err="Get \"https://136.65.98.190/api/v1/namespaces/cross-site-proxysql-28589/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-vqznf&resourceVersion=1782958950308746000&timeoutSeconds=364&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-vqznf to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-56fd5498cd-vqznf condition met E0702 02:23:01.035783 1556 reflector.go:227] "Failed to watch" err="Get \"https://136.65.98.190/api/v1/namespaces/cross-site-proxysql-28589/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-vqznf&resourceVersion=1782958976659935000&timeoutSeconds=507&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-vqznf to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- destroy cluster/operator and all other resources ----------------------------------------------------------------------------------- perconaxtradbclusterbackup.pxc.percona.com "backup-minio-source" deleted from cross-site-proxysql-28589 namespace perconaxtradbclusterbackup.pxc.percona.com "backup-minio-replica" deleted from cross-site-proxysql-replica-29606 namespace perconaxtradbcluster.pxc.percona.com/cross-site-source patched perconaxtradbcluster.pxc.percona.com/cross-site-replica patched perconaxtradbcluster.pxc.percona.com "cross-site-source" deleted from cross-site-proxysql-28589 namespace perconaxtradbcluster.pxc.percona.com "cross-site-replica" deleted from cross-site-proxysql-replica-29606 namespace perconaxtradbclusterrestore.pxc.percona.com "backup-minio" deleted from cross-site-proxysql-28589 namespace perconaxtradbclusterrestore.pxc.percona.com "backup-minio" deleted from cross-site-proxysql-replica-29606 namespace validatingwebhookconfiguration.admissionregistration.k8s.io "percona-xtradbcluster-webhook" deleted namespace "pxc-operator" force deleted Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. namespace "cross-site-proxysql-28589" force deleted Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. ----------------------------------------------------------------------------------- destroy cluster/operator and all other resources ----------------------------------------------------------------------------------- No resources found Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): namespaces "pxc-operator" not found namespace "cross-site-proxysql-replica-29606" force deleted Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. ----------------------------------------------------------------------------------- test passed -----------------------------------------------------------------------------------