Log: /mnt/jenkins/workspace/cloud-pxc-operator_PR-2521/e2e-tests/logs/sst-retry-limit-8-0.log Warning: version difference between client (1.36) and server (1.33) exceeds the supported minor version skew of +/-1 Warning: version difference between client (1.36) and server (1.33) exceeds the supported minor version skew of +/-1 No resources found No resources found No resources found error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces pxc-operator ----------------------------------------------------------------------------------- Error from server (NotFound): namespaces "pxc-operator" not found waiting for namespace/pxc-operator to be deletederror: resource(s) were provided, but no name was specified Error from server (NotFound): namespaces "pxc-operator" not found ----------------------------------------------------------------------------------- create namespace pxc-operator ----------------------------------------------------------------------------------- namespace/pxc-operator created Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2521-bb594380-1-cluster16" modified. ----------------------------------------------------------------------------------- start PXC operator ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterbackups.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterrestores.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusters.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusters.pxc.percona.com condition met E0611 04:13:00.032086 17417 reflector.go:227] "Failed to watch" err="Get \"https://34.66.15.31/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dperconaxtradbclusters.pxc.percona.com&resourceVersion=1781151177782063000&timeoutSeconds=323&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterbackups.pxc.percona.com condition met E0611 04:13:04.502423 17976 reflector.go:227] "Failed to watch" err="Get \"https://34.66.15.31/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dperconaxtradbclusterbackups.pxc.percona.com&resourceVersion=1781151182782091000&timeoutSeconds=559&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterrestores.pxc.percona.com condition met E0611 04:13:07.840387 18623 reflector.go:227] "Failed to watch" err="Get \"https://34.66.15.31/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dperconaxtradbclusterrestores.pxc.percona.com&resourceVersion=1781151182782091000&timeoutSeconds=505&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" clusterrole.rbac.authorization.k8s.io/percona-xtradb-cluster-operator unchanged serviceaccount/percona-xtradb-cluster-operator created clusterrolebinding.rbac.authorization.k8s.io/service-account-percona-xtradb-cluster-operator unchanged deployment.apps/percona-xtradb-cluster-operator created service/percona-xtradb-cluster-operator created pod/percona-xtradb-cluster-operator-774dd4b69-cxdjh condition met E0611 04:13:33.322619 22266 reflector.go:227] "Failed to watch" err="Get \"https://34.66.15.31/api/v1/namespaces/pxc-operator/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpercona-xtradb-cluster-operator-774dd4b69-cxdjh&resourceVersion=1781151212964420000&timeoutSeconds=463&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" pod/percona-xtradb-cluster-operator-774dd4b69-cxdjh condition met E0611 04:13:40.488366 23321 reflector.go:227] "Failed to watch" err="Get \"https://34.66.15.31/api/v1/namespaces/pxc-operator/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpercona-xtradb-cluster-operator-774dd4b69-cxdjh&resourceVersion=1781151217907904000&timeoutSeconds=531&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/percona-xtradb-cluster-operator-774dd4b69-cxdjh to become Ready.Ok error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces sst-retry-limit-21289 ----------------------------------------------------------------------------------- Error from server (NotFound): namespaces "sst-retry-limit-21289" not found waiting for namespace/sst-retry-limit-21289 to be deletederror: resource(s) were provided, but no name was specified Error from server (NotFound): namespaces "sst-retry-limit-21289" not found ----------------------------------------------------------------------------------- create namespace sst-retry-limit-21289 ----------------------------------------------------------------------------------- namespace/sst-retry-limit-21289 created Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2521-bb594380-1-cluster16" modified. ----------------------------------------------------------------------------------- create secrets for cloud storages ----------------------------------------------------------------------------------- secret/minio-secret created secret/aws-s3-secret created secret/do-spaces-secret created secret/gcp-cs-secret created secret/azure-secret created ----------------------------------------------------------------------------------- start 3-node cluster with SST retry limit ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- create first PXC cluster ----------------------------------------------------------------------------------- secret/my-cluster-secrets created secret/some-name-ssl created secret/some-name-ssl-internal created deployment.apps/pxc-client created perconaxtradbcluster.pxc.percona.com/some-name created ----------------------------------------------------------------------------------- waiting pods to be running for some-name ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- Waiting for 3 running pods for some-name-pxc ----------------------------------------------------------------------------------- pod/some-name-pxc-0 condition met waiting for pod/some-name-pxc-0 to become Ready.Ok pod/some-name-pxc-1 condition met waiting for pod/some-name-pxc-1 to become Ready.Ok pod/some-name-pxc-2 condition met waiting for pod/some-name-pxc-2 to become Ready.Ok ----------------------------------------------------------------------------------- No resource provided to wait for running pods ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/some-name to be ready ----------------------------------------------------------------------------------- write data ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-lkfnl condition met E0611 04:18:52.823378 2376 reflector.go:227] "Failed to watch" err="Get \"https://34.66.15.31/api/v1/namespaces/sst-retry-limit-21289/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-lkfnl&resourceVersion=1781151530887868000&timeoutSeconds=337&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-lkfnl to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-lkfnl condition met E0611 04:19:06.047671 4375 reflector.go:227] "Failed to watch" err="Get \"https://34.66.15.31/api/v1/namespaces/sst-retry-limit-21289/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-lkfnl&resourceVersion=1781151544381083000&timeoutSeconds=349&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-lkfnl to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- compare initial data ----------------------------------------------------------------------------------- pod/pxc-client-67fc4995bb-lkfnl condition met E0611 04:19:46.915236 9323 reflector.go:227] "Failed to watch" err="Get \"https://34.66.15.31/api/v1/namespaces/sst-retry-limit-21289/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-lkfnl&resourceVersion=1781151585497163000&timeoutSeconds=465&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-lkfnl to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-lkfnl condition met E0611 04:19:51.505992 9854 reflector.go:227] "Failed to watch" err="Get \"https://34.66.15.31/api/v1/namespaces/sst-retry-limit-21289/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-lkfnl&resourceVersion=1781151590752492000&timeoutSeconds=561&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-lkfnl to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-67fc4995bb-lkfnl condition met E0611 04:19:59.321188 10706 reflector.go:227] "Failed to watch" err="Get \"https://34.66.15.31/api/v1/namespaces/sst-retry-limit-21289/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-67fc4995bb-lkfnl&resourceVersion=1781151597665689000&timeoutSeconds=391&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-67fc4995bb-lkfnl to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- scale to 4 nodes to trigger SST ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/some-name patched ----------------------------------------------------------------------------------- verify the joiner pod has SST retry support wired in ----------------------------------------------------------------------------------- waiting for pod/some-name-pxc-3 container/pxc to start........ ----------------------------------------------------------------------------------- interrupt SST twice so the joiner exhausts allowed retries ----------------------------------------------------------------------------------- waiting for pod/some-name-pxc-3 restartCount=0 SST attempt.. waiting to detect active donorinterrupting donor pod/some-name-pxc-1 during SST attempt 1 pod "some-name-pxc-1" force deleted from sst-retry-limit-21289 namespace Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. waiting for pod/some-name-pxc-3 restartCount to reach 1.............................................. pod/some-name-pxc-1 condition met E0611 04:22:49.182682 32578 reflector.go:227] "Failed to watch" err="Get \"https://34.66.15.31/api/v1/namespaces/sst-retry-limit-21289/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-pxc-1&resourceVersion=1781151766567583014&timeoutSeconds=582&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/some-name-pxc-1 to become Ready.Ok waiting for pod/some-name-pxc-3 restartCount=1 SST attempt waiting to detect active donorinterrupting donor pod/some-name-pxc-1 during SST attempt 2 pod "some-name-pxc-1" force deleted from sst-retry-limit-21289 namespace Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. waiting for pod/some-name-pxc-3 restartCount to reach 2............................................ pod/some-name-pxc-1 condition met E0611 04:25:28.636559 22446 reflector.go:227] "Failed to watch" err="Get \"https://34.66.15.31/api/v1/namespaces/sst-retry-limit-21289/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name-pxc-1&resourceVersion=1781151925812239014&timeoutSeconds=353&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/some-name-pxc-1 to become Ready.Ok ----------------------------------------------------------------------------------- verify the joiner stops restarting after the retry limit is reached ----------------------------------------------------------------------------------- waiting for pod/some-name-pxc-3 logs to contain 'SST retry limit reached.' ----------------------------------------------------------------------------------- verify retries stop after the limit ----------------------------------------------------------------------------------- [2026-06-11T04:25:39+0000] sleeping for 120 seconds ----------------------------------------------------------------------------------- verify donor candidates remain healthy ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- manually recover the joiner and verify the cluster becomes ready ----------------------------------------------------------------------------------- removing SST retry marker files from pod/some-name-pxc-3 waiting for pxc/some-name to become ready with 4 PXC pods......... ----------------------------------------------------------------------------------- destroy cluster/operator and all other resources ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/some-name patched perconaxtradbcluster.pxc.percona.com "some-name" deleted from sst-retry-limit-21289 namespace No resources found validatingwebhookconfiguration.admissionregistration.k8s.io "percona-xtradbcluster-webhook" deleted namespace "pxc-operator" force deleted Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. namespace "sst-retry-limit-21289" force deleted Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. ----------------------------------------------------------------------------------- test passed -----------------------------------------------------------------------------------