Log: /mnt/jenkins/workspace/cloud-pxc-operator_PR-2476/e2e-tests/logs/users-scheduler-8-4.log Warning: version difference between client (1.36) and server (1.33) exceeds the supported minor version skew of +/-1 Warning: version difference between client (1.36) and server (1.33) exceeds the supported minor version skew of +/-1 No resources found + kubectl patch pxc -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: resource(s) were provided, but no name was specified No resources found No resources found No resources found error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces pxc-operator ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified namespace "pxc-operator" deleted waiting for namespace/pxc-operator to be deletedError from server (NotFound): namespaces "pxc-operator" not found ----------------------------------------------------------------------------------- create namespace pxc-operator ----------------------------------------------------------------------------------- namespace/pxc-operator created Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2476-a8b01a39-5-cluster5" modified. ----------------------------------------------------------------------------------- start PXC operator ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterbackups.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterrestores.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusters.pxc.percona.com serverside-applied clusterrole.rbac.authorization.k8s.io/percona-xtradb-cluster-operator unchanged serviceaccount/percona-xtradb-cluster-operator created clusterrolebinding.rbac.authorization.k8s.io/service-account-percona-xtradb-cluster-operator unchanged deployment.apps/percona-xtradb-cluster-operator created service/percona-xtradb-cluster-operator created pod/percona-xtradb-cluster-operator-8548fd5788-p27c9 condition met E0516 21:44:53.003448 3303 reflector.go:227] "Failed to watch" err="Get \"https://34.173.120.16/api/v1/namespaces/pxc-operator/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpercona-xtradb-cluster-operator-8548fd5788-p27c9&resourceVersion=1778967892644871000&timeoutSeconds=466&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" pod/percona-xtradb-cluster-operator-8548fd5788-p27c9 condition met E0516 21:44:57.702594 4091 reflector.go:227] "Failed to watch" err="Get \"https://34.173.120.16/api/v1/namespaces/pxc-operator/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpercona-xtradb-cluster-operator-8548fd5788-p27c9&resourceVersion=1778967895869645000&timeoutSeconds=542&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/percona-xtradb-cluster-operator-8548fd5788-p27c9 to become Ready.Ok error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces users-scheduler-18191 ----------------------------------------------------------------------------------- Error from server (NotFound): namespaces "users-scheduler-18191" not found waiting for namespace/users-scheduler-18191 to be deletederror: resource(s) were provided, but no name was specified Error from server (NotFound): namespaces "users-scheduler-18191" not found ----------------------------------------------------------------------------------- create namespace users-scheduler-18191 ----------------------------------------------------------------------------------- namespace/users-scheduler-18191 created Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-2476-a8b01a39-5-cluster5" modified. ----------------------------------------------------------------------------------- create secrets for cloud storages ----------------------------------------------------------------------------------- secret/minio-secret created secret/aws-s3-secret created secret/do-spaces-secret created secret/gcp-cs-secret created secret/azure-secret created ----------------------------------------------------------------------------------- create PXC cluster with 1-password secret ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- create first PXC cluster ----------------------------------------------------------------------------------- secret/my-cluster-secrets created secret/some-name-ssl created secret/some-name-ssl-internal created deployment.apps/pxc-client created perconaxtradbcluster.pxc.percona.com/some-name created ----------------------------------------------------------------------------------- check if all 3 Pods started ----------------------------------------------------------------------------------- error: no matching resources found ----------------------------------------------------------------------------------- wait for running cluster ----------------------------------------------------------------------------------- pod/some-name-proxysql-0 condition met waiting for pod/some-name-proxysql-0 to become Ready.Ok ----------------------------------------------------------------------------------- wait for running cluster ----------------------------------------------------------------------------------- pod/some-name-pxc-0 condition met waiting for pod/some-name-pxc-0 to become Ready.Ok pod/some-name-pxc-1 condition met waiting for pod/some-name-pxc-1 to become Ready.Ok pod/some-name-pxc-2 condition met waiting for pod/some-name-pxc-2 to become Ready.Ok ----------------------------------------------------------------------------------- write data ----------------------------------------------------------------------------------- pod/pxc-client-56fd5498cd-czhzd condition met E0516 21:49:19.280715 32197 reflector.go:227] "Failed to watch" err="Get \"https://34.173.120.16/api/v1/namespaces/users-scheduler-18191/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-czhzd&resourceVersion=1778968157517903000&timeoutSeconds=331&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-czhzd to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-56fd5498cd-czhzd condition met E0516 21:49:28.879879 1122 reflector.go:227] "Failed to watch" err="Get \"https://34.173.120.16/api/v1/namespaces/users-scheduler-18191/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-czhzd&resourceVersion=1778968167149930000&timeoutSeconds=352&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-czhzd to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-56fd5498cd-czhzd condition met E0516 21:50:09.174823 6251 reflector.go:227] "Failed to watch" err="Get \"https://34.173.120.16/api/v1/namespaces/users-scheduler-18191/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-czhzd&resourceVersion=1778968207667209000&timeoutSeconds=375&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-czhzd to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-56fd5498cd-czhzd condition met E0516 21:50:16.624402 7216 reflector.go:227] "Failed to watch" err="Get \"https://34.173.120.16/api/v1/namespaces/users-scheduler-18191/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-czhzd&resourceVersion=1778968214984736000&timeoutSeconds=449&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-czhzd to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-56fd5498cd-czhzd condition met E0516 21:50:24.376352 8544 reflector.go:227] "Failed to watch" err="Get \"https://34.173.120.16/api/v1/namespaces/users-scheduler-18191/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-czhzd&resourceVersion=1778968223535485000&timeoutSeconds=506&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-czhzd to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok Unable to use a TTY - input is not a terminal or the right kind of file ----------------------------------------------------------------------------------- test missing passwords were created and present in internal secrets ----------------------------------------------------------------------------------- Checking root Running compare for root pod/pxc-client-56fd5498cd-czhzd condition met E0516 21:50:38.827963 10417 reflector.go:227] "Failed to watch" err="Get \"https://34.173.120.16/api/v1/namespaces/users-scheduler-18191/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-czhzd&resourceVersion=1778968237183219000&timeoutSeconds=347&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-czhzd to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok Checking xtrabackup Running compare for xtrabackup pod/pxc-client-56fd5498cd-czhzd condition met E0516 21:50:50.161412 11923 reflector.go:227] "Failed to watch" err="Get \"https://34.173.120.16/api/v1/namespaces/users-scheduler-18191/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-czhzd&resourceVersion=1778968248794938000&timeoutSeconds=447&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-czhzd to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok Checking monitor Running compare for monitor pod/pxc-client-56fd5498cd-czhzd condition met E0516 21:51:01.360289 13324 reflector.go:227] "Failed to watch" err="Get \"https://34.173.120.16/api/v1/namespaces/users-scheduler-18191/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-czhzd&resourceVersion=1778968259040621000&timeoutSeconds=348&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-czhzd to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok Checking proxyadmin Running compare for proxyadmin Checking operator Running compare for operator pod/pxc-client-56fd5498cd-czhzd condition met E0516 21:51:15.982738 15460 reflector.go:227] "Failed to watch" err="Get \"https://34.173.120.16/api/v1/namespaces/users-scheduler-18191/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-czhzd&resourceVersion=1778968274039006000&timeoutSeconds=580&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-czhzd to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok Checking replication Running compare for replication pod/pxc-client-56fd5498cd-czhzd condition met E0516 21:51:28.984018 17349 reflector.go:227] "Failed to watch" err="Get \"https://34.173.120.16/api/v1/namespaces/users-scheduler-18191/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-czhzd&resourceVersion=1778968287124524000&timeoutSeconds=300&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-czhzd to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- test root ----------------------------------------------------------------------------------- secret/my-cluster-secrets patched ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/some-name to be ready pod/pxc-client-56fd5498cd-czhzd condition met E0516 21:52:20.963842 23570 reflector.go:227] "Failed to watch" err="Get \"https://34.173.120.16/api/v1/namespaces/users-scheduler-18191/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-czhzd&resourceVersion=1778968339101489000&timeoutSeconds=556&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-czhzd to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- test proxyadmin ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/some-name patched ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/some-name to be ready secret/my-cluster-secrets patched ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/some-name to be ready.. ----------------------------------------------------------------------------------- test xtrabackup ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/some-name patched secret/my-cluster-secrets patched ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/some-name to be ready......................... ----------------------------------------------------------------------------------- test monitor ----------------------------------------------------------------------------------- secret/my-cluster-secrets patched pod/pxc-client-56fd5498cd-czhzd condition met E0516 21:57:21.996597 22128 reflector.go:227] "Failed to watch" err="Get \"https://34.173.120.16/api/v1/namespaces/users-scheduler-18191/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-czhzd&resourceVersion=1778968641169247011&timeoutSeconds=478&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-czhzd to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok {"additional_password": "$A$005$\\u001cz-JB\\u00056!\\u0016\\u001d\\u0017K6)4>x\\u001d\\u000b#Caust8K0b5fCM2plWUq9HS8Wkg0HArVGxmrIeu6EZVC"} Old password successfully discarded after 7 retries ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/some-name to be ready pod/pxc-client-56fd5498cd-czhzd condition met E0516 21:59:02.233889 561 reflector.go:227] "Failed to watch" err="Get \"https://34.173.120.16/api/v1/namespaces/users-scheduler-18191/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-czhzd&resourceVersion=1778968740640186000&timeoutSeconds=324&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-czhzd to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- test operator ----------------------------------------------------------------------------------- secret/my-cluster-secrets patched ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/some-name to be ready pod/pxc-client-56fd5498cd-czhzd condition met E0516 21:59:51.634748 4401 reflector.go:227] "Failed to watch" err="Get \"https://34.173.120.16/api/v1/namespaces/users-scheduler-18191/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-czhzd&resourceVersion=1778968790880327000&timeoutSeconds=523&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-czhzd to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- change secret name ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/some-name patched ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/some-name to be ready......................... ----------------------------------------------------------------------------------- test new operator ----------------------------------------------------------------------------------- secret/my-cluster-secrets-2 patched ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for pxc/some-name to be ready.... pod/pxc-client-56fd5498cd-czhzd condition met E0516 22:04:05.157375 18579 reflector.go:227] "Failed to watch" err="Get \"https://34.173.120.16/api/v1/namespaces/users-scheduler-18191/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-czhzd&resourceVersion=1778969044417957000&timeoutSeconds=399&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-czhzd to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- test new users sync ----------------------------------------------------------------------------------- pod/pxc-client-56fd5498cd-czhzd condition met E0516 22:04:09.456559 18883 reflector.go:227] "Failed to watch" err="Get \"https://34.173.120.16/api/v1/namespaces/users-scheduler-18191/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-czhzd&resourceVersion=1778969048704874000&timeoutSeconds=582&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-czhzd to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok pod/pxc-client-56fd5498cd-czhzd condition met E0516 22:05:33.916835 21379 reflector.go:227] "Failed to watch" err="Get \"https://34.173.120.16/api/v1/namespaces/users-scheduler-18191/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dpxc-client-56fd5498cd-czhzd&resourceVersion=1778969133176568000&timeoutSeconds=534&watch=true\": context canceled" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured" waiting for pod/pxc-client-56fd5498cd-czhzd to become ReadyDefaulted container "pxc-client" out of: pxc-client, backup .Ok ----------------------------------------------------------------------------------- check secret without operator ----------------------------------------------------------------------------------- secret/my-cluster-secrets-2 configured Warning: resource secrets/my-cluster-secrets-2 is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.