Log: /mnt/jenkins/workspace/cloud-pxc-operator_PR-1706/e2e-tests/logs/scheduled-backup-5-7.log WARNING: version difference between client (1.30) and server (1.26) exceeds the supported minor version skew of +/-1 WARNING: version difference between client (1.30) and server (1.26) exceeds the supported minor version skew of +/-1 No resources found + kubectl patch pxc -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: resource(s) were provided, but no name was specified No resources found No resources found No resources found error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces pxc-operator ----------------------------------------------------------------------------------- Error from server (NotFound): namespaces "pxc-operator" not found namespace/pxc-operator - Error from server (NotFound): namespaces "pxc-operator" not found ----------------------------------------------------------------------------------- create namespace pxc-operator ----------------------------------------------------------------------------------- Error from server (Forbidden): namespaces "default" is forbidden: this namespace may not be deleted namespace/pxc-operator created Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-1706-347969f5-1-cluster8" modified. ----------------------------------------------------------------------------------- start PXC operator ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterbackups.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterrestores.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusters.pxc.percona.com serverside-applied clusterrole.rbac.authorization.k8s.io/percona-xtradb-cluster-operator unchanged serviceaccount/percona-xtradb-cluster-operator created clusterrolebinding.rbac.authorization.k8s.io/service-account-percona-xtradb-cluster-operator unchanged deployment.apps/percona-xtradb-cluster-operator created service/percona-xtradb-cluster-operator created pod/percona-xtradb-cluster-operator-795676b7f9-6q5fr condition met pod/percona-xtradb-cluster-operator-795676b7f9-6q5fr condition met percona-xtradb-cluster-operator-795676b7f9-6q5fr.Ok error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces scheduled-backup-2932 ----------------------------------------------------------------------------------- Error from server (NotFound): namespaces "scheduled-backup-2932" not found namespace/scheduled-backup-2932 - Error from server (Forbidden): namespaces "default" is forbidden: this namespace may not be deleted Error from server (NotFound): namespaces "scheduled-backup-2932" not found ----------------------------------------------------------------------------------- create namespace scheduled-backup-2932 ----------------------------------------------------------------------------------- namespace/scheduled-backup-2932 created Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-1706-347969f5-1-cluster8" modified. ----------------------------------------------------------------------------------- create secrets for cloud storages ----------------------------------------------------------------------------------- secret/minio-secret created secret/aws-s3-secret created secret/gcp-cs-secret created secret/azure-secret created "hashicorp" already exists with the same configuration, skipping "minio" already exists with the same configuration, skipping Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "minio" chart repository ...Successfully got an update from the "hashicorp" chart repository ...Successfully got an update from the "chaos-mesh" chart repository ...Successfully got an update from the "percona" chart repository Update Complete. ⎈Happy Helming!⎈ ----------------------------------------------------------------------------------- install Minio ----------------------------------------------------------------------------------- Error: uninstall: Release not loaded: minio-service: release: not found NAME: minio-service LAST DEPLOYED: Thu May 2 02:35:50 2024 NAMESPACE: scheduled-backup-2932 STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: MinIO can be accessed via port 9000 on the following DNS name from within your cluster: minio-service.scheduled-backup-2932.svc.cluster.local To access MinIO from localhost, run the below commands: 1. export POD_NAME=$(kubectl get pods --namespace scheduled-backup-2932 -l "release=minio-service" -o jsonpath="{.items[0].metadata.name}") 2. kubectl port-forward $POD_NAME 9000 --namespace scheduled-backup-2932 Read more about port forwarding here: http://kubernetes.io/docs/user-guide/kubectl/kubectl_port-forward/ You can now access MinIO server on http://localhost:9000. Follow the below steps to connect to MinIO server with mc client: 1. Download the MinIO mc client - https://min.io/docs/minio/linux/reference/minio-mc.html#quickstart 2. export MC_HOST_minio-service-local=http://$(kubectl get secret --namespace scheduled-backup-2932 minio-service -o jsonpath="{.data.rootUser}" | base64 --decode):$(kubectl get secret --namespace scheduled-backup-2932 minio-service -o jsonpath="{.data.rootPassword}" | base64 --decode)@localhost:9000 3. mc ls minio-service-local pod/minio-service-76ffcfd45-kfgdt condition met minio-service-76ffcfd45-kfgdt.Ok make_bucket: operator-testing pod "aws-cli" deleted If you don't see a command prompt, try pressing enter. warning: couldn't attach to pod/aws-cli, falling back to streaming logs: unable to upgrade connection: container aws-cli not found in pod aws-cli_scheduled-backup-2932 priorityclass.scheduling.k8s.io/high-priority created ----------------------------------------------------------------------------------- create first PXC cluster ----------------------------------------------------------------------------------- secret/my-cluster-secrets created secret/some-name-ssl created secret/some-name-ssl-internal created deployment.apps/pxc-client created perconaxtradbcluster.pxc.percona.com/scheduled-backup created ----------------------------------------------------------------------------------- check if all 3 Pods started ----------------------------------------------------------------------------------- error: no matching resources found ----------------------------------------------------------------------------------- wait for running cluster ----------------------------------------------------------------------------------- pod/scheduled-backup-proxysql-0 condition met scheduled-backup-proxysql-0.Ok ----------------------------------------------------------------------------------- wait for running cluster ----------------------------------------------------------------------------------- pod/scheduled-backup-pxc-0 condition met scheduled-backup-pxc-0.Ok pod/scheduled-backup-pxc-1 condition met scheduled-backup-pxc-1.Ok pod/scheduled-backup-pxc-2 condition met scheduled-backup-pxc-2.Ok ----------------------------------------------------------------------------------- write data ----------------------------------------------------------------------------------- Unable to use a TTY - input is not a terminal or the right kind of file pod/pxc-client-64b479df95-f79kz condition met pxc-client-64b479df95-f79kz.Ok pod/pxc-client-64b479df95-f79kz condition met pxc-client-64b479df95-f79kz.Ok pod/pxc-client-64b479df95-f79kz condition met pxc-client-64b479df95-f79kz.Ok pod/pxc-client-64b479df95-f79kz condition met pxc-client-64b479df95-f79kz.Ok pod/pxc-client-64b479df95-f79kz condition met pxc-client-64b479df95-f79kz.Ok Unable to use a TTY - input is not a terminal or the right kind of file ----------------------------------------------------------------------------------- add backups schedule for pvc storage ----------------------------------------------------------------------------------- Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-1706-347969f5-1-cluster8" modified. perconaxtradbcluster.pxc.percona.com/scheduled-backup configured node/gke-jen-pxc-1706-347969f-default-pool-3b814c7f-4bvp labeled perconaxtradbcluster.pxc.percona.com/scheduled-backup configured ----------------------------------------------------------------------------------- wait for running cluster ----------------------------------------------------------------------------------- error: timed out waiting for the condition on pods/scheduled-backup-pxc-0 scheduled-backup-pxc-0.Ok error: timed out waiting for the condition on pods/scheduled-backup-pxc-1 scheduled-backup-pxc-1.Ok error: timed out waiting for the condition on pods/scheduled-backup-pxc-2 scheduled-backup-pxc-2.Ok NAME CLUSTER STORAGE DESTINATION STATUS COMPLETED AGE cron-scheduled-backup-pvc-2024522432-q6fav scheduled-backup pvc pvc/xb-cron-scheduled-backup-pvc-2024522432-q6fav Running 24s NAME READY STATUS RESTARTS AGE minio-service-76ffcfd45-kfgdt 1/1 Running 0 7m35s pxc-client-64b479df95-f79kz 1/1 Running 0 6m18s scheduled-backup-proxysql-0 3/3 Running 1 (2m44s ago) 6m16s scheduled-backup-proxysql-1 3/3 Running 0 5m56s scheduled-backup-pxc-0 1/1 Running 0 6m16s scheduled-backup-pxc-1 1/1 Running 0 5m32s scheduled-backup-pxc-2 1/1 Running 0 4m19s xb-cron-scheduled-backup-pvc-2024522432-q6fav-tk562 1/1 Running 0 25s ----------------------------------------------------------------------------------- add backups schedule for aws s3 storage ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/scheduled-backup configured perconaxtradbcluster.pxc.percona.com/scheduled-backup configured ----------------------------------------------------------------------------------- wait for running cluster ----------------------------------------------------------------------------------- error: timed out waiting for the condition on pods/scheduled-backup-pxc-0 scheduled-backup-pxc-0.Ok pod/scheduled-backup-pxc-1 condition met scheduled-backup-pxc-1.Ok error: timed out waiting for the condition on pods/scheduled-backup-pxc-2 scheduled-backup-pxc-2.Ok NAME CLUSTER STORAGE DESTINATION STATUS COMPLETED AGE cron-scheduled-backup-aws-s3-20245224452-q6fav scheduled-backup aws-s3 s3://operator-testing/scheduled-backup-2024-05-02-02:44:52-full Running 18s cron-scheduled-backup-pvc-2024522432-q6fav scheduled-backup pvc pvc/xb-cron-scheduled-backup-pvc-2024522432-q6fav Succeeded 87s 2m8s NAME READY STATUS RESTARTS AGE minio-service-76ffcfd45-kfgdt 1/1 Running 0 9m19s pxc-client-64b479df95-f79kz 1/1 Running 0 8m2s scheduled-backup-proxysql-0 3/3 Running 1 (4m28s ago) 8m scheduled-backup-proxysql-1 3/3 Running 0 7m40s scheduled-backup-pxc-0 1/1 Running 0 8m scheduled-backup-pxc-1 1/1 Running 0 7m16s scheduled-backup-pxc-2 1/1 Running 0 6m3s xb-cron-scheduled-backup-aws-s3-20245224452-q6fav-crplf 1/1 Running 0 19s xb-cron-scheduled-backup-pvc-2024522432-q6fav-tk562 0/1 Completed 0 2m9s ----------------------------------------------------------------------------------- wait for running cluster ----------------------------------------------------------------------------------- error: timed out waiting for the condition on pods/scheduled-backup-pxc-0 scheduled-backup-pxc-0.Ok error: timed out waiting for the condition on pods/scheduled-backup-pxc-1 scheduled-backup-pxc-1.Ok error: timed out waiting for the condition on pods/scheduled-backup-pxc-2 scheduled-backup-pxc-2.Ok NAME CLUSTER STORAGE DESTINATION STATUS COMPLETED AGE cron-scheduled-backup-aws-s3-20245224452-q6fav scheduled-backup aws-s3 s3://operator-testing/scheduled-backup-2024-05-02-02:44:52-full Running 56s cron-scheduled-backup-pvc-2024522432-q6fav scheduled-backup pvc pvc/xb-cron-scheduled-backup-pvc-2024522432-q6fav Succeeded 2m5s 2m46s NAME READY STATUS RESTARTS AGE minio-service-76ffcfd45-5sjpl 0/1 ContainerCreating 0 21s minio-service-76ffcfd45-kfgdt 0/1 Completed 0 9m57s pxc-client-64b479df95-f79kz 0/1 Error 0 8m40s pxc-client-64b479df95-x5zfx 1/1 Running 0 5s scheduled-backup-proxysql-0 3/3 Running 1 (5m6s ago) 8m38s scheduled-backup-proxysql-1 0/3 Init:0/1 0 20s scheduled-backup-pxc-0 0/1 Pending 0 6s scheduled-backup-pxc-1 1/1 Running 0 7m54s scheduled-backup-pxc-2 1/1 Running 0 6m41s xb-cron-scheduled-backup-aws-s3-20245224452-q6fav-crplf 0/1 Completed 0 57s xb-cron-scheduled-backup-pvc-2024522432-q6fav-tk562 0/1 Completed 0 2m47s ----------------------------------------------------------------------------------- wait for running cluster ----------------------------------------------------------------------------------- error: timed out waiting for the condition on pods/scheduled-backup-pxc-0 scheduled-backup-pxc-0.Name: scheduled-backup-pxc-0 Namespace: scheduled-backup-2932 Priority: 0 Service Account: default Node: Labels: app.kubernetes.io/component=pxc app.kubernetes.io/instance=scheduled-backup app.kubernetes.io/managed-by=percona-xtradb-cluster-operator app.kubernetes.io/name=percona-xtradb-cluster app.kubernetes.io/part-of=percona-xtradb-cluster controller-revision-hash=scheduled-backup-pxc-798f687584 statefulset.kubernetes.io/pod-name=scheduled-backup-pxc-0 Annotations: cloud.google.com/cluster_autoscaler_unhelpable_since: 2024-05-02T02:45:45+0000 cloud.google.com/cluster_autoscaler_unhelpable_until: Inf percona.com/configuration-hash: d41d8cd98f00b204e9800998ecf8427e percona.com/ssl-hash: cfd6a52398268173b51d7cdb1331c09a percona.com/ssl-internal-hash: 751e59a1c1d94ac67f9d6bb6e756acdc Status: Pending IP: IPs: Controlled By: StatefulSet/scheduled-backup-pxc Init Containers: pxc-init: Image: perconalab/percona-xtradb-cluster-operator:PR-1706-347969f5 Port: Host Port: Command: /pxc-init-entrypoint.sh Limits: cpu: 50m memory: 50M Requests: cpu: 50m memory: 50M Environment: Mounts: /var/lib/mysql from datadir (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9vgsf (ro) Containers: pxc: Image: perconalab/percona-xtradb-cluster-operator:main-pxc5.7 Ports: 3306/TCP, 4444/TCP, 4567/TCP, 4568/TCP, 33062/TCP, 33060/TCP Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP Command: /var/lib/mysql/pxc-entrypoint.sh Args: mysqld Liveness: exec [/var/lib/mysql/liveness-check.sh] delay=300s timeout=5s period=10s #success=1 #failure=3 Readiness: exec [/var/lib/mysql/readiness-check.sh] delay=15s timeout=15s period=30s #success=1 #failure=5 Environment Variables from: scheduled-backup-env-vars-pxc Secret Optional: true Environment: PXC_SERVICE: scheduled-backup-pxc-unready MONITOR_HOST: % MYSQL_ROOT_PASSWORD: Optional: false XTRABACKUP_PASSWORD: Optional: false MONITOR_PASSWORD: Optional: false CLUSTER_HASH: 2043657 OPERATOR_ADMIN_PASSWORD: Optional: false LIVENESS_CHECK_TIMEOUT: 5 READINESS_CHECK_TIMEOUT: 15 DEFAULT_AUTHENTICATION_PLUGIN: mysql_native_password Mounts: /etc/my.cnf.d from auto-config (rw) /etc/mysql/init-file from mysql-init-file (rw) /etc/mysql/mysql-users-secret from mysql-users-secret-file (rw) /etc/mysql/ssl from ssl (rw) /etc/mysql/ssl-internal from ssl-internal (rw) /etc/mysql/vault-keyring-secret from vault-keyring-secret (rw) /etc/percona-xtradb-cluster.conf.d from config (rw) /tmp from tmp (rw) /var/lib/mysql from datadir (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9vgsf (ro) Conditions: Type Status PodScheduled False Volumes: datadir: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: datadir-scheduled-backup-pxc-0 ReadOnly: false tmp: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: config: Type: ConfigMap (a volume populated by a ConfigMap) Name: scheduled-backup-pxc Optional: true ssl-internal: Type: Secret (a volume populated by a Secret) SecretName: some-name-ssl-internal Optional: true ssl: Type: Secret (a volume populated by a Secret) SecretName: some-name-ssl Optional: false auto-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: auto-scheduled-backup-pxc Optional: true vault-keyring-secret: Type: Secret (a volume populated by a Secret) SecretName: scheduled-backup-vault Optional: true mysql-users-secret-file: Type: Secret (a volume populated by a Secret) SecretName: internal-scheduled-backup Optional: false mysql-init-file: Type: Secret (a volume populated by a Secret) SecretName: scheduled-backup-mysql-init Optional: true kube-api-access-9vgsf: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal NotTriggerScaleUp 31s cluster-autoscaler pod didn't trigger scale-up: Warning FailedScheduling 17s (x5 over 33s) default-scheduler 0/3 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/not-ready: }, 2 node(s) didn't match pod anti-affinity rules. preemption: 0/3 nodes are available: 1 Preemption is not helpful for scheduling, 2 No preemption victims found for incoming pod.. Defaulted container "pxc" out of: pxc, pxc-init (init) Error from server: Get "https://10.215.0.112:10250/containerLogs/pxc-operator/percona-xtradb-cluster-operator-795676b7f9-6q5fr/percona-xtradb-cluster-operator": http2: client connection lost