Log: /mnt/jenkins/workspace/cloud-pxc-operator_PR-1729/e2e-tests/logs/scheduled-backup-5-7.log WARNING: version difference between client (1.30) and server (1.26) exceeds the supported minor version skew of +/-1 WARNING: version difference between client (1.30) and server (1.26) exceeds the supported minor version skew of +/-1 No resources found + kubectl patch pxc -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: resource(s) were provided, but no name was specified No resources found No resources found No resources found error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces pxc-operator ----------------------------------------------------------------------------------- Error from server (NotFound): namespaces "pxc-operator" not found namespace/pxc-operator - Error from server (NotFound): namespaces "pxc-operator" not found ----------------------------------------------------------------------------------- create namespace pxc-operator ----------------------------------------------------------------------------------- Error from server (Forbidden): namespaces "default" is forbidden: this namespace may not be deleted namespace/pxc-operator created Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-1729-aeb7b936-1-cluster6" modified. ----------------------------------------------------------------------------------- start PXC operator ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterbackups.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterrestores.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusters.pxc.percona.com serverside-applied clusterrole.rbac.authorization.k8s.io/percona-xtradb-cluster-operator unchanged serviceaccount/percona-xtradb-cluster-operator created clusterrolebinding.rbac.authorization.k8s.io/service-account-percona-xtradb-cluster-operator unchanged deployment.apps/percona-xtradb-cluster-operator created service/percona-xtradb-cluster-operator created pod/percona-xtradb-cluster-operator-76495f68fb-2n8m5 condition met pod/percona-xtradb-cluster-operator-76495f68fb-2n8m5 condition met percona-xtradb-cluster-operator-76495f68fb-2n8m5.Ok error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces scheduled-backup-19443 ----------------------------------------------------------------------------------- Error from server (NotFound): namespaces "scheduled-backup-19443" not found namespace/scheduled-backup-19443 - Error from server (NotFound): namespaces "scheduled-backup-19443" not found ----------------------------------------------------------------------------------- create namespace scheduled-backup-19443 ----------------------------------------------------------------------------------- Error from server (Forbidden): namespaces "default" is forbidden: this namespace may not be deleted namespace/scheduled-backup-19443 created Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-1729-aeb7b936-1-cluster6" modified. ----------------------------------------------------------------------------------- create secrets for cloud storages ----------------------------------------------------------------------------------- secret/minio-secret created secret/aws-s3-secret created secret/gcp-cs-secret created secret/azure-secret created "hashicorp" already exists with the same configuration, skipping "minio" already exists with the same configuration, skipping Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "chaos-mesh" chart repository ...Successfully got an update from the "hashicorp" chart repository ...Successfully got an update from the "percona" chart repository ...Successfully got an update from the "minio" chart repository Update Complete. ⎈Happy Helming!⎈ ----------------------------------------------------------------------------------- install Minio ----------------------------------------------------------------------------------- Error: uninstall: Release not loaded: minio-service: release: not found NAME: minio-service LAST DEPLOYED: Thu Jun 13 02:56:15 2024 NAMESPACE: scheduled-backup-19443 STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: MinIO can be accessed via port 9000 on the following DNS name from within your cluster: minio-service.scheduled-backup-19443.svc.cluster.local To access MinIO from localhost, run the below commands: 1. export POD_NAME=$(kubectl get pods --namespace scheduled-backup-19443 -l "release=minio-service" -o jsonpath="{.items[0].metadata.name}") 2. kubectl port-forward $POD_NAME 9000 --namespace scheduled-backup-19443 Read more about port forwarding here: http://kubernetes.io/docs/user-guide/kubectl/kubectl_port-forward/ You can now access MinIO server on http://localhost:9000. Follow the below steps to connect to MinIO server with mc client: 1. Download the MinIO mc client - https://min.io/docs/minio/linux/reference/minio-mc.html#quickstart 2. export MC_HOST_minio-service-local=http://$(kubectl get secret --namespace scheduled-backup-19443 minio-service -o jsonpath="{.data.rootUser}" | base64 --decode):$(kubectl get secret --namespace scheduled-backup-19443 minio-service -o jsonpath="{.data.rootPassword}" | base64 --decode)@localhost:9000 3. mc ls minio-service-local pod/minio-service-76ffcfd45-krw8n condition met minio-service-76ffcfd45-krw8n.Ok make_bucket: operator-testing pod "aws-cli" deleted priorityclass.scheduling.k8s.io/high-priority created ----------------------------------------------------------------------------------- create first PXC cluster ----------------------------------------------------------------------------------- secret/my-cluster-secrets created secret/some-name-ssl created secret/some-name-ssl-internal created deployment.apps/pxc-client created perconaxtradbcluster.pxc.percona.com/scheduled-backup created ----------------------------------------------------------------------------------- check if all 3 Pods started ----------------------------------------------------------------------------------- error: no matching resources found ----------------------------------------------------------------------------------- wait for running cluster ----------------------------------------------------------------------------------- pod/scheduled-backup-proxysql-0 condition met scheduled-backup-proxysql-0.Ok ----------------------------------------------------------------------------------- wait for running cluster ----------------------------------------------------------------------------------- pod/scheduled-backup-pxc-0 condition met scheduled-backup-pxc-0.Ok pod/scheduled-backup-pxc-1 condition met scheduled-backup-pxc-1.Ok pod/scheduled-backup-pxc-2 condition met scheduled-backup-pxc-2.Ok ----------------------------------------------------------------------------------- write data ----------------------------------------------------------------------------------- Unable to use a TTY - input is not a terminal or the right kind of file pod/pxc-client-64b479df95-5sj66 condition met pxc-client-64b479df95-5sj66.Ok pod/pxc-client-64b479df95-5sj66 condition met pxc-client-64b479df95-5sj66.Ok pod/pxc-client-64b479df95-5sj66 condition met pxc-client-64b479df95-5sj66.Ok pod/pxc-client-64b479df95-5sj66 condition met pxc-client-64b479df95-5sj66.Ok pod/pxc-client-64b479df95-5sj66 condition met pxc-client-64b479df95-5sj66.Ok Unable to use a TTY - input is not a terminal or the right kind of file ----------------------------------------------------------------------------------- add backups schedule for pvc storage ----------------------------------------------------------------------------------- Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-1729-aeb7b936-1-cluster6" modified. perconaxtradbcluster.pxc.percona.com/scheduled-backup configured node/gke-jen-pxc-1729-aeb7b93-default-pool-f6401504-712n labeled perconaxtradbcluster.pxc.percona.com/scheduled-backup configured ----------------------------------------------------------------------------------- wait for running cluster ----------------------------------------------------------------------------------- error: timed out waiting for the condition on pods/scheduled-backup-pxc-0 scheduled-backup-pxc-0.Ok error: timed out waiting for the condition on pods/scheduled-backup-pxc-1 scheduled-backup-pxc-1.Ok error: timed out waiting for the condition on pods/scheduled-backup-pxc-2 scheduled-backup-pxc-2.Ok NAME CLUSTER STORAGE DESTINATION STATUS COMPLETED AGE cron-scheduled-backup-pvc-20246133357-q6fav scheduled-backup pvc pvc/xb-cron-scheduled-backup-pvc-20246133357-q6fav Running 21s NAME READY STATUS RESTARTS AGE minio-service-76ffcfd45-krw8n 1/1 Running 0 8m1s pxc-client-64b479df95-5sj66 1/1 Running 0 6m47s scheduled-backup-proxysql-0 3/3 Running 0 6m43s scheduled-backup-proxysql-1 3/3 Running 0 6m26s scheduled-backup-pxc-0 1/1 Running 0 6m43s scheduled-backup-pxc-1 1/1 Running 0 5m27s scheduled-backup-pxc-2 1/1 Running 0 4m14s xb-cron-scheduled-backup-pvc-20246133357-q6fav-8qk5k 1/1 Running 0 22s ----------------------------------------------------------------------------------- add backups schedule for aws s3 storage ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/scheduled-backup configured perconaxtradbcluster.pxc.percona.com/scheduled-backup configured ----------------------------------------------------------------------------------- wait for running cluster ----------------------------------------------------------------------------------- error: timed out waiting for the condition on pods/scheduled-backup-pxc-0 scheduled-backup-pxc-0.Ok error: timed out waiting for the condition on pods/scheduled-backup-pxc-1 scheduled-backup-pxc-1.Ok error: timed out waiting for the condition on pods/scheduled-backup-pxc-2 scheduled-backup-pxc-2.Ok NAME CLUSTER STORAGE DESTINATION STATUS COMPLETED AGE cron-scheduled-backup-aws-s3-20246133543-q6fav scheduled-backup aws-s3 s3://operator-testing/scheduled-backup-2024-06-13-03:05:43-full Running 22s cron-scheduled-backup-pvc-20246133357-q6fav scheduled-backup pvc pvc/xb-cron-scheduled-backup-pvc-20246133357-q6fav Succeeded 94s 2m8s NAME READY STATUS RESTARTS AGE minio-service-76ffcfd45-krw8n 1/1 Running 0 9m49s pxc-client-64b479df95-5sj66 1/1 Running 0 8m35s scheduled-backup-proxysql-0 3/3 Running 0 8m31s scheduled-backup-proxysql-1 3/3 Running 0 8m14s scheduled-backup-pxc-0 1/1 Running 0 8m31s scheduled-backup-pxc-1 1/1 Running 0 7m15s scheduled-backup-pxc-2 1/1 Running 0 6m2s xb-cron-scheduled-backup-aws-s3-20246133543-q6fav-4w9pn 1/1 Running 0 24s xb-cron-scheduled-backup-pvc-20246133357-q6fav-8qk5k 0/1 Completed 0 2m10s ----------------------------------------------------------------------------------- add backups schedule for minio storage ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/scheduled-backup configured perconaxtradbcluster.pxc.percona.com/scheduled-backup configured ----------------------------------------------------------------------------------- wait for running cluster ----------------------------------------------------------------------------------- error: timed out waiting for the condition on pods/scheduled-backup-pxc-0 scheduled-backup-pxc-0.Ok error: timed out waiting for the condition on pods/scheduled-backup-pxc-1 scheduled-backup-pxc-1.Ok error: timed out waiting for the condition on pods/scheduled-backup-pxc-2 scheduled-backup-pxc-2.Ok NAME CLUSTER STORAGE DESTINATION STATUS COMPLETED AGE cron-scheduled-backup-aws-s3-20246133543-q6fav scheduled-backup aws-s3 s3://operator-testing/scheduled-backup-2024-06-13-03:05:43-full Succeeded 89s 2m7s cron-scheduled-backup-minio-20246133731-q6fav scheduled-backup minio s3://operator-testing/scheduled-backup-2024-06-13-03:07:31-full Running 19s cron-scheduled-backup-pvc-20246133357-q6fav scheduled-backup pvc pvc/xb-cron-scheduled-backup-pvc-20246133357-q6fav Succeeded 3m19s 3m53s NAME READY STATUS RESTARTS AGE minio-service-76ffcfd45-krw8n 1/1 Running 0 11m pxc-client-64b479df95-5sj66 1/1 Running 0 10m scheduled-backup-proxysql-0 3/3 Running 0 10m scheduled-backup-proxysql-1 3/3 Running 0 9m57s scheduled-backup-pxc-0 1/1 Running 0 10m scheduled-backup-pxc-1 1/1 Running 0 8m58s scheduled-backup-pxc-2 1/1 Running 0 7m45s xb-cron-scheduled-backup-aws-s3-20246133543-q6fav-4w9pn 0/1 Completed 0 2m7s xb-cron-scheduled-backup-minio-20246133731-q6fav-sg9qn 1/1 Running 0 19s xb-cron-scheduled-backup-pvc-20246133357-q6fav-8qk5k 0/1 Completed 0 3m53s ----------------------------------------------------------------------------------- add backups schedule for gcs storage ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/scheduled-backup configured perconaxtradbcluster.pxc.percona.com/scheduled-backup configured ----------------------------------------------------------------------------------- wait for running cluster ----------------------------------------------------------------------------------- error: timed out waiting for the condition on pods/scheduled-backup-pxc-0 scheduled-backup-pxc-0.Ok error: timed out waiting for the condition on pods/scheduled-backup-pxc-1 scheduled-backup-pxc-1.Ok error: timed out waiting for the condition on pods/scheduled-backup-pxc-2 scheduled-backup-pxc-2.Ok NAME CLUSTER STORAGE DESTINATION STATUS COMPLETED AGE cron-scheduled-backup-aws-s3-20246133543-q6fav scheduled-backup aws-s3 s3://operator-testing/scheduled-backup-2024-06-13-03:05:43-full Succeeded 3m16s 3m54s cron-scheduled-backup-gcp-cs-20246133916-q6fav scheduled-backup gcp-cs s3://operator-testing/scheduled-backup-2024-06-13-03:09:16-full Running 21s cron-scheduled-backup-minio-20246133731-q6fav scheduled-backup minio s3://operator-testing/scheduled-backup-2024-06-13-03:07:31-full Succeeded 100s 2m6s cron-scheduled-backup-pvc-20246133357-q6fav scheduled-backup pvc pvc/xb-cron-scheduled-backup-pvc-20246133357-q6fav Succeeded 5m6s 5m40s NAME READY STATUS RESTARTS AGE minio-service-76ffcfd45-krw8n 1/1 Running 0 13m pxc-client-64b479df95-5sj66 1/1 Running 0 12m scheduled-backup-proxysql-0 3/3 Running 0 12m scheduled-backup-proxysql-1 3/3 Running 0 11m scheduled-backup-pxc-0 1/1 Running 0 12m scheduled-backup-pxc-1 1/1 Running 0 10m scheduled-backup-pxc-2 1/1 Running 0 9m33s xb-cron-scheduled-backup-aws-s3-20246133543-q6fav-4w9pn 0/1 Completed 0 3m55s xb-cron-scheduled-backup-gcp-cs-20246133916-q6fav-cgvxh 1/1 Running 0 22s xb-cron-scheduled-backup-minio-20246133731-q6fav-sg9qn 0/1 Completed 0 2m7s xb-cron-scheduled-backup-pvc-20246133357-q6fav-8qk5k 0/1 Completed 0 5m41s ----------------------------------------------------------------------------------- add backups schedule for azure storage ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/scheduled-backup configured perconaxtradbcluster.pxc.percona.com/scheduled-backup configured ----------------------------------------------------------------------------------- wait for running cluster ----------------------------------------------------------------------------------- error: timed out waiting for the condition on pods/scheduled-backup-pxc-0 scheduled-backup-pxc-0.Ok error: timed out waiting for the condition on pods/scheduled-backup-pxc-1 scheduled-backup-pxc-1.Ok error: timed out waiting for the condition on pods/scheduled-backup-pxc-2 scheduled-backup-pxc-2.Ok NAME CLUSTER STORAGE DESTINATION STATUS COMPLETED AGE cron-scheduled-backup-aws-s3-20246133543-q6fav scheduled-backup aws-s3 s3://operator-testing/scheduled-backup-2024-06-13-03:05:43-full Succeeded 5m2s 5m40s cron-scheduled-backup-azure-blob-20246133111-q6fav scheduled-backup azure-blob azure://operator-testing/scheduled-backup-2024-06-13-03:11:01-full Running 22s cron-scheduled-backup-gcp-cs-20246133916-q6fav scheduled-backup gcp-cs s3://operator-testing/scheduled-backup-2024-06-13-03:09:16-full Succeeded 95s 2m7s cron-scheduled-backup-minio-20246133731-q6fav scheduled-backup minio s3://operator-testing/scheduled-backup-2024-06-13-03:07:31-full Succeeded 3m26s 3m52s cron-scheduled-backup-pvc-20246133357-q6fav scheduled-backup pvc pvc/xb-cron-scheduled-backup-pvc-20246133357-q6fav Succeeded 6m52s 7m26s NAME READY STATUS RESTARTS AGE minio-service-76ffcfd45-krw8n 1/1 Running 0 15m pxc-client-64b479df95-5sj66 1/1 Running 0 13m scheduled-backup-proxysql-0 3/3 Running 0 13m scheduled-backup-proxysql-1 3/3 Running 0 13m scheduled-backup-pxc-0 1/1 Running 0 13m scheduled-backup-pxc-1 1/1 Running 0 12m scheduled-backup-pxc-2 1/1 Running 0 11m xb-cron-scheduled-backup-aws-s3-20246133543-q6fav-4w9pn 0/1 Completed 0 5m42s xb-cron-scheduled-backup-azure-blob-20246133111-q6fav-k5z5q 1/1 Running 0 24s xb-cron-scheduled-backup-gcp-cs-20246133916-q6fav-cgvxh 0/1 Completed 0 2m9s xb-cron-scheduled-backup-minio-20246133731-q6fav-sg9qn 0/1 Completed 0 3m54s xb-cron-scheduled-backup-pvc-20246133357-q6fav-8qk5k 0/1 Completed 0 7m28s ----------------------------------------------------------------------------------- Check backup existence ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- Check that KEEP option saves correct backup's amount (1 for our settings) ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- add backups schedule for gcs storage ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/scheduled-backup configured perconaxtradbcluster.pxc.percona.com/scheduled-backup configured ----------------------------------------------------------------------------------- add backups schedule for azure storage ----------------------------------------------------------------------------------- perconaxtradbcluster.pxc.percona.com/scheduled-backup configured perconaxtradbcluster.pxc.percona.com/scheduled-backup configured ----------------------------------------------------------------------------------- wait for running cluster ----------------------------------------------------------------------------------- error: timed out waiting for the condition on pods/scheduled-backup-pxc-0 scheduled-backup-pxc-0.Name: scheduled-backup-pxc-0 Namespace: scheduled-backup-19443 Priority: 0 Service Account: default Node: Labels: app.kubernetes.io/component=pxc app.kubernetes.io/instance=scheduled-backup app.kubernetes.io/managed-by=percona-xtradb-cluster-operator app.kubernetes.io/name=percona-xtradb-cluster app.kubernetes.io/part-of=percona-xtradb-cluster controller-revision-hash=scheduled-backup-pxc-7ffc5fbcb4 statefulset.kubernetes.io/pod-name=scheduled-backup-pxc-0 Annotations: cloud.google.com/cluster_autoscaler_unhelpable_since: 2024-06-13T03:15:24+0000 cloud.google.com/cluster_autoscaler_unhelpable_until: Inf percona.com/configuration-hash: d41d8cd98f00b204e9800998ecf8427e percona.com/ssl-hash: cfd6a52398268173b51d7cdb1331c09a percona.com/ssl-internal-hash: 751e59a1c1d94ac67f9d6bb6e756acdc Status: Pending IP: IPs: Controlled By: StatefulSet/scheduled-backup-pxc Init Containers: pxc-init: Image: perconalab/percona-xtradb-cluster-operator:PR-1729-aeb7b936 Port: Host Port: Command: /pxc-init-entrypoint.sh Limits: cpu: 50m memory: 50M Requests: cpu: 50m memory: 50M Environment: Mounts: /var/lib/mysql from datadir (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-dbfcw (ro) Containers: pxc: Image: perconalab/percona-xtradb-cluster-operator:main-pxc5.7 Ports: 3306/TCP, 4444/TCP, 4567/TCP, 4568/TCP, 33062/TCP, 33060/TCP Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP Command: /var/lib/mysql/pxc-entrypoint.sh Args: mysqld Liveness: exec [/var/lib/mysql/liveness-check.sh] delay=300s timeout=5s period=10s #success=1 #failure=3 Readiness: exec [/var/lib/mysql/readiness-check.sh] delay=15s timeout=15s period=30s #success=1 #failure=5 Environment Variables from: scheduled-backup-env-vars-pxc Secret Optional: true Environment: PXC_SERVICE: scheduled-backup-pxc-unready MONITOR_HOST: % MYSQL_ROOT_PASSWORD: Optional: false XTRABACKUP_PASSWORD: Optional: false MONITOR_PASSWORD: Optional: false CLUSTER_HASH: 1953677 OPERATOR_ADMIN_PASSWORD: Optional: false LIVENESS_CHECK_TIMEOUT: 5 READINESS_CHECK_TIMEOUT: 15 DEFAULT_AUTHENTICATION_PLUGIN: mysql_native_password Mounts: /etc/my.cnf.d from auto-config (rw) /etc/mysql/init-file from mysql-init-file (rw) /etc/mysql/mysql-users-secret from mysql-users-secret-file (rw) /etc/mysql/ssl from ssl (rw) /etc/mysql/ssl-internal from ssl-internal (rw) /etc/mysql/vault-keyring-secret from vault-keyring-secret (rw) /etc/percona-xtradb-cluster.conf.d from config (rw) /tmp from tmp (rw) /var/lib/mysql from datadir (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-dbfcw (ro) Conditions: Type Status PodScheduled False Volumes: datadir: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: datadir-scheduled-backup-pxc-0 ReadOnly: false tmp: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: config: Type: ConfigMap (a volume populated by a ConfigMap) Name: scheduled-backup-pxc Optional: true ssl-internal: Type: Secret (a volume populated by a Secret) SecretName: some-name-ssl-internal Optional: true ssl: Type: Secret (a volume populated by a Secret) SecretName: some-name-ssl Optional: false auto-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: auto-scheduled-backup-pxc Optional: true vault-keyring-secret: Type: Secret (a volume populated by a Secret) SecretName: scheduled-backup-vault Optional: true mysql-users-secret-file: Type: Secret (a volume populated by a Secret) SecretName: internal-scheduled-backup Optional: false mysql-init-file: Type: Secret (a volume populated by a Secret) SecretName: scheduled-backup-mysql-init Optional: true kube-api-access-dbfcw: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal NotTriggerScaleUp 45s cluster-autoscaler pod didn't trigger scale-up: Warning FailedScheduling 31s (x5 over 48s) default-scheduler 0/3 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/not-ready: }, 2 node(s) didn't match pod anti-affinity rules. preemption: 0/3 nodes are available: 1 Preemption is not helpful for scheduling, 2 No preemption victims found for incoming pod.. Defaulted container "pxc" out of: pxc, pxc-init (init) Error from server (NotFound): pods "gke-jen-pxc-1729-aeb7b93-default-pool-f6401504-712n" not found