Log: /mnt/jenkins/workspace/cloud-pxc-operator_PR-1751/e2e-tests/logs/monitoring-2-0-8-0.log WARNING: version difference between client (1.30) and server (1.26) exceeds the supported minor version skew of +/-1 WARNING: version difference between client (1.30) and server (1.26) exceeds the supported minor version skew of +/-1 E0708 20:57:30.843825 32755 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:30.960559 32755 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:31.069791 32755 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:31.176908 32755 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:31.284485 32755 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request error: the server doesn't have a resource type "pxc" + kubectl patch pxc -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' E0708 20:57:32.511180 389 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:32.640397 389 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:32.749638 389 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:32.856611 389 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:33.361492 389 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:33.485999 389 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:33.627759 389 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request error: the server doesn't have a resource type "pxc" E0708 20:57:34.800975 519 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:34.908827 519 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:35.014563 519 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:35.123746 519 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:35.462647 519 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:35.575915 519 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:35.683538 519 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request error: the server doesn't have a resource type "pxc" E0708 20:57:36.732768 618 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:36.950813 618 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:37.059283 618 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:37.167445 618 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:37.508540 618 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:37.709513 618 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:37.819527 618 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request error: the server doesn't have a resource type "pxc-backup" E0708 20:57:38.903352 737 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:39.213395 737 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:39.319364 737 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:39.425602 737 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:39.756589 737 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:39.868987 737 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:39.976867 737 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request error: the server doesn't have a resource type "pxc-restore" E0708 20:57:41.490805 931 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:41.705676 931 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:41.811947 931 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:41.918158 931 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:43.056807 1032 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:43.272920 1032 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:43.380243 1032 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:43.486751 1032 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request error: resource(s) were provided, but no name was specified E0708 20:57:44.569310 1160 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:44.786515 1160 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:44.893783 1160 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:44.999910 1160 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:46.153292 1322 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:46.363161 1322 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:46.469570 1322 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:46.575756 1322 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request error: resource(s) were provided, but no name was specified E0708 20:57:47.162144 1365 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:47.475974 1365 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:47.581850 1365 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:47.687958 1365 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:48.865325 1449 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:49.118613 1449 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:49.225688 1449 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:49.335650 1449 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request error: resource(s) were provided, but no name was specified E0708 20:57:50.509611 1550 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:50.714123 1550 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request error: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:51.778779 1622 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:52.003492 1622 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:52.111922 1622 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:52.220507 1622 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:53.427119 1699 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:53.744216 1699 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:53.851324 1699 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:53.960970 1699 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request error: resource(s) were provided, but no name was specified E0708 20:57:55.106316 1780 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:55.308861 1780 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:55.415239 1780 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:55.521563 1780 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:56.886371 1867 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:56.992323 1867 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:57.099235 1867 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:57.205533 1867 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request error: resource(s) were provided, but no name was specified E0708 20:57:57.790535 1916 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:58.008550 1916 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:58.116568 1916 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:58.223936 1916 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:59.607294 1999 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:59.820106 1999 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:57:59.926749 1999 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:58:00.032831 1999 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces pxc-operator ----------------------------------------------------------------------------------- E0708 20:58:01.199026 2083 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:58:01.319058 2083 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:58:01.428940 2083 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:58:01.535406 2083 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request Error from server (NotFound): namespaces "pxc-operator" not found namespace/pxc-operator - E0708 20:58:01.101649 2082 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:58:01.332348 2082 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:58:01.441355 2082 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:58:01.549354 2082 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:58:02.894082 2238 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:58:03.005405 2238 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:58:03.115894 2238 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:58:03.241030 2238 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request Error from server (NotFound): namespaces "pxc-operator" not found ----------------------------------------------------------------------------------- create namespace pxc-operator ----------------------------------------------------------------------------------- Error from server (Forbidden): namespaces "default" is forbidden: this namespace may not be deleted namespace/pxc-operator created Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-1751-f9555a6f-11-cluster1" modified. ----------------------------------------------------------------------------------- start PXC operator ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterbackups.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterrestores.pxc.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusters.pxc.percona.com serverside-applied E0708 20:58:05.655400 2478 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:58:05.767618 2478 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request clusterrole.rbac.authorization.k8s.io/percona-xtradb-cluster-operator created serviceaccount/percona-xtradb-cluster-operator created clusterrolebinding.rbac.authorization.k8s.io/service-account-percona-xtradb-cluster-operator created E0708 20:58:08.833602 2622 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:58:09.045425 2622 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request deployment.apps/percona-xtradb-cluster-operator created service/percona-xtradb-cluster-operator created E0708 20:58:12.234709 2892 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0708 20:58:12.353553 2892 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request pod/percona-xtradb-cluster-operator-7888f9874f-6hwb2 condition met pod/percona-xtradb-cluster-operator-7888f9874f-6hwb2 condition met percona-xtradb-cluster-operator-7888f9874f-6hwb2.Ok error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces monitoring-2-0-13601 ----------------------------------------------------------------------------------- Error from server (NotFound): namespaces "monitoring-2-0-13601" not found namespace/monitoring-2-0-13601 - Error from server (Forbidden): namespaces "default" is forbidden: this namespace may not be deleted Error from server (NotFound): namespaces "monitoring-2-0-13601" not found ----------------------------------------------------------------------------------- create namespace monitoring-2-0-13601 ----------------------------------------------------------------------------------- namespace/monitoring-2-0-13601 created Context "gke_cloud-dev-112233_us-central1-a_jen-pxc-1751-f9555a6f-11-cluster1" modified. ----------------------------------------------------------------------------------- create secrets for cloud storages ----------------------------------------------------------------------------------- secret/minio-secret created secret/aws-s3-secret created secret/gcp-cs-secret created secret/azure-secret created "hashicorp" has been added to your repositories "minio" has been added to your repositories Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "minio" chart repository ...Successfully got an update from the "hashicorp" chart repository Update Complete. ⎈Happy Helming!⎈ ----------------------------------------------------------------------------------- install PMM Server ----------------------------------------------------------------------------------- "percona" has been added to your repositories Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "minio" chart repository ...Successfully got an update from the "percona" chart repository ...Successfully got an update from the "hashicorp" chart repository Update Complete. ⎈Happy Helming!⎈ Error: uninstall: Release not loaded: monitoring: release: not found NAME: monitoring LAST DEPLOYED: Mon Jul 8 20:58:42 2024 NAMESPACE: monitoring-2-0-13601 STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: PMM server can be accessed via HTTPS (port 443) on the following DNS name from within your cluster: endpoint: https://monitoring-service.monitoring-2-0-13601.svc.cluster.local:443 login: admin password: admin pod/monitoring-0 condition met logger=settings t=2024-07-08T20:59:59.362292811Z level=info msg="Starting Grafana" version= commit= branch= compiled=1970-01-01T00:00:00Z logger=settings t=2024-07-08T20:59:59.36266105Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini logger=settings t=2024-07-08T20:59:59.362682229Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini logger=settings t=2024-07-08T20:59:59.362691672Z level=info msg="Path Home" path=/usr/share/grafana logger=settings t=2024-07-08T20:59:59.362699915Z level=info msg="Path Data" path=/srv/grafana logger=settings t=2024-07-08T20:59:59.36273129Z level=info msg="Path Logs" path=/srv/logs logger=settings t=2024-07-08T20:59:59.362742337Z level=info msg="Path Plugins" path=/srv/grafana/plugins logger=settings t=2024-07-08T20:59:59.362750595Z level=info msg="Path Provisioning" path=/usr/share/grafana/conf/provisioning logger=settings t=2024-07-08T20:59:59.3627575Z level=info msg="App mode production" logger=sqlstore t=2024-07-08T20:59:59.362872986Z level=info msg="Connecting to DB" dbtype=postgres logger=migrator t=2024-07-08T20:59:59.404730851Z level=info msg="Starting DB migrations" logger=migrator t=2024-07-08T20:59:59.415655855Z level=info msg="migrations completed" performed=0 skipped=452 duration=1.179708ms logger=secrets t=2024-07-08T20:59:59.418612388Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 logger=plugin.finder t=2024-07-08T20:59:59.520798724Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled logger=plugin.signature.validator t=2024-07-08T20:59:59.812679174Z level=warn msg="Permitting unsigned plugin. This is not recommended" pluginID=pmm-app pluginDir=/srv/grafana/plugins/pmm-app/dist logger=plugin.pmm-app t=2024-07-08T20:59:59.823102365Z level=warn msg="Included dashboard is missing a UID field" logger=plugin.signature.validator t=2024-07-08T20:59:59.823548806Z level=warn msg="Permitting unsigned plugin. This is not recommended" pluginID=pmm-pt-summary-datasource pluginDir=/srv/grafana/plugins/pmm-app/dist/pmm-pt-summary/datasource logger=plugin.signature.validator t=2024-07-08T20:59:59.82359693Z level=warn msg="Permitting unsigned plugin. This is not recommended" pluginID=pmm-pt-summary-panel pluginDir=/srv/grafana/plugins/pmm-app/dist/pmm-pt-summary/panel logger=plugin.signature.validator t=2024-07-08T20:59:59.823646095Z level=warn msg="Permitting unsigned plugin. This is not recommended" pluginID=pmm-qan-app-panel pluginDir=/srv/grafana/plugins/pmm-app/dist/pmm-qan logger=plugin.signature.validator t=2024-07-08T20:59:59.823671425Z level=warn msg="Permitting unsigned plugin. This is not recommended" pluginID=grafana-polystat-panel pluginDir=/srv/grafana/plugins/grafana-polystat-panel logger=plugin.loader t=2024-07-08T20:59:59.823805908Z level=info msg="Plugin registered" pluginID=camptocamp-prometheus-alertmanager-datasource logger=plugin.loader t=2024-07-08T20:59:59.823830825Z level=info msg="Plugin registered" pluginID=grafana-clickhouse-datasource logger=plugin.grafana-clickhouse-datasource t=2024-07-08T20:59:59.856442327Z level=warn msg="Plugin process is running with elevated privileges. This is not recommended" logger=plugin.loader t=2024-07-08T20:59:59.856486652Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel logger=plugin.loader t=2024-07-08T20:59:59.856501542Z level=info msg="Plugin registered" pluginID=grafana-worldmap-panel logger=plugin.loader t=2024-07-08T20:59:59.856531273Z level=info msg="Plugin registered" pluginID=petrslavotinek-carpetplot-panel logger=plugin.loader t=2024-07-08T20:59:59.856542436Z level=info msg="Plugin registered" pluginID=pmm-app logger=plugin.loader t=2024-07-08T20:59:59.856552182Z level=info msg="Plugin registered" pluginID=pmm-pt-summary-datasource logger=plugin.loader t=2024-07-08T20:59:59.856564112Z level=info msg="Plugin registered" pluginID=pmm-pt-summary-panel logger=plugin.loader t=2024-07-08T20:59:59.856574883Z level=info msg="Plugin registered" pluginID=jdbranham-diagram-panel logger=plugin.loader t=2024-07-08T20:59:59.856588075Z level=info msg="Plugin registered" pluginID=natel-discrete-panel logger=plugin.loader t=2024-07-08T20:59:59.856608272Z level=info msg="Plugin registered" pluginID=pmm-qan-app-panel logger=plugin.loader t=2024-07-08T20:59:59.856618769Z level=info msg="Plugin registered" pluginID=grafana-polystat-panel Admin password changed successfully ✔ ----------------------------------------------------------------------------------- create PXC cluster ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- create first PXC cluster ----------------------------------------------------------------------------------- secret/my-cluster-secrets created deployment.apps/pxc-client created perconaxtradbcluster.pxc.percona.com/monitoring created ----------------------------------------------------------------------------------- check if all 3 Pods started ----------------------------------------------------------------------------------- error: no matching resources found ----------------------------------------------------------------------------------- wait for running cluster ----------------------------------------------------------------------------------- Error from server (NotFound): pods "monitoring-haproxy-0" not found monitoring-haproxy-0......................................................Defaulted container "pmm-client" out of: pmm-client, haproxy, pxc-monit, pxc-init (init) .Ok ----------------------------------------------------------------------------------- wait for running cluster ----------------------------------------------------------------------------------- pod/monitoring-pxc-0 condition met monitoring-pxc-0.Ok pod/monitoring-pxc-1 condition met monitoring-pxc-1.Ok pod/monitoring-pxc-2 condition met monitoring-pxc-2.Ok ----------------------------------------------------------------------------------- write data ----------------------------------------------------------------------------------- pod/pxc-client-6644d8898f-vtzjx condition met pxc-client-6644d8898f-vtzjx.Ok pod/pxc-client-6644d8898f-vtzjx condition met pxc-client-6644d8898f-vtzjx.Ok pod/pxc-client-6644d8898f-vtzjx condition met pxc-client-6644d8898f-vtzjx.Ok pod/pxc-client-6644d8898f-vtzjx condition met pxc-client-6644d8898f-vtzjx.Ok pod/pxc-client-6644d8898f-vtzjx condition met pxc-client-6644d8898f-vtzjx.Ok Unable to use a TTY - input is not a terminal or the right kind of file ----------------------------------------------------------------------------------- add PMM API key to secret ----------------------------------------------------------------------------------- % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 155 100 119 100 36 261 79 --:--:-- --:--:-- --:--:-- 339 100 155 100 119 100 36 261 79 --:--:-- --:--:-- --:--:-- 339 secret/my-cluster-secrets patched Waiting for sts/monitoring-pxc to reach generation 2... Resource sts/monitoring-pxc is at generation 1. Waiting... Resource sts/monitoring-pxc has reached generation 2. Waiting for sts/monitoring-haproxy to reach generation 2... Resource sts/monitoring-haproxy has reached generation 2. pod/monitoring-haproxy-0 condition met pod/monitoring-haproxy-1 condition met pod/monitoring-pxc-0 condition met pod/monitoring-pxc-1 condition met pod/monitoring-pxc-2 condition met ----------------------------------------------------------------------------------- wait cluster consistency ----------------------------------------------------------------------------------- waiting for cluster readyness waiting for cluster readyness waiting for cluster readyness waiting for cluster readyness waiting for cluster readyness ----------------------------------------------------------------------------------- compare statefulset/monitoring-pxc--no-prefix ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- return true if kubernetes version equal or greater than desired ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- return true if kubernetes version equal or greater than desired ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- return true if kubernetes version equal or greater than desired ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- return true if kubernetes version equal or greater than desired ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- return true if kubernetes version equal or greater than desired ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- compare statefulset/monitoring-haproxy--no-prefix ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- return true if kubernetes version equal or greater than desired ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- return true if kubernetes version equal or greater than desired ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- return true if kubernetes version equal or greater than desired ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- return true if kubernetes version equal or greater than desired ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- return true if kubernetes version equal or greater than desired ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- apply my-env-var-secrets to add PMM_PREFIX ----------------------------------------------------------------------------------- secret/my-env-var-secrets created ----------------------------------------------------------------------------------- add new PMM API key to secret ----------------------------------------------------------------------------------- % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 40 0 0 100 40 0 94 --:--:-- --:--:-- --:--:-- 94 100 167 100 127 100 40 277 87 --:--:-- --:--:-- --:--:-- 365 secret/my-cluster-secrets patched ----------------------------------------------------------------------------------- delete old PMM key ----------------------------------------------------------------------------------- % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 602 100 602 0 0 1365 0 --:--:-- --:--:-- --:--:-- 1365 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 29 100 29 0 0 65 0 --:--:-- --:--:-- --:--:-- 65 {"message":"API key deleted"}Waiting for sts/monitoring-pxc to reach generation 3... Resource sts/monitoring-pxc has reached generation 3. Waiting for sts/monitoring-haproxy to reach generation 3... Resource sts/monitoring-haproxy has reached generation 3. pod/monitoring-haproxy-0 condition met pod/monitoring-haproxy-1 condition met pod/monitoring-pxc-0 condition met pod/monitoring-pxc-1 condition met pod/monitoring-pxc-2 condition met ----------------------------------------------------------------------------------- check if pmm-client container enabled ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- compare statefulset/monitoring-pxc- ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- return true if kubernetes version equal or greater than desired ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- return true if kubernetes version equal or greater than desired ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- return true if kubernetes version equal or greater than desired ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- return true if kubernetes version equal or greater than desired ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- return true if kubernetes version equal or greater than desired ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- compare statefulset/monitoring-haproxy- ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- return true if kubernetes version equal or greater than desired ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- return true if kubernetes version equal or greater than desired ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- return true if kubernetes version equal or greater than desired ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- return true if kubernetes version equal or greater than desired ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- return true if kubernetes version equal or greater than desired ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- verify clients agents statuses ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- check mysql metrics ----------------------------------------------------------------------------------- "1720472024" "1720472024" "155" "215" ----------------------------------------------------------------------------------- check haproxy metrics ----------------------------------------------------------------------------------- "0" "0" "1" "1" ----------------------------------------------------------------------------------- check QAN data ----------------------------------------------------------------------------------- null perconaxtradbcluster.pxc.percona.com/monitoring patched pod/monitoring-pxc-0 - ..........................Error from server (NotFound): pods "monitoring-pxc-0" not found ----------------------------------------------------------------------------------- check for passwords leak ----------------------------------------------------------------------------------- secrets=WEo0K0hITFVQMDJJN0daTm14RWhESDJGT0liTW9sZ0pYU1d0cmw2Vw== K1N6aDhFYjBJWjBvMjV0SXZBei9yN1NzZG1EeWN0M09ENUNMMzIzblZvbFpZdlorWGV3ZFkxM2ZYRW9HM2pHc1o3L05uSWovdTl2ZStBU3R5TEtRa2c9PQo= VEtVS0N0dkFZRC9uR1p3dEYxa0hBOFFuTXpoNlFHMFlrUlZJK0ZLSQ== ZjErZUEzNTNvUFcsOTxiST8hNw== b3BlcmF0b3JhZG1pbg== YWRtaW4= ZXlKcklqb2lZVXBHY2pnMFlrWjFNVFoyV1RaVGRFNVplVkpCVDA5cE0xcGhWbkZMWmtRaUxDSnVJam9pYjNCbGNtRjBiM0l0Ym1WM0lpd2lhV1FpT2pGOQ== YWRtaW5fcGFzc3dvcmQ= cmVwbF9wYXNzd29yZA== cm9vdF9wYXNzd29yZA== YmFja3VwX3Bhc3N3b3Jk c29tZS1zZWNyZXQta2V5 ZjErZUEzNTNvUFcsOTxiST8hNw== b3BlcmF0b3JhZG1pbg== YWRtaW4= ZXlKcklqb2lZVXBHY2pnMFlrWjFNVFoyV1RaVGRFNVplVkpCVDA5cE0xcGhWbkZMWmtRaUxDSnVJam9pYjNCbGNtRjBiM0l0Ym1WM0lpd2lhV1FpT2pGOQ== YWRtaW5fcGFzc3dvcmQ= cmVwbF9wYXNzd29yZA== cm9vdF9wYXNzd29yZA== YmFja3VwX3Bhc3N3b3Jk cHhjLXByZWZpeC0= passwords=XJ4+HHLUP02I7GZNmxEhDH2FOIbMolgJXSWtrl6W +Szh8Eb0IZ0o25tIvAz/r7SsdmDyct3OD5CL323nVolZYvZ+XewdY13fXEoG3jGsZ7/NnIj/u9ve+AStyLKQkg== TKUKCtvAYD/nGZwtF1kHA8QnMzh6QG0YkRVI+FKI f1+eA353oPW,9