Log: /mnt/jenkins/workspace/cloud-psmdb-operator_PR-1569/e2e-tests/logs/monitoring-2-0.log E0611 12:02:22.622195 2169 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:22.729305 2169 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:22.837829 2169 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:22.945708 2169 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request WARNING: version difference between client (1.30) and server (1.26) exceeds the supported minor version skew of +/-1 WARNING: version difference between client (1.30) and server (1.26) exceeds the supported minor version skew of +/-1 WARNING: version difference between client (1.30) and server (1.26) exceeds the supported minor version skew of +/-1 ----------------------------------------------------------------------------------- get and delete old CRDs and RBAC ----------------------------------------------------------------------------------- E0611 12:02:24.953332 2272 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:25.128004 2272 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:26.863092 2330 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:27.117239 2330 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:27.224592 2330 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:27.331384 2330 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:27.682089 2330 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:27.865757 2330 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:27.975994 2330 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:28.085616 2330 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:28.193315 2330 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request error: the server doesn't have a resource type "perconaservermongodbbackups" + kubectl patch perconaservermongodbbackups.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' E0611 12:02:28.779215 2393 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:29.003408 2393 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:29.109828 2393 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:29.216795 2393 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:29.546849 2393 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:29.746857 2393 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:29.854904 2393 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:29.961543 2393 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:30.068315 2393 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request error: the server doesn't have a resource type "perconaservermongodbbackups" E0611 12:02:30.760319 2428 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:30.913916 2428 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:31.023698 2428 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:31.130272 2428 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:31.940768 2455 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:32.141910 2455 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:32.248476 2455 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:32.354978 2455 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:32.689303 2455 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:32.891171 2455 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:32.999201 2455 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:33.105803 2455 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:33.213227 2455 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request error: the server doesn't have a resource type "perconaservermongodbrestores" + kubectl patch perconaservermongodbrestores.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' E0611 12:02:33.899788 2495 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:34.112728 2495 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:34.220967 2495 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:34.328751 2495 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:34.659484 2495 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:34.867482 2495 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:34.978561 2495 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:35.086020 2495 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:35.193333 2495 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request error: the server doesn't have a resource type "perconaservermongodbrestores" E0611 12:02:35.877596 2534 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:35.985831 2534 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:36.091482 2534 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:36.205235 2534 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:36.893581 2562 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:37.111778 2562 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:37.218818 2562 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:37.325829 2562 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:37.665918 2562 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:37.858225 2562 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:37.966060 2562 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:38.071904 2562 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:38.178091 2562 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request error: the server doesn't have a resource type "perconaservermongodbs" + kubectl patch perconaservermongodbs.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' E0611 12:02:38.768731 2620 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:39.083368 2620 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:39.191045 2620 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:39.299886 2620 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:39.658567 2620 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:39.887062 2620 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:39.996451 2620 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:40.104909 2620 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:40.212073 2620 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request error: the server doesn't have a resource type "perconaservermongodbs" E0611 12:02:41.309925 2695 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:41.715882 2695 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:41.822471 2695 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:41.929070 2695 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:42.735962 2764 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:42.847101 2764 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:44.400904 2809 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:44.516211 2809 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:44.624383 2809 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found E0611 12:02:45.425430 2832 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:45.636812 2832 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:45.746595 2832 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found E0611 12:02:51.034552 2929 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:51.234453 2929 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:51.340405 2929 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found E0611 12:02:51.034552 2929 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:51.234453 2929 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0611 12:02:51.340405 2929 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "null" not found ----------------------------------------------------------------------------------- destroy chaos-mesh ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces psmdb-operator ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- create namespace psmdb-operator ----------------------------------------------------------------------------------- namespace/psmdb-operator created Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-1569-eb818ce7-7-cluster1" modified. ----------------------------------------------------------------------------------- start PSMDB operator ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io/perconaservermongodbbackups.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbrestores.psmdb.percona.com serverside-applied customresourcedefinition.apiextensions.k8s.io/perconaservermongodbs.psmdb.percona.com serverside-applied clusterrole.rbac.authorization.k8s.io/percona-server-mongodb-operator created serviceaccount/percona-server-mongodb-operator created clusterrolebinding.rbac.authorization.k8s.io/service-account-percona-server-mongodb-operator created deployment.apps/percona-server-mongodb-operator created waiting for pod/percona-server-mongodb-operator-8d99b8f59-l7j52 to be ready...OK ----------------------------------------------------------------------------------- destroy chaos-mesh ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- cleaned up all old namespaces ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- cleaned up old namespaces monitoring-2-0-18818 ----------------------------------------------------------------------------------- error: resource(s) were provided, but no name was specified ----------------------------------------------------------------------------------- create namespace monitoring-2-0-18818 ----------------------------------------------------------------------------------- namespace/monitoring-2-0-18818 created Context "gke_cloud-dev-112233_us-central1-a_jen-psmdb-1569-eb818ce7-7-cluster1" modified. ----------------------------------------------------------------------------------- deploy cert manager ----------------------------------------------------------------------------------- namespace/cert-manager created namespace/cert-manager labeled namespace/cert-manager configured customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io created serviceaccount/cert-manager-cainjector created serviceaccount/cert-manager created serviceaccount/cert-manager-webhook created clusterrole.rbac.authorization.k8s.io/cert-manager-cainjector created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-issuers created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificates created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-orders created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-challenges created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created clusterrole.rbac.authorization.k8s.io/cert-manager-cluster-view created clusterrole.rbac.authorization.k8s.io/cert-manager-view created clusterrole.rbac.authorization.k8s.io/cert-manager-edit created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests created clusterrole.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-cainjector created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-issuers created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificates created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-orders created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-challenges created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews created role.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created role.rbac.authorization.k8s.io/cert-manager:leaderelection created role.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created rolebinding.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created rolebinding.rbac.authorization.k8s.io/cert-manager:leaderelection created rolebinding.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created service/cert-manager created service/cert-manager-webhook created deployment.apps/cert-manager-cainjector created deployment.apps/cert-manager created deployment.apps/cert-manager-webhook created mutatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created Warning: resource namespaces/cert-manager is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically. pod/cert-manager-5658d944df-h97jx condition met pod/cert-manager-cainjector-cb99ff845-nhd5j condition met pod/cert-manager-webhook-7fd74b8dc7-qmdh9 condition met ----------------------------------------------------------------------------------- install PMM Server ----------------------------------------------------------------------------------- Error: uninstall: Release not loaded: monitoring: release: not found Error: no repositories configured "stable" has been added to your repositories NAME: monitoring LAST DEPLOYED: Tue Jun 11 12:05:57 2024 NAMESPACE: monitoring-2-0-18818 STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: PMM server can be accessed via HTTPS (port 443) on the following DNS name from within your cluster: endpoint: https://monitoring-service.monitoring-2-0-18818.svc.cluster.local:443 login: admin password: admin error: unable to upgrade connection: container not found ("monitoring") error: unable to upgrade connection: container not found ("monitoring") error: unable to upgrade connection: container not found ("monitoring") error: unable to upgrade connection: container not found ("monitoring") Retry 2 error: unable to upgrade connection: container not found ("monitoring") error: unable to upgrade connection: container not found ("monitoring") error: unable to upgrade connection: container not found ("monitoring") error: unable to upgrade connection: container not found ("monitoring") Retry 3 ----------------------------------------------------------------------------------- create secrets and start client ----------------------------------------------------------------------------------- secret/some-users created secret/some-users unchanged deployment.apps/psmdb-client created ----------------------------------------------------------------------------------- create first PSMDB cluster monitoring ----------------------------------------------------------------------------------- perconaservermongodb.psmdb.percona.com/monitoring created waiting for pod/monitoring-rs0-0 to be ready.........................OK waiting for pod/monitoring-rs0-1 to be ready..................OK waiting for pod/monitoring-rs0-2 to be ready.........................OK Waiting for cluster readyness....................... ----------------------------------------------------------------------------------- check if pmm-client container is not enabled ----------------------------------------------------------------------------------- Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://monitoring-mongos.monitoring-2-0-18818.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb {"t":{"$date":"2024-06-11T12:11:25.617Z"},"s":"I", "c":"NETWORK", "id":5490002, "ctx":"thread1","msg":"Started a new thread for the timer service"} Implicit session: session { "id" : UUID("3d376cec-858f-420c-bb5d-0def4cdbe904") } Percona Server for MongoDB server version: v7.0.11-6 WARNING: shell and server versions do not match Successfully added user: { "user" : "myApp", "roles" : [ { "db" : "myApp", "role" : "readWrite" } ] } bye Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://monitoring-mongos.monitoring-2-0-18818.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb {"t":{"$date":"2024-06-11T12:11:27.724Z"},"s":"I", "c":"NETWORK", "id":5490002, "ctx":"thread1","msg":"Started a new thread for the timer service"} Implicit session: session { "id" : UUID("5c011b08-0438-4a8b-8d8c-5c324e862300") } Percona Server for MongoDB server version: v7.0.11-6 WARNING: shell and server versions do not match { "ok" : 1, "$clusterTime" : { "clusterTime" : Timestamp(1718107887, 8), "signature" : { "hash" : BinData(0,"5t4gyovN/KnhQNX1H8fS1xd8LLA="), "keyId" : NumberLong("7379216842067279895") } }, "operationTime" : Timestamp(1718107887, 2) } bye Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://monitoring-mongos.monitoring-2-0-18818.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb {"t":{"$date":"2024-06-11T12:11:29.767Z"},"s":"I", "c":"NETWORK", "id":5490002, "ctx":"thread1","msg":"Started a new thread for the timer service"} Implicit session: session { "id" : UUID("b073eb7d-2444-40db-bcfb-452e83531e88") } Percona Server for MongoDB server version: v7.0.11-6 WARNING: shell and server versions do not match switched to db myApp WriteResult({ "nInserted" : 1 }) bye Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://monitoring-mongos.monitoring-2-0-18818.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb {"t":{"$date":"2024-06-11T12:11:31.885Z"},"s":"I", "c":"NETWORK", "id":5490002, "ctx":"thread1","msg":"Started a new thread for the timer service"} Implicit session: session { "id" : UUID("1d101bf3-3e7c-4be1-892f-0daa10f78d33") } Percona Server for MongoDB server version: v7.0.11-6 WARNING: shell and server versions do not match switched to db myApp WriteResult({ "nInserted" : 1 }) bye Percona Server for MongoDB shell version v4.4.29-28 connecting to: mongodb://monitoring-mongos.monitoring-2-0-18818.svc.cluster.local:27017/admin?compressors=disabled&gssapiServiceName=mongodb {"t":{"$date":"2024-06-11T12:11:33.837Z"},"s":"I", "c":"NETWORK", "id":5490002, "ctx":"thread1","msg":"Started a new thread for the timer service"} Implicit session: session { "id" : UUID("ff5c3fe9-51d0-4240-8991-8f310de206f7") } Percona Server for MongoDB server version: v7.0.11-6 WARNING: shell and server versions do not match switched to db myApp WriteResult({ "nInserted" : 1 }) bye ----------------------------------------------------------------------------------- add PMM_SERVER_API_KEY for secret some-users ----------------------------------------------------------------------------------- % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 155 100 119 100 36 257 78 --:--:-- --:--:-- --:--:-- 335 secret/some-users patched ----------------------------------------------------------------------------------- check if all 3 Pods started ----------------------------------------------------------------------------------- waiting for pod/monitoring-rs0-0 to be ready.OK waiting for pod/monitoring-rs0-1 to be ready.OK waiting for pod/monitoring-rs0-2 to be ready.OK Waiting for cluster readyness..................................................................................................................... ----------------------------------------------------------------------------------- check if pmm-client container enabled ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- check mongod metrics ----------------------------------------------------------------------------------- "1718107064" "1718107064" "0" "0" ----------------------------------------------------------------------------------- check mongo config metrics ----------------------------------------------------------------------------------- "1718107064" "1718107064" "0" "0" ----------------------------------------------------------------------------------- check mongos metrics ----------------------------------------------------------------------------------- "1718107064" "1718107064" ----------------------------------------------------------------------------------- check QAN data ----------------------------------------------------------------------------------- Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found command terminated with exit code 1 command terminated with exit code 1 command terminated with exit code 1 command terminated with exit code 1 Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found command terminated with exit code 1 command terminated with exit code 1 command terminated with exit code 1 command terminated with exit code 1 Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found command terminated with exit code 1 command terminated with exit code 1 command terminated with exit code 1 command terminated with exit code 1 Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found command terminated with exit code 1 command terminated with exit code 1 command terminated with exit code 1 command terminated with exit code 1 Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found command terminated with exit code 1 command terminated with exit code 1 command terminated with exit code 1 command terminated with exit code 1 Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found command terminated with exit code 1 command terminated with exit code 1 command terminated with exit code 1 command terminated with exit code 1 Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found command terminated with exit code 1 command terminated with exit code 1 command terminated with exit code 1 command terminated with exit code 1 Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found command terminated with exit code 1 command terminated with exit code 1 command terminated with exit code 1 command terminated with exit code 1 Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found command terminated with exit code 1 command terminated with exit code 1 command terminated with exit code 1 command terminated with exit code 1 perconaservermongodb.psmdb.percona.com/monitoring patched pod/monitoring-mongos-0 - .................................Error from server (NotFound): pods "monitoring-mongos-0" not found Error from server (NotFound): pods "monitoring-mongos-0" not found Error from server (NotFound): pods "monitoring-mongos-0" not found Error from server (NotFound): pods "monitoring-mongos-0" not found pod/monitoring-rs0-0 - ............Error from server (NotFound): pods "monitoring-rs0-0" not found Error from server (NotFound): pods "monitoring-rs0-0" not found Error from server (NotFound): pods "monitoring-rs0-0" not found Error from server (NotFound): pods "monitoring-rs0-0" not found pod/monitoring-cfg-0 - ..........Error from server (NotFound): pods "monitoring-cfg-0" not found Error from server (NotFound): pods "monitoring-cfg-0" not found Error from server (NotFound): pods "monitoring-cfg-0" not found Error from server (NotFound): pods "monitoring-cfg-0" not found ----------------------------------------------------------------------------------- check if services are not deleted ----------------------------------------------------------------------------------- NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE monitoring-rs0 ClusterIP None 27017/TCP 16m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE monitoring-cfg ClusterIP None 27017/TCP 16m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE monitoring-mongos ClusterIP 10.236.85.3 27017/TCP 14m Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found command terminated with exit code 1 command terminated with exit code 1 command terminated with exit code 1 command terminated with exit code 1 Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found command terminated with exit code 1 command terminated with exit code 1 command terminated with exit code 1 command terminated with exit code 1 Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found command terminated with exit code 1 command terminated with exit code 1 command terminated with exit code 1 command terminated with exit code 1 Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found command terminated with exit code 1 command terminated with exit code 1 command terminated with exit code 1 command terminated with exit code 1 Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found command terminated with exit code 1 command terminated with exit code 1 command terminated with exit code 1 command terminated with exit code 1 Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found command terminated with exit code 1 command terminated with exit code 1 command terminated with exit code 1 command terminated with exit code 1 Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found command terminated with exit code 1 command terminated with exit code 1 command terminated with exit code 1 command terminated with exit code 1 Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found command terminated with exit code 1 command terminated with exit code 1 command terminated with exit code 1 command terminated with exit code 1 Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found Error from server (NotFound): perconaservermongodbs.psmdb.percona.com "monitoring-service" not found command terminated with exit code 1 command terminated with exit code 1 command terminated with exit code 1 command terminated with exit code 1 error: error from server (NotFound): pods "monitoring-rs0-0" not found in namespace "monitoring-2-0-18818" error: error from server (NotFound): pods "monitoring-rs0-0" not found in namespace "monitoring-2-0-18818" error: error from server (NotFound): pods "monitoring-rs0-0" not found in namespace "monitoring-2-0-18818" error: error from server (NotFound): pods "monitoring-rs0-0" not found in namespace "monitoring-2-0-18818" ----------------------------------------------------------------------------------- check for passwords leak ----------------------------------------------------------------------------------- secrets=YmFja3VwMTIzNDU2 Y2x1c3RlckFkbWluMTIzNDU2 Y2x1c3Rlck1vbml0b3IxMjM0NTY= ZGF0YWJhc2VBZG1pbjEyMzQ1Ng== dXNlckFkbWluMTIzNDU2 YmFja3VwMTIzNDU2 Y2x1c3RlckFkbWluMTIzNDU2 Y2x1c3Rlck1vbml0b3IxMjM0NTY= ZGF0YWJhc2VBZG1pbjEyMzQ1Ng== dXNlckFkbWluMTIzNDU2 passwords=backup123456 clusterAdmin123456 clusterMonitor123456 databaseAdmin123456 userAdmin123456 backup123456 clusterAdmin123456 clusterMonitor123456 databaseAdmin123456 userAdmin123456 YmFja3VwMTIzNDU2 Y2x1c3RlckFkbWluMTIzNDU2 Y2x1c3Rlck1vbml0b3IxMjM0NTY= ZGF0YWJhc2VBZG1pbjEyMzQ1Ng== dXNlckFkbWluMTIzNDU2 YmFja3VwMTIzNDU2 Y2x1c3RlckFkbWluMTIzNDU2 Y2x1c3Rlck1vbml0b3IxMjM0NTY= ZGF0YWJhc2VBZG1pbjEyMzQ1Ng== dXNlckFkbWluMTIzNDU2 pods=monitoring-0 psmdb-client-6cd48df8b6-wrr6b logs saved in: /tmp/tmp.LhztpBHtl6/logs_output-monitoring-0-monitoring.txt logs saved in: /tmp/tmp.LhztpBHtl6/logs_output-psmdb-client-6cd48df8b6-wrr6b-psmdb-client.txt logs saved in: /tmp/tmp.LhztpBHtl6/logs_output-percona-server-mongodb-operator-8d99b8f59-l7j52-percona-server-mongodb-operator.txt release "monitoring" uninstalled ----------------------------------------------------------------------------------- destroy cluster/operator and all other resources ----------------------------------------------------------------------------------- ----------------------------------------------------------------------------------- get and delete old CRDs and RBAC ----------------------------------------------------------------------------------- customresourcedefinition.apiextensions.k8s.io "perconaservermongodbbackups.psmdb.percona.com" deleted customresourcedefinition.apiextensions.k8s.io "perconaservermongodbrestores.psmdb.percona.com" deleted customresourcedefinition.apiextensions.k8s.io "perconaservermongodbs.psmdb.percona.com" deleted error: the server doesn't have a resource type "perconaservermongodbbackups" + kubectl patch perconaservermongodbbackups.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbbackups" error: the server doesn't have a resource type "perconaservermongodbrestores" + kubectl patch perconaservermongodbrestores.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbrestores" error: the server doesn't have a resource type "perconaservermongodbs" + kubectl patch perconaservermongodbs.psmdb.percona.com -n sh --type=merge -p '{"metadata":{"finalizers":[]}}' error: the server doesn't have a resource type "perconaservermongodbs" clusterrole.rbac.authorization.k8s.io "percona-server-mongodb-operator" deleted clusterrolebinding.rbac.authorization.k8s.io "service-account-percona-server-mongodb-operator" deleted namespace "cert-manager" deleted customresourcedefinition.apiextensions.k8s.io "certificaterequests.cert-manager.io" deleted customresourcedefinition.apiextensions.k8s.io "certificates.cert-manager.io" deleted customresourcedefinition.apiextensions.k8s.io "challenges.acme.cert-manager.io" deleted customresourcedefinition.apiextensions.k8s.io "clusterissuers.cert-manager.io" deleted customresourcedefinition.apiextensions.k8s.io "issuers.cert-manager.io" deleted customresourcedefinition.apiextensions.k8s.io "orders.acme.cert-manager.io" deleted serviceaccount "cert-manager-cainjector" deleted serviceaccount "cert-manager" deleted serviceaccount "cert-manager-webhook" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-cainjector" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-controller-issuers" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-controller-clusterissuers" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-controller-certificates" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-controller-orders" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-controller-challenges" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-controller-ingress-shim" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-cluster-view" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-view" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-edit" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-controller-approve:cert-manager-io" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-controller-certificatesigningrequests" deleted clusterrole.rbac.authorization.k8s.io "cert-manager-webhook:subjectaccessreviews" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-cainjector" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-issuers" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-clusterissuers" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-certificates" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-orders" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-challenges" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-ingress-shim" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-approve:cert-manager-io" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-controller-certificatesigningrequests" deleted clusterrolebinding.rbac.authorization.k8s.io "cert-manager-webhook:subjectaccessreviews" deleted role.rbac.authorization.k8s.io "cert-manager-cainjector:leaderelection" deleted role.rbac.authorization.k8s.io "cert-manager:leaderelection" deleted role.rbac.authorization.k8s.io "cert-manager-webhook:dynamic-serving" deleted rolebinding.rbac.authorization.k8s.io "cert-manager-cainjector:leaderelection" deleted rolebinding.rbac.authorization.k8s.io "cert-manager:leaderelection" deleted rolebinding.rbac.authorization.k8s.io "cert-manager-webhook:dynamic-serving" deleted deployment.apps "cert-manager-cainjector" deleted mutatingwebhookconfiguration.admissionregistration.k8s.io "cert-manager-webhook" deleted validatingwebhookconfiguration.admissionregistration.k8s.io "cert-manager-webhook" deleted namespace "cert-manager" deleted ----------------------------------------------------------------------------------- test passed -----------------------------------------------------------------------------------