Monitoring vSphere IaaS Control Plane with VMware Aria Operations Management Pack for Kubernetes.

As a vSphere administrator, you can activate existing vSphere clusters for the vSphere IaaS Control Plane, formerly known as vSphere with Tanzu. This creates a Kubernetes control plane layer within the clusters’ ESXi hosts. vSphere clusters activated for the vSphere IaaS control plane are called Supervisors.

Enabling the vSphere IaaS Control Plane transforms vSphere into a platform that allows organizations to run Kubernetes workloads natively on the vSphere hypervisor layer and within dedicated namespaces called vSphere or Supervisor Namespace. This creates a developer-ready platform where DevOps engineers and application developers can run Kubernetes applications in a self-service manner.

The vSphere Kubernetes workloads are applications deployed in one of the following ways:

• Applications that consist of containers running inside vSphere Pods running directly on the Supervisor.
• Virtual Machines that are provisioned through the Supervisor VM Service.
• Upstream vSphere Kubernetes Clusters, formerly known as TKG or Guest Clusters, are deployed using the Supervisor vSphere Kubernetes Service (VKS), formerly known as Tanzu Kubernetes Grid Service.
• Applications that run inside the vSphere Upstream Kubernetes Clusters.
• Supervisor services that extend the vSphere IaaS Control Plane functionality (such as Harbor, Data Services Manager (DSM), Contour, and ArgoCD, to name a few). Click here for the complete list of vSphere Supervisor services.

The OOTB vCenter integration in VMware Aria Operations can discover and monitor the vSphere IaaS Control Plane constructs. However, the VMware Aria Operations Management Pack for Kubernetes can provide end-to-end visibility for Kubernetes and its resources.

With Kubernetes becoming the platform of choice for running enterprise applications, organizations must have the required tools for IT teams to operationalize any Kubernetes platform. 

The Kubernetes management pack can monitor multiple Kubernetes solutions such as Red Hat Openshift, TKGm, TKGi, TKGs or any open-source Kubernetes Clusters. However, in this blog, we will focus on configuring the Kubernetes management pack to monitor the Supervisor itself and any vSphere Kubernetes Clusters running on it using Auto Discovery. This means we do not have to create a Kubernetes adaptor manually to monitor every upstream vSphere Kubernetes Cluster that gets provisioned on the Supervisor.

Requirements

• Download and install the VMware Aria Operations Management Pack for Kubernetes 2.2 from the marketplace. Link
• You have a working Supervisor accessible and resolvable as part of VMware Cloud Foundation (VCF) 5.2.0 or higher.
• You have a working VMware Aria Operations 8.18.0 or higher where you have installed the VMware Aria Operations Management Pack for Kubernetes 2.2.

Note here that cAdvisor or Prometheus will not be needed for data collection in this specific use case.

Before configuring the Kubernetes adaptor, we must get the Supervisor’s Control Plane URL and the Client Certificate Authorization the adaptor needs to establish a successful connection with the Supervisor. When configuring the adaptor, you can see three different credentials types under Credential from which we can choose.

You really want to use Client Certificate Auth since Basic Auth for Aria Operations has been deprecated since Kubernetes 1.19, and The Token Auth is only valid for 10 Hours, which would stop data collection when it expires. Obviously, we can’t expect the Administrator to update the Kubernetes adaptor Token Auth every 10 hours; that would just be not practical at all.

To get that information, we need to do the following:

• SSH into the vCenter appliance, managing the Supervisor.
• Run the decryptK8sPwd python script /usr/lib/vmware-wcp/decryptK8Pwd.py
• Login to the Supervisor via SSH using the provided floating IP and password.

Control Plane URL

Option 1 (Not Recommended): Based on the previous screenshot, your Control Plane URL would be https://192.168.111.120:6443 where 192.168.111.120 is the Supervisor floating IP provided by the decryptK8sPwd Python script.

Option 2: (Recommended): Login into the vSphere Client -> Shortcuts -> Inventories -> Workload Management -> Supervisors and capture the Control Plane Node address. Your Control Plane URL would be https://192.168.150.2:6443

I personally recommend using Option 2 since unlike the floating IP of the Supervisor the Control Plane Node IP will never change specially after Supervisor upgrade operations.

Client Certificate Authorization

Next, on the same prompt type kubectl config view --flatten to get the Certificate Authority Data, Client Certificate Data and Client Key Data that we will need for the Client Certificate Authorization.

Configuring the Kubernetes Adaptor

Login as an administrator into VMware Aria Operations and click Administration -> Integrations -> ADD to add a new adaptor.

Select Kubernetes as the cloud account you want to configure.

Provide the needed information while making sure to select Kubelet as the Collector Service as per the screenshot below

Now, scroll down and make sure we configure the following Advanced Settings as well, as shown in the screenshot below.

We are activating the auto-discovery of Tanzu Kubernetes Grid Workload/Guest clusters and the Auto-deleting when the Kubernetes Clutters no longer exist. We also provide the vCenter Server FQDN so the Kubernetes nodes can be mapped to their respective Virtual Machine Objects. Finally, we are deactivating the cAdvisor install check since it’s not being used.

Now click on VALIDATE CONNECTION to test the adaptor configuration and connection.

Review and Accept the Certificate.

Verify that the connection is successful and click Ok.

Finally, click ADD to add the Kubernetes cloud account.

After one of two collection cycles, we should see the status of the Kubernetes Adaptor monitoring the Supervisor is OK, which means that we are successfully monitoring the Supervisor and collecting data.

If we search now for the name of our Supervisor, which also happens to be called Supervisor, we can see the Supervisor Cluster being monitored via the vCenter Cloud account and the Kubernetes Cluster being monitored by the VMware Aria Operations Management Pack for Kubernetes 2.2

If we click on the Kubernetes Cluster -> Supervisor -> Metrics, we can see the Supervisor’s end-to-end relationship with all its components and available metrics.

Suppose we now provision an upstream Kubernetes Cluster on the Supervisor to deploy, for example, a Modern Application using Cloud Consumption Interface within VMware Aria Automation. In that case, we can see that the configured Kubernetes Adaptor automatically discovers the new provisioned Kubernetes Cluster and starts to monitor and collect its performance metrics.

We can first see it under Administration -> Integrations -> Accounts -> Kubernetes where we see a Kubernetes adaptor for the new provisioned Kubernetes cluster.

The Auto-discovery capability also automatically injects and uses the Client Certificate Auth data of the newly provisioned and discovered Kubernetes Cluster. If we edit the newly added Kubernetes adaptor and check its credentials, we can see all the information related to the Client Certificate Auth was automatically populated, which is fantastic.

If we search again for the Kubernetes Cluster called Supervisor like we did initially and click Metrics, we can see that the newly provisioned Kubernetes Cluster oc-tkg-cluster is now part of the end-to-end Supervisor relationship topology where it’s being monitored and data collected.

The VMware Aria Operations Management Pack for Kubernetes 2.2 includes many dashboards and reports for monitoring and managing your Supervisor’s Kubernetes Infrastructure.

For example we are showing the Infrastructure Inventory dashboard, which shows an overview of the inventory of the nodes in the Kubernetes clusters. Selecting a cluster allows you to see the configuration of the nodes in it, Total Pods, Services, Deployments, etc., with the distribution of CPU and memory.

Summary

In this blog, we explored how to set up the Kubernetes management pack, utilizing VMware Aria Operations to monitor the Supervisor and the performance of any upstream Kubernetes clusters running on vSphere.

This solution equips organizations with the tools to help IT teams effectively operationalize the Supervisor Kubernetes platform within VMware Cloud Foundation (VCF).

If you liked this content, please like and subscribe to my Blog and YouTube channel so you don’t miss any of my future posts.