While trying to Connect my VMware Cloud Director to my Tanzu Kubernetes Grid environment (TKGS) in vSphere 7 update 2, I kept hitting a certificate error. The error presented itself during the step to configure the Kubernetes policy for my Provider VDC. After following the wizard at:

Resources ==> Cloud Resources ==> Provider VDCs ==> Kubernetes.

The Wizard goes properly till I get to the Machine Classes tab which is where VMware Cloud Director get to check on the certificate of the supervisor cluster. That when the below error present itself:

“VMware Cloud Director cannot verify the Kubernetes API endpoint certificate on this Supervisor Cluster. It might be a vSphere generated default self-signed certificate or another invalid certificates. For the steps to install your own certificate, see Change the Kubernetes API Endpoint Certificate. For trusting the certificate, see https://kb.vmware.com/s/article/80996”

The cause of the issue is the certificate structure of Tanzu Kubernetes in vCenter, The certificate of the Supervisor Cluster is not automatically trusted by VCD. Calls made to the Supervisor Cluster by VCD fail due to SSL errors. While I have this issue with VCD 10.2.2, it actually affected previous versions as well, especially when using self-signed certificates.

KB80996 article on how to fix this need to be updated with step marked in red below, but for those who needs to resolve the issue today, I wanted to document the steps in here. Thanks to my colleague Stefan Berner for helping find the solution for it.

You will need to follow the below steps manually to set certificates to be trusted by VMware Cloud Director. Let’s start by the process to export the Supervisor Cluster Certs:

1. Login to the vCenter UI
2. Navigate to Menu > Developer Center > API Explorer > GET /api/vcenter/namespace-management/clusters/{cluster}
3. In the value for the cluster parameter, input the moref of the vCenter Cluster containing the Kubernetes Supervisor Cluster. Note: The moref is located in the URL of the H5 UI when clicking on the cluster. It is in the format: domain-xx
4. Click execute to execute the API, then copy the curl command displayed in the VC UI into a linux shell. Example:
   • curl -X GET ‘https://bos1-vcloud-static-179-8.eng.vmware.com/api/vcenter/namespace-management/clusters/domain-c8’ -H ‘vmware-api-session-id: 0884879709888982bc2daf63a758a5fd’
5. Copy and amend this curl command by adding “|jq ‘.tls_management_endpoint_certificate’|sed ‘s/\\n/\n/g’” to a linux shell. <== This is the step that was missing in the original public KB
6. Optionally, you may add the curl option “–insecure” if vCenter’s endpoint is associated with a self-signed certificate (non production use case).
7. Example of running the command, and output is shown below.
   • Output:   curl -X GET ‘https://bos1-vcloud-static-179-8.eng.vmware.com/api/vcenter/namespace-management/clusters/domain-c8‘ -H ‘vmware-api-session-id: 0884879709888982bc2daf63a758a5fd’ | jq ‘.tls_management_endpoint_certificate’|sed ‘s/\\n/\n/g’
   • Executing this command should output something like:

“—–BEGIN CERTIFICATE—–
MIIC5zCCAc+gAwIBAgIBADANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwprdWJlcm5ldGVzMB4XDTIxMDMxNzA1NTUxN1oXDTMxMDMxNTA1NTUxN1owFTETMBEGA1UEAxMKa3ViZXJuZXRlczCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALKfxtn8isJaZiYGGPw+2XU4rrNeZe6OeUPFJSHv7AedoEi5ZuVNJCxKZJkWskCwbgEDyfoNu0gu1ReVhrHmzz9K52+fn0rUFaR6K7sTTVZRsVgo5Lhfs7kIYXbEqY+AINjSN4YecFtzMzrDknhiNWKMox43d9jNSweyGoD1CwaQjD3LAWxqzdAAXXE3WvN7svqn2oHvn3x8KOce2X7LsIBe478rVq66oZlaICl9ihKg7n4vGQnBWvCcIMsia1HYBclGfACtCi6gZCEPFTORb0m6N7S6CQzJlzKWeiZzy7FEQlda+oZri9Rm/DsQ05TRMgYspBNQ7SJamZrjMbXN65cCAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgKkMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFET3ffQTvJTxRjUbwK+9/Hn0wIKMA0GCSqGSIb3DQEBCwUAA4IBAQAWOL+hZfO2EgIqTOMZRaytTQNhHpFOREqHWEPgyTokXXk861HDwv/9Z4TvWR9sSUAWy5bXQuXNW3kbQMsgmfZBe2jITEPtKkplEoxh10Ma9FZ+ulpY8H5/Rzny5OSZIJiBOJe4NhZ3kx5ImaN1/FlvaJeqorx97CKF7TUr4yFkuEyNxg1o60xyvPVClpdGA7StcpUQw3Cv28JI3oJPXrQESwIlKvyvraefC+fDFade9L8++VJ8lT07k2b0EVu8uynnhgEXMZcGWWgB9q76qg+0ORGVMrquq9hFe7zC3iHfszruDNFP5ByXjSEd5ZBC9JkSfSjGbP8TVtLQ9Sal3Bzb
—–END CERTIFICATE—–
“

Here is a screenshot of me running the command in my lab and the exact output I have gotten.
 

Now that we have the Supervisor Cluster, we will need to copy the text of it in a text file and save it with .pem extension and import it in VMware Cloud Director. Below is the steps to do that.

1. Copy this output content, without the double-quotes into a file with suffix “.pem”.
2. Open a browser to vCD provider UI, and navigate to tab “Administration“, and select menu “Trusted Certificates“
3. Click on “IMPORT”, then click on “SELECT FILE”, and input the PEM file created above.
4. Verify the details displayed and click the “IMPORT” button.
5. In the VCD provider UI, navigate to the Tanzu Kubernetes vCenter.
6. Go to Infrastructure Resources > vCenter Server Instances > (specific vCenter). Click on the vCenter Server.
7. Click “REFRESH” to refresh the vCenter Server.
8. Wait a short period of time (about a minute) and confirm that the policy was generated (or refreshed) by navigating to vCD provider UI “Monitor > Events”. where you should observe an event such as: “Provider VDC Kubernetes Policy ‘[name-of-PVDC]-[name-of-host-cluster]-KubernetesPolicy’ (c7167717-9484-4a59-bae3-7464b276212f) modification”
9. navigate to “Provider VDCs”, select your PVDC, then select “Policies / Kubernetes”, and confirm that the corresponding policy was auto generated as shown in the below screenshot.

This should fix the issue and allows you to move forward. Below is a screenshot of the certs parameters in my environment in case some might need to check if the cert they have imported into VCD is inline with what expected.