While delivering a distributed install of vCloud Automation Center using exactly the same steps I have used for few previous engagements, & while the setup completed perfectly without any errors, accessing the infrastructure tab in vCAC has continuously reported the following error:
—————————————————————————-
Service Unreachable
A required service cannot be reached at the expected address.
Please contact your system Administrator for Assistance.
Reference error REPO404.
—————————————————————————–
I was quite certain I have done the certs right, as I followed my certs guide that I had followed in few other engagements previously and posted it on my blog before at: vCloud Automation Center 6 Certificates A to Z. Just for your reference if you have not read that post all the certs was generated by Active Directory CA.
As the above error can be caused by few different causes, I have went into checking my different vCAC logs and the error that helped me identify the cause was in the vCAC IaaS Web Server Windows Event log and below is a copy of that error (I marked in red the part that gave it away)
——————————— vCAC 6 SSL/TLS secure channel Error Start ——————
Log Name: Application
Source: VMware GUI Administration
Date: 05/28/2014 5:36:52 PM
Event ID: 0
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: IAASWEB1.vt.com
Description:
Timestamp: 5/28/2014 9:36:52 PM
Message: Error occurred writing to the respository tracking log
System.Net.WebException: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel. —> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.
at System.Net.Security.SslState.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, Exception exception)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Net.TlsStream.ProcessAuthentication(LazyAsyncResult result)
at System.Net.TlsStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.PooledStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.ConnectStream.WriteHeaders(Boolean async)
— End of inner exception stack trace —
at System.Net.HttpWebRequest.GetRequestStream(TransportContext& context)
at System.Net.HttpWebRequest.GetRequestStream()
at System.Data.Services.Client.DataServiceContext.SaveResult.BatchRequest(Boolean replaceOnUpdate)
at System.Data.Services.Client.DataServiceContext.SaveChanges(SaveChangesOptions options)
at DynamicOps.Repository.RepositoryServiceContext.SaveChanges(SaveChangesOptions options)
at DynamicOps.Repository.Tracking.RepoLoggingSingleton.WriteExceptionToLogs(String message, Exception exceptionObject, Boolean writeAsWarning)
INNER EXCEPTION: System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.
at System.Net.Security.SslState.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, Exception exception)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Net.TlsStream.ProcessAuthentication(LazyAsyncResult result)
at System.Net.TlsStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.PooledStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.ConnectStream.WriteHeaders(Boolean async)
——————————— vCAC 6 SSL/TLS secure channel Error End ——————
Sorry for posting the full lengthy error, I just thought it might get handy for you to compare and ensure you are having exactly the same problem as well for people searching with a particular part of the error.
This problem seems to be caused by having the CA not being accessible by the different vCAC Machines due to it being blocked by firewall. The easy solution and better way of doing this is to create the firewall rules that allow the vCAC components to reach out the the Active Directory CA to be able to validate the certificates, but this was not an option at my customer and might not be an option at your customer, so let me show you the work around I have used to go around this.
1. On the IaaS Web Server (in my case IAASWEB1) Edit the following two files:
– E:\Program Files (x86)\VMware\vCAC\Server\Website\web.config
– E:\Program Files (x86)\VMware\vCAC\Web API\web.config
2. Change <servicePointManager checkCertificateName=”false” checkCertificateRevocationList=”true” />
To
<servicePointManager checkCertificateName=”false” checkCertificateRevocationList=”false” />
The below screenshot show the file after it has been edited:
3. After completing the above changes, Restart IIS, the vSphere agent, the DEM-Worker and DEM-Orchestrator and the vCAC service.
By the way if you only fix the following file E:\Program Files (x86)\VMware\vCAC\Server\Website\web.config, you will pass that stage and the infrastructure tab work, but as soon you try to deploy a blueprint, you get the following error:
This should get you up and running again, & should allow you to use the Infrastructure tab. Please leave me a comment if this has fixed your problem, or if you have fixed your problem differently to allow others to benefit.
12 responses to “vCAC 6 Service Unreachable – Reference error REPO404”
I have this exact issue, but the listed settings aren’t found in any of the web.config files. I’m guessing you guys changed something with 6.0.1
Hi Mike,
I believe these were there in 6.0.1 as well, though I might failed to mention in the article that I had marked the “Suppress certificate mismatch” when deploying the vCAC IaaS Web component certificate which should added this line. You can still get away with this by adding this line even if you did not have it as you that probably mean you did not check mark the “Suppress certificate mismatch” when you have done your installation.
Hope that help,
Eiad
[…] before to get it working but could not get past this error. I found references to it at places like https://www.virtualizationteam.com/cloud/vcac-6-service-unreachable-reference-error-repo404.html and in the release notes but the fixes documented were not working for me. I knew it was related […]
Hi Eiad,
Thanks for posting this.
Also, a note to others who are implementing this fix and are adding the relevant section to the web.config file because you did not check the “suppress certificate mismatch” checkbox on install… Don’t simply copy and paste the text from Eiad’s blog post into notepad, the formatting of the double quotes gets jacked up and the portal will throw a HTTP 500 error because of that. Make sure you *type* the additional configuration lines into the file, or if you prefer to copy/paste, make sure you delete and re-type the double quotes around the “false” lines.
Now for my comment…
I just ran into it at a customer site. We are also doing a distributed install, and we also installed signed certificates on all systems in the vCAC stack during the initial installation (identity appliance, cafe appliances, iaas web servers, and iaas manager servers). None of our browsers complain about invalid certs, but we got the same errors that you have listed here in our iaas web server windows event logs.
I’ve done other distributed installs with no issues, so I’m wondering why this one in particular had a problem. The only thing I can think of is that at this particular client they are using a 3rd party CA and the cert was signed by an intermediate CA. We chained the certs to include the base cert, the intermediate CA and root CA, and like I said above we get absolutely no browser errors and all servers trust the entire chain, but we still ran into this problem. In other environments they were all using an internal Microsoft CA and I never had a problem there. What type of CA were you using in the environment you discovered this problem and was there an intermediate CA in the middle between the base cert and the root?
The problem is the event log gives no indication as to what certificate it thinks is invalid, which is frustrating. I’d prefer to not have to implement the workaround, and would better like to understand what cert it believes to be invalid and fix the root problem. Any ideas?
Will
Will, Actually the environment I had the problem with was using Microsoft CA. For me it seemed as the cause was the vCAC machines running the certs were not able to access the CA to validate the certs.
I got this but the cause appeared to by my backend SQL DB didn’t start after a windows update.
Hi!
I fix this problem (Reference error REPO404) by updateding cert for SQL DB.
Open a command prompt as an administrator and navigate to the Cafe directory on the Model Manager Data installation machine.
C:\Program Files (x86)\VMware\vCAC\Server\Model Manager Data\Cafe
2
Type the following command to update the IaaS database with the certificate information in one step. Supply the IaaS database name (vcac, by default) and the fully qualified domain name of the database server.
vcac-Config.exe UpdateServerCertificates -d vcac_database -s sql_database_server -v
For example:
vcac-Config.exe UpdateServerCertificates -d vCAC -s tr-w2008-13.eng.mycompany -v
Thanks Kirill, I guess that can be another cause for the REPO404 and the solution for it :).
Hello;
The same symptom and evidence in error Logs had manifested itself in my vCAC 5.2 Windows Server instance. In my case though, I had to generate a new “Self-Signed Certificate” in IIS because the one that was created during the initial vCAC installation had “expired”.
As background info, because the original Certificate had expired, the Windows (vCAC) Services would simply not start!! This is a nasty default behaviour particularly since there weren’t many signs or references indicating this as a possible source.
To generate a new Self-Signed Cert in IIS 7, follow the simple instructions here http://technet.microsoft.com/en-us/library/cc753127%28v=ws.10%29.aspx and the rebind the newly-created Cert in IIS Manager under the Default Website. Restart IIS and you should be good to go.
I tried all of the above fixes, but was unable to resolve the REPO404 error. After digging a bit in the logs I noticed that there were some errors regarding FIPS encryption. Over the holiday, my vCAC box had been “hardened” to gov spec by another team and FIPS encryption was set to “enabled” in the local security policy. FIPS encryption breaks the connectivity between the vCAC appliance and the IaaS server.
If all of the above fixes do not resolve your issue check this setting:
Control Panel -> Administrative Tools -> Local Security Policy -> Local Policies -> Security Options -> System Cryptography: Use FIPS compliant algorithms (set to “disabled)
For me this fixed the problem: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2095768
Symptoms
•In the Infrastructure tab, you see the error:
Service Unreachable – A required service cannot be reached at the expected address. Contact your system administrator for assistance. Reference error REPO404
•In the C:\Program Files (x86)\VMware\vCAC\Server\Website\Logs\Web_Admin.log files, you see the error:
Bad Request – Request Too Long – HTTP Error 400. The size of the request headers is too long
Cause
This issue occurs because the default VMware vRealize Automation (formerly known as vCloud Automation Center) headers when searching LDAP or performing actions on a user’s behalf are too large for the default Windows Kerberos token size.
[…] https://www.virtualizationteam.com/cloud/vcac-6-service-unreachable-reference-error-repo404.html […]