Threat status: Active threat (exploited in the wild)
Attacker utility: Network infrastructure compromise
Update June 3, 2021: Remote code execution (RCE) proof-of-concept (PoC) details were made public on June 2. Community and Rapid7 researchers have noted the PoC’s use in the wild, making CVE-2021-21985 an active threat.
Description
On Tuesday, May 25, 2021, VMware published security advisory VMSA-2021-0010, which includes details on CVE-2021-21985, a critical remote code execution vulnerability in the vSphere Client (HTML5) component of vCenter Server and VMware Cloud Foundation. The vulnerability arises from lack of input validation in the Virtual SAN Health Check plug-in, which is enabled by default in vCenter Server. Successful exploitation requires network access to port 443 and allows attackers to execute commands with unrestricted privileges on the underlying operating system that hosts vCenter Server. CVE-2021-21985 carries a CVSSv3 base score of 9.8.
VMware has released a blog post and a supplemental FAQ for VMSA-2021-0010, which highlights the elevated threat of ransomware, including against organizations running vCenter Server. As of May 26, 2021, there are no reports of exploitation in the wild—this, however, is unlikely to last.
Affected products
- vCenter Server 6.5
- vCenter Server 6.7
- vCenter Server 7.0
- Cloud Foundation (vCenter Server) 3.x
- Cloud Foundation (vCenter Server) 4.x
For information on fixed versions, see the matrix of affected products and updates in VMware’s advisory: https://www.vmware.com/security/advisories/VMSA-2021-0010.html
Rapid7 analysis
As with previous vCenter Server vulnerabilities, we classify CVE-2021-21985 as an impending threat: It is a high-value attack target for both advanced and commodity threat actors, and we expect exploitation to occur quickly and at scale. As of May 26, 2021, Rapid7 Labs identified roughly 6,000 vCenter Server instances exposed to the public internet.
Patch
The following changes add authentication to the Virtual SAN Health Check plugin’s /rest/* endpoints:
--- a/unpatched/src/h5-vsan-context.jar/WEB-INF/web.xml
+++ b/patched/src/h5-vsan-context.jar/WEB-INF/web.xml
@@ -5,6 +5,21 @@
<display-name>h5-vsan-service</display-name>
+ <context-param>
+ <param-name>contextConfigLocation</param-name>
+ <param-value>/WEB-INF/spring/bundle-context.xml</param-value>
+ </context-param>
+
+ <!-- The application context needs to be OSGI-enabled in order to look up services -->
+ <context-param>
+ <param-name>contextClass</param-name>
+ <param-value>org.eclipse.virgo.web.dm.ServerOsgiBundleXmlWebApplicationContext</param-value>
+ </context-param>
+
+ <listener>
+ <listener-class>org.springframework.web.context.ContextLoaderListener</listener-class>
+ </listener>
+
<!-- Processes application requests -->
<servlet>
<servlet-name>springServlet</servlet-name>
@@ -12,7 +27,7 @@
<init-param>
<param-name>contextConfigLocation</param-name>
- <param-value>/WEB-INF/spring/bundle-context.xml</param-value>
+ <param-value>/WEB-INF/spring/empy-context.xml</param-value>
</init-param>
<!-- The application context needs to be OSGI-enabled in order to look up services -->
@@ -40,4 +55,14 @@
<url-pattern>/*</url-pattern>
</filter-mapping>
+ <filter>
+ <filter-name>authenticationFilter</filter-name>
+ <filter-class>com.vmware.vsan.client.services.AuthenticationFilter</filter-class>
+ </filter>
+
+ <filter-mapping>
+ <filter-name>authenticationFilter</filter-name>
+ <url-pattern>/rest/*</url-pattern>
+ </filter-mapping>
+
</web-app>package com.vmware.vsan.client.services;
import com.vmware.vise.usersession.UserSessionService;
import java.io.IOException;
import javax.servlet.Filter;
import javax.servlet.FilterChain;
import javax.servlet.FilterConfig;
import javax.servlet.ServletException;
import javax.servlet.ServletRequest;
import javax.servlet.ServletResponse;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.config.AutowireCapableBeanFactory;
import org.springframework.web.context.WebApplicationContext;
import org.springframework.web.context.support.WebApplicationContextUtils;
public class AuthenticationFilter implements Filter {
private static final Logger logger = LoggerFactory.getLogger(AuthenticationFilter.class);
@Autowired
private UserSessionService userSessionService;
public void init(FilterConfig filterConfig) {
WebApplicationContext context = WebApplicationContextUtils.getWebApplicationContext(filterConfig.getServletContext());
AutowireCapableBeanFactory factory = context.getAutowireCapableBeanFactory();
factory.autowireBean(this);
}
public void doFilter(ServletRequest request, ServletResponse response, FilterChain filterChain) throws IOException, ServletException {
if (this.userSessionService.getUserSession() == null) {
HttpServletRequest httpRequest = (HttpServletRequest)request;
HttpServletResponse httpResponse = (HttpServletResponse)response;
logger.warn(String.format("Null session detected for a %s request to %s", new Object[] { httpRequest.getMethod(), httpRequest.getRequestURL() }));
httpResponse.setStatus(401);
return;
}
filterChain.doFilter(request, response);
}
public void destroy() {}
}Furthermore, additional input validation was added to the com.vmware.vsan.client.services.ProxygenController class:
--- a/unpatched/src/h5-vsan-service.jar/com/vmware/vsan/client/services/ProxygenController.java
+++ b/patched/src/h5-vsan-service.jar/com/vmware/vsan/client/services/ProxygenController.java
@@ -1,151 +1,152 @@
package com.vmware.vsan.client.services;
import com.google.common.collect.ImmutableMap;
import com.google.gson.Gson;
+import com.vmware.proxygen.ts.TsService;
import com.vmware.vim.binding.vmodl.LocalizableMessage;
import com.vmware.vim.binding.vmodl.MethodFault;
import com.vmware.vim.binding.vmodl.RuntimeFault;
import com.vmware.vsphere.client.vsan.util.MessageBundle;
import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import org.apache.commons.lang.StringUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.BeansException;
import org.springframework.beans.factory.BeanFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.ResponseBody;
import org.springframework.web.multipart.MultipartFile;
@Controller
@RequestMapping({"/proxy"})
public class ProxygenController extends RestControllerBase {
private static final Logger logger = LoggerFactory.getLogger(ProxygenController.class);
@Autowired
private BeanFactory beanFactory;
@Autowired
private MessageBundle messages;
@RequestMapping(value = {"/service/{beanIdOrClassName}/{methodName}"}, method = {RequestMethod.POST}, consumes = {"application/json"}, produces = {"application/json"})
@ResponseBody
public Object invokeServiceWithJson(@PathVariable("beanIdOrClassName") String beanIdOrClassName, @PathVariable("methodName") String methodName, @RequestBody Map<String, Object> body) throws Exception {
List<Object> rawData = null;
try {
rawData = (List<Object>)body.get("methodInput");
} catch (Exception e) {
logger.error("service method failed to extract input data", e);
return handleException(e);
}
return invokeService(beanIdOrClassName, methodName, null, rawData);
}
@RequestMapping(value = {"/service/{beanIdOrClassName}/{methodName}"}, method = {RequestMethod.POST}, consumes = {"multipart/form-data"}, produces = {"application/json"})
@ResponseBody
public Object invokeServiceWithMultipartFormData(@PathVariable("beanIdOrClassName") String beanIdOrClassName, @PathVariable("methodName") String methodName, @RequestParam("file") MultipartFile[] files, @RequestParam("methodInput") String rawData) throws Exception {
List<Object> data = null;
try {
Gson gson = new Gson();
data = (List<Object>)gson.fromJson(rawData, List.class);
} catch (Exception e) {
logger.error("service method failed to extract input data", e);
return handleException(e);
}
return invokeService(beanIdOrClassName, methodName, files, data);
}
private Object invokeService(String beanIdOrClassName, String methodName, MultipartFile[] files, List<Object> data) throws Exception {
try {
Object bean = null;
String beanName = null;
Class<?> beanClass = null;
try {
beanClass = Class.forName(beanIdOrClassName);
beanName = StringUtils.uncapitalize(beanClass.getSimpleName());
} catch (ClassNotFoundException classNotFoundException) {
beanName = beanIdOrClassName;
}
try {
bean = this.beanFactory.getBean(beanName);
} catch (BeansException beansException) {
bean = this.beanFactory.getBean(beanClass);
}
byte b;
int i;
Method[] arrayOfMethod;
for (i = (arrayOfMethod = bean.getClass().getMethods()).length, b = 0; b < i; ) {
Method method = arrayOfMethod[b];
- if (!method.getName().equals(methodName)) {
+ if (!method.getName().equals(methodName) || !method.isAnnotationPresent((Class)TsService.class)) {
b++;
continue;
}
ProxygenSerializer serializer = new ProxygenSerializer();
Object[] methodInput = serializer.deserializeMethodInput(data, files, method);
Object result = method.invoke(bean, methodInput);
Map<String, Object> map = new HashMap<>();
map.put("result", serializer.serialize(result));
return map;
}
} catch (Exception e) {
logger.error("service method failed to invoke", e);
return handleException(e);
}
logger.error("service method not found: " + methodName + " @ " + beanIdOrClassName);
return handleException(null);
}
private Object handleException(Throwable t) {
if (t instanceof InvocationTargetException)
return handleException(((InvocationTargetException)t).getTargetException());
if (t instanceof java.util.concurrent.ExecutionException && t.getCause() != t)
return handleException(t.getCause());
if (t instanceof com.vmware.vise.data.query.DataException && t.getCause() != t)
return handleException(t.getCause());
if (t instanceof com.vmware.vim.vmomi.client.common.UnexpectedStatusCodeException)
return ImmutableMap.of("error", this.messages.string("util.dataservice.notRespondingFault"));
if (t instanceof VsanUiLocalizableException) {
VsanUiLocalizableException localizableException = (VsanUiLocalizableException)t;
return ImmutableMap.of("error", this.messages.string(
localizableException.getErrorKey(), localizableException.getParams()));
}
LocalizableMessage[] faultMessage = null;
String vmodlMessage = null;
if (t instanceof MethodFault) {
faultMessage = ((MethodFault)t).getFaultMessage();
vmodlMessage = ((MethodFault)t).getMessage();
} else if (t instanceof RuntimeFault) {
faultMessage = ((RuntimeFault)t).getFaultMessage();
vmodlMessage = ((RuntimeFault)t).getMessage();
}
if (faultMessage != null) {
byte b;
int i;
LocalizableMessage[] arrayOfLocalizableMessage;
for (i = (arrayOfLocalizableMessage = faultMessage).length, b = 0; b < i; ) {
LocalizableMessage localizable = arrayOfLocalizableMessage[b];
if (localizable.getMessage() != null && !localizable.getMessage().isEmpty())
return ImmutableMap.of("error", localizeFault(localizable.getMessage()));
if (localizable.getKey() != null && !localizable.getKey().isEmpty())
return ImmutableMap.of("error", localizeFault(localizable.getKey()));
b++;
}
}
if (StringUtils.isNotBlank(vmodlMessage))
return ImmutableMap.of("error", vmodlMessage);
return ImmutableMap.of("error", this.messages.string("vsan.common.generic.error"));
}
private String localizeFault(String key) {
return key;
}
}Which appears to be vulnerable to Java unsafe reflection:
unpatched/src/h5-vsan-service.jar/com/vmware/vsan/client/services/ProxygenController.java
severity:warning rule:java.lang.security.audit.unsafe-reflection.unsafe-reflection: If an attacker can supply values that the application then uses to determine which class to instantiate or which method to invoke,
the potential exists for the attacker to create control flow paths through the application
that were not intended by the application developers.
This attack vector may allow the attacker to bypass authentication or access control checks
or otherwise cause the application to behave in an unexpected manner.
73: beanClass = Class.forName(beanIdOrClassName);PoC
Affected endpoints are under /ui/h5-vsan/rest/proxy/service/ and respond to POST request:
wvu@kharak:~$ curl -kv https://192.168.161.2/ui/h5-vsan/rest/proxy/service/CLASS/METHOD -H "Content-Type: application/json" -d {}
* Trying 192.168.161.2...
* TCP_NODELAY set
* Connected to 192.168.161.2 (192.168.161.2) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: CN=192.168.161.2; C=US
* start date: May 28 00:29:04 2021 GMT
* expire date: May 23 00:29:02 2031 GMT
* issuer: CN=CA; DC=vsphere; DC=local; C=US; ST=California; O=photon-machine; OU=VMware Engineering
* SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
> POST /ui/h5-vsan/rest/proxy/service/CLASS/METHOD HTTP/1.1
> Host: 192.168.161.2
> User-Agent: curl/7.64.1
> Accept: */*
> Content-Type: application/json
> Content-Length: 2
>
* upload completely sent off: 2 out of 2 bytes
< HTTP/1.1 200
< Set-Cookie: JSESSIONID=57366FD1A729FCB43AA08B8304B1B4B6; Path=/ui/h5-vsan; Secure; HttpOnly
< Content-Type: application/json;charset=UTF-8
< Transfer-Encoding: chunked
< Date: Fri, 28 May 2021 15:45:14 GMT
< Server: Anonymous
<
* Connection #0 to host 192.168.161.2 left intact
{"error":"CLASS cannot be found by com.vmware.vsphere.client.h5vsan-6.7.0.20000-com.vmware.vsan.client.h5-vsan-service_6.5.0.11397901-storage-main in KernelBundleClassLoader: [bundle=com.vmware.vsphere.client.h5vsan-6.7.0.20000-com.vmware.vsan.client.h5-vsan-service_6.5.0.11397901-storage-main]"}* Closing connection 0
wvu@kharak:~$Note that this PoC does not achieve RCE on its own, as validation is performed against CLASS and METHOD. Supplemental analysis can be found here.
IOCs
The default log location for Virtual SAN health check plugin is /var/log/vmware/vsan-health. And user can change it by modifying the configuration item “logdir” in the configuration file under /usr/lib/vmware-vpx/vsan-health. On the vCenter Server for Windows, the file is located in %VMWARE_LOG_DIR%\vsan-health. No security related information is logged in the log file.
https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/products/products/vsan/vmw-gdl-vsan-health-check.pdf
Testing the PoC, only /var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log contained suspicious log entries:
==> /var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log <==
[2021-05-28T15:45:14.391Z] [ERROR] http-nio-5090-exec-5 com.vmware.vsan.client.services.ProxygenController service method failed to invoke org.eclipse.virgo.kernel.osgi.framework.ExtendedClassNotFoundException: CLASS cannot be found by com.vmware.vsphere.client.h5vsan-6.7.0.20000-com.vmware.vsan.client.h5-vsan-service_6.5.0.11397901-storage-main in KernelBundleClassLoader: [bundle=com.vmware.vsphere.client.h5vsan-6.7.0.20000-com.vmware.vsan.client.h5-vsan-service_6.5.0.11397901-storage-main]
at org.eclipse.virgo.kernel.userregion.internal.equinox.KernelBundleClassLoader.loadClass(KernelBundleClassLoader.java:150)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at com.vmware.vsan.client.services.ProxygenController.invokeService(ProxygenController.java:69)
at sun.reflect.GeneratedMethodAccessor532.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.vmware.vise.security.SessionManagementFilter.doFilter(SessionManagementFilter.java:201)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:685)
at org.eclipse.virgo.web.tomcat.support.ApplicationNameTrackingValve.invoke(ApplicationNameTrackingValve.java:33)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:800)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:800)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1471)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: CLASS cannot be found by com.vmware.vsphere.client.h5vsan-6.7.0.20000-com.vmware.vsan.client.h5-vsan-service_6.5.0.11397901-storage-main
at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader.java:501)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:421)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:412)
at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.loadClass(DefaultClassLoader.java:107)
at org.eclipse.virgo.kernel.userregion.internal.equinox.KernelBundleClassLoader.loadClass(KernelBundleClassLoader.java:146)
... 47 common frames omittedThe vCenter Server logs are placed in a different directory on disk depending on vCenter Server version and the deployed platform:
- vCenter Server 6.x and higher versions on Windows server: C:\ProgramData\VMware\vCenterServer\Logs\
- vCenter Server Appliance 6.x: /var/log/vmware/
- vCenter Server Appliance 6.x flash: /var/log/vmware/vsphere-client
- vCenter Server Appliance 6.x HTML5: /var/log/vmware/vsphere-ui
https://kb.vmware.com/s/article/1021804
This article provides steps to increase the size and number of the hostd, vpxa, and vpxd logs so that additional data is saved. This data may be useful for troubleshooting purposes.
https://kb.vmware.com/s/article/1004795
Guidance
Organizations should update to an unaffected version of vCenter Server immediately, without waiting for their regular patch cycles. Those with emergency patch or incident response procedures should consider invoking them, particularly if their implementations of vCenter Server are (or were recently) exposed to the public internet. If you are unable to patch immediately, VMware has instructions on disabling the Virtual SAN Health Check plugin here. Note that while disabling the plugin may mitigate exploitability, it does not remove the vulnerability.
Network administrators should ensure that vCenter Server is not exposed to the internet.
References
- https://www.vmware.com/security/advisories/VMSA-2021-0010.html
- https://blogs.vmware.com/vsphere/2021/05/vmsa-2021-0010.html
- https://core.vmware.com/resource/vmsa-2021-0010-faq
