Scrutinizing the Scrutinizer
While conducting an assessment for a client earlier this year we encountered the Plixer Scrutinizer application in use on the internal network. Having never seen this particular application before, a quick search provided the following description:
Plixer Scrutinizer is a network monitoring and analysis appliance that collects, interprets, and contextualizes data from every digital exchange and transaction to deliver insightful network intelligence and security reports.The product documentation also provided deployment guides for multiple virtual machine platforms, including KVM with a
link to download an image (https://docs.plixer.com/projects/plixer-scrutinizer-docs/en/latest/deployment_guides/deploy_virtual/virtual_kvm.html).
Extracting the file system from the KVM QCOW disk can be done a few ways. I chose to utilize the nbd module from
qemu-utils, the generic process for doing this is as follows:
# apt-get install qemu-utils
# modprobe nbd max_part=16
# qemu-nbd -c /dev/nbd0 /path/to/image.qcow2With the new device setup, the partition table can be dumped to identify the disk layout:
# fdisk -l /dev/nbd0
Disk /dev/nbd0: 100 GiB, 107374182400 bytes, 209715200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x000a89ae
Device Boot Start End Sectors Size Id Type
/dev/nbd0p1 * 2048 2099199 2097152 1G 83 Linux
/dev/nbd0p2 2099200 209715199 207616000 99G 8e Linux LVMThe disk image contains two partitions, the first is for system boot and contains the bootloader, kernel, initial file
system, while the second contains the system's root file system. The second partition type is Linux LVM, meaning it
cannot be mounted directly and requires LVM utilities to access. The first step is to activate the LVM target using the
pvscan command:
# pvscan --cache /dev/nbd0p2
pvscan[1340564] PV /dev/nbd0p2 online.With the LVM partition activated, the physical volumes can be listed using pvdisplay:
# pvdisplay /dev/nbd0p2
--- Physical volume ---
PV Name /dev/nbd0p2
VG Name vg_scrut
PV Size <99.00 GiB / not usable 3.00 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 25343
Free PE 0
Allocated PE 25343
PV UUID qgr177-hDNb-efLX-Y8AB-lPuE-jUvU-ejn2t0The output shows that the Volume Group (VG) is vg_scrut, vgdisplay can then be used to list the volumes within the
VG:
# lvdisplay /dev/vg_scrut
--- Logical volume ---
LV Path /dev/vg_scrut/lv_swap
LV Name lv_swap
VG Name vg_scrut
LV UUID glfyh1-2iiy-K2Ki-h6ii-exyR-Lqda-0qETJy
LV Write Access read/write
LV Creation host, time localhost, 2022-03-16 17:53:56 +0000
LV Status available
# open 0
LV Size 4.00 GiB
Current LE 1024
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:1
--- Logical volume ---
LV Path /dev/vg_scrut/lv_root
LV Name lv_root
VG Name vg_scrut
LV UUID uatqDs-i3wS-yHVw-4qe1-hLuD-vfwR-nIBkMe
LV Write Access read/write
LV Creation host, time localhost, 2022-03-16 17:53:56 +0000
LV Status available
# open 0
LV Size 20.00 GiB
Current LE 5120
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:2
--- Logical volume ---
LV Path /dev/vg_scrut/lv_db
LV Name lv_db
VG Name vg_scrut
LV UUID ArDzWb-ncPf-1mgJ-TD1u-2Dg1-NKEh-zI42kS
LV Write Access read/write
LV Creation host, time localhost, 2022-03-16 17:53:57 +0000
LV Status available
# open 0
LV Size <75.00 GiB
Current LE 19199
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:3In this case we are looking for the root file system which is contained within lv_root. This partition
can be mounted directly using the LV Path value:
# mount /dev/vg_scrut/lv_root tmp
# ll tmp
total 88
dr-xr-xr-x. 19 root root 4096 Apr 21 2022 ./
drwxrwxr-x 3 chris chris 4096 Oct 19 18:18 ../
lrwxrwxrwx. 1 root root 7 Mar 16 2022 bin -> usr/bin/
drwxr-xr-x. 2 root root 4096 Mar 16 2022 boot/
drwxr-xr-x. 2 root root 4096 Mar 16 2022 dev/
drwxr-xr-x. 85 root root 4096 Apr 21 2022 etc/
drwxr-xr-x. 5 root root 4096 Apr 21 2022 home/
lrwxrwxrwx. 1 root root 7 Mar 16 2022 lib -> usr/lib/
lrwxrwxrwx. 1 root root 9 Mar 16 2022 lib64 -> usr/lib64/
drwx------. 2 root root 16384 Mar 16 2022 lost+found/
drwxr-xr-x. 2 root root 4096 Apr 11 2018 media/
drwxr-xr-x. 2 root root 4096 Apr 11 2018 mnt/
drwxr-xr-x. 4 root root 4096 Apr 21 2022 opt/
drwxr-xr-x. 2 chris chris 4096 Apr 21 2022 plxr_spool/
drwxr-xr-x. 2 root root 4096 Mar 16 2022 proc/
dr-xr-x---. 4 root root 4096 Apr 21 2022 root/
drwxr-xr-x. 2 root root 4096 Mar 16 2022 run/
lrwxrwxrwx. 1 root root 8 Mar 16 2022 sbin -> usr/sbin/
drwxr-xr-x. 2 root root 4096 Apr 11 2018 srv/
drwxr-xr-x. 2 root root 4096 Mar 16 2022 sys/
drwxrwxrwt. 7 root root 4096 Apr 21 2022 tmp/
drwxr-xr-x. 14 root root 4096 Apr 21 2022 usr/
drwxr-xr-x. 20 root root 4096 Apr 21 2022 var/With the root file system mounted it is now possible to inspect the application content in hopes of identifying vulnerabilities
that can be used on the target within the client environment. Initial inspection of the system identified that the application
is utilizing Apache with FastCGI. This was identified by reviewing the configuration file /home/scrutinizer/files/conf/httpd-plixer.conf:
# This will hold all the configurations for apache that Plixer makes.
# We will no longer be editing the default httpd.conf file.
...
## FASTCGI SETUP ##
ErrorLogFormat "[%t] [%l] %F: %E: %M"
FcgidIOTimeout 600
FcgidBusyTimeout 600
FcgidMaxProcesses 100
FcgidIdleTimeout 1800
FcgidProcessLifeTime 1800
FcgidMaxRequestLen 52428800
FcgidMinProcessesPerClass 5
FcgidMaxProcessesPerClass 100
FcgidInitialEnv PGDATABASE plixer
FcgidInitialEnv PGHOST localhost
FcgidInitialEnv PGUSER plixer
FcgidInitialEnv PGSSLKEY timber_badger:/usr/share/httpd/.postgresql/postgresql.key
AddType application/x-httpd-fcgi .fcgi
...
...
Alias /fcgi "/home/plixer/scrutinizer/html/fcgi"
<Directory "/home/plixer/scrutinizer/html/fcgi">
RewriteEngine Off
Options +ExecCGI
AllowOverride None
Order allow,deny
Allow from all
</Directory>Within the directory specified inside the Apache configuration file, a single 12mb file was found (scrut_fcgi.fcgi).
The file contents can be seen in the following excerpt:
#!/opt/perl-5.34.0/bin/perl
#line 2 "/opt/perl/bin/par.pl"
eval 'exec /usr/bin/perl -S $0 ${1+"$@"}'
if 0; # not running under some shell
package __par_pl;
# --- This script must not use any modules at compile time ---
# use strict;
...
...
CORE::exit($1) if ($::__ERROR =~/^_TK_EXIT_\((\d+)\)/);
die $::__ERROR if $::__ERROR;
1;
#line 1006
__END__
PK<BINARY CONTENT>This application is written in Perl using the Perl Archive Toolkit (PAR) (https://metacpan.org/pod/PAR) as well as the
PAR Crypto filter (https://metacpan.org/pod/PAR::Filter::Crypto).
In practice, this file uses Perl to extract the zip contents
attached at the bottom of the file, unpacking to a directory in /tmp/. For instance, the application
is extracted to /tmp/par-726f6f74 in the following example:
$ ll /tmp/par-726f6f74/cache-0f9488d5891e440457464a09412b8fd4a393c4a3
total 24
drwxr-xr-x 3 root root 4096 Oct 27 21:03 ./
drwxr-xr-x 3 root root 4096 Oct 27 20:57 ../
-rw-r--r-- 1 root root 178 Oct 26 21:03 _CANARY_.txt
-rw-r--r-- 1 root root 3322 Oct 27 21:03 d4787e12.pl
-rw-r--r-- 1 root root 657 Oct 27 21:03 e52e8794.pl
drwxr-xr-x 4 root root 4096 Oct 27 21:03 inc/
-rw-r--r-- 1 root root 0 Oct 27 21:03 inc.lockThe actual application contents are encrypted using the use Filter::Crypto::Decrypt module:
package main;
#line 1 "script/scrut_fcgi.pl"
use Filter::Crypto::Decrypt;
460aecfc30146bb6acd3f326e386638f66ba2f653bc6b.......The module responsible for decrypting the application ships within the archive and can be found inside the inc
directory:
$ ll /tmp/par-726f6f74/cache-0f9488d5891e440457464a09412b8fd4a393c4a3/inc/lib/auto/Filter/Crypto/Decrypt/
total 28
-r-xr-xr-x 1 root root 24728 May 9 18:09 Decrypt.soWhile the source of the Perl module for the Crypto filter is available, I decided to take the approach of analyzing the extracted binary statically, as we often encounter instances where we are forced to analyze binary content that applies encryption and/or obfuscation (practice makes progress).
Within the shared object the function FilterCrypto_FilterDecrypt handles decryption by passing a hardcoded key filter_crypto_pswd into PKCS5_PBKDF2_HMAC_SHA1 with a known 'random' salt value to recreate a known unique password for each call:
EVP_CIPHER_CTX_init(ctx_1);
if ( EVP_CipherInit_ex(ctx_1, aes_256_cbc, 0LL, 0LL, 0LL, enc) )
{
if ( EVP_CIPHER_CTX_set_key_length(ctx_1, 32LL) )
{
if ( PKCS5_PBKDF2_HMAC_SHA1(&filter_crypto_pswd, 32LL, in_pass, in_salt, 2048LL, 32LL) == 1 )
{
out_buf = 0LL;
if ( EVP_CipherInit_ex(ctx_1, 0LL, 0LL, hmac_key, iv, enc) )The hardcoded key material filter_crypto_pswd is stored within the library at offset 0x3A20:
.rodata:0000000000003A20 filter_crypto_pswd db 4Bh, 44h, 0B4h, 75h, 7Eh, 0EEh, 9, 1Dh, 0E6h, 72h, 0FDh; 0
.rodata:0000000000003A20 ; DATA XREF: FilterCrypto_FilterDecrypt+6B2↑o
.rodata:0000000000003A2B db 85h, 0EAh, 73h, 0B9h, 19h, 7Fh, 0F9h, 84h, 2Ah, 9Eh; 0Bh
.rodata:0000000000003A35 db 0B3h, 5Ch, 0BBh, 38h, 80h, 9Eh, 49h, 0E7h, 13h, 0E2h; 15h
.rodata:0000000000003A3F db 4Eh ; 1Fh
.rodata:0000000000003A40 rng_seed dq 405FC00000000000h ; DATA XREF: FilterCrypto_PRNGInit+A0↑rThere are a few ways to proceed to retrieve the encrypted content, the documentation page for the module explicitly calls out the shortcomings (https://metacpan.org/pod/Filter::Crypto#WARNING):
None of the above checks are infallible, however, because unless the source code decryption filter module is statically
linked against the Perl executable then users can always replace the Perl executable being used to run the script with
their own version, perhaps hacked in such a way as to work around the above checks, and thus with debugging/deparsing
capabilities enabled. Such a hacked version of the Perl executable can certainly be produced since Perl is open source
itself.Looking at how the library works internally; the easiest solution was to hook the SSL import calls using LD_PRELOAD. The LD_PRELOAD environment variable allows users to specify additional shared libraries to be loaded before others, enabling the override of function calls in those later-loaded libraries with custom implementations provided in the LD_PRELOAD libraries. The following example code implements a simple shared object that will print the key material as it is used as well as the decrypted Perl code:
#define _GNU_SOURCE
#include <dlfcn.h>
#include <openssl/conf.h>
#include <openssl/evp.h>
#include <openssl/err.h>
#include <string.h>
#include <syslog.h>
#include <stdio.h>
// gcc evphook.c -o evphook.so -fPIC -shared -ldl -lcrypto
int key_len = 0;
void printHexString(const char* str) {
int i;
// Iterate over each character in the string
for (i=0; i<key_len; i++) {
// Print the hexadecimal representation of the character
printf("%02X ", (unsigned char)str[i]);
}
printf("\n");
}
//function prototype - int EVP_CipherUpdate(EVP_CIPHER_CTX *ctx, unsigned char *out,int *outl, const unsigned char *in, int inl);
int EVP_CipherUpdate(EVP_CIPHER_CTX *ctx, unsigned char *out,int *outl, const unsigned char *in, int inl) {
int (*original_target)(EVP_CIPHER_CTX *ctx, unsigned char *out,int *outl, const unsigned char *in, int inl);
int ret;
original_target = dlsym(RTLD_NEXT, "EVP_CipherUpdate");
ret = original_target(ctx,out,outl,in,inl);
printf("%s",out);
return ret;
}
//function prototype - int EVP_CipherInit_ex(EVP_CIPHER_CTX *ctx, const EVP_CIPHER *type,ENGINE *impl, const unsigned char *key, const unsigned char *iv, int enc);
int EVP_CipherInit_ex(EVP_CIPHER_CTX *ctx, const EVP_CIPHER *type,ENGINE *impl, const unsigned char *key, const unsigned char *iv, int enc) {
int (*original_target)(EVP_CIPHER_CTX *ctx, const EVP_CIPHER *type,ENGINE *impl, const unsigned char *key, const unsigned char *iv, int enc);
*(void **)(&original_target) = dlsym(RTLD_NEXT, "EVP_CipherInit_ex");
if(key != '\x00'){
printf("### Decrypt Init:\n#### Key: ");
printHexString(key);
printf("#### IV: ");
printHexString(iv);
}
return((*original_target)(ctx,type,impl,key,iv,enc));
}
//function prototype - int EVP_CIPHER_CTX_set_key_length(EVP_CIPHER_CTX *x, int keylen);
int EVP_CIPHER_CTX_set_key_length(EVP_CIPHER_CTX *x, int keylen) {
int (*original_target)(EVP_CIPHER_CTX *x, int keylen);
key_len = keylen;
*(void **)(&original_target) = dlsym(RTLD_NEXT, "EVP_CIPHER_CTX_set_key_length");
return((*original_target)(x,keylen));
}
//function prototype - int EVP_CipherFinal_ex(EVP_CIPHER_CTX *ctx, unsigned char *outm, int *outl);
int EVP_CipherFinal_ex(EVP_CIPHER_CTX *ctx, unsigned char *outm, int *outl) {
int (*original_target)(EVP_CIPHER_CTX *ctx, unsigned char *outm, int *outl);
int ret;
*(void **)(&original_target) = dlsym(RTLD_NEXT, "EVP_CipherFinal_ex");
ret = original_target(ctx,outm,outl);
printf(" %s\n##### CipherFinal\n",outm);
return ret;
}The compiled shared object is loaded using the LD_PRELOAD environment variable to hook the defined calls and output the
decrypted application content:
# LD_PRELOAD="/home/plixer/evphook.so" perl /home/plixer/scrutinizer/html/fcgi/scrut_fcgi.fcgi
### Decrypt Init:
#### Key: 5B 1F 31 FC 73 F8 C5 5F E2 52 DA A2 3C 76 EA DC 0E AB 3A A9 9F 73 C1 E3 49 32 73 D5 17 2F D1 FC
#### IV: AC D3 F3 26 E3 86 63 8F 66 BA 2F 65 3B C6 BA 93 00 FB C2 01 00 00 00 00 61 02 00 00 00 00 00 00
#!/usr/bin/perl
#START #UTF-8#
# http://www.perl.com/pub/2012/04/perlunicook-standard-preamble.html #UTF-8#
use utf8; # so literals and identifiers can be in UTF-8 #UTF-8#
use v5.16; # or later to get "unicode_strings" feature #UTF-8#
use strict; # quote strings, declare variables #UTF-8#
use warnings; # on by default #UTF-8#
use warnings qw(FATAL utf8); # fatalize encoding glitches #UTF-8#
use open qw(:std :utf8); # undeclared streams in UTF-8 #UTF-8#
#END #UTF-8#
# sanitize known environment variables.
use Plixer::Util::Taint qw( untaint_environment );
BEGIN {
# Bug 24156 - force LANG=en_US.UTF-8 in Scrutinizer
$ENV{LANG} = 'en_US.UTF-8';
untaint_environment();
}With access to the decrypted application content further testing identified multiple vulnerabilities, which resulted in unauthenticated users being able to compromise the application server and pivot further into the environment. The details of the vulnerabilities can be found in our public disclosure repository:
https://github.com/atredispartners/advisories/blob/master/ATREDIS-2023-0001.md
It is worth noting that Plixer made the disclosure process effortless and were communicative during the process, it was refreshing to work with a vendor who was accepting of our report and prioritized the remediation process.
