Skip to Main Content
January 11, 2020

NetScaler Remote Code Execution Forensics

Written by Tyler Hudak

With the recent Citrix ADC (NetScaler) CVE-2019-19781 Remote Code Execution vulnerability, the TrustedSec Incident Response team has been working closely with our offensive and research teams as they created a working exploit. This has allowed us to create a list of locations and indicators to search for on potentially compromised Citrix ADC hosts.

Based on the exploit designed by TrustedSec, and our analysis of an internally compromised Citrix NetScaler Gateway 12.0 host, the following files and locations may contain evidence of a compromise.

Note: The Citrix ADC/NetScaler Gateway hosts that we have examined are running FreeBSD 8.4. Any FreeBSD-specific forensic information can be used as well.

Vulnerability Information

The vulnerability exploits a directory traversal attack on the /vpn directory provided by NetScaler. By specifying the correct files, remote code execution can be achieved. Note that code will execute as user nobody.

Accessing the Command Prompt

In order to access the FreeBSD command prompt, investigators will have to log in to the NetScaler command prompt (typically via SSH) and run the system command. This should place them at a root command prompt.

We recommend obtaining a disk image of the system when possible. If your gateway is running as a virtual machine, pause it and export all disks.

Log Files

The /var/log directory contains multiple log files for various system activities. The following log files have been found to be useful.

Note: FreeBSD will use gzip compression for many of these logs after they reach a specific size. Therefore, on active systems, you may have to examine the archived .gz versions of the log files as well.

bash.log: This file contains information on commands run through the Bash interpreter, /usr/bin/bash. TrustedSec has found that this file will still contain command executions even if the HISTFILE environment variable has been unset (a common anti-forensics technique).

Search for any suspicious executables such as curl, hostname, uname, or whoami, or commands run by user nobody.

Jan 10 13:35:47
<local7.notice> ns bash[63394]: nobody on /dev/pts/3

The number within the brackets (63394 in the example above) is the process ID of the process that executed it. Make sure to search for that in order to find additional information on what was executed.

Additionally, TrustedSec has also seen commands executed with the phrase '(null) on' where the username should be. This should also be searched for to find suspicious activity.

Jan 10 15:18:18  ns bash[66134]: (null) on (null) shell_command="Trying…"
 Jan 10 15:18:18  ns bash[66134]: (null) on (null) return_code="127"
 Jan 10 15:18:18  ns bash[66134]: (null) on (null) shell_command="Connected to"
 Jan 10 15:18:18  ns bash[66134]: (null) on (null) return_code="127"
 Jan 10 15:18:18  ns bash[66134]: (null) on (null) shell_command="Escape character is '^]'."
 Jan 10 15:18:18  ns bash[66134]: (null) on (null) return_code="127"
 Jan 10 15:18:24  ns bash[66134]: (null) on (null) shell_command="whoami"

Note: The file sh.log contains logs executed from the Bourne shell located at /bin/sh. This may also include useful data.

Notice.log: The notice.log file contains all Notice severity level messages from the system. Many commands, such as the Bash commands in bash.log, are logged with the Notice severity level, which will also be in this location.

Search for any suspicious executables such as curl, hostname, uname, or whoami, or commands run by user nobody.

httpaccess.log, httperror.log: These two files contain HTTP log messages from the Citrix server. Since the attack occurs over the web interface, these files will contain the IP addresses of the attacking systems.

Look for any HTTP log messages that contain directory traversal attacks with /vpn. Note that seeing a POST followed by a GET to an XML file is a great indicator for a successful attack. We have also seen successful executions with a direct request logged to /vpns  without the xml specified. - - [10/Jan/2020:13:23:51 +0000] "POST /vpn/../vpns/portal/scripts/ HTTP/1.1" 200 143 "" "USERAGENT " - - [10/Jan/2020:13:23:53 +0000] "GET /vpn/../vpns/portal/backdoor.xml HTTP/1.1" 200 941 "-" "USERAGENT" - - [10/Jan/2020:16:12:31 +0000] "POST /vpns/portal/scripts/ HTTP/1.1" 200 143 "" "USERAGENT"

File Locations

There are a few locations in which backdoors may initially be placed on the system after it is exploited. Search the following directories for unusual files:

  • /netscaler/portal/templates
  • /var/tmp/netscaler/portal/templates

Running Processes

Since the web server runs as user nobody, any initial commands (before escalation) will also run as user nobody. Therefore, you can search running processes for those owned by user nobody. From our analysis so far, few things outside of /bin/httpd should be running. Any processes running as a child of httpd, as shown below, should be investigated as suspicious .

 root@ns# ps auxd | grep nobody
 nobody    61092  0.0  0.3 111528  4844  ??  I    12:00PM   0:00.17 | |-- /bin/httpd
 nobody    63390  0.0  0.0  8320    16  ??  I     1:35PM   0:00.00 | | `-- sh -c uname & curl -o -
 nobody    63393  0.0  0.2 33204  3240  ??  I     1:35PM   0:00.04 | |   `-- python (python2.6)
 nobody    63394  0.0  0.1  8264  1544   3  Is+   1:35PM   0:00.00 | |     `-- bash
 nobody    64120  0.0  1.2 110496 19708  ??  S     2:00PM   0:00.15 | |-- /bin/httpd
 nobody    64121  0.0  0.0 105352     0  ??  IW   -         0:00.00 | |-- /bin/httpd
 nobody    64122  0.0  0.0 105352     0  ??  IW   -         0:00.00 | |-- /bin/httpd
 nobody    64123  0.0  0.0 105352     0  ??  IW   -         0:00.00 | |-- /bin/httpd
 nobody    64124  0.0  0.0 105352     0  ??  IW   -         0:00.00 | `-- /bin/httpd

Since FreeBSD uses a procfs virtual file system that contains information on processes, investigators can go into /proc/<PID> and possibly obtain additional information on running processes.

Additionally, running the lsof program (lsof -p <PID>) on the process ID of suspicious programs may also give information and clues.

Cron Jobs

Attackers are using cron to place their backdoors onto the system with this vulnerability as well. Since the exploit runs as user nobody, you will have to check nobody’s cron jobs to see if anything is present. By default, there should be no scheduled cron jobs present.

Run the following command to check nobody’s cron jobs:

# crontab -l -u nobody


To prevent this vulnerability from being exploited, we recommend following the mitigation steps published by Citrix: