Just before the holidays, Citrix announced that their Citrix Application Delivery Controller (ADC) and Citrix Gateway are prone to a vulnerability which can allow remote unauthenticated attackers to execute code on vulnerable gateways. This led to a wave of alarming headlines about “80,000 firms” being exposed to hacking due to this flaw. What’s more interesting is that Citrix has not released a fix for this issue but has rather published just mitigation steps that can help guard against the possibility of attacks. Citrix has not said a lot about the specifics of the vulnerability, leaving many admins wondering if they are actually affected and unsure of how to forensically identify attacks. In this post, I will briefly attempt to clear the air somewhat about this advisory, and I will also share original data about the prevalence of unpatched systems 20 days after the initial advisory. To begin this investigation, we must look at the mitigation steps advised by Citrix. The core of this fix is a NetScaler responder policy to reject certain requests: add responder policy ctx267027 "HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\"/vpns/\") && (!CLIENT.SSLVPN.IS_SSLVPN || HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\"/../\"))" respondwith403 We can readily infer from this that the exploit attempts would involve a path traversal likely related to the /vpns/ path on Citrix. I reviewed the Apache configuration on an ADC appliance and found that the /vpns/ path is home to some Perl content including a directory without an authorization requirement.
Looking into this Perl code, I quickly remembered how much I hate doing anything with Perl, but before long, I spotted a file path being constructed from a request header without sanitization. I could now write a ‘.xml’ file with limited user-controlled content in any location the web server process can write. Despite this, there was no clear way to go from an XML file to a PHP or Perl web shell, but I was able to load the crafted XML files through the Perl Template Toolkit. The EVAL_PERL option was disabled, but this still enabled some interesting functionality like enumerating valid usernames or even crawling directories to reveal active administrative session IDs. Perhaps most interestingly though was that the Template Toolkit also had an ‘undocumented feature’ which allowed arbitrary command execution when processing a crafted directive. To fully understand the situation and the possible exploitation vectors though, it is necessary to take a step back and discuss the high-level operation of NetScaler/ADC. I have minimal experience with these products, but I will try my best to explain the relevant bits as best I can. The ADC/NS product is designed to straddle multiple networks to provide load balancing, SSL offloading, secure gateways, etc. The system is configured using a management IP address referred to in Citrix nomenclature as the ADC or NS IP. Additional ‘Virtual IP’ addresses can be bound to HTTPS based VPN gateways. HTTPS requests directed to these virtual addresses are handled by the NetScaler Packet Processing Engine (NSPPE) which will redirect most requests to the VPN login page. Although the vulnerable Perl scripts can be directly referenced via the ADC/NS IP, requests on the virtual IP will be handled by NSPPE rather than being directed to Apache where the Perl handler would run. If you haven’t guessed yet, this is where the path traversal comes into play. The packet engine contains a mistake I’ve seen before in several other embedded network appliances. The system performs path comparisons without first canonicalizing the path and uses the result to determine the disposition of an incoming request. This aspect of the flaw is not actually a path traversal but rather an insufficient access control. When NSPPE receives a request for /vpn/index.html, it is expected to send this to Apache httpd, but in this case, the request has /vpn/../vpns/. Apache transforms this into simply /vpns/ and handles the request normally. The complete exploit chain requires just two HTTPS requests to achieve command execution. The first request establishes the crafted template, and the second invokes the command when the template is processed. It is important to note, however, that certain payloads will cause NetScaler to excessively log errors until it fills up the /var partition.
How Many Affected Organizations?
With several media reports citing possibly 80,000 affected firms, I was curious just how many NetScaler VPN portals are exposed to the Internet and how many of these have yet to enable to mitigation. To do this without revealing details of the vulnerability or potentially effecting affected devices, I came up with the following simple test: curl -vk --path-as-is https://$TARGET/vpn/../vpns/ 2>&1 | grep "You don't have permission to access /vpns/" >/dev/null && echo "VULNERABLE: $TARGET" || echo "MITIGATED: $TARGET" This is a simple direct test to see whether the service responds with an error text from Apache. I then exported data from Shodan using a variety of queries to identify possibly vulnerable VPN gateways. In total, this turned up more than 60,000 IP addresses. I used masscan to identify all IPs from my exports with a currently active service on port 443 and to exclude any IPs where the owner has previously requested exclusion from scanning. At this point, I was left with 58,620 IP addresses of “likely” NetScaler or ADC VPN portals. The scans, which were conducted 21 days after the advisory was released, showed that less than a third of these very exposed Citrix appliances had the mitigation enabled. (39,378 of the 58,620 scanned IPs were apparently vulnerable.) To put this in perspective, I correlated the IP addresses with their certificate data and found more than 26,000 unique subject common name values. The list contains countless high value targets across a swath of verticals including finance, government, and healthcare. In total, there were 141 distinct domain names ending .gov plus another 351 distinct names containing .gov. in the domain. (These country specific covered 24 different ccTLD but were predominantly from *.gov.uk and *.gov.au) It is alarming that so many organizations are currently at risk in such a sensitive part of their organization. Each one of these devices is an opportunity for criminals or spies to gain access to restricted networks and impersonate authorized users. I would strongly advise all organizations with NetScaler/ADC to apply the mitigation immediately to avoid compromise. This is not a particularly difficult flaw to exploit, and it’s almost hard to believe that nobody is exploiting it against real targets yet.
Mastering Security Configuration Management
Master Security Configuration Management with Tripwire's guide on best practices. This resource explores SCM's role in modern cybersecurity, reducing the attack surface, and achieving compliance with regulations. Gain practical insights for using SCM effectively in various environments.
