Back in June, Robert Hansen posted an interesting write-up[1] on his Smartphone Exec blog about outsourced web development that was returned with multiple embedded PHP backdoors. While this betrayal of trust by a freelance web developer shouldn’t have been surprising, it was, and it prompted Tripwire’s Vulnerability and Exposure Research Team (VERT) to ask: How widespread is this issue? Tripwire’s VERT secured funding from the marketing team and started hiring developers from a pair of outsourcing websites. The plan was simple: create a non-technical persona, and as that person, hire a number of developers to create a website. Each developer received the same email with the same criteria (Figure 1).
Figure 1: Letter Sent to Developers While we weren’t sure what to expect, we had a series of goals in mind. First, we wanted to detail the interactions with the developers. How was the communication? Did they attempt to scam us right off the bat? How many people accepted the job, and how many turned it down or attempted to increase the price? While these were all interesting questions to answer, they weren’t our ultimate goal. Instead, we had two main objectives. First, we wanted to identify backdoors, hardcoded passwords, and intentionally malicious actions taken by the developers. Secondly, we wanted to find all the flaws and vulnerabilities in their code. If Hansen saw a backdoor with a single sample, we were interested in a full review of multiple samples. So, Tripwire’s VERT took to the freelance marketplace websites, posted the jobs, and interacted with the developers. What follows are details on the interactions and the results of our review.
Contractor Interactions
Across the freelance websites, there were 25 freelancers who made contact offering to take the job, with bids going up to $250. Several of the respondents requested direct payment and communication outside of the hiring system to avoid paying a commission to the website. These candidates were immediately excluded for terms of service violations. A handful of other respondents were only interested in the job if we could provide an example website to copy. Finally, a few more candidates were ruled out because they would not work within our budget. All of the remaining 17 candidates were hired and given seven to nine days to complete the functionality. (Interestingly, several of the programmers requested to have the payment split across two to three jobs, perhaps to inflate their “completed jobs” statistics and appear more experienced.) The developers were told that their sole job was to provide source code for a website with the required functions and utilizing a particular technology stack. We also specified that someone else would fill in all the content. None of the developers thought to ask which distribution or specific server versions would be used. During the course of the project, seven of the contracts had to be terminated for various reasons as follows:
- Contractor provided a basic WordPress install without support for clients to upload documents. They refused to provide a refund and falsely claimed (six times) that the work was done to our specifications. Customer support had to be invoked to force a refund.
- Contractor provided initial files and then requested to cancel the order because the developer “got busy” a few days into the project.
- Contractor asked for more money immediately after accepting the initial contract. They claimed a misunderstanding, believing the project was for updating an existing site rather than creating a new website. They did not have a problem providing a refund.
- Contractor repeatedly provided links to a WordPress website with configuration issues that led to page errors. Two weeks after the deadline, they agreed to provide a refund.
- Contractor requested the URL for the control panel of the website and indicated that it was difficult to make a website without this. The contract was canceled because the developer said it would take 20-25 days rather than the agreed upon seven days and that the price would need to be renegotiated.
- Contractor provided an HTML+JS page template and was unable to provide the full feature set.
- Contractor was unable to provide any deliverables after three weeks.
If this were a real-world project, these conflicts would be a serious problem, causing the project to run over budget or past the deadline. In the end, only 10 of the 17 commissioned projects resulted in completed websites. Deceptive practices and communication problems were also prevalent throughout our interactions with the contractors. One prominent example: With the deadline looming, many of the contractors delivered partial work, screen shots, or a simple message saying, “almost done.” Some of the contractors doing this claimed that it was simply to prevent a deadline slip from adversely affecting their profile ranking, but this tactic could also be used to fleece the customer. In the case where screen shots were delivered, it later became clear that the screen shots were mock-ups without real code behind them. One of the freelancing websites also has a policy that if a delivered project is not accepted or rejected within three days, it will automatically be considered as accepted. This makes it difficult for the customer to obtain a refund, and the buyer may at this point only have the option to accept the job or use one of the contractually-limited project revisions to avoid automatic acceptance. When we rejected fake deliveries (i.e., requested revisions) in two of the cases, the contractors responded with hostility, accusing us of trying to hurt their ratings and immediately delivered the same content. In another case, the contractor delivered a zip file containing nothing more than a login screen and said it was basically a finished product. We inquired about how to perform the basic functions on the website, such as uploading documents for translation, marking transactions as paid, and submitting feedback. The developer, though, insisted that we accept the delivery immediately, saying, “okay boss?? would you accep now. i told you i have the code ready but i havent checked that. if you accept now i will test and add them and deliver you before the time limit. actually i want this order to be completed ASAP to reach the level 1 thas why i requested you to accept this now.... [sic] ” The biggest barrier in the process was the limited English proficiency of the developers. We received messages like:
- “Actually I need to see the system. I think this type system is not available. If available then you can show me. You cant show me. So I think Possible functionalities I created. Now we can some modify it. Please try to understand. [sic]”
- “Please check this.I have sent you the whole code. After extracting it click on index.html file. I wanted to know would you be able to install database script on your system because all user accounts and admin data.All things are done with the help of database.Please let me know would you be able.Should i also send it to you? thanx [sic]”
- “Hello sir i need to ask you that would it be better that when the customer will pa-y you then you will upload the translated document back to him? [sic]”
In general, it was possible to work through the language barrier; however, at times, it was clear that the contractors were agreeing to the specifications without comprehending what was being asked.
Technical Results
As the project deliverables began coming in, it was clear there would be some security issues. For example, the first code that was delivered contained a clearly recognizable SQL injection vector, where a query string was being used directly to construct a query without sanitizing any potentially hostile input. Skill levels of the contractors varied greatly, but as expected for low-budget contract jobs, none of them were exactly subject matter experts on web development or security. The projects were mixed 50-50 between framework-based and custom PHP applications. Three of the projects used the CodeIgniter framework, and two others opted for WordPress applications. Most of the developers were using Windows development environments and had little to no knowledge of Linux. In several cases, critical errors arose when migrating the applications from the development environment to our LAMP system. This was due to case sensitivity and differences in how file paths are interpreted. With four of the developers, it was necessary to load their applications onto a virtual private server (VPS) because the contractors accused us of making up problems. The contractors were then provided credentials for accessing the system using SSH (Secure Shell) which is the de facto standard for remote administration of Linux servers. In every case, they had a problem understanding what to do:
- Contractor was provided a SSH key pair but did not know initially what it was or how to use it. (Key pairs are commonly distributed by service providers as a way to allow secured login without sharing a password.)
- Contractor was provided a root password for SSH and MySQL but insisted that only FTP access was available and therefore could not update the database. (FTP was not running on the server.)
- Contractor was provided SSH root password but did not know what to do with it because there was no site panel in which to enter the password. (The developer seemed to be referring to the cPanel web-based hosting control panel.)
- Contractor was provided SSH root password but could not access the server because it didn’t have cPanel installed.
Of the contractors hired for this work, none provided code to implement all the requested features in their first delivery attempt. In eight cases, the contractor submitted websites that had little to no functionality implemented. Despite the lack of a functional product, each of these contractors claimed they had completed all of the requirements and demanded immediate payment. Five of these contracts had to be terminated because the developers wanted to renegotiate the contract or it became clear that they did not have the competency to develop the full application. In one of the remaining cases, the contractor explained they had misunderstood the initial requirements and that our job would take three times as long, costing an additional $300. Although there was no problem issuing a refund, this exchange happened five days into a seven-day contract. If this were a real business project, a change like this would be difficult to manage. In light of these problems, we recommend that buyers of freelance development should have contractors prepare a clear bulleted list of the required deliverables. If they have provided such a list, it will be harder for them to later claim there was a misunderstanding. Here are some of the common failures we observed during the development cycle, along with tips for prevention or early identification:
- Queries not working because our MySQL was configured for strict mode, whereas the developer clearly used more permissive settings.
- State up front that the site must work with restrictive database configuration.
- PHP include directives failing because of improper file paths for Linux.
- Hire a developer who will test on the same OS as your server.
- CodeIgniter routes not working due to references with the wrong capitalization.
- Provide a development environment for the contractor.
- Broken content because absolute URLs reference the developer’s system.
- Test the site in an isolated environment. (This will also reveal if scripts are sourced from a CDN and therefore dependent on services outside of your control.)
- Broken content due to database entries referencing paths on the developer’s system.
- Search SQL dumps for paths and raise any concerns with the developer.
- Using similar development and production systems minimizes this risk.
Security Review
It was no surprise to find that every single website was plagued with critical security failures. One of the key requirements for this website was that clients could upload their documents and download translations only after having paid for the translation, yet every single website failed to protect any documents from unauthorized users. Additionally, none of the delivered websites effectively prevented us from uploading a PHP webshell. (A PHP webshell can be used by an attacker to access and modify data as well as to use the server as infrastructure for other attacks.) Several websites also had authentication bypass through basic SQL injection, which made it trivial for a completely anonymous user to gain access and take over the server. The following sections provide a summary report of each web site produced in this project.
Recommendations
Relying on low-budget freelance site development is not a good idea. While it may be possible to find someone producing quality work for bottom dollar, it is not the norm. Design tools and tutorials have made it much simpler to produce a professional looking site, but an understanding of how to use these tools is needed to make a secure product. When it’s necessary to outsource work in this way remember the following:
- Before starting a contract, get a sense of whether the candidate will be well-suited for the job.
- After you have decided on a developer, you will want to discuss appropriate project milestones so that you may review the work to see that it is progressing appropriately.
- The finished product should at a minimum be scanned by a web application vulnerability scanner and ideally evaluated by a professional penetration tester before final payments are made.
