CloudFlare has patched an issue in its HTML parser chain that caused a buffer overrun and returned memory containing private information.
According to CloudFlare CTO John Graham-Cumming, the Internet performance and security company first learned of the bug on 17 February. Tavis Ormandy, a Google Project Zero researcher who’s previously found holes in several anti-virus programs, contacted CloudFlare and said he was seeing some HTTP requests that had run through the company returning corrupted web pages. A closer look revealed the organization’s servers were running past the end of a buffer and returning memory that contained private information like authentication cookies as well as HTTP cookies and POST bodies.
Fortunately, the issue did not leak CloudFlare customers’ SSL private keys. The company has also found no evidence to suggest anyone maliciously exploited the bug.
So what caused the buffer overrun?
It all boils down to how the security firm employed an old parser written using Ragel. For years, the underlying bug had been present in the parser’s code. But the way it used its internal NGINX buffers prevented any memory leakage. That changed, however, when the company began to develop another parser named cf-html.
You can view the bug for yourself below:
/* generated code */ if ( ++p == pe ) goto _test_eof;
Graham-Cumming provides some context in a blog post:
“The root cause of the bug was that reaching the end of a buffer was checked using the equality operator and a pointer was able to step past the end of the buffer. This is known as a buffer overrun. Had the check been done using >= instead of == jumping over the buffer end would have been caught. The equality check is generated automatically by Ragel and was not part of the code that we wrote. This indicated that we were not using Ragel correctly.”
The bug essentially allowed in-flight HTML chunks exchanged between CloudFlare customers to be dumped in memory. This information should ideally remain private. To make matters worse, the company found that Google and other search engines had cached some of the leaked details as part of their normal processes.
CloudFlare didn’t waste any time in fixing the problem. 47 minutes after learning of the bug, the firm disabled email obfuscation, Server-side Excludes, and Automatic HTTPS Rewrites that were all using the same HTML parser. It also created a cross-functional team based in London and San Francisco to fix the problem.
In total, it took the team all of seven hours to develop a patch and deploy it worldwide.
But the security team isn’t done. As CloudFlare explains:
“Unfortunately, it was the ancient piece of software that contained a latent security problem and that problem only showed up as we were in the process of migrating away from it. Our internal infosec team is now undertaking a project to fuzz older software looking for potential other security problems.”
The Internet performance and security company went on to say it was “grateful that [the bug] was found by one of the world’s top security research teams and reported to us.”