Heroku Hacked? Dissecting Heroku’s Critical Security Flaw
Today, Heroku announced a serious security flaw that I discovered in their platform:
The vulnerability was a window through which an unauthorized user could potentially gain read-only access to an app’s deployed code and configuration variables… We confirmed the vulnerability, determining that it was introduced on December 28th.
Source: Heroku Status
Discovery
In 2008, my friend Todd Eichel sent me an invitation to try out this fledgling Platform as a Service beta. It was a piece of cake to deploy my apps. And, best of all, I thought the days of worrying about server security were behind me.
Like thousands of other Heroku customers, I relied on the promise of a carefully curated platform that I never had to worry about. But earlier this month, while working on one of my 20+ apps on Heroku, I discovered something terribly wrong:
I realized my Heroku “slug”, or the compressed archive of my app, was readable by anyone on this Linux server that was shared by many. It was a file named disk.img in a folder that seemed to be uniquely named for my dyno.
I did some digging around.
So though I could see my own slug disk.img, I could not list the slugs belonging to other people’s apps. And since the directory name for each was a long generated string, I could not guess the name of the others either. You can see here that the disk.img permission is 644, which means any user on the system can read it if it’s in a directory they can get to.
I did some more digging around and I realized that the same ID used in the directory name was mentioned by a process named ps-watch.
Aha! Now that I know one of the IDs, I could look for someone else’s disk.img.
And we’re in! Now I have read access to someone else’s disk.img. Let’s see what’s in it.
Complete shock and disbelief.
In addition to this configuration file for each app, I could also see the entire source code.
If I had malicious intent, I could anonymously deploy a number of free apps on Heroku to harvest the slugs of other Heroku apps. Source code. Databases. Credentials. Unbelievable.
I took a moment to think about the impact of what I just discovered, and then I notified Heroku.
Clouds are not immune from basic security risks.
As someone who has managed servers for thousands of production applications, I can tell you that the worst nightmare for an administrator is discovering that you have been hacked. Because by the time you find out, you’ve already lost. No file can be trusted, no credential can be left unchanged. It begins a process of recovering from failure and salvaging whatever you can.
It’s no different for administrators of cloud computing platforms. There are two strategies for providers to partition the resources for each user I would like to discuss: virtualization, or operating system/runtime privileges. In the first, each user is provided with a contained copy of what appears to be a complete machine dedicated to their use. In the latter, many users share one machine (which may be virtualized itself). Heroku has chosen the latter approach, though this is likely not apparent to many users.
The system/runtime privileges approach applied here is basically many users on a server, with filesystem privileges and quotas regulating what a user can do. Each running dyno on Heroku runs with its own user on a Linux server that is shared with the dynos of many other users.
Heroku is responsible for ensuring that users don’t access other user’s privileged information. This is similar to the challenges of shared hosting or even the shell accounts ISP provided in the 90’s. The cloud is just a bunch of regular servers basically. And the same potential security problems can happen.
What does this mean for Heroku customers? What does this mean for cloud computing?
I would venture to say most Platform as a Service users behave like they are powerless with regard to system security. Basically they have totally and blindly entrusted the power of securing the servers with their provider.
Fortunately, Heroku has a very good team that makes a good effort to ensure that all necessary precautions are taken and they are ready to respond to a security incident quickly.
But as more and more Platform as a Service providers pop up (including my own), I have noticed that many have a poor understanding of the importance of security and how to develop an effective security plan. For example, one Node.js platform provider that has been in the news recently was hacked in the past few days and all of their user databases were deleted. The reason why? They accidentally published their database password on GitHub. Oops.
What can you do?
Take security into your own hands. Users should be responsible for verifying that their providers meet their needs. Ask questions. Ultimately, you are responsible for your app and it’s up to you to find out what your provider really does for your app’s security behind the shallow promises of marketing materials.
Follow me at @daverecycles on Twitter. In my next post, I’ll show you how Heroku responded when I reported the vulnerability, and the 5 most important things you should know about your Platform as a Service provider before you entrust them with your app.
