learning from a grave mistake- the Apache.org attack

On April 13th, Apache announced that some of their servers were successfully compromised last week. They wrote a very detailed article about every step of the attack and what they did to avoid it again. This is a recap of that article and some of the responses in the community.

It was a very complicated attack that combined XSS, brute force password guessing, and social engineering. According to Apache, all users that were logged on between April 6 and 9 should consider their password broken and change it as soon as possible.

what they did right

While it is tempting to enumerate the bad things that were done, Apache did some right things to which we can learn from:

1. 1.OTPs were used on some of their servers.
   This  kept the damage to only the server with Jira, Bugzilla, and their wiki.  

2. 2.Fail2Ban was used on many other servers.
   This kept those servers from being compromised in the same way as the one with Jira. 

3. 3.They responded well.
   Once the attacks were revealed the services on it were moved to another server and the boxes they used to be on were considered untrustworthy.   

4. 4.They were transparent.
   Because they were so open, we can all learn from their mistakes. 

    

who was involved

It wasn't just Apache. The application, Jira, was written by  Altassian, it was hosted on an Ubuntu box, and it was hosted at Slicehost. Altassian responded well, Ubuntu wasn't part of the problem, but Slicehost didn't respond at all. That worries me. I'll talk about that more in just a minute.

the attack on passwords

Passwords were hashed instead of encrypted. While hashing is almost like a one way encryption,  it is not as strong as full encryption.

The passwords weren't correctly salted. Apache's write-up was a little vague about this. Some people read it to mean that the passwords were salted, but with a static salt, but I read that it wasn't salted at all. The idea is that while a salt that doesn't change avoids a dictionary attack, it is still open to a rainbow attack if the salt is known. However, the rainbow table needs to be rebuilt every time there is a new salt and if the salt constantly changes then the rainbow table is useless.

The passwords were also cached on subversion some passwords were stored in plain text. That allowed the attackers to get into a server with shell access. This problem, however, has been corrected.

Then there were some password issues that Apache has little if any control of.   Some administrators were using weak passwords that are easy to break.  Also, a user with sudo access was using the same password in his accounts on other services. While this wasn't in Apaches control completely, a sure with SUDO access should be under stricter rules. This is what allowed the attackers to break into Bugzilla and get shell access.

Most of these attacks could have been avoided if One-Time-Passwords were used more consistently, and SSH passwords weren't used. The OTPs were required on several servers, but not on the server that hosted Jira. The policy to require anybody with super-user rights to use an OTP is now system wide.

From their write-up I got that SSH was setup to not allow password based logins, but the attackers were able to get around this with PAM passwords being enabled. Again, that has been corrected.

Cross Site Scripting attack

The attack started on April 5 with a new issue that contained a shortened URL pointing to a XSS on the web application. Then on April 6 both the XSS became active and they started doing a brute-force password guessing attack.

Part of the attack involved using javascript to steal the session cookies which includes a hashed copy of the password.  That could have been mitigated with either another way to track sessions or using a flag in the HTTP response header.

Brute Force attack

At the same time as the XSS attack, they started a brute for attack trying to guess passwords. Hundreds of thousands passwords were sent. I have a hard time believing that this wasn't noticeable, but then if the attack was distributed over hundreds or thousands of IP addresses in a slow manner then it would be difficult for their server to distinguish what an attack from popularity.

social engineering attack

On April 9th, an email was faked that administrators and infrastructure users alike fell for. The attackers sent an email to all users saying that there was an error and gave them a temporary password for them to reset their password to the original.

This was only aggravated by the fact that a signed email wasn't used so account holders had no way of knowing whether the email was real or not. However, the attackers already had shell access which I believe means that they could have sent the email signed with Apache's current certificate. So while signing emails such as this could have reduced the threat, I doubt if it would help in this instance.

stages of attack

1. 1.Get some administrative credentials
   This was done with a XSS vulnerability and a brute-force password guessing. 

2. 2.Stop all notifications.
   This is a simple administrative setting. 

3. 3.Allow execution of uploaded files.
   This was done by directing attachments of new issues to a subdirectory with executable rights and adding new issues with the malicious programs attached. 

4. 4.Copy user homes. 

5. 5.Install a backdoor for future access. 

6. 6.Get as many passwords as possible 

   1. 1.Install a program to log all passwords successfully used. 

   2. 2.Convince users to login and reset their password. 

    

what we learned (both hosts and their "clients")

1. 1.We need to worry about XSS.
   Several years ago we heard that XSS was not dangerous. After all, what could they do but run some javascript? Well, combined with the other attacks it can be deadly. From the write-up we don't know if the XSS or the brute-force was successful. But the XSS was an important part of the whole process. Then there are the many other attacks to point to. 

2. 2.URL shortening helps social engineering.
   URL shortening has made me nervous for a long time. This only solidifies the fact that it is a problem. However, both Cooliris and TinyURL have ways of showing the real location. Both clients and servers alike can use it.  

3. 3.Clients must be diligent about watching where a link will take them.
   With a combination of watching the left side of their task bar and using a preview option they can make sure that a link takes them where they say. There are firefox extensions that will automatically change any URL for to do the preview.
   Whenever possible I avoid "click me" links and shortened URL's. Of course this isn't always possible when you only have 140 characters to work with. The fact is that whole URLs could be obfuscated . 

4. 4.Notifications and logging should be used at all times. 

   There is a reason that turning notifications off was the first thing that was done. The last thing hackers do is wipe the logs so no one knows they were there. If you get a notice that something happened on your site or you see something suspicious in your log files then you need to worry. The hard part of this is separating the noise from an indication of something that really needs your attention. 

5. 5.Users passwords are important.
   For years we've heard that passwords are no longer relevant. If that is true then someone needs to let me know. While there may be better options than passwords, passwords are still the first layer of defense and not  having strong passwords are letting your defenses down. This is true of both servers and clients alike. 

6. 6.Running a service/daemon without limited permissions is asking for trouble.
   I've seen this one too often. It is convenient to give a service/daemon all the rights. But what if that service or daemon is compromised? 

7. 7.Use an anti-XSS on the browsers.
   This includes no-script on firefox and the anti-XSS filter on IE8.  

8. 8.Sanitize input on web-apps.
   One of the easiest mistakes for a web-app programmer to make is to output data that is inside a HTTP header. What if that data printed includes scripting commands or it is crafted in a way to create a server crash? The easiest way to handle that is to not allow certain characters (< or > on an HTTP server and ' on a SQL server. Only allowing a specific set of characters would help.  

9. 9.Use an attack detection of some kind.
   This would include a daemon that runs in the background and if too many failed login attempts are performed then the IP is added to the hosts.deny file. The daemons 'hostdeny' and 'failtoban' are two such applications. Or slow the log in response time down so that a brute-force is not feasible. The daemons 'tarpit' would do this.
   Apache now uses 'fail2ban' on all their servers.  

10. 10.Don't keep track of sessions with cookies or use HTTPONLY.
    There are other ways to handle session tracking, but each of them has their own set of negatives. There is a wikipedia article below that explains it very well. Another thing you can do is to set the HTTPONLY flag so that javascript has no access to that cookie. 

11. 11.Expect a response from vendors and hosts alike when your site is under attack.
    The vendor of the web app patched their problem in only a couple of days after the attack. When Apache let them know wasn't enumerated. However, I would be very weary of the hosting company at this point. Even if the attack turned out to not be their problem, they should have been working with Apache when the attack was under way.  

12. 12.Use OTP whenever possible.
    They did stop hackers from spreading the maliciousness even further. If Apache hadn't been using them on other servers then the attack would have been even more successful. 

13. 13.Passwords should be stored encrypted.
    Hashing is better than storing them in plain text but it is not good enough. 

14. 14.Application hashes should be considered.
    Like in this case where passwords were found in plain text in the cache for a versioning software. This problem was corrected.  

other solutions

The community responded to this with the religious zeal of the OS war and with the cool  headed approach to figuring out what could be done better. Not only was the article at Apache very informative, but so were the comments at security sites. At least they were if you could ignore the noise from the religious wars. One way of learning involves seeing examples of how other sites avoid these attacks.

1.  

2. 1.Use a one time number sent to a cell phone. 

3. 2.Require administrator's permission for access each and every time.
   And if your session goes over 30 minutes you have to ask again.
   This can be done by requiring a manual edit of host.access and a chron job that clears the file every 30 minutes. 

4. 3.IT assigns password so they  have more control over how strong the password is.  

5. 4.Another uses a challenge/response with another server which calculates a token for them. Actually, the way this is used sounds a little iffy to me. Couldn't a bot do the same thing a human could? 

I have a couple questions myself:

1. 1.Was it a good idea to use Ubuntu?
   Yes it is used for other server sites, one of the biggest being Wikipedia, but it is basically known as a desktop OS. They use SUDO in a way that may be ok for desktops but isn't for servers. They also don't have SELINUX enabled like Fedora and Centos does. 

2. 2.Why wasn't SELinux used?
   I can answer part of that; their OS didn't support it. However, if it was used then the attack could go no further than the www directories. 

3. 3.What about flood detection?
   While this isn't really a part of the attack, maybe it could have noticed the attack earlier and warned the administrators that something was amiss.  

4. 4.Should Jira been used at all?
   Bugzilla has an option to not allow attachments to be uploaded into files with executable privileges. It would easy to point fingers at their programmers and scream "XSS", but that is a mistake many programmers make so I'll let them off this time. 

Final observations

A lot of people when they heard about this went to security blogs and started the debate about other OSes. First of all, this is was a problem with the web apps, social engineering, and administrative settings. All OSes are susceptible to these. The fact that it was sitting on top of Linux, FOSS was used, and they were the king of FOSS HTTP services doesn't change the fact that it wasn't any of their faults. Now, the FOSS defenders need to recognize that some of the arguments used against them were the same as the arguments they used against other OSes. Only the FOSS evangelicals were saying that Linux was immune to security attacks.  That doesn't negate the fact that Windows seems to include more attacks on their kernel and more web attacks affect the kernel. Linux is not perfect, but it is close compared to Windows IMHO.

sources

https://blogs.apache.org/infra/entry/apache_org_04_09_2010

http://talkback.zdnet.com/5208-12691-0.html?forumID=1&threadID=79083&messageID= 1529704&tag=col1;tbTools

http://www.darknet.org.uk/2010/04/hackers-penetrate-apache-org-in-direct-targeted-attack/

http://en.wikipedia.org/wiki/HTTP_cookie

http://tomcat.apache.org/security-5.html