This article will not go into detailed technicalities on identifying, troubleshooting, tracing, and fixing hacks. These topics are far too broad to cover in a single article, and is outside the scope of this article’s intent. This guide is to help you, the system admin, come up with a plan to handle situations involving breached security.

FTP Attacks on the Rise

A lot of hosts have reported continual issues with FTP hacks, where hackers are logging into FTP accounts with the user’s credentials and uploading malicious or “spammy” code. Here are a few examples:

http://support.inmotionhosting.com/ftp_exploits.html
http://mediumcube.com/mctalk/tag/ftp-password-hack/
http://www.34sp.com/blog/official-news/ftp-exploits-and-account-hacks/

No one seems to have identified the exact cause of the hacks, other than that they originate from the client’s computer. Originally this was attributed to viruses like Gumblar, but it’s hard to grasp that thousands of users all had the same problem – many of which had antivirus software capable of detecting the virus, or that were running various operating systems that didn’t have Adobe products installed. All we know is that someone hackers were accessing user login information from end users.

After a bit of research, it was found out that many popular FTP clients, like FileZilla, don’t encrypt the passwords when they’re stored, so it was recommended by security professionals to avoid storing passwords in FTP clients and browsers.

Need help on this? See: Securing FTP Access on a cPanel Server

Verifying the Extent of Damage

Most hosting providers are lucky to have never really had to deal with a major security breach. Most hacked site reports I’ve gotten in the last three years were standalone attacks or attacks targeted to vulnerabilities in specific software. Then there’s the few actual server issues that you have to take a look at…the ones involving root exploits and destroyed servers. Your job is to take that hacked site report and determine whether it’s an application issue, a server issue, or a client issue.

The Script Kiddie Attacks (Application Issues)

I categorize any targeted attack to a specific type of software, a script kiddie attack. In this situation, a software (usually open-source) has a discovered vulnerability that is posted on an advisory site, with a full synopsis of how the attack is performed. Then, an ignorant self-proclaimed “hacker” uses something like Google code search to look for sites running the vulnerable software, sometimes only targeting a specific server, and launches the attack against them.

These tend to be easier to handle since they’re easier to track. When someone comes to me about their hacked OsCommerce site, the first thing I know to check is the version, the security advisory for that software, and Google for similar attacks. Outdated software is the #1 factor in hacked sites nowadays, and they are also the easiest to deal with. Your action in this case only really needs to be as far as recommending to the user to keep their software up to date.

The Server Issue

You know something was out of place when you saw those suspicious processes running from /tmp on your server. You already traced it, and you may or may now know where it came from. That doesn’t mean your server was rooted, but don’t quite rule it out. If you see any malicious processes running as root on your server, you should assume that your server was rooted. And based on what you know about the powers of the root user, you probably already know that you need evacuate the users from the machine and reinstall the OS. Don’t take a chance by not doing this – you have no idea what the hacker could have done to your server, and it’s a lot less painful to suck up a few hours of moving data than having something ten times worse happen the second time around. Hopefully you already have a failover or backup plan in place to do emergency migrations in situations of a security breach or server failure.

The Initial Reaction / A Little Bit of Background

Quite honestly, I get really tired of the typical customer reaction when their site gets hacked. Since you’re the hosting provider, the customer almost always assumes that it’s your fault. “What can you do to protect my site” or “Did you fix the problem with your server” or “How did you let this happen”.

While you’d be doing your customers an excellent service to come up with ways to counteract attacks, it’s an unrealistic expectation that any hosting provider can prevent and/or detect every possible attack that can occur, especially when a majority of them are at the fault of the user. What does this mean? Well, let’s look at some simply statistics shall we? Out of 1,890 hacked sites (from various hosting providers) we studied in 2009 (not specific to a certain hosting provider):

• 31% were affected by the Gumblar virus or its variants (586 total)
• 42% were running severely outdated open-source software (794 total)
• 23% were running custom or unidentified applications that were exploited due to insecure coding (435 total)
• 4% were affected by a script executed on the server itself, or a server-side exploit (75 total)

Even in the 7% category, where the problem on the server end resulted in a hacked site, 52 out of the 75 users were using self-managed hosting services and [after a little prodding] self-admittedly failed to secure their server properly. The others in this category were determined as residing on a shared hosting server that was not secured properly to prevent rogue scripts in a common folder (like /tmp) from executing against everyone else on the server. Even then, this seemed to only affect servers running PHP as an Apache module, where there existed a ton of files owned by the webserver user or set to 777 permissions.

Dealing with your customers

If you’re part of a larger hosting organization you probably have a department that deals with these kinds of things. How far you need to go into communication depends on the particular issue, but here’s a metric that I feel is appropriate:

• If single, unrelated sites are hacked, deal with them separately
• If multiple sites running the same kind of software are getting similar attacks, consider adding a security advisory to your support center to notify users of a potential exploit
• If a server is hacked but only a handful of sites are hacked, deal with those users only, and keep it under wraps assuming you are able to fix the problem or seamlessly move everyone to another server
• If there’s a major issue going around, consider sending your customer base a notification, or making an official statement in your support center, blog, or mailing list

Basically, the key is communication, but you also don’t want to overshare. Be informative but brief. Those of you who have dealt with things like this before know that there’s really only three things the the customer wants to know:

• How did it happen
• Who did it
• How do we prevent this in the future

Whether you’re dealing with one customer or 50,000, you want to be able to answer those questions. Here is an excellent example:

http://www.34sp.com/blog/official-news/ftp-exploits-and-account-hacks/

Put simply, you have to assume that most of your customers are going to be idiots less knowledgeable, and realize that when they don’t know what’s going one, they either pretend like they know everything, or blame someone else. And while us hot-headed sysadmins just love to break down the ego of a know-it-all, the last thing you want to do is provoke a rash response. However, you also don’t want to beat around the bush.