Securing your website or dealing with the fallout

- Fallout: Half-Sploded Pip-Boy, - NickLangleyInnovate, Artist Extraordinary, see his work on Deviant Art

Some time ago I was called forth to write an article on securing a web hosting server; As web hosting Systems Administrators here at SingleHop this is a topic with which we are all fairly familiar. After enough CSF installs, SSH port changes, and package updates you find that it all becomes muscle memory allowing your fingers to do the work while your mind gets to ponder on why that 32 core server is overloaded in spite of having no network traffic and only one major website. Maybe it needs a faster hard drive to store the MySQL data? Oh look, CSF is done installing. Sadly this approach fails pretty miserably when exploring the topic of site or account level security. Where beefing up server level security means following some rather regimented and well documented steps, site level security can mean ensuring that completely bespoke code doesn’t allow raw data to be injected or execute remote scripts. It can also mean ensuring that open source code, being quite the opposite of bespoke, doesn’t contain vulnerabilities that seem to be well known amongst all the wrong people. As with any security topic, there is absolutely no means whatsoever to cover all that is necessary in a single blog article. Furthermore there are some realities regarding the rights and responsibilities of account level security, and I’ll take some time to discuss these alongside the technical nitty-gritty.

No rights with plenty of responsibility: Reacting to a successful attack

Here is an all too familiar scenario with which most SysAdmins have dealt plenty of times:  The boys over at WordPress put out a fantastic product, and it runs most of the websites on your server fairly reliably.  You have many resellers, each of whom has many clients, and profits are good.  Suddenly, your yacht party is interrupted with a most dire phone call:, one of your client’s sites, has been compromised!  Working diligently with your good friends here at SingleHop we find that a WordPress plugin that was written by some random 14 year old from Switzerland has been leveraged to gain privileged access to your database.  Access to the WordPress database has allowed the attacker to inject code turning your server into a spam cannon, and now you have your resellers calling you angrily because they each have 20 customers who are getting bounce backs when they try to send mail.  Rather than selling the yacht, changing your name, and moving to Portugal to live out the rest of your days as a mime, Let’s buck up and see about getting this fixed.

Step one:  Zen. Seriously.

Who do you blame?  Who deserves your wrath?  You are quite legitimately concerned about the future of your business, so where does the yelling go?  You will never know the name or location of the person who actually compromised the website, so they’re out of the picture in spite of being the obvious choice.  Another (all too common) instinct is to blame your hosting provider; It’s their server, shouldn’t it be more secure?  Well, your hosting provider has zero control over what code you allow onto this server, and half the time neither do you.  Clients sign up through resellers and upload their code without any auditing, and who really looks at the code outside of the initial design?  Designers are, well, design oriented.  They don’t put much effort into ensuring that their (very pretty) site doesn’t have any unsanitized input fields.  How do we find peace among this madness?  Peace means resolving ourselves to fix the problem, not yell at it.  This is the most objectively efficient means of dealing with any issue; I’ve yelled at servers plenty, it hasn’t fixed a single one.  With that down, let’s move on.

Step two:  Nullify

How you render the attack inert depends somewhat on your server layout.  If you are running a shared server (as is the case in our case study here) then you want to suspend the site with extreme prejudice.  Right now this server is sending out spam, and we need to stop that post haste.  Most Real-time Blackhole lists (the guys who block your email when your server starts spamming) don’t allow you to remove the listing until they have stopped seeing spam activity for a few days.  The sooner you stop the spam, the sooner you can go back to sending mail properly.  You also stand the possibility of a defaced page, meaning that thousands of hapless tomato enthusiasts are greeted with a politically charged message about freeing whomever attacker has deemed to be an oppressed people rather than good advice for enriching their lives with home grown produce.  Making the pain stop is step one, and here are your most common options:

  1. Suspend the account in cPanel/Plesk.  Quick, easy, and allows you to immediately delegate corrective action to the site’s developer as nobody isn’t going to notice the giant “suspended” page on the front of their site.  It’s harsh, but effective.
  2. Chmod 000 any files that are currently perpetrating malicious activity.  We are glad to help you with this task, and can provide advice if you aren’t managed. We are here to help!
  3. Restore the site to a known good version.  Remember:  In all things, backups, backups, backups.

Step three:  Realize that you haven’t actually fixed anything yet.

This is crucial; I see so many people get this far down the line and walk away thinking that they’re good to go.  In reality you have only treated the symptoms; The real issue is that the site’s code was weak, not just that these weaknesses were actually leveraged.  Many attackers keep a database of vulnerable websites, so if you simply remove or chmod the malicious files they may be back within hours.  This is also where the above-mentioned rights and responsibilities come into play as we have to carefully delegate what needs to be done.

Next time we will discuss the final steps of reacting to a successful attack: delegation and follow-up.