By default most computers are left vulnerable to attack the moment they are attached to a network because they either have an inadequate firewall or no firewall configured on their machine. This is the case, in many situations, for both servers and your own desktop at home. It is for this reason that I am often asked the question "How good is an operating system-based firewall?" The answer is different for each situation.
In the home situation, it can be used as your primary firewall (for example, you do not have a router between your DSL, cable, or FiOS modem hardware) provided you have configured it correctly (or at least allow the operating system to manage it for you). In a dedicated server hosting environment, however, it is extremely important to have a firewall of some type enabled and to manually configure it to filter your incoming traffic.
The operating system firewall is, in most cases, a sufficient way to protect your server from being attacked. The Windows firewall service allows for you to designate what applications should be allowed to connect to the Internet, and allows for you to configure what services should be accessible from the Internet.
The Advanced Firewall in Windows Server 2008 and the Linux firewall (ipchains on kernel 2.4.x and iptables on kernel 2.6.x) allows for a very advanced firewall based upon matching traffic coming in and going out of the server based upon a set of pre-configured rules. The default Advanced firewall in Windows Server 2008 provides an easy to use interface, where as the Linux firewall either requires a 3rd party application to manage the firewall or extensive knowledge of the firewall command line options to configure it.
Of the 3rd party options for Linux, CSF (ConfigServer Firewall), Firehol, and ASF allow for a much easier to use configuration scheme, and CSF even provides a GUI Interface via the web when installed on a server with cPanel/WHM on it.
However there are some fall-backs to using a software based-firewall. For starters, the firewall is on the server itself and may be unable to handle the amount of traffic that your more busy networks can bring to the server. In the case of web hosting, this can be detrimental to your server's ability to handle the incoming traffic because it has to spend time processing traffic coming into the server through the firewall before it can handle the actual connection.
If your operating system firewall is mis-configured it can leave your server completely inaccessible or even more vulnerable to attacks. In theory, a malicious user could exploit a web site hosted on your server to gain access to an administrator's account and possibly modify your firewall to allow them through. In all of these cases, this is where a hardware firewall can prove much more reliable and secure than your software equivalent.
For starters, a hardware firewall takes the load of processing firewall rules, blocking, and permitting traffic, as well as providing logging functionality (which can be sent to a syslog server either on a Linux server behind the firewall or to the memory of the firewall itself) off of your servers. Hardware firewalls tend to be much more robust in their capability to block certain types of traffic as well as being more friendly to other Internet protocols. They also allow for you to have a single authority when it comes to filtering the traffic on your network. Hardware firewalls can handle the traffic for multiple servers and can differentiate between what traffic is allowed to one server but not to another. Your software firewall is not able to be managed as easily in terms of an enterprise-level deployment without extensive scripting or the implementation of a centralized domain controller in the case of a Windows-based network.
In the case of the Cisco PIX or ASA firewalls, they can provide a 1-to-1 NAT-based firewall solution, where machines behind the firewall have internal IP addresses, and the external interface of the firewall is configured with the individual globally-accessible IP addresses forwarding to the appropriate internal IP address. This can further secure your network by obfuscating your internal network's topology, making it harder for an intruder to map out your network and plan his or her attack.
Furthermore, the Cisco ASA series of firewall appliances can even provide a transparent firewall functionality, which operates on a level similar to a software-based firewall in that you do not have to configure your protected machines with an internal IP – the appliance will filter the traffic transparently and allow what you define as acceptable traffic through and reject or silently drop the rest of the traffic.
Another benefit of using a hardware firewall is their proven security in terms of access restriction to the management of the appliance itself. The Cisco firewall appliances come with a web-based GUI as well as software that can be installed on any machine you wish to delegate as your management terminal for your firewall.
Lastly, many firewall appliances allow for further functionality as a VPN access point, which provides access to an internal network in a secure fashion, and takes yet another load off of the servers on the inside.
All in all, your best option will depend on the individual needs for your dedicated hosting solution. If you are an end-user without much experience in securing a server for use on the Internet, or the administrator of a relatively low-traffic website, a software-based firewall may be the right choice for you. But if you are an experienced system administrator of a network of 2 or more machines with high security requirements for your network, a hardware firewall would be the best solution for your situation.For added security, you can implement both solutions and have something to fall back on in the event that a personal computer accessing your VPN connection should get compromised and try to infect your servers behind your firewall!