Email Server Vulnerability and Anti-Spam Protection Techniques

The role of cyber security in modern business is hard to overstate. Almost all business processes are automated to a degree and thus need to be thoroughly protected from any potential tampering. Vendors use anti-malware and anti-reverse engineering techniques to protect their products, but they can’t possibly weed out every vulnerability. One particularly vulnerable area is communications. Your company probably intensively uses emails in your business communications just like most other companies out there. It means that crucially important data gets stored on email servers and transferred via the internet. Thus, the problem of securing those email servers becomes extremely important. However, this issue is not as simple as it sounds. One part of the problem is to protect outgoing data, which can be done by using encryption and making sure that you’re sending it to the correct recipient. But even more important is to protect your server from incoming emails – spam with malware-ridden attachments, as well as denial-of-service attacks. In this article, we will look at various ways to protect email servers from spam and other cyber security threats and will give you some tips on how to detect, evaluate, and fix vulnerabilities.

Dealing with vulnerabilities

Every cyber security breach is a result of particular vulnerability. As mentioned above, it’s impossible to weed out all of them, but nevertheless, we should do what we can. A great way to limit the number of vulnerabilities is to actively follow the best cyber security practices for email server setup and maintenance. This will allow you to avoid the most common issues and make sure there are no obvious holes in your defenses.

• Store the minimum amount of data – Any unnecessary data stored on the server simply widens potential attack surface and contributes to damage costs in case of an attack. Make sure that you aren’t using any unnecessary software and that all opened ports are in use and thoroughly protected (for example, via authorization requirements).
• Make sure that your server is up to date – Software always contains vulnerabilities, and when one is discovered, vendors usually issue a patch. You need to make sure that all the components of the server are always up-to-date with all the latest security patches and fixes.
• Employ a strong authentication procedure – Set complex password requirements for any account used to access the server. This will prevent a brute-force attack, which is one the easiest ways to crack a password. Other security measures depend on your specific hardware and software configurations, such as type of the server and OS in use, etc.

One more way to protect a server from unauthorized login is to use SMTP authentication. This is something that we will cover in more detail down the line. Another basic cyber security measure is to make sure that all your emails are thoroughly encrypted so as to protect the data from being intercepted via a man-in-the-middle (MitM) type attack. You must encrypt SMTP, POP3 and IMAP protocols with SSL/TLS type encryption. The problem of spam emails Apart from general vulnerabilities described above, probably the biggest problem email servers face today is spam emails. This problem can be further divided into two categories:

• Incoming spam messages – Spam messages from the outside that are sent to the server’s own clients
• Outgoing spam messages – Spam sent from clients to other parties, where the server acts as an Open Relay.

The best way to fight spam is to use content filtering. Such filters should be configured either on the server itself or via a proxy application, such as a firewall, that protects access to the server. Besides filtering, you can also blacklist known spam sending servers. There are a number of local IP black lists as well as DNS based lists such as DNSBL and SURBL out there. And to prevent an Open Relay, you should configure Mail Relay parameters for email server. One of the biggest problems of spam is that it often carries malware via attachments or links in the body of the email that infect the whole system when clicked. An infected email server is a threat to the stability of the whole system, not to mention the risk of someone compromising customers private data. There are, however, a myriad of tools, both built-in and third party, designed to protect email servers from malicious software.

Server stability and performance

Another concern with regards to email servers is their stability and server performance. And when we think about performance, the first thing that comes to mind is load balancing. With this regard, Denial of Service (or DoS) attacks can prove extremely damaging, as they can render the whole service out-of-commission for long periods of time. This can have double costs in remediation as well as lost reputation and customer loyalty. To prevent DoS attacks, you need to limit the amount of both general overtime and simultaneous connections to the SMTP server. Another type of DoS attack is sending high number of Send requests. To protect from it, you may want to enable SMTP authentication. When enabled, each time someone wants to send an email to the server, a set of credentials is required. Other ways to protect the server from large quantities of Send messages include Mail Relay and Reverse DNS. While the former allows you to specify IP addresses from which the server can send mail, the latter allows to compare IP addresses with domain and host names. Also, as a general rule of thumb, if your server doesn’t work, regardless of the reason, you need to have a reserve server ready. You can do this by having two MX records for each domain.

Security assessment

First, you need to make sure that you have a way to assess server security. Often times, the best thing to do is to take to solutions that are already there, such as cyber security audit services. However, if this is not an option, then you need to design your own process, choosing yourself how formal and flexible it should be.

Initial preparations

First things first, you need to determine the scope of your audit. To do this, you need to answer three simple questions:

• "What do I need to check?": List every piece of data (e.g., user names, attachments, contacts, etc.) and every parameter (e.g., uptime, performance, etc.) that you consider important. Subdivide it into a separate checklists within each area of responsibility (such as operating system, server, network). Weigh each entry on your lists according to the potential impact that a problem with this entry could cause.
• "How do I need to check it?": There are two approaches to this question:
  • Find the tools necessary to check whether the components on your list are vulnerable or not. Each entry on the list should correspond to a specific way of checking its vulnerability.
  • Derive additional controls from your list of potential vulnerabilities and add them to the list of monitored objects. Repeated entries are market rather than deleted.
• "Why do I need to check it?": Target priority is derived from its weight and from the effectiveness and scope of the check. We assign priorities for every entry and remove repeated ones only when they are fully covered by another control or a combination of controls.

The next step is to assess the worth of each security check with regards to resources necessary to run it. This includes estimating the costs of buying software, hiring or training personnel, and conducting the checks themselves. If the cost of a low priority entry is too high, it can be moved further down the list but not fully removed because the situation can always change. The checklist from NIST SP 800-45 is a great asset for creating your own list of objects to check. When the list is fully formed, all that’s left to do is to set a scope and designate resources for an audit. As a part of this preparation process, you can also make a detailed plan covering each procedure from top to bottom up to the point when the data is included into final report.

Checking for vulnerabilities

Now all you need to do is to conduct all the necessary checks. When a strict time limit is involved, you should get high priority items out of the way first. However, if there is no time limit, it may often be best to group checks out of convenience based on their scope – this approach can help you save both time and money. Sometimes, executing a check can take more time than was initially designated. In this case, it is often better to skip a check and move it into a separate group while trying to find a way to optimize the process. Any incident regarding data and server settings should be logged. In this early stages you don’t need to investigate each and every detail but instead make sure that your checks cover as much as possible in the designated time.

Final tips

The field of cyber security is constantly evolving, and email server security is no exception. However, as a conclusion we wanted to list a set of basic tips that should definitely be followed by everyone who wants to secure an email server. First, you need to make sure that security is on the table as early as possible. Many problems can be solved by setting up a server initially with security in mind, not to mention that this is probably the most cost-effective way to do things. The things you need to consider include:

1. The type of data that will go through the server and the type of services it will support
2. What level of security is required for the server
3. Who will use the server and what level of privilege will they have
4. What method of authentication are you planning to employ
5. How the server will integrate into existing network infrastructure
6. What other software needs to be installed
7. How the server will be maintained and managed

You also need to consider the required cyber security level and potential vectors from which your email server can be attacked. Another important security factor is an operating system the server is installed on. So, to sum things up, here are the most basic recommendations on email server security:

• Make sure that attack surface of your server is as small as possible. The best way to do this is to establish a network perimeter that will protect your corporate network. A proxy application within the perimeter (for Exchange Server it can be Edge Transport server) can be linked to an email server and used to transfer emails from and within your corporate network.
• Always apply encryption on any stage of data transfer. Never use self-written certificates and instead carefully select an SSL certificate for each component of the server.
• Don’t forget about the basics – The fact that the email server has built-in anti-malware capabilities is not a reason to drop third-party anti-viruses and anti-malware solutions. Using them will only help to reinforce your protection.
• Don’t forget about updates. Microsoft, for example, issues Security Bulletins with all the latest patches.
• And last but not least, set two MX DNS records, and don’t forget to back up your data.