We have recently dealt with an encryption virus infection at one of our contracted customers. Although these infections have fallen from the News, the kind of viruses which infected the NHS last year are still very prevalent and can quickly cripple a business. To give you an idea of how this kind of attack plays out, I thought it would be helpful to run through the incident to see where lessons can be learned.
During routine checks one morning, KCS identified that a large number of files and folders were displaying a last modified date very close together. The timestamp was late the previous day at around 5pm. Further investigation showed that file names throughout the folder structure were appended with .rapid. With alarm bells ringing, we determined that this was a typical presentation of Rapid Ransomware, including the tell-tale file ‘How Recovery Files.txt’.
KCS contacted the customer who were discussing the problem internally. Under instruction, they immediately shut down all PCs and an engineer prepped to go to site.
Point #1 – Immediate action and uncertainty.
At this point, PCs in the building cannot be trusted. As it can be difficult to tell where the virus entered the network, we have to err on the side of caution. There’s also a risk that the virus has infected multiple computers so we have to assume that everything is a potential vector of re-infection and remove all of them from the network. All PCs should be shut down until they have been cleared to re-join the network, often after a full re-install of the operating system.
The next step is to determine how we are going to recover from the incident. The first port of call here is of course the backups. In this case we hit our second problem, the virus had found, infected and corrupted the backups. This meant the latest usable backup would be the offsite drive from the previous week. Potentially a week of data could be lost. Luckily, due to a belt and braces configuration set in place by KCS, we were able to recover data from the previous morning. In another stroke of good luck, email data on the server was unaffected, meaning that this could be moved to a new email system without loss.
Point #2 – Backups can be a lifeline, invest in them.
Unfortunately the old advice regarding backups, (offsite backups are only for Enterprises, Windows Server Backup is fine for small businesses) are no longer true. Windows Server Backup (as this company was using.) is particularly vulnerable to this kind of attack and can’t always be trusted to get you out of trouble. Automated cloud backups are now within the price range of everyone and can be installed quickly and easily to protect servers and workstations. These can provide ransomware protection designed to combat exactly this kind of incident.
At this point the engineer arrived on site. All PCs were collected, along with a copy of all recovered data from the previous day. The PCs were returned to the workshop for rebuilding and the data kept in a safe location until the server rebuild could begin.
A decision has to be made at this point. Do we recover the old server as it was or rebuild as new and copy the data across.
As the backups had been corrupted, we could either restore from the previous week and copy over this week’s recovered data, or rebuild from scratch. A rebuild can have its bonuses of clearing out old configurations and retired software, but is time consuming. On the other hand, if the server is getting long in the tooth then replacement hardware may be the preferable option. In this case, that is exactly what was decided upon.
A two stage project began. First, migrate email to Office 365. Second, source and install a replacement server.
Given the time constraints, the email migration project had to begin immediately. Through experience and elbow grease, we were able to lay the groundworks for this overnight and by the following morning we were able to get the mail flowing to the new location.
Simultaneously, back in the workshop, two PCs were turned around as quickly as possible to get back to site. This did unfortunately leave the site without a computer for around a day from the beginning of the incident. Once on site the following morning, a selection of data was copied from the recovered files to allow users to continue working at limited capacity until the new server was put back in place. When returned to site, they were connected to the new email provider allowing communication to be restored and for customers to be warned of the IT problems.
At this point, the company was able to function with assistance from KCS until the server was put in place. The server was on site 3 days after the incident which fell on a Friday. The decision was made not to rock the boat before a weekend and final migration took place the following Monday morning. Ongoing work to bring the till back up to full functionality continued after that point, but all important services could now either work as intended or work with some manual intervention.
Point #3 – Reflection and minimising risk going forward.
User education must be a priority in the aftermath of these incidents. Be sure you know who a sender is before engaging them or opening their links and attachments. Someone who normally signs off with ‘Thanks, Jo’ is now saying ‘Sincerely, Joanne’. An email looks like it’s come from a professional company but they’re using a gmail email address. Use these odd signs to identify the con artists, and if in doubt, ask.
You should always look to close the gaps in security that caused problems previously. If the backups failed you, invest in something better. If the network was hacked from outside, upgrade your firewall. Whenever you’re not sure, seek advice and ask questions.
So how did the virus get in? And more importantly, why didn’t the anti-virus catch it?
The most common vector for these viruses is email. A message supposedly from DHL or Parcelforce stating you’ve missed a delivery gets your attention and has a convenient link to arrange redelivery. Few take the time to wonder whether they’ve ordered anything recently, or to realise that the office is open all day and any delivery would have been received. A historical configuration decision was to allow all users to install their own software. While we don’t allow this on any new installs, it used to be the norm. The security risks were seen as small compared to the inconvenience of having to call someone to install software.
Unfortunately, anti-virus can’t stop you installing software or opening a malicious email link, and it is via users that the majority of these infections take hold.
Regardless of your business size, you have data that dishonest people want. Don’t leave it to chance, stay one step ahead with a solid backup regime, an updated firewall and centrally monitored anti-virus.