Threat Hunting Toolset and Operational Security Considerations

Photo by Markus Spiske on Unsplash

In my last blog, I wrote how to detect fileless malware, but out-of-scope was how to hunt if you have an indication that someone is poking around in your network. With this blog, I want to tackle how to build an Open-Source Hunting Toolset, which artifacts and logs to collect, what to hunt for, and discuss Operational Security considerations.

Why Threat Hunt?

For me, a Threat Hunt is mostly something you do because you suspect someone has hacked your network. It is something else than log analysis in a SOC (Security Operations Center) based on an alert. Of course, an alert might be the starting point and after some deeper investigation could instigate a Threat Hunt. Same goes for intel from a CERT or a three- or four-letter agency that there’s something wrong in your network. Whatever the cause may be, you start by forming a hypothesis, defining a starting point, and then creating a plan of attack.

Do note that a Threat Hunt requires highly skilled and experienced people to be effective. Besides that, if you do not have the right tools and processes in place, it will be very difficult to do yourself. Now, I can’t help with the people part, however, I can help with tools and general directions. If you want a complete deep-dive into the subject, I highly recommend you read this book: https://www.packtpub.com/product/practical-threat-intelligence-and-data-driven-threat-hunting/9781838556372

Hypothesis

I already stated that forming a hypothesis is key to a Threat Hunt. Assume breach, look for weak links in your internal IT infrastructure and work out a plausible attack scenario. A hypothesis may be: Our Exchange Server was exposed to the internet and unpatched for six months so we assume it has been breached. Now, it is up to you to collect evidence to support or refute this hypothesis. Remember, both supporting and refuting the hypothesis are equally important. Beware of tunnel vision or confirmation bias!

Logs and Artifacts

Before selecting your tools, it should be clear what artifacts and logs you need to collect, based on your hypothesis. Taking a generic network into account there is quite an extensive list. However, there are some sources I would always collect:

Windows logs

Essential for a Threat Hunt in my opinion. However, there are some important considerations here. Do you want ALL the logs? Probably not. And which Event IDs do you want to collect in what manner and from which servers/clients? Regarding the log policy I can highly recommend this GitHub from the Dutch Security and Intelligence Service, with the addition of script block logging.

As for where to collect the logs from, it depends on your hypothesis. Where would you find the evidence for a compromise? As a rule of thumb, I would say to at least log the Domain Controllers (DC), File Servers and any place where your Crown Jewels are, because the threat actor might be after those (more about threat actors and OpSEC later). Don’t forget your exchange server audit logs if you have it on premise, especially if you’ve forgotten to patch it.

Not Windows logs

The choice here is vast, but there are some logs that I would always try to collect. These include:

• Firewall logs;
• Proxy logs;
• IDS/IPS logs if you have them;
• Syslog;
• Netflow.

Together with the Windows logs these will provide you with a comprehensive set of data to hunt. It might seem like a lot because it is. To quote one of the only good Coldplay songs, nobody said it was easy.

Artifacts

Now this is the trickiest part, because this really depends on where you’re looking and what you’re looking for. If we still assume we’re talking about mostly a Windows based network, you’ll find a list of artifacts here that you could collect:

• Browser history;
• Registry hives;
• Schedules tasks;
• Prefetch files (not always present on clients with SSD or on Windows Server);
• Downloads;
• Windows Event Logs (EVTX files) if you have no way to collect them with something like Windows Event Collection (WEC).

There are many, many more and I highly recommend checking this SANS poster for more artifacts. However, the above list should already provide you with a lot of information. Especially the registry can give you a lot of insights.

Photo by Nina Mercado on Unsplash

Tools

So, you’ve identified what data you need to conduct a Threat Hunt. Now what? Well, you need the correct tools for the job to assist you in the Threat Hunt. There are some great tools that I use in a Threat Hunt and that I know work.

Elastic

These guys should almost start paying me money given how much I mention them ;). However, if you don’t already have a log collector or SIEM, setting up Elastic is a quick and easy way to have one, in combination with Logstash if needed. There are several ways to get logs to Elastic, however OpSEC is a very important driver here which I will discuss later.

Suricata

You need something that captures network traffic, creates metadata, and has some great detection out of the box. Suricata checks all those boxes and is relatively easy to install. There is a standard Filebeat module for elastic so you can send the logs to Elastic. Depending on how big your network is, you should consider installing this on a different machine than Elastic.

Thor Lite/Loki

These tools from Nextron provide you with a way to scan remote systems with YARA rules on Indicators of Compromise (IOCs). Absolutely necessary to check for anomalies on hosts, and because you can do this remotely, great for your OpSEC.

Chainsaw

Chainsaw is a new kid on the block, and provides you with a way to quicky search through evtx, either based on EventID, regex and strings or with Sigma rules. I’ve tested this on some real logs and it was able to detect lateral movement and Empire backdoors, so quite impressed so far.

NfSen

If you plan on collecting Netflow, NfSen is a GUI that lets you search through Netflow created with nfdump. Alternatively, if you’re using Elasting, you can just enable the Filebeat Netflow module and directly feed your netflow into Elastic.

Powershell

You will need some Powershell Fu to collect and script some things for collecting artifacts. If you need to collect registry files, browser history and Downloads from several clients, you don’t want to do this by hand. Not only is it a lot of work, collecting this by hand or with something like FTK is tedious and not very handy from an OpSEC perspective. If you can, ask your Windows sysadmin, they will probably know how to do this.

What to hunt for?

Very difficult question, but as a rule of thumb you can use the Cyber Kill Chain and you have your hypothesis. Focus on numbers 3 to 7 from the Kill Chain and combine those with Modi Operandi (MO) that you know from threat actors. This of course requires that you or your staff keep up to date with the latest MO’s, for example by reading relevant sites or by using Threat Intel feeds.

Here are a few examples to make things more tangible:

• Unidentifiable PowerShell usage;
• Unexplainable Scheduled Tasks;
• DNS domains that look out of the ordinary;
• Divergent netflows;
• Suspicious emails and/or attachments;
• (Sysadmin) activity out of business hours;
• Unexplainable created accounts or groups in Active Directory;
• Commercial Cloud Software usage, like OneDrive or Google Drive which might not be allowed.

Of course, the list is quite extensive, but I do recommend checking out MITRE ATT&CK if you need more inspiration. Ultimately, this for the most part falls on the shoulders of the hunter performing this hunt. He or she should know what is normal and what is not and what to look for.

OpSEC

One of the pitfalls of a Threat Hunt is alerting the baddies that you’re on to them. This can have all kind of consequences, depending on the type of threat actor. A ransomware gang might decide to encrypt all your data or publish sensitive information faster than they initially intended. But a state actor might go deeper into your network or do some other shady stuff. If you perform a threat hunt, you’ll need to be very cautious. Do you just want to stop the attack, then you might not even need a Threat Hunt and just remediate the incident and perform forensics.

However, if you want to examine the impact before you kick out the opponent (or let them do their thing in a controlled manner), you need to be very cautious during the hunt. There are some steps you can take to minimize the noise you make while conducting your investigation. Now, these things might come across as a little paranoid, but you’re doing a Treat Hunt and not some kind of CTF (Capture The Flag). Just like hunting in real life, you do not want to alert the prey!

Communication

Use side-channels to communicate about the incident, for example a Signal group. Try to avoid company e-mail because the actor might control this and happily read along. You can also consider using a codename for the hunt, to make it easier to talk about. You should also compartmentalise who knows about the threat hunt. Remember, “loose lips sink ships” is still very much valid in this day and age.

Business hours

Don’t start working frantically but use normal operation hours and procedures to cover your tracks. If you need IT changes done, do them during the time windows that are normal for IT to do changes in. If it’s normal that people leave at 5 p.m., keep doing that. If there’s a need to make haste, then a Threat Hunt might not be the right course of action. Remediation might be a better one.

Gathering data

Suddenly logging in and gathering data might stand out. Luckily, many things can be dealt with remotely. Try to use WEC for event logs and just send them to a WEC server or Filebeat somewhere. Do you need artifacts from a Windows client or server? Check if it’s a Virtual Machine (VM). You can copy it and extract the artifacts somewhere else. The same goes for memory, just copy the VMEM and use Volatility or Mandiant REDLINE to examine the memory.

If it’s not a VM, that’s where PowerShell comes in. You can login remotely and copy the registry, browser history and other artifacts to a remote server. This can be a bit difficult, since especially the registry is hard to copy on a live system. You can use Volume Shadow Copy for this and for the rest is Google your friend (But here’s a tip)And again, do this in small steps and during normal business hours. Be sure to test scripts elsewhere before letting them loose on the target!

For non-Windows machines, there usually is some option to send syslog, so have a syslog server at the ready and configure the systems to send syslog there.

The hunting server

If you started using a server just for the purpose of hunting, make sure you secure it correctly. Use a host-based firewall and only open ports you need to. Consider using a VPN to connect to the server and place it in a logical network segment. Also, give it a logical hostname that adheres to your naming convention (leaving out the “this is a hunting server”-part. In the case of Elastic, try to use TLS for communication between the components. Use new SSH keys to log into the box and collect logs from the box itself with Filebeat to see if it has been attacked.

Conclusion

A Threat Hunt is a very resource intensive endeavour and I have barely scratched the surface. However, I do hope that these tips and tricks can help you build a threat hunting capability if you desire one in your company or institution. Should you have any questions, you can hit me up on twitter and LinkedIn.