I wanted to write a quick post after a few recent security experiences that I’ve had involving log monitoring and analysis…It turned out a little longer than expected as I trailed off down a few side streams of thought. Anyhow, I wanted to cover a few of the low/no cost important practices around security monitoring and share a couple of the tools that helped me. These experiences dealt with failed authentication attempts against publicly exposed Microsoft services. In each case, the services did not need to be as open as they were, and the system owners were unaware of the exposure. Two of the situations were with the SMTP service on Exchange 2010 servers, and one was the RDP service of a Windows 2008 server. In all cases, the system owners did manage to detect a spike in the authentication failure events in the Domain Controller Security logs, but the logs themselves did not give them the actual source IP address of the attacker.
The first practice I wanted to share was that it is important to monitor the connections that are traversing the firewall. This can be done fairly efficiently using flows or logs. Start with digging through the traffic and methodically filtering out everything that is known good traffic, then moving on to the things that are unknown, and finally keeping track of any changes and investigate the unknowns in more depth. Do this for both the incoming and the outgoing traffic. There are a variety of great open source tools for this including Graylog, Elastic Stack, and SiLK.
The second practice is to leverage port scans against the public facing IP real-estate. I like to include a port scan as part of the change management process as well as regular scheduled scans. If port scans are occurring as part of change management, it ensures that changes aren’t inadvertently introducing vulnerabilities. I’ve seen situations where a host was decommissioned, but the corresponding ACL and/or NAT was not removed from the firewall. A while later the real (internal) IP of the old host was recycled. This immediately and inadvertently exposed the host to the internet. I’m going to diverge on a tangent here. A poor practice that I see far too often is that a system is connected to a live network before it has been patched and because the system itself isn’t going into production yet, a weak, easy to remember password is used for the initial setup with the thought that later, before it is moved to production, it will be integrated with AD and given a strong password, maybe by the final system owner once everything is up and running. < – Please do not do this! The result is that the host is now public facing, unpatched, with weak credentials. This is a perfect target of opportunity, and at the rate that things are being scanned and compromises are automated, it really presents a significant risk. That tangent aside, the port scan will catch something like this if the scan is done after any changes (a good case for including proof of concepts in change management), when previously; because there was no live host, the scan would not detect the port as open. Manual port scanning can be taken a step further by automating the process via commercial services such as Shodan, or by scripting out your own scanning systems and diffing their results against a known good baseline. Both are ways to be alerted when a new service appears on the public facing IP space. Smaller organizations can leverage a free instance in AWS using NMAP. Just make sure and let the AWS security team know about the intent to scan. It is a good idea to give them a copy of the permission slip that authorized the scanning signed by management of the organization.
The third practice is to make sure and monitor things like the SMTP logs and IIS logs on public facing services in addition to firewall logs, AD logs, and others. These will have a wealth of information that once baselined, will provide clues about misconfigurations, as well as attacks. I also thoroughly enjoy looking at the before and after logs from years past. Things like how much log volume, how many unique hosts, what versions were running, etc. remind me how far we can travel in short periods of time. I’m a fan of logging as much as I can get away with and afford as well.
With regards to the tools, there were two tools that really helped me on these incidents. First were the SMTP logs, and second was enabling debug logging for the Netlogon service. The SMTP logs were useful because they gave me a clear look at the source IP addresses of the attackers, as well as some good information about the attackers. I could see that some of them were just spraying AUTH LOGIN at the services, while some would try AUTH NTLM after AUTH LOGIN was rejected, and then send some usernames and passwords. This tells me that I should look more closely at those sources. Maybe they are enumerating users, and maybe they are password spraying as well. The AUTH LOGIN attempts won’t show up in the Domain Controller logs because they never get the opportunity to try credentials, but the AUTH NTLM attempts will. If only the Domain Controller logs are being monitored, all those other attempts will be missed. We may not care about those directly, but as I mentioned, if there is a progression from one to the other, that should elevate concern because the attacker is showing more sophistication. The debug logging for the Netlogon service is enabled using the command “nltest /DBFlag:2080FFFF” from an Administrative command prompt on the Domain Controller. Turn it back off when done using the “nltest /dbflag:0x0” command, or leave it on if there are resources available to do so and it is useful to have in the logs that are being collected. The debug logging was key in the incident with the RDP server because the logs on the Domain Controller were not giving the source of the authentication failure. I would like to thank Jack Stromberg, whose post I ran across when researching this initially.
Hopefully this helps someone else! Thanks for reading!