Ask me about Enterprise Security, Incident Response (IR), and APTs

The Something Awful Forums > Discussion > Ask/Tell > Ask me about Enterprise Security, Incident Response (IR), and APTs

Redshifted Ghost: Jan 12, 2016

About Myself
I�m a computer forensic investigator who has been doing enterprise incident response (IR) investigations for about four years. Within those four years I�ve done around 30 investigations for organizations ranging in size from small (couple hundred endpoints) to huge (100k+ endpoints). These organizations fall in categories such as defense contractors, merchants, healthcare, software, manufacturers, biochemical companies, law firms, research, media, government, and a bunch more that I can�t think of offhand. I primarily investigate breaches by state sponsored or financial threat actors.

What the hell is an APT?
APTs (Advanced Persistent Threats) was a term made popular by the IR company Mandiant to describe state sponsored and other threat actors who target organizations for the purpose of breaking into their network, maintaining long term persistence (months to years), and stealing information such as intellectual property, privileged communications, personal identifiable information (PII), credit card data, etc. I want to emphasize that this is not just malware, but people behind a keyboard somewhere on the internet actively doing things in your network. The best way to describe how serious this is, picture an administrator with domain admin moving around the network like an admin normally would, accessing servers and workstations with legitimate credentials, and is actively stealing data and you can�t stop them.

Nowadays security marketing pretty much murdered the APT terminology where every security product provides some sort of �APT� protection where the producing company has no experience dealing APT threats. Instead of using the terminology APT I prefer using �threat actor� and tossing in a category such as �state-sponsored threat actor� or �financial threat actor�, which brings us to the next topic of what kind of threats are out there? Here�s a high level categorization of various threats:

Botnets/mass malware/commodity malware � This is your run of the mill malware whose target is everyone, so you�ll see this stuff on both personal computers and on enterprise workstations. Ad-ware, banking Trojans, fake AV, crypto locker, etc. These are definitely things that an enterprise will want to take care of, but in most situations an incident of commodity malware isn�t going to amount more than cleaning up the system and moving on with your life.
Hacktivists � Hacktivists are threats such as anonymous whose primary goal is publicity, so they will do things like website defacement or stumbling across a vulnerable internet facing server that they break into, steal some data, and post it on pastebin. Typically these groups aren�t very sophisticated and tend to just take advantage of a situation they stumble across.
Opportunistic Threats � Like hacktivists these are threats that take advantage of a situation they stumble across, though instead of hacking for publicity, they will take advantage of a situation for money or to make the target serve malware in a watering hole attack.
Financial Threats � Financial threats primary goal is money, and we are not talking about small game like banking Trojans. Financial threats will target organizations to steal some sort of data to sell (credit cards, bank accounts) or to siphon money from (bank accounts, escrow accounts). One thing I want to emphasize is that in most cases, the financial threat actor has been within the target organization�s environment in order to access whatever data they are going for. Often financial threat actors break into the corporate environment, maintain persistence, and laterally moved from the corporate environment to the PCI environment for the purpose of mass deploying something like memory scraping credit card harvesting malware. They�ll then come back every few days and pull out the harvested credit card data back through the corporate environment.
Targeted Threats � Targeted threats are often state sponsored (examples being Chinese, Russian, Iranian) that target a specific organization for a specific reason. Maybe your company makes some super cool stealth widget. Some Chinese state owned company also wants to make the same super cool stealth widget. They could spend a lot of time and money on R&D or they can contract some hackers, for much cheaper than it would cost to R&D, to break into your company�s network and steal the schematics for the super cool stealth widgets in drastically less time (rather than years of R&D this could be achieved in days if the threat actor is good). They then begin production of your super cool stealth widget and then kill off your market share in China since it costs them way less to (not) R&D the technology and less to produce. The worst thing is, targeted threats typically won�t stop until they meet their end goal, so unless your company is hyper vigilant with the best security tech, fort Knox network segmentation, and an IR A-team, the threat actor will likely succeed in the end. Unfortunately, it takes drastically less time for a threat actor to do their thing than it takes to effectively investigate and react, putting the defensive blue team at an automatic disadvantage.

There are some topics I cannot touch on, specifically mentioning breached company names, discussing specific investigations whether public or non-public, or discussing specific threat actors (which to be honest I�m no intel expert). However, there are tons of other things I can talk about! Here are some starter topics I can hit in addition to general questions people might have:

How do threat actors gain access to an environment?
Malware persistence � The boring, the cool, and the really hard to find
Security �Oh god, please don�t do this in your environment� suggestions with real world explanations of why (tl;dr you did something stupid and I�ve seen a threat actor take advantage of it)
Nifty forensic artifacts
If I did X, Y, and Z on a computer, could you figure out what I did through forensics?
How do threat actors laterally move within an environment?
Why is AV such poo poo when it comes to stopping threat actors?
My hate for media reporting of breaches
Is this breach due to an insider threat? (probably not)
How common are breaches?
How to and how to not kick a threat actor out of your environment
How can I get into computer forensics and incident response?

# ? Mar 9, 2016 02:25

Adbot: ADBOT LOVES YOU

# ? Apr 16, 2024 19:24

denzelcurrypower: Jan 28, 2011

I'm interested to know if there's any crossover between your field and software development in general. I'm taking a course in enterprise software development which focuses on java and c# as well as SQL, but I find security work sounds really interesting. However, my networking knowledge is limited.

I'm a bit confused how one gets into this field because I was told by developers to not take a course in cyber security because it's more of an advanced topic to get into once you already know coding pretty well. But lately I've been hearing a lot about specialized courses for security analysts which require no prior experience.

Forgive me if it's a foolish question, I'm rather new to the field as a whole.

# ? Mar 9, 2016 16:09

Lincoln: May 12, 2007; Ladies.

So that one time when Captain Picard

# ? Mar 10, 2016 00:52

TheresNoThyme: Nov 23, 2012

How'd you get into the field? Do you ever do proactive work looking for threats or do people just call you up for consulting after they think they have a problem?

You can't talk about your job experience but you gotta dish on something... like how much stolen assets are we talking? In general if you find a breach are you successful tracing it back to the original actor? Is there any incentive to go after, like, foreigners who might be more difficult to prosecute or do people just patch up and move on with life?

# ? Mar 10, 2016 00:59

Redshifted Ghost: Jan 12, 2016

Ornithology posted:

I'm interested to know if there's any crossover between your field and software development in general. I'm taking a course in enterprise software development which focuses on java and c# as well as SQL, but I find security work sounds really interesting. However, my networking knowledge is limited.

I'm a bit confused how one gets into this field because I was told by developers to not take a course in cyber security because it's more of an advanced topic to get into once you already know coding pretty well. But lately I've been hearing a lot about specialized courses for security analysts which require no prior experience.

Forgive me if it's a foolish question, I'm rather new to the field as a whole.

That's not a foolish question, I'm happy you asked it because it gives me a good transition into general tips to get involved with security (with a heavy emphasis on incident response). For your situation, it depends on what in security you want to do. If you want to do dev work on some sort of security product, then yeah coding skills will be very important. If you want to reverse malware, coding skills and being intimately familiar with assembly will be crucial. Honestly outside of those two areas, coding skills are a secondary skill that is not crucial to your success in security.

I'll use myself as an example. For my day to day I'm either performing host based forensics, network based forensics, and/or running an investigation. During no part of my day to day do I need to break out my rudimentary programming skills. Frankly, the only times I write any sort of code is if I need some python script to help automate analysis or data parsing, and even then I do it infrequently enough that I usually need to use Google to get me to the finish line. Knowing a scripting language would be a boon, but it's not gonna shut you out of the market if you are not a programming/scripting pro.

So what skills do you need to break into security?

It depends on what you would like to do since there are many different areas. I'll touch on things directly related to forensics and incident response since those are my specialties. I'm sure there are others who can jump in with how you can get involved with red teaming, pen testing, web app assessing, etc.

Forensics/Incident Response
I do interviews for applicants looking to do IR, so I can tell you exactly the kind of things I'm looking for. There are two main realms, host based forensics and network based forensics. For host based forensics, here are some things you will want to become familiar with:

NTFS file system - You don't need to get into the weeds with the NTFS file system, but you should be familiar with what is stored in an NTFS master file table (MFT). For instance, the MFT contains file information such as file path, size, file timestamps (hint: there's 12, though you'll mostly be using 8 of the 12). You won't need to worry about knowing how to parse the MFT since there are tons of freely available parsers on the internet such as mft2csv.
Windows registry - Most systems will have hundreds of thousands of registry keys and no one expects you to memorize them, but there are some things in the registry that you want to be familiar with. For instance, there are tons of ways to persist malware across reboots through the registry. For user's who are interactively logged on, the user's NTUSER.DAT and USRCLASS.DAT will contain a bunch of information related to their logon session, allowing you to figure out some of what a user has done in the past. You can leverage something like regripper to auto pull out a lot of useful registry keys.
Malware Persistence - If you are doing incident response, you will be dealing with malware. Malware persistence is the leveraging of some sort of mechanism to ensure that a backdoor will start if the system were to be rebooted. There are an astounding numbers of ways to persist malware, though the most common being Windows services and registry "Run" keys. Know ways malware can persist is also helpful from the point of view of finding malware quickly. There are plenty of times where I've had evidence on the network of some sort of backdoor beaconing originating from a system and rather than sifting through the file system (which will be futile without knowledge of what to look for upfront) I'll short circuit the process by looking at unusual binaries persisting and then taking it from there. I can elaborate on further on malware persistence if there is an interest because there are some really cool and obscure ways you can have malware persist.
Windows Event Logs - Windows event logs often contain information crucial for an investigation. There are three main event logs: Application, system, and security. The application event log contains events logged from you guessed it, applications. Things like MSSQL and AV will make entries in this log. The system event log contains important items like service installations and service starts and stop which can be useful if a threat actor is using a tool or malware that leverages a unique service name. And then there's the security event log which will contain your logon events which are crucial for investigating lateral movement. If you find evidence of an attacker on a server that in only accessible internally, you will want to find out where the hell the attacker was coming from to get to that server. For Windows Vista and Server 2008 and above, there are additional event logs for things like terminal services, scheduled tasks, and powershell.
Web log analysis - This can be fairly straightforward and is mostly leveraged during cases when you know a threat actor is leveraging web shells, or web based backdoors. Web shells can sometimes be really cool (like a full featured backdoor in a less than 100 bytes!) which I can delve deeper into if there is interest.
How attackers laterally move - Or more simply put "if I wanted to execute X file on remote host Y, how can I do that?" A big thing that I want to emphasize with this is to think of how a legit admin could go about executing a file on a remote system. Literally, threat actors leverage the same methods as a legit admin once they have the credentials to do so. Threat actors will use things like scheduled tasks, PsExec, WMI, powershell, and RDP to laterally move in an environment.

If you are familiar with the above, you would pass my interview, regardless of education background. One extremely good book on forensics and IR is Incident Response & Computer Forensics written by a bunch of Mandiant folks. That book will teach you more about forensics and incident response than any class would.

When it comes to network forensics, you don't need to be a network engineer in order to succeed (though it wouldn't hurt). Here are some topics I recommend becoming familiar with:

DNS - Backdoors often use domains like "evil.com" for command and control (C2). Knowing how the DNS process functions will assist in determining where backdoor beaconing is truly originating from (hint: if your DNS logging sucks, just resolve the domain to its IP and look for hosts making a connection to that IP)
DHCP - Dynamic Host Configuration Protocol. You don't need to know the nitty gritty of DHCP, but just know that it exists and that it's used to dynamically assign IP address to hosts as they join a network. You'll need DHCP logs if you run into a situation where you have a workstation IP address from 6 months ago that was beaconing to a threat actor's C2 server. It's unlikely that the workstation currently at the IP address was the same one at the IP address 6 months ago.
TCP/UDP/SSL protocols - A high level understanding of these protocols is sufficient for entry level
What ports are used by what services - This is very useful if you are analyzing netflow data. Examples being HTTP - 80, SSL - 443, SMB (used by PsExec and scheduled tasks) - 445, SSH - 22, RDP - 3389, Powershell remoting - 5985 & 5986
Be familiar with packet analysis - Starting out, just use wireshark and do right click -> follow TCP stream for quick analysis. Once you get more comfortable you can start using some additional tools like Bro.
Be familiar with HTTP headers in requests and responses. Backdoors often mimic HTTP traffic but often times threat actors will gently caress it up by typoing a header field or not formatting it properly, making the backdoor traffic stick out like a sore thumb.

If you are familiar with the above, you would pass my interview for entry level.

Malware Reversing
I'll be blunt in I'm no malware reversing expert. I can do dynamic analysis and rudimentary static analysis. If you want to ask questions about malware reversing, I highly recommend checking out tekproxy's thread. If you want to start getting into malware reversing, I recommend picking up a copy of Practical Malware Analysis and checking out open security training's Introductory Intel x86-64.

Hopefully that should give you a good idea of where to start. If you have additional questions please continue to either ask them in this thread or you can send me PMs.

# ? Mar 10, 2016 02:58

Redshifted Ghost: Jan 12, 2016

TheresNoThyme posted:

How'd you get into the field? Do you ever do proactive work looking for threats or do people just call you up for consulting after they think they have a problem?

You can't talk about your job experience but you gotta dish on something... like how much stolen assets are we talking? In general if you find a breach are you successful tracing it back to the original actor? Is there any incentive to go after, like, foreigners who might be more difficult to prosecute or do people just patch up and move on with life?

Great questions!

How'd you get into the field?

I started working in the forensic/IR field straight out of college. My degree in college was a cyber security degree, though looking back I wish the degree had more things relevant to deep dive forensics and incident response. I was involved with a couple security clubs in college and interned with a company that did forensics work. I also did some elementary dabbling with malware reversing. When I interviewed at my current company, I ended up relying a lot on being able to logically walk through IR questions even if I wasn't an expert in the details, along with a lot of dumb luck.

Do you ever do proactive work looking for threats or do people just call you up for consulting after they think they have a problem?

I do both, depending on what kind of work is available at the time my schedule is free. We often have organizations come to us knowing they have a problem or maybe they received a law enforcement notification telling them they have a problem. If that kind of work is slow, we also do proactive assessments to see if an organization has been or is currently compromised, which will often convert to an investigation.

You can't talk about your job experience but you gotta dish on something... like how much stolen assets are we talking?

Here's a high level overview of some things I've seen stolen:

Code signing certificates and private keys. Perfect for code signing targeted malware.
Credit card data
PII data
Quarterly corporate earnings reports before their public release (insider trading done by an outsider!)
Company financials in general
Business strategies for China
Intellectual property like blueprints, schematics, etc
Information on an organization's business processes
Internal communications discussing negotiating strategy with China. It's funny when a Chinese company gives you upfront the exact number of the lowest bid you would consider.
Source code
Research data
Privileged legal communications

One thing I want to point out on the list above, the only things that are required to be reported by law to the effected people are credit card data an PII data. For everything else, the organization doesn't have to do a public release, and they never do. For things that are not required by law to report, the people who will know about the breach are the investigators, the organization's security team (and even then some companies compartmentalize it to a need to know only basis), and the company's C level execs. No one else will hear about it. You see all kinds of news reports nowadays about company A was breached and lost credit card data or company B was breached and lost PII data, but these reports are just the tip of the ice berg. Of the 30 or so breaches I've worked on involving a targeted threat actor, I can only think of 3 or so that have been public.

In general if you find a breach are you successful tracing it back to the original actor? Is there any incentive to go after, like, foreigners who might be more difficult to prosecute or do people just patch up and move on with life?

This is an intel question that is bit out of my expertise. I know our intel team has tied back certain threat actors to individuals, but often figuring out exactly who the individuals are is outside of what we can do. We often work with law enforcement since they have the power to seize C2 servers or to work with other governments to obtain information. Plus law enforcement has the power to prosecute individuals if they are confirmed and can be extradited/captured. In most situations, all an organization can do is just kick the attacker out, try to increase their security posture, and move on with their lives.

One thing I want to point out that while we might not know the exact individuals behind a breach, we will often see the same threat actors over and over across multiple organizations. This gives us an advantage because certain groups like to do certain things in certain ways. If you are already familiar with what the group does, you can short circuit an investigation to immediately hone in on what the threat actor typically does. For instance, if I were to do an investigation for an organization who is breached by a threat actor that in the past we've seen primarily use the corporate VPN to access the environment, we know that day 1 we need to begin analyzing VPN logs.

# ? Mar 10, 2016 04:06

Optimus_Rhyme: Apr 15, 2007; are you that mainframe hacker guy?

So many questions:

How many breaches did you see that were the cause of insiders? How many vs APT?

When speaking of APT, how A were they compared to what we see at DEFCON/BlackHat?

I'm curious to hear about your thoughts on media reporting, specifically the OPM breach and them blaming it on COBOL.

What are your thoughts on mainframes and why no one is targeting them?

With the influx of BYOD (ugh) how do you see advances in Android/iOS malware threatening the loosening of BYOD policies?

What conferences do you go to, why? which are your favorite?

What podcasts do you listen to?

Kali Linux or roll your own distro?

Whats the stupidest thing you saw a company do? Whats the smartest? You can be vague.

Assuming you're a contractor, on average what was your billable rate vs how much they paid you?

wrt threat actors, are nation states worse than organized mobs?

what are your thoughts on dormant malware/apt which waits and lays dormant until specific conditions are met (i.e. it lands on a ICS, or a specific ad domain) how common are these threats?

Optimus_Rhyme fucked around with this message at 05:43 on Mar 10, 2016

# ? Mar 10, 2016 05:33

Kaislioc: Feb 14, 2008

Redshifted Ghost posted:

I do both, depending on what kind of work is available at the time my schedule is free. We often have organizations come to us knowing they have a problem or maybe they received a law enforcement notification telling them they have a problem. If that kind of work is slow, we also do proactive assessments to see if an organization has been or is currently compromised, which will often convert to an investigation.

Law enforcement notification? Do you know how that usually happens? I can sort of imagine how such a situation might come about, but how does law enforcement end up finding out about a problem before the company does? Do they catch someone and start sending out info to their suspected targets? Do the the FBI send them a letter telling they suspect that China has been naughty in their general vicinity and that they might want to look into that?

# ? Mar 10, 2016 06:15

Redshifted Ghost: Jan 12, 2016

Optimus_Rhyme posted:

So many questions:

How many breaches did you see that were the cause of insiders? How many vs APT?

When speaking of APT, how A where they compared to what we see at DEFCON/BlackHat?

I'm curious to hear about your thoughts on media reporting, specifically the OPM breach and them blaming it on COBOL.

What are your thoughts on mainframes and why no one is targeting them?

With the influx of BYOD (ugh) how do you see advances in Android/iOS malware threatening the loosening of BYOD policies?

What conferences do you go to, why? which are your favorite?

What podcasts do you listen to?

Kali Linux or roll your own distro?

Whats the stupidest thing you saw a company do? Whats the smartest? You can be vague.

Assuming you're a contractor, on average what was your billable rate vs how much they paid you?

wrt threat actors, are nation states worse than organized mobs?

what are your thoughts on dormant malware/apt which waits and lays dormant until specific conditions are met (i.e. it lands on a ICS, or a specific ad domain) how common are these threats?

How many breaches did you see that were the cause of insiders? How many vs APT?

I've never done an insider case, and I've only heard of two instances of a legit insider case being investigated by my company. It's funny, one of the most common questions that I get is "could this be an insider?!" and the answer is almost always no. Sometimes companies believe this because they see an employee's account being used for malicious purposes in the environment and they assume that only their employee would have access to that account. However when a threat actor gets into an environment, one of the first things they will beeline for is credentials, typically domain admin, that will grant them access to the environment at large. They will then leverage those credentials throughout their attack to log into systems, access/steal data, deploy backdoors. One thing that I shudder thinking about is how many companies out there might have been hit by a targeted threat, decided to do the investigation themselves, be terrible at it, and then conclude it must be an insider and shitcan some innocent person. I have no stats or anecdotes to back that up, but I would not be shocked if that is a thing that happens.

When speaking of APT, how A where they compared to what we see at DEFCON/BlackHat?

Not sure what you mean by this question, do you have a specific example?

I'm curious to hear about your thoughts on media reporting, specifically the OPM breach and them blaming it on COBOL.

I can talk about my thoughts but I can't discuss any kind of breach in the news because either I have no knowledge of the breach and would be talking out of my rear end or I cannot talk about it for legal reasons. So, my hate relationship for media reporting. There have been investigations where I've had insider knowledge of a breach that is in the news so I can compare what really happened vs how the media reports it. The media is consistently wrong. Reporters will not be getting insider information from anyone actually involved with investigating the breach because everyone is tight lipped, even with legal reporting requirements (which means only the minimally required information will be disclosed). So what reporters will do is start with whatever press release the breached company has made public and go to other security experts not involved with the investigation to get their opinion. These other security experts won't know what happened because they are not involved with the investigation but they sure do like to speculate, which the reporter will use for their story because hell they have nothing else to go with. The reporter is also probably not technically savvy when it comes to security, so the final write-up will be filled with poorly written speculation. So basically anytime the media discusses details of a breach, take it with a huge grain of salt because it's more than likely incorrect.

With the influx of BYOD (ugh) how do you see advances in Android/iOS malware threatening the loosening of BYOD policies?

From a targeted threat perspective, I don't really see this really changing much. The easiest way for a targeted threat to get into an environment will still be a phishing email which at least one person will fall for. That or popping a vulnerable web server. Those are the top two ways I see companies initially get popped and both are gonna be less effort for a targeted threat than breaching a company via mobile malware.

What conferences do you go to, why? which are your favorite?

I don't do many security conferences to be honest, and when I do I spend more time hanging out with friends than actually seeing talks. The conferences I've been to were Blackhat Vegas, DEFCON, and BSides DC. Of those three, I enjoyed DEFCON the most since there were some really good talks related to blue team topics.

What podcasts do you listen to?

I don't listen to any podcasts. I usually get random articles from the hive mind that is my coworkers.

Kali Linux or roll your own distro?

Nothing fancy, just Windows 7 with a variety of VMs for specific purposes.

Whats the stupidest thing you saw a company do? Whats the smartest? You can be vague.

Stupid things? Here's a few highlights that I can think of right now:

Exposed RDP to the internet
Domain controllers routable and accessible inbound from the internet
Creating AV exclusions by extension out of laziness (why yes lets exclude jar files from AV scans)
Using the same local admin password across the entire environment. This means as soon as a workstation is popped, the threat actor right off the bat has access to all systems in the environment without needing to hit a domain controller (though they will still hit it).
Deciding to remediate early when a compromise is not fully scoped (which means we don't know all the ways the attacker has access, like unknown backdoors), which tips off the threat actor and leads to the them becoming active and further ingraining themselves in the environment. gently caress dealing with an active threat who knows you are trying to kick them out.
Saving passwords in plain text documents on a local system or in documentation on a sharepoint site. Threat actors love hitting sharepoint for this reason.

Smartest? There was one company who had Bit9 + Carbon black rolled out to a large chunk of their environment which helped them shut down a particularly skilled threat group within two days from the initial breach. I'm still impressed with that situation since the threat group usually trounces organizations with little effort.

Assuming you're a contractor, on average what was your billable rate vs how much they paid you?

I'm paid well, but I can't really elaborate any further than that.

wrt threat actors, are nation states worse than organized mobs?

Not sure what you mean by this question. Do you mean nation state skillsets vs financial threat skillsets? If that was your question, it honestly depends on the group. In both categories, there are groups who are really good at what they do. I've also seen groups in both categories that are total amateurs. The sad thing is, it doesn't really matter in the end since the worst threat groups I've seen were still successful in their mission of stealing whatever they wanted. Many organizations do not have the skills, resources, and/or technology to even handle the most fledgling threat group.

what are your thoughts on dormant malware/apt which waits and lays dormant until specific conditions are met (i.e. it lands on a ICS, or a specific ad domain) how common are these threats?

I think you might be confusing APT malware as self propagating. The only situation I can think of that involves the untargeted propagating of APT malware is with a watering hole attack. A watering hole attack is the compromising of some legitimate web site that a target you want to break into might go to. For instance, if you want to get into Widget Company A, you might compromise the non-profit Widget Convention site in the hope that some employees from Widget Company A would visit it. There would be some collateral infections of people not from Widget Company A that the threat actor won't bother with, but once Widget Company A is infected they will actively leverage their backdoors at Widget Company A to dig in and then do their typical attack lifecycle. These kinds of attacks are less common, but they do happen.

If you stumble across a piece of APT malware in your environment, it's not there by chance and it's not a one off dormant piece of malware. If you find a single copy of APT malware in your environment, you almost definitely have a much broader, very serious issue that needs to be looked into. Even if the malware is stupidly old, you should still look into it. I've seen threat actors maintain persistence for long lengths of time. The longest I've seen a threat active (and by active I mean deploying new backdoors, stealing data, dumping passwords) in an environment was about five years.

# ? Mar 10, 2016 07:15

Redshifted Ghost: Jan 12, 2016

Kaislioc posted:

Law enforcement notification? Do you know how that usually happens? I can sort of imagine how such a situation might come about, but how does law enforcement end up finding out about a problem before the company does? Do they catch someone and start sending out info to their suspected targets? Do the the FBI send them a letter telling they suspect that China has been naughty in their general vicinity and that they might want to look into that?

I'm not ex-law enforcement so I don't know the exact details for how they come about, but my understanding is they will seize an APT server, like a C2 server for example. The C2 server will have APT malware checking into it. Law enforcement can just check to see where the malware is checking in from and they have a nice list of owned organizations.

# ? Mar 10, 2016 07:21

Wiggly Wayne DDS: Sep 11, 2010

What countries do you operate in primarily? Do you primarily do forensics after acquisition, or have you been part of the acquisition process? What's your process look like in a hostile environment (I.e. active attackers)? Do you have experience in creating or maintaining forensic logs? Have you been asked to act as an expert witness regarding your IR work, or has your work been used in court proceedings to your knowledge?

# ? Mar 10, 2016 11:42

Shooting Blanks: Jun 6, 2007; Real bullets mess up how cool this thing looks.

-Blade

What is the appropriate response to a threat actor (and how to/not to get them out)? And, if you can discuss it, what would change your response?

Interesting/funny stories you can share (that won't compromise your ID/job) about a specific target or threat?

Can you give a rough estimate of private vs. public targets - i.e. how often are people going after Chase bank as opposed to the DoD?

Without going into specifics, how quickly do threat vectors change? There are always old standbys (phishing, etc.) but how often are new and unexpected threats uncovered? I'm thinking specifically of Heartbleed in this case. On that subject, that was pretty widespread - how serious was it, in your opinion?

# ? Mar 10, 2016 12:04

Lain Iwakura: Aug 5, 2004; The body exists only to verify one's own existence.; Taco Defender

Which anti-virus product do you recommend and why?

# ? Mar 10, 2016 18:13

Serjeant Buzfuz: Dec 5, 2009

Are there any tools that you use that are openly available that you would be willing to talk about?

# ? Mar 10, 2016 18:25

New Zealand can eat me: Aug 29, 2008

What's a Redshifted Ghost? :iiam:

# ? Mar 10, 2016 23:18

adorai: Nov 2, 2002; 10/27/04 Never forget; Grimey Drawer

please describe a few traits of organization that you more or less literally LOL at. Basically, I want to compare your response to my organization and make sure we aren't total scrubs.

# ? Mar 11, 2016 06:26

spankmeister: Jun 15, 2008

adorai posted:

please describe a few traits of organization that you more or less literally LOL at. Basically, I want to compare your response to my organization and make sure we aren't total scrubs.

I hope you're doing better than the one I handled an incident for a while back, whose admin I had to explain that yes, pdf's and other documents can be infection vectors and that indeed you need to keep adobe reader and office and stuff up to date. Yes sir, drive by downloads are also a thing and you need to keep your flash and other plugins up to date. A patch policy with monthly patch cycle and provisions for out of band critical patching is indeed a good practice, probably a good idea to start implementing that. Etc...

Sometimes I still get surprised at the incompetence of some organizations.

# ? Mar 11, 2016 08:20

Gromit: Aug 15, 2000; I am an oppressed White Male, Asian women wont serve me! Save me Campbell Newman!!!!!!!

I've been doing computer forensics for nearly 16 years now for various government agencies, but never touch on IR or vulnerability testing etc so it's interesting to read about a related but different field. Thanks for posting, Ghost.

# ? Mar 12, 2016 14:33

Dazzo: Jun 22, 2006

wrong topic

# ? Mar 12, 2016 17:57

Redshifted Ghost: Jan 12, 2016

What countries do you operate in primarily?

I primary operate in the US but I've done some investigations in other countries before.

Do you primarily do forensics after acquisition, or have you been part of the acquisition process?

My company has a proprietary tool that allows us to on the fly data parsing of sources like the MFT, event logs, registry, etc remotely and return to a centralized server for analysis. We rarely deal with forensic images since that process does not scale if you have say 300+ systems compromised by a threat actor. The only time we do forensic images are for systems with suspected data theft/data staging or major pivot systems used by a threat actor. Whenever we get forensic images, we have the client collect the image via their imaging process. If they don't have a process we provide them instructions. In my four years I've never had to personally collect an image.

What's your process look like in a hostile environment (I.e. active attackers)?

It depends on how far into the attack life cycle the attacker is, and if PCI data is at risk, but one thing stays consistent, scoping is key. What is scoping of a compromise? Scoping is figuring out what systems the attacker accessed and trying to figure out all the ways the attacker has access to the environment (backdoors, VPN access, Citrix access, etc). This is important because kicking the attacker out will not be successful unless the attacker cannot access the environment any longer. If you miss a single backdoor, the attacker will just come in through it and set your remediation and investigation back to square one because at that point, the attacker knows that they have been found and they will take steps to strengthen their foothold in the environment by deploy new back doors that you don't have host or network based indicators for. If you do a premature remediation, you will be playing attacker whack-a-mole.

If you are early in the life cycle and you know for a fact that the attacker got in the day before or a few days ago, you'll want to investigate systems touched by the attacker (where you will likely have security event logs or netflow data to pretty quickly determine this information) as quickly as possible to figure out what was done, primarily looking for backdoors. Once the breach is scoped and you are confident you know all the ways the attacker has access to the environment, you want to remove all ways of access the attacker has in one fell swoop to kick the attacker out. If you catch a breach in the first few days, you can do the investigation and kick them out in a matter of a few days.

If you find a breach and the breach is already a few months to a few years in you will have to play the long game even if the attacker is active. Lets say you find out about a breach by stumbling across suspicious activity on server X. At first, you know only of activity on one system and lord knows how many more the attacker accessed or deployed backdoors to over the period of months to years. If the activity is old, you likely won't have security event logs to figure out where the activity on Server X originated from. So what you have to do is analyze server X to gather host based indicators of compromise. Lets say that during your analysis of Server X, you see the attacker used unnamed scheduled tasks (or "At" jobs created by the Windows binary "at.exe") to execute the files "C:\wmpub\evil.exe" and "C:\wmpub\bad.exe". This is a great start for host based indicators to search the enterprise for, since we know the following:

The attacker likes to use unnamed scheduled tasks. Let's search the environment for unnamed scheduled tasks to see if there are additional attacker created ones out there
The attacker used files named "evil.exe" and "bad.exe". Are there files named "evil.exe" or "bad.exe" sitting on a system anything else in the environment? If yes are they sitting in directories other than "C:\wmpub"?
Is there forensic evidence of execution of files "evil.exe" or "bad.exe" that are no longer present, anywhere in the environment? We can leverage data sources like the Windows Application Compatibility Cache, Windows amcache.hve, Windows prefetch files, Windows UserAssist registry keys, and Windows MUICACHE registry keys for the search for file execution
The attacker used the directory "C:\wmpub" as a working directory. This directory typically has very few files in it and isn't actively used. Are there other systems in the environment that have unusual files in this directory? This might yield new files and filenames that we can leverage for additional searches
Do sources of file execution have records of files in the directory "C:\wmpub" being executed? These kind of searches might help us find new attacker files and filenames that we can use for additional searches

Let's say we do these searches and find 20 more compromised systems. We will want to perform analysis on those 20 systems to harvest new host based and network based indicators which we will then leverage to do additional searches across the environment. We will want to continue this "search -> find compromised systems -> analyze compromised systems -> search with new indicators -> find more compromised systems -> etc" until we are no longer finding anything new. At that point the compromise is as scoped as it will get at which point we can remove all known avenues of access the attacker has. If the attacker is active, you will want a small team tracking the stuff the attacker is actively doing and a team investigating the historic stuff. If the attacker is active, you need to resist the urge to start removing backdoors or resetting the passwords for compromised accounts because "we can't let the attacker just do things in our environment without doing anything" because if you tip off the attacker about the investigation before the appropriate time to remediate, you are shooting yourself in the foot. If you have an attacker that has been in your environment for a couple of months to a couple of years, the investigation will take weeks to months to complete.

Do you have experience in creating or maintaining forensic logs?

I have experience maintaining chain of custody for the uncommon instances where we are handling forensic images, but beyond that no.

Have you been asked to act as an expert witness regarding your IR work, or has your work been used in court proceedings to your knowledge?

I have not been an expert witness and I'm not aware of my work being used in the courtroom. Most of my cases never see a courtroom because they rarely become public for the general public to sue the breached company and the attackers are usually unknown individuals in countries abroad so prosecution is not common.

What is the appropriate response to a threat actor (and how to/not to get them out)? And, if you can discuss it, what would change your response?

I've answered part 1 above, but for your second question, the methodology can change in certain circumstances. If you are doing an investigation into a PCI breach, the organization is required by the card brands to stop the loss of credit cards immediately, so in a matter of days the investigation has to figure out how credit card loss is occurring and then remediate it. The investigation then continues after that to scope the compromise. Thankfully often times when a financial threat actor knows they've been found they cut their losses and move onto their next target since time is money and fighting a blue team can waste a lot of time. The same can't be said about a targeted threat who knows they've been found, they will fight back (not in a destructive manner thankfully) to maintain access.

Interesting/funny stories you can share (that won't compromise your ID/job) about a specific target or threat?

I've seen some interesting things that I can think of off the top of my head:

Attackers using a compromised email account to email IT about the location of the data they want to steal. IT obliged the request
Attackers calling an HR help desk. Full disclaimer, I don't really remember why they did this.
Attackers deploying keyloggers and then keylogging themselves. Why yes thank you for capturing all of your commands for me
Seeing forensic evidence that shows an attacker spending six days fiddling with a backdoor trying to get it to work. By day 7 the backdoor was operational.
Watching attackers attempt to get around application whitelisting is always entertaining. They'll move their binaries to different directories and attempt to execute. When that fails they start renaming their binaries to try to get around the whitelisting. Eventually you can see them give up assuming none of those attempts were successful

Can you give a rough estimate of private vs. public targets - i.e. how often are people going after Chase bank as opposed to the DoD?

I don't have any sort of realistic statistics, but I have the anecdote that my company does more investigation for the private sector than for the public sector.

Without going into specifics, how quickly do threat vectors change? There are always old standbys (phishing, etc.) but how often are new and unexpected threats uncovered? I'm thinking specifically of Heartbleed in this case. On that subject, that was pretty widespread - how serious was it, in your opinion?

I've seen attackers leverage new vectors while they are available until they are patched and then go back to the traditional phishing or exploiting a web server. When you mention heartbleed, I've seen that used on one occasion by a state sponsored targeted threat as the initial compromise vector. It's pretty interesting seeing how the heartbleed vulnerability is logged in the VPN logs. You'll see say Bob's account in the VPN logs originating from some US ISP and then randomly you'll start seeing Bob's account also start coming from say China during the same VPN session without there being a login from china. In the logs you'll see Bob's source IP address flapping between the US ISP and china. What the attacker did was leverage heartbleed to pull back memory chunks from the VPN concentrator until they pulled back the VPN session ID for an account which they then replayed to the VPN concentrator, giving them VPN access without needing the credentials. So yeah, given that I've seen a threat use heartbleed to break into an environment, heartbleed was pretty serious.

Which anti-virus product do you recommend and why?

I'm not a great source on this because I don't really know the efficacy of the various AV products vs. mass malware. I can attest however that all AV is poo poo at stopping a targeted threat. The reason being, targeted threats create malware specific to your environment that they will pre-test with AV products to see if they would be detected before deployment. Even if something does get detected and deleted by AV while the attacker is actively in your environment, unless you have someone watching AV logs (which never happens) who can then spin up an investigation, the attacker will just shrug, change the binary a bit, and then execute it successfully. From the perspective of an investigator who specifically investigates targeted threats, I've found McAfee the easiest to work with during an investigation.

# ? Mar 13, 2016 02:02

B33rChiller: Aug 18, 2011

Interesting thread, thanks for posting it. I've got a couple questions for you
1. You've mentioned how the media often gets their story wrong. What's your opinion on Brian Krebs' reporting.
2. What's your take on phishing drills?
For example, at my workplace, we've been sent several memos explaining what phishing attacks are, and reminded to not click on suspicious links, etc. Now, we've been told that the IT dept will be sending out phishing emails, and if you click on the link, you will have to take remedial security training. I'm not sure this is the best idea, as at least one co-worker has told me he's just going to click on all the links, because it's like the boy who cried wolf kind of thing. "What are they going to do? Pay me to take another course? Oh well." (This particular individual is a loving moron, and for unrelated reasons I'd love to see him sacked)

# ? Mar 13, 2016 23:29

Redshifted Ghost: Jan 12, 2016

Are there any tools that you use that are openly available that you would be willing to talk about?

Yes! There are a shitton of tools out there to do various kinds of data parsing. I'll give a break down of tools I'll typically use if I'm given just a forensics image (because gently caress doing analysis with Encase).

MFT Parsing - I like using mft2csv and then using a combination of grep and excel to work through the data. Another option is TZWorks ntfswalk (License Required)
Event log parsing - Event Log Explorer (License Required), evtwalk (Licensed Required). Additionally if you are doing event log carving from unallocated space, I highly recommend Willi Ballenthin's tools LfLe and EVTXtract
Registry - For a straight registry browser I like WRR. If you want to do timelining of the registry, Access Data Registry Viewer (License Required) and TZWorks yaru (License Required) are both great. If you want to do a quick extract of certain kinds of data of interest, RegRipper is handy.
Registry Application Compatibility Shim Cache (AppCompat Cache) - The AppCompat Cache is a nifty forensic artifact that gives you a historic record of files executed on a system. I'll make a future post going into more detail about the nuances of this source of evidence, but for parsing, Mandiant's ShimCacheParser reigns supreme.
Registry Shell Bags - Another neat source of evidence that can be used to prove directory access by a user which I'll make a future post about. For a parser, Willi Ballenthin's Shellbags.py is great.
If you want to collect live response data vs an image, Mandiant's Redline is a go to tool. Also, if you hate reviewing your event log/registry/file listing data in Redline like I do, you can use audit_parser.py to parse the XML data to csv.
amcache.hve - AppCompat Cache's beefier brother and another topic for another time. When it comes to parsing, Willi Ballenthin's amcache.py is my go to.

What's a Redshifted Ghost?

http://scienceblogs.com/startswithabang/2009/11/20/falling-into-a-black-hole-suck/

please describe a few traits of organization that you more or less literally LOL at. Basically, I want to compare your response to my organization and make sure we aren't total scrubs.

I could go on and on for this subject, but I'll go over some of the more common recommendations I give clients.

Use two factor authentication for all remote access with either a hard token or a phone token (desktop tokens are poo poo don't use them because attackers will steal them). This includes VPN, Citrix, OWA, everything. If you aren't using two factor for all remote access, attacker will take advantage of this and access the environment via VPN/Citrix and read emails via OWA. There's actually some threat actors who prefer using the VPN and is one of the first things they target. An attacker on your VPN/Citrix/OWA sucks because their activity will blend in more with legitimate activity vs if they were accessing the environment via backdoors. Implementing two factor makes it more difficult for attackers to leverage legitimate remote access methods. Speaking of two factor, here's my next point:
Put your two factor server behind a jump box that requires said two factor and don't attach the two factor server to the domain. Why is this? There are attackers that will specifically target two factor infrastructure to either find accounts in PIN only mode or to steal the respective two factor database so they can replicate it on their end to start using accounts that require tokens. I want to emphasize, I've seen on many occasions of servers that support two factor auth getting compromised specifically to use remote access methods.
Don't have you DMZ on the same domain as your corporate domain. Doing so defeats the purpose of having a DMZ because AD requires the SMB port 445 open to function. The attacker can laterally move over 445 via PsExec and scheduled tasks. So you might have some locked down ACLs between the DMZ and CORP, but that's not gonna stop an attacker from popping a web server in the DMZ, and then going from DMZ -> Corp Domain controller -> Corp environment.
Please do network segmentation and a liberal use of jump servers. You don't need workstation to workstation communications. Force your desktop admins to use a locked down jump server that requires two factor in order to access the workstation environment. Ensure that the workstation IP space does not have free access to everything in the environment. Karen for HR should not be able to RDP to an engineering server from her workstation.
Don't give service accounts domain admin privileges. It's rarely necessary. Domain admin service accounts are top targets by attackers. Why? These account rarely have password expirations which means they can leverage the same credentials for a really really long time.
Liberally use application whitelisting. At a minimum have app whitelisting on domain controllers, exchange servers, file servers, Citrix servers, and any other locations that might contain a vast amount of user credentials. Application whitelisting will make credential harvesting more difficult for an attacker (but does not make it impossible). In an ideal world, if every system in an environment had app whitelisting, you will make life for a threat actor very difficult since it makes it way harder to harvest credentials and to deploy backdoors. Also, have someone monitoring your logs for app whitelisting because if you have a block alert for "pwdump.exe" on a domain controller, you have a serious problem that needs to be addressed.

Honestly, if you do all of the above and have a competent security team, you will be in much better shape than most organizations I've seen.

1. You've mentioned how the media often gets their story wrong. What's your opinion on Brian Krebs' reporting.

I know of two breaches where Brian Krebs' reported inaccurate information. While Brian Krebs is more technical than most of the media and he writes interesting articles (don't get me wrong, I like reading his stuff), I would take breach reporting from him with a grain of salt.

2. What's your take on phishing drills?
For example, at my workplace, we've been sent several memos explaining what phishing attacks are, and reminded to not click on suspicious links, etc. Now, we've been told that the IT dept will be sending out phishing emails, and if you click on the link, you will have to take remedial security training. I'm not sure this is the best idea, as at least one co-worker has told me he's just going to click on all the links, because it's like the boy who cried wolf kind of thing. "What are they going to do? Pay me to take another course? Oh well." (This particular individual is a loving moron, and for unrelated reasons I'd love to see him sacked)

I think phishing drills are great because it gets people thinking about emails they receive vs blindly trusting them. You might still have assholes like your coworker who fucks it up for everyone, but people like that will be a fuckup when it comes to security regardless. You can tell your coworker that if he falls for a phish by a targeted threat, his system is patient 0, and an investigation finds that his system is patient 0, the CEO and other C level execs will know that he hosed up. Breaches get visibility all the way to the top of the food chain and the number 1 question asked is "how did this happen?". I'm sure he doesn't want his CEO knowing him by name as the guy who hosed up.

# ? Mar 18, 2016 03:02

Gromit: Aug 15, 2000; I am an oppressed White Male, Asian women wont serve me! Save me Campbell Newman!!!!!!!

Redshifted Ghost posted:

[*]Registry - For a straight registry browser I like WRR. If you want to do timelining of the registry, Access Data Registry Viewer (License Required) and TZWorks yaru (License Required) are both great. If you want to do a quick extract of certain kinds of data of interest, RegRipper is handy.

I used to work with the guy who wrote the free Registry Browser and it worked pretty well when I needed it. With the stuff I do these days, though, I don't really need to pull the Registry apart.

# ? Mar 18, 2016 09:13

I would blow Dane Cook: Dec 26, 2008

How much hosed up pron do you see?

# ? Mar 19, 2016 11:20

Adbot: ADBOT LOVES YOU

# ? Apr 16, 2024 19:24

Gromit: Aug 15, 2000; I am an oppressed White Male, Asian women wont serve me! Save me Campbell Newman!!!!!!!

Me? Not a lot these days. But back when I worked for the police it would be fair to say that 80% of our work was child abuse related. On a relative "hosed up" scale, most of it was your run-of-the-mill CP - nudes and posing. But there's always a bunch of truly depraved material right around the corner.

The most I ever saw on one computer was maybe quarter of a million photos. I had a colleague who examined a computer with 2 or 3 times that amount. Back then we had to look at all of it to make sure it was correctly identified, but the processes have changed now so you don't have to go to those lengths.

So yeah, you see some crazy stuff. It's almost a relief to come across your common or garden scat and vomit porn.

Gromit fucked around with this message at 11:32 on Mar 19, 2016

# ? Mar 19, 2016 11:28

The Something Awful Forums > Discussion > Ask/Tell > Ask me about Enterprise Security, Incident Response (IR), and APTs