Recognising a security breach on the network

Why this matters

Not every fault is an accident. A server that is suddenly slow, a service that keeps stopping, or network traffic at three in the morning may be the sign of an attacker, not a failing disk. Server administrators are often the first to see these signs, because they notice when something about their servers is not normal, long before any report reaches the security team.

What happens in the first hour after a breach is suspected shapes everything after it. Acting too slowly lets an attacker spread; acting carelessly destroys the evidence needed to understand what happened. This lesson covers recognising the signs of compromise, isolating a server, preserving evidence, working out how far a breach reaches, and handing it on through the incident response plan introduced in the SIEM lesson.

The lesson

Signs of compromise: unusual traffic, new accounts, disabled security tools

Evidence that a system has been compromised is called an indicator of compromise (IoC). Some of the most common on servers:

Unusual network traffic.

• connections to unfamiliar addresses or countries, especially from servers that have no reason to talk to the internet;
• large outbound transfers, particularly at unusual hours, which may be data being stolen, called exfiltration;
• regular, small connections at fixed intervals to one external address, typical of malware checking in with a command and control server;
• traffic on unexpected ports, or a server scanning other machines on the network;
• unusual volumes of DNS queries, sometimes used to smuggle data out.

Unexpected account activity.

• new accounts that nobody created, especially administrative ones;
• existing accounts added to privileged groups;
• logons at unusual times, from unusual places, or to servers the account never normally uses;
• many failed logons followed by a success;
• service accounts logging on interactively.

Changes to the system itself.

• security tools disabled: antivirus or EDR stopped, firewall rules loosened, or logging turned off, since attackers try to blind defenders;
• logs cleared, which on Windows itself generates an event recording that the log was cleared;
• new services, scheduled tasks or startup entries that nobody installed;
• unfamiliar processes, or known programs running from unusual locations;
• files encrypted or renamed en masse, the signature of ransomware;
• unexplained performance problems, such as high processor use from cryptocurrency mining software.

No single sign proves a breach, and many have innocent explanations. But several together, or any sign on a critical server, should be treated as a possible incident until shown otherwise.

Isolating a compromised server

Once a compromise is suspected, the priority is containment: stopping the attacker from spreading to other systems, continuing to steal data, or doing further damage.

The usual approach is to isolate the server from the network while leaving it running:

• use EDR's network isolation feature, which cuts the server off from everything except the security tools managing it;
• or move its switch port to an isolated quarantine VLAN, or disable the port;
• or, for a virtual machine, disconnect its virtual network adapter;
• or apply firewall rules that block its traffic.

Why not just switch it off? Powering down destroys the contents of memory: running processes, network connections, encryption keys and malware that exists only in memory. That evidence may be essential to understanding the attack. Some malware also reacts to shutdown, and some ransomware leaves files more recoverable while the system is still running.

There are exceptions. If the server is actively encrypting or destroying data, stopping the damage may matter more than preserving memory, and pulling the network or power is justified. These decisions belong to the incident response plan and the security team, which is why the first call is to them.

Containment also covers accounts: disable or reset credentials that may have been compromised, remembering that an attacker with a domain administrator's password is not contained by isolating one server.

Preserving evidence before rebuilding

The instinct after a compromise is to wipe the server and rebuild it. That is usually the right end point, but done first it destroys the evidence needed to answer vital questions: how did the attacker get in, what did they take, and are they still in other systems?

Evidence is collected in order of volatility, most short-lived first:

1. memory contents, captured with dedicated memory acquisition tools;
2. running processes, network connections and logged-on users;
3. the disks, captured as a forensic image, a bit-for-bit copy, usually taken with a write blocker or from a snapshot so the original is not altered;
4. logs, from the server and from central collectors;
5. backups and archived data.

For virtual machines, a snapshot that includes memory captures much of this quickly.

Evidence must be handled so it can be trusted, and possibly used in legal proceedings:

• record hashes of images when they are taken, so it can be shown later that they have not changed;
• keep a chain of custody, the same principle as in the media sanitisation lesson, recording who handled the evidence and when;
• work on copies, never the original evidence;
• record every action, with times, including actions taken on the live server.

Many organisations bring in specialist forensic investigators for serious incidents. The administrator's job is usually to avoid destroying evidence before they arrive, and to help them collect it.

Scoping the breach from logs and the SIEM

A compromised server is rarely the whole story. Scoping works out how far the breach reaches: when it started, how the attacker got in, which systems and accounts they used, and what data they accessed.

The SIEM and central logs are the main tools, which is why the SIEM lesson insisted on collecting logs centrally, keeping them long enough and synchronising clocks:

• search for the indicators found on the first server, such as addresses, file names, file hashes and account names, across every system;
• trace the accounts involved: where else did they log on, and when?
• look for lateral movement, connections from the compromised server to other servers, especially using administrative protocols such as RDP, SSH, or remote management;
• establish a timeline, working back to the first sign of the attacker and forward to the present;
• check firewall and proxy logs for exfiltration, and estimate what data may have left.

Scope determines everything that follows: which systems must be cleaned or rebuilt, which passwords reset, and whether the organisation has legal duties to notify regulators or the people whose data was affected. Underestimating scope is the common failure, since cleaning one server while the attacker remains on another only starts the incident again.

Escalating through the incident response plan

A suspected breach is not something an administrator should handle alone. The incident response plan, described in the SIEM lesson, sets out who is told, who decides, and what happens next, and following it is the most important single action.

When escalating:

• report immediately to the security team or incident contact, through the channel the plan names, with what was seen, where and when;
• consider that the attacker may be watching email or chat on a compromised network, and use out-of-band communication, such as phone, where the plan says so;
• do not investigate alone or tip off the attacker, for example by deleting their account before the scope is known, which may prompt them to use other access they already hold;
• follow instructions from the incident lead on containment and evidence;
• record everything you did and observed, with times.

The plan also covers what happens beyond IT: management decisions, legal advice, notification of regulators and affected people within legal deadlines, and communication with customers. Recovery follows, restoring systems from backups known to predate the compromise, which the backup lessons in this domain cover. Afterwards, the lessons learned review turns the incident into better defences.