Troubleshooting security

Why this matters

Security failures have a distinctive quality: the error message is almost never about security. SELinux denies a read and the application reports a missing file. An expired certificate produces "connection failed". A cipher mismatch says "handshake failure". The symptom points somewhere else entirely, which is why people spend an hour on the wrong layer.

There is also a specific temptation this objective exists to inoculate against. Every problem in this lesson can be "solved" by turning the protection off — setenforce 0, chmod 777, PermitRootLogin yes, --insecure. Each of those makes the symptom disappear and leaves the machine worse than it was. The difference between an administrator and someone who makes errors go away is almost entirely in this lesson.

The lesson

SELinux issues

The signature: a permission error that the file permissions do not explain. The user owns the file, the mode is right, and the read still fails.

getenforce                        # Enforcing, Permissive, or Disabled
sestatus
ausearch -m avc -ts recent        # the denials, which is the actual answer
ausearch -m avc -ts today | audit2why
journalctl -t setroubleshoot

ausearch -m avc is the command. An AVC denial names the source context, the target context and the operation, and audit2why explains it in English.

The confirming test — and this is a diagnostic, not a fix:

setenforce 0                      # temporarily permissive
# retry the operation
setenforce 1                      # PUT IT BACK, now

If it works in permissive mode, it is SELinux. Then go and fix the label, because a machine left at setenforce 0 has lost the protection and nobody will notice until it matters. Worse is SELINUX=disabled in /etc/selinux/config, which makes it permanent and requires a full filesystem relabel to undo.

Context is the label on files and processes, and mislabelling is the most common cause:

ls -Z /var/www/html/index.html    # the file's context
ps -eZ | grep nginx               # the process's context
restorecon -Rv /var/www/html      # reset to the POLICY's expectation
semanage fcontext -a -t httpd_sys_content_t "/srv/web(/.*)?"
restorecon -Rv /srv/web
matchpathcon /srv/web/index.html  # what SHOULD this be labelled?

The classic case: a file moved with mv keeps its original context, while a file copied with cp inherits the destination's. So moving a page from your home directory into /var/www/html gives a 403 that the permissions do not explain — and restorecon fixes it in one command.

For a non-standard location, restorecon alone is not enough: the policy has no rule for /srv/web, so you add one with semanage fcontext and then relabel. restorecon without that just reapplies the default, which is the wrong label.

Booleans are policy switches — on/off settings that permit whole classes of behaviour:

getsebool -a | grep httpd
getsebool httpd_can_network_connect
setsebool -P httpd_can_network_connect on     # -P = persistent
semanage boolean -l | grep httpd

Forgetting -P is the classic mistake: the setting works and reverts at reboot, producing a fault that appears months later with no apparent cause.

httpd_can_network_connect is the one to remember — a web server acting as a reverse proxy cannot make outbound connections until it is on, and the error is an opaque 502.

Ports are labelled too, so a service on a non-standard port is denied until the port is labelled:

semanage port -l | grep http_port_t
semanage port -a -t http_port_t -p tcp 8443

The policy itself is the whole rule set — targeted (the default, confining specific services) or mls. Custom policy modules can be built from denials:

ausearch -m avc -ts recent | audit2allow -M myapp
semodule -i myapp.pp

Use audit2allow with care. It generates a rule permitting exactly what was denied, which is fine when the denial is legitimate and terrible when the denial was catching something that should not be happening. Read what it generates before installing it — it will happily write you a policy allowing a web server to read /etc/shadow.

The other family, on Debian and Ubuntu, is AppArmor — path-based rather than label-based:

aa-status
aa-complain /etc/apparmor.d/usr.sbin.nginx
journalctl | grep -i apparmor

Same class of symptom, different tooling.

File and directory permission issues

ls -l file                        # mode, owner, group
ls -ld /var/www /var/www/html     # the DIRECTORIES on the path matter too
namei -l /var/www/html/index.html # every component, with its permissions
stat file
sudo -u nginx cat /var/www/html/index.html    # test AS the failing user
id nginx

namei -l is the underused one. Permission denied on a file whose mode is 644 is nearly always a directory on the path lacking execute for that user — and namei -l shows every component at once, which is faster than walking up by hand.

The rules that produce most of these failures:

• A directory needs x to be traversed and r to be listed. x without r means you can open a known filename and cannot list the directory — which is a deliberate and useful pattern, and confusing when unintentional.
• Write permission on the directory, not the file, controls deletion. You can delete a file you cannot write, if you can write its directory. That surprises people every time.
• The sticky bit on /tmp (drwxrwxrwt) exists for exactly that: everyone can write, only the owner can delete their own files.
• SUID/SGID — find / -perm -4000 lists SUID binaries, and an unexpected one is a genuine finding, not a curiosity.

Testing as the failing user is the fastest diagnosis. sudo -u nginx and retry: it either reproduces, which confirms permissions, or it works, which sends you to SELinux.

And the temptation to name explicitly: chmod 777 is not a fix. It makes the symptom disappear by making the file world-writable, which is a different and worse problem. Find the user, find the group, set the mode it should have.

ACLs

When the owner/group/other model is not enough:

getfacl /srv/data
setfacl -m u:alice:rw /srv/data/report.txt
setfacl -m g:developers:rx /srv/data
setfacl -d -m g:developers:rwx /srv/data      # DEFAULT: inherited by new files
setfacl -x u:alice /srv/data/report.txt
setfacl -b /srv/data                          # remove all ACLs

The tells that an ACL is involved:

A + at the end of the mode string — -rw-r--r--+ — means the file has an ACL, and ls -l alone is now misleading. This is the single most useful thing to know here: if the permissions do not explain the behaviour, look for the plus.

The mask limits everything. getfacl shows a mask:: line, and it caps the effective permissions of every named user and group entry. An ACL granting rw under a mask of r-- yields read only, and getfacl marks it #effective:r--. Running chmod on a file with ACLs changes the mask, which silently reduces the ACL — a genuinely surprising interaction.

Default ACLs apply to newly created files in a directory, and only to new ones. Setting a default does not fix the files already there; you need a recursive setfacl -R as well.

ACLs also need filesystem support — mount with acl where it is not the default.

Attributes

A layer below permissions, and invisible to ls -l:

lsattr file
chattr +i /etc/resolv.conf        # immutable: cannot be modified, deleted or renamed
chattr -i /etc/resolv.conf
chattr +a /var/log/audit.log      # append only

The symptom is unmistakable once you have seen it: root cannot modify the file. "Operation not permitted" as root, with correct permissions, no SELinux denial, and a read-write filesystem. That is the immutable attribute, and lsattr is the only thing that reveals it.

Legitimate uses exist — pinning /etc/resolv.conf so NetworkManager cannot regenerate it, making an audit log append-only. It is also used by attackers to protect a backdoor, so an unexpected i on a system file is worth investigating rather than simply clearing.

Account access

id alice
getent passwd alice               # exists at all? (includes LDAP/SSSD)
passwd -S alice                   # password status: P (set), L (locked), NP (none)
chage -l alice                    # expiry dates
grep alice /etc/shadow            # a leading ! or * = locked
lastb | head                      # failed logins
faillock --user alice             # lockout state
journalctl -u sshd | grep alice

The causes, distinguished by what they look like:

• Password expired — the user is prompted to change it, or refused if the inactive period has passed. chage -l shows both dates.
• Account expired — chage -E. A different field, and a common confusion. Expiry has been set, not the password's.
• Locked — passwd -l or usermod -L prefixes the hash with !. Unlock with passwd -u. Note that locking the password does not block SSH key authentication — a locked account with an authorised key still logs in, which is a real gap. usermod --expiredate 1 or setting the shell to /sbin/nologin is what actually stops access.
• Shell is /sbin/nologin or /bin/false — authentication succeeds and the session ends immediately. Correct for service accounts, and confusing when unintended.
• faillock lockout after repeated failures. faillock --user alice --reset.
• Home directory missing or wrong ownership — the login succeeds and lands in / with errors.
• SSH-specific: AllowUsers/DenyUsers/AllowGroups in sshd_config, or permissions on ~/.ssh. sshd refuses keys if ~/.ssh is not 700 or authorized_keys is not 600, or if the home directory is group-writable. It logs the reason and the client is told only "permission denied", which is the single most common SSH key failure.

sshd -T | grep -i "allowusers\|permitrootlogin\|pubkeyauthentication"
ssh -vvv user@host                 # the client's view of what was tried
journalctl -u sshd -f              # the server's view -- where the reason is

Read the server's log, not the client's message. The client is told nothing useful on purpose.

Remote access issues

ssh -vvv user@host                 # verbose: see exactly where it stops
ssh -o PreferredAuthentications=publickey user@host
ssh-add -l                         # what keys does the agent hold?
ssh-keygen -lf ~/.ssh/id_ed25519.pub
sshd -t                            # validate config BEFORE reloading

The recurring ones:

• Key permissions, above. chmod 700 ~/.ssh; chmod 600 ~/.ssh/authorized_keys.
• Wrong key offered. The agent presents keys in order and the server may cut you off after too many attempts. -o IdentitiesOnly=yes -i key.
• Host key changed — the loud warning about a possible man-in-the-middle. Usually a rebuilt server, occasionally not. Verify the new fingerprint out of band before removing the old entry with ssh-keygen -R host; blindly clearing it is how you accept a real attack.
• PermitRootLogin or PasswordAuthentication no — the policy is working as configured.
• Locked out by your own change. Always run sshd -t before systemctl reload sshd, and keep the current session open while you test a new one. A syntax error in sshd_config plus a restart plus a closed session equals a console trip.

Certificate issues

openssl x509 -in cert.pem -noout -text
openssl x509 -in cert.pem -noout -dates -subject -issuer
openssl s_client -connect example.com:443 -servername example.com </dev/null
echo | openssl s_client -connect example.com:443 2>/dev/null | openssl x509 -noout -enddate
openssl verify -CAfile chain.pem cert.pem
openssl x509 -noout -modulus -in cert.pem | openssl md5     # must match the key's
openssl rsa -noout -modulus -in key.pem | openssl md5

The distinct failures, each with a different fix:

Expired. The commonest outage in this list, and entirely preventable — monitor expiry at 30 and 7 days, and automate renewal. certbot renew in a timer, with a reload hook.

Hostname mismatch. The certificate is for www.example.com and you connected to example.com. Check the Subject Alternative Name list; modern clients ignore the Common Name entirely, so a certificate with only a CN fails everywhere even though openssl shows the right name.

Incomplete chain. This one is nasty because it works in a browser and fails from curl and from other servers. Browsers cache intermediates from previous sites; nothing else does. The server must send the full chain, and openssl s_client showing "unable to get local issuer certificate" is the diagnosis.

Self-signed or unknown CA. Add the CA to the trust store — /etc/pki/ca-trust/source/anchors/ then update-ca-trust on RHEL, /usr/local/share/ca-certificates/ then update-ca-certificates on Debian. Not curl --insecure, which disables the check that would tell you about a real attack.

Certificate and key do not match — the two modulus hashes above differ. The service refuses to start, usually with an unhelpful message.

Clock skew. A certificate valid from tomorrow, on a machine whose clock is wrong, is "not yet valid". Check timedatectl — this is the fourth or fifth time in the course that a wrong clock has produced a failure that looks like something else.

Unpatched vulnerable systems

dnf check-update
dnf updateinfo list security
dnf update --security
apt list --upgradable
unattended-upgrade --dry-run

rpm -qa --last | head              # what was installed, and when
needs-restarting -r                # does a reboot LIVE-affect anything?

The point most often missed: installing a patch does not apply it. A library update does nothing for a process still running with the old version mapped, and a kernel update does nothing until reboot. needs-restarting -r says whether a reboot is required and needs-restarting -s lists the services to restart — a patched-but-not-restarted machine is still vulnerable while reporting as patched.

Where patching is blocked — an application certified against an old library, a machine that cannot take downtime — the honest answer is compensating controls and a recorded exception with a date, not silence.

Exposed or misconfigured services

ss -tlnp                          # everything LISTENING, and on what address
ss -tlnp | grep -v '127.0.0.1\|::1'    # what is exposed beyond loopback
nmap -sV localhost                # from outside, what does it look like?
firewall-cmd --list-all
systemctl list-units --type=service --state=running

ss -tlnp is the audit. Anything bound to 0.0.0.0 that does not need to be is exposure — a database, an admin interface, a metrics endpoint, a development server someone left running.

The specific things to look for: default credentials, an admin interface reachable from the internet, a database listening on all interfaces, directory listing enabled, verbose errors leaking paths and versions, and debug endpoints in production.

The fix is usually one line of configuration — bind to 127.0.0.1 and put a reverse proxy in front, or restrict by firewall. ss before and after confirms it.

Misconfigured package repository

dnf repolist all
dnf config-manager --set-disabled unstable-repo
cat /etc/yum.repos.d/*.repo
rpm -q gpg-pubkey --qf '%{summary}\n'

apt policy
cat /etc/apt/sources.list /etc/apt/sources.list.d/*
apt-key list                      # deprecated; see /etc/apt/keyrings/

The failures:

• gpgcheck=0. Packages are installed without signature verification, which removes the only protection against a compromised mirror. It is frequently set to silence an error, and it is the wrong fix — import the correct key instead.
• A missing or wrong GPG key, giving "NOKEY" or a signature error.
• Mixed major versions — an EL8 repository on an EL9 machine — which produces dependency chaos that looks like a broken package.
• Third-party repositories displacing distribution packages, so a security update from the vendor never arrives because a third party ships a higher version number. dnf repoquery --installed --qf '%{name} %{from_repo}\n' shows where each installed package actually came from, which is worth checking on a machine with surprising behaviour.
• HTTP rather than HTTPS for the repository. Signatures still protect integrity, but the metadata and your package list are visible in transit.

Obsolete protocols and ciphers

nmap --script ssl-enum-ciphers -p 443 example.com
openssl s_client -connect example.com:443 -tls1_1     # should FAIL
testssl.sh https://example.com
sshd -T | grep -i "ciphers\|macs\|kexalgorithms"
update-crypto-policies --show                          # RHEL: DEFAULT, FUTURE, LEGACY

Use of obsolete or insecure protocols and ciphers is the finding; the list of what is obsolete is worth knowing:

• SSLv2, SSLv3, TLS 1.0, TLS 1.1 — all deprecated. TLS 1.2 minimum, 1.3 preferred.
• RC4, DES, 3DES, MD5, SHA-1, and any cipher suite marked EXPORT or NULL.
• SSHv1, and ssh-rsa with SHA-1 signatures — which OpenSSH 8.8 disabled by default, breaking connections to old devices and generating a support ticket that looks like a bug.
• Telnet, FTP, rsh, rlogin — plaintext credentials on the wire. Replace with SSH, SFTP, HTTPS.

RHEL's crypto policies set this system-wide in one place, which is much better than editing every service:

update-crypto-policies --set DEFAULT:NO-SHA1
update-crypto-policies --set FUTURE          # strict; will break old clients

Cipher negotiation issues

The failure mode of the above: client and server share no acceptable cipher suite, and the connection fails at the handshake.

error:0A000102:SSL routines::unsupported protocol
no matching key exchange method found
sshd: Unable to negotiate with 10.0.0.5: no matching cipher found

openssl s_client -connect host:443 -cipher 'ECDHE-RSA-AES256-GCM-SHA384'
ssh -vv host                                  # shows the offered and accepted lists
ssh -o KexAlgorithms=+diffie-hellman-group1-sha1 host    # temporary, for old kit
nmap --script ssl-enum-ciphers -p 443 host    # what the server ACTUALLY accepts

The failure is usually a success. A modern server hardened to TLS 1.3 only, or a FUTURE crypto policy, correctly refuses an old embedded device, an ancient Java client, or a legacy appliance. Nothing is broken; the two ends genuinely disagree about what is acceptable.

So the decision is not technical but a risk judgement: upgrade the client, isolate it on a network segment where weak crypto is contained, or — knowingly, temporarily, and documented — re-enable a specific algorithm for that one service. Re-enabling weak crypto system-wide to fix one legacy device is the wrong trade, and it is exactly the shape of question the exam asks.

ssh -vv is the practical tool here: it prints what each side offered and where they failed to intersect, which turns "no matching cipher" into a specific missing algorithm.

On the exam

• The error is rarely about security. A permission denied that permissions do not explain is SELinux; root unable to write is chattr +i.
• ausearch -m avc finds denials; restorecon fixes labels; a moved file keeps its old context where a copied one inherits. setsebool -P — without -P it reverts at reboot. setenforce 0 is a diagnostic, never a fix.
• namei -l shows every path component; a directory needs x to be traversed, and directory write permission controls deletion.
• A + in ls -l means an ACL. The mask caps every named entry, and chmod changes the mask.
• lsattr reveals immutable (+i) — the reason root cannot modify a file.
• Account access: password expiry and account expiry are different fields; a locked password does not block SSH keys; ~/.ssh must be 700 and authorized_keys 600. Read the server log, not the client message.
• Certificates: check expiry, SAN not CN, full chain, key match, and the clock. An incomplete chain works in a browser and fails everywhere else. Never --insecure.
• Installing a patch is not applying it — needs-restarting -r.
• ss -tlnp is the exposure audit; bind to 127.0.0.1 and proxy.
• gpgcheck=0 is not a fix for a signature error — import the key.
• Obsolete: SSLv3, TLS 1.0/1.1, RC4, 3DES, MD5, SHA-1, telnet, FTP.
• A cipher negotiation failure is usually correct behaviour. Fix the old client or isolate it; do not weaken system-wide crypto for one device.