Regular Expressions & Log Parsing

Master Python's re module to write patterns that extract IPs, timestamps, usernames, and anomalies from real-world log files.

Medium 70m 3 tasks
Prerequisites: File I/O & JSON

Learning Objectives

  • → Write and compile regex patterns using the re module
  • → Use re.search, re.match, re.findall, and re.sub
  • → Extract named capture groups from log lines
  • → Parse Apache, nginx, and auth.log format entries
  • → Aggregate findings into structured reports

Logs are where attacks leave fingerprints. A brute-force attempt is a burst of Failed password lines in auth.log. A directory scan is hundreds of 404s from one IP in an nginx access log. Data exfiltration shows up as unusually large responses. A SIEM finds these patterns at scale, but every analyst still ends up writing a quick script to answer a question the dashboard can't: "which IPs failed SSH login more than 20 times last night, and did any of them later succeed?"

Regular expressions are the tool for that job. Log lines are semi-structured text: the same layout repeated millions of times with different values. A regex describes the layout once and pulls out the fields you care about. This lesson covers Python's re module from the basics to a working anomaly detector, plus the mistakes that make log parsers silently miss events.

Before you start: you should be comfortable reading files line by line and using dictionaries, as covered in File I/O & JSON and Data Structures.

The re Module

Python's regex engine lives in the standard library. There is nothing to install:

import re

Five functions cover almost every log-parsing task:

Function Description
re.search() Find first match anywhere in string
re.match() Match only at start of string
re.findall() Return all non-overlapping matches
re.sub() Replace matches with a string
re.compile() Compile a pattern for repeated use

Always write patterns as raw strings (r'...'). In a normal string, Python interprets backslashes before the regex engine sees them: '\b' becomes a backspace character instead of a word boundary, and your pattern silently stops matching. The r prefix passes the backslashes through untouched.

Basic Patterns

Most security indicators have a recognisable shape. These three patterns come up constantly:

# IP address
ip_pat = re.compile(r'\b(?:\d{1,3}\.){3}\d{1,3}\b')

# Email
email_pat = re.compile(r'[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}')

# URL
url_pat = re.compile(r'https?://[^\s"]+')

Two details are worth noticing:

  • \b is a word boundary. It stops the IP pattern from matching the middle of a longer number such as a version string 1.2.3.4.5.
  • (?:...) is a non-capturing group. It groups \d{1,3}\. so {3} repeats it, without creating a capture. This matters with findall(): if a pattern contains a capturing group (...), findall() returns only the group's text instead of the full match, so you'd get '1.' instead of '192.168.1.1'.

These patterns check shape, not validity. \d{1,3} happily accepts 999.999.999.999. For log triage that's usually fine, because the log wrote a real address. When validity matters, extract with the regex, then confirm with ipaddress.ip_address(), which raises ValueError on anything that isn't a real IPv4 or IPv6 address.

search and match

re.search() scans the whole line for the first match. re.match() only succeeds if the pattern matches at the very start of the string. Confusing the two is a classic bug: a parser using match() on syslog lines finds nothing, because each line starts with a timestamp, not the text you're looking for.

line = "Failed password for root from 192.168.1.50 port 22 ssh2"

m = re.search(r'from (\d+\.\d+\.\d+\.\d+)', line)
if m:
    print(m.group(1))   # 192.168.1.50
    print(m.group(0))   # full match: 'from 192.168.1.50'

Both functions return a match object on success and None on failure. Always test the result before calling .group(), otherwise the first line that doesn't match raises AttributeError and the whole script stops halfway through the file.

findall

When a line can contain several indicators, findall() returns every match as a list:

log = "10.0.0.1 accessed /api, 10.0.0.2 accessed /login, 10.0.0.1 tried /admin"
ips = re.findall(r'\d{1,3}(?:\.\d{1,3}){3}', log)
print(ips)   # ['10.0.0.1', '10.0.0.2', '10.0.0.1']
print(set(ips))  # unique

Converting to a set gives the unique IPs. When you need how often each one appears, which is usually the question in an investigation, use collections.Counter(ips) instead.

Named Groups

Numbered groups (group(1), group(2)…) become unreadable once a pattern has five or six fields, and they break when you add a group in the middle. Named groups, written (?P<name>...), label each field. groupdict() then turns a log line straight into a dictionary:

pattern = re.compile(
    r'(?P<ip>\d+\.\d+\.\d+\.\d+) - - '
    r'\[(?P<time>[^\]]+)\] '
    r'"(?P<method>\w+) (?P<path>[^"]+) HTTP/[^"]+" '
    r'(?P<status>\d+)'
)

line = '192.168.1.1 - - [10/Jan/2024:10:23:45 +0000] "GET /admin HTTP/1.1" 403'
m = pattern.search(line)
if m:
    print(m.group("ip"))      # 192.168.1.1
    print(m.group("status"))  # 403
    print(m.groupdict())

The pattern is split across several raw strings. Python joins adjacent string literals, so you can comment and align each field without changing the regex. Note the use of [^\]]+ ("anything except a closing bracket") rather than .+. That keeps the match inside the brackets and avoids the greedy-matching problem described below.

re.sub — Redaction

Logs often contain data that shouldn't leave the system, such as card numbers, tokens or email addresses. Before you share a log extract in a ticket or paste it into an external tool, redact it with re.sub():

# Redact credit card numbers
text = "Card: 4111 1111 1111 1111 charged"
redacted = re.sub(r'\b\d{4}[\s-]?\d{4}[\s-]?\d{4}[\s-]?\d{4}\b', '[REDACTED]', text)
print(redacted)   # Card: [REDACTED] charged

[\s-]? allows an optional space or dash between groups, so the same pattern catches 4111111111111111, 4111 1111 1111 1111 and 4111-1111-1111-1111. Redaction patterns should err on the side of matching too much. A false positive costs you a few characters of context; a false negative leaks a card number.

Parsing auth.log

On Debian and Ubuntu, SSH authentication events go to /var/log/auth.log (on RHEL-family systems, /var/log/secure). Each failed login produces a line like:

Jan 10 03:14:07 web01 sshd[2214]: Failed password for invalid user admin from 203.0.113.45 port 51122 ssh2

The function below counts failures per source IP and per targeted username. Those are the two questions you ask first about a brute-force attempt: who is attacking, and which accounts are they guessing.

import re
from collections import Counter

def parse_ssh_failures(path):
    pat = re.compile(r'Failed password for (?:invalid user )?(\S+) from (\S+)')
    ip_counts = Counter()
    user_counts = Counter()

    with open(path) as f:
        for line in f:
            m = pat.search(line)
            if m:
                user, ip = m.group(1), m.group(2)
                ip_counts[ip] += 1
                user_counts[user] += 1

    return ip_counts, user_counts

# ip_c, usr_c = parse_ssh_failures("/var/log/auth.log")
# print("Top attacker:", ip_c.most_common(1))

(?:invalid user )? makes that phrase optional. sshd adds it when the username doesn't exist on the system, and a pattern without it would silently skip exactly the lines produced by password-spraying tools guessing common names like admin or oracle. A lot of invalid user hits means someone is guessing blindly; failures against real accounts are more serious and worth escalating.

The function reads the file line by line instead of calling f.read(). Production logs can be several gigabytes, and iterating keeps memory use constant.

Parsing Apache Access Log

Apache and nginx both default to the Combined Log Format: client IP, identity, user, timestamp, request line, status code and response size, followed by referrer and user agent:

APACHE_PAT = re.compile(
    r'(?P<ip>[\d.]+) \S+ \S+ \[(?P<time>[^\]]+)\] '
    r'"(?P<method>\w+) (?P<path>\S+)[^"]*" '
    r'(?P<status>\d+) (?P<size>\d+|-)'
)

def parse_apache(path):
    hits = []
    with open(path) as f:
        for line in f:
            m = APACHE_PAT.match(line)
            if m:
                hits.append(m.groupdict())
    return hits

match() is correct here because every access-log line starts with the client IP. (?P<size>\d+|-) handles responses with no body, which Apache logs as - rather than 0. Forgetting that alternative is enough to drop every 304 Not Modified from your results.

Count what doesn't match. Real logs contain malformed lines, often because of attacks: scanners send garbage request lines, and injection payloads include quotes that break naive patterns. A parser that silently skips non-matching lines can hide exactly the events you're hunting for. In production code, count misses and print a sample of them.

Common Pitfalls

Greedy quantifiers. .* matches as much as it can. Against "GET /a" 200 "GET /b", the pattern "(.*)" captures GET /a" 200 "GET /b, not GET /a. Use a lazy .*? or, better, a negated class like [^"]* that cannot cross the delimiter.

Catastrophic backtracking. Nested quantifiers such as (\w+\s?)+$ can take exponential time on a line that almost matches. One crafted log line can freeze your parser. This is the same weakness attackers exploit in ReDoS (Regular expression Denial of Service). Keep patterns specific and avoid nesting repeated groups.

Recompiling in a loop. Python caches recently used patterns, but compiling once with re.compile() outside the loop is clearer and guarantees no repeated work over millions of lines.

Timezones. Access logs include an offset (+0000), but syslog timestamps usually don't, and they omit the year. When correlating events across sources, normalise everything to UTC before comparing times.

Mini Project – Log Anomaly Detector

This detector combines the techniques above. It flags IPs that generate many client errors (4xx), which is the typical signature of directory brute-forcing, vulnerability scanners and credential-stuffing attempts. It also reports the most-requested paths, which shows what the scanners were looking for.

import re, json
from collections import Counter, defaultdict

def analyze_log(path, threshold=10):
    pat = re.compile(r'(\d+\.\d+\.\d+\.\d+).*?"(GET|POST|HEAD) (\S+)[^"]*" (\d{3})')
    ip_4xx   = defaultdict(int)
    path_hit = Counter()

    with open(path) as f:
        for line in f:
            m = pat.search(line)
            if not m:
                continue
            ip, method, url_path, status = m.groups()
            path_hit[url_path] += 1
            if status.startswith("4"):
                ip_4xx[ip] += 1

    suspects = {ip: c for ip, c in ip_4xx.items() if c >= threshold}
    return {
        "suspects":  suspects,
        "top_paths": dict(path_hit.most_common(5)),
    }

Ideas to extend it:

  • Add a time window. Ten 404s in a day is noise; ten in a second is a scanner. Parse the timestamp and count per IP per minute.
  • Check for success after failure. An IP with many 401s followed by a 200 on /login may have guessed a password, which is far more urgent than the failures alone.
  • Keep the status anchored. The pattern reads the status code right after the closing quote of the request line. A looser .* (\d{3}) would grab the last three-digit number on the line, which is often the response size (512 instead of 404), and the detector would silently flag nothing.
  • Output JSON with json.dumps(result, indent=2) so the result can feed a ticket, a dashboard or another script.

From Script to SIEM

A Python script is the right tool for a one-off question, a quick triage on a single host, or prototyping a detection. At scale, a SIEM such as Splunk, Elastic or Wazuh does the same work continuously across every server. It still relies on parsing rules, and those are usually regular expressions much like the ones in this lesson. Writing your own parser is the best way to understand what those rules match and, just as importantly, what they miss.

To see how log analysis fits into day-to-day defence, continue with Security Operations and Incident Response. For writing production-grade detections, see Detection Engineering. For the Linux side of log collection, see Linux Networking Tools & Log Management.

FAQ

What's the difference between re.match() and re.search()?
re.match() only matches at the beginning of the string. re.search() looks anywhere in it. For log lines, search() is the safer default; use match() only when your pattern describes the line from its first character.

Why does findall() return pieces instead of full matches?
Because your pattern contains a capturing group. findall() returns the groups' contents when groups exist. Make the group non-capturing with (?:...), or use finditer() and call .group(0) on each match.

Is regex fast enough for large log files?
Yes, for most investigations. A compiled pattern processes hundreds of thousands of lines per second. Bottlenecks usually come from badly written patterns (heavy backtracking) or loading the whole file into memory, not from the regex engine itself.

Should I parse JSON logs with regex?
No. If a log is structured JSON, as many cloud and container logs are, parse each line with json.loads(). Regex is for semi-structured text formats like syslog and access logs.

Next lesson: Subprocess & OS Automation. You'll run system commands from Python, which lets your log scripts collect their own data.

Write regex patterns to extract: (1) all IPv4 addresses from a string, (2) email addresses, (3) HTTP status codes (3 digits at end of log line).

✦ Answer the questions to complete this task

What re function returns all non-overlapping matches as a list?

What does \\d match in a regex?

Write the Apache log parser from the lesson. Create a sample log file with 5 lines and parse them into a list of dicts. Print only lines where status is 403.

✦ Answer the questions to complete this task

How do you access a named group 'ip' from a match object m?

Implement parse_ssh_failures() using Counter. Create a sample auth.log with 20 lines (some with 'Failed password'). Find the top attacking IP.

✦ Answer the questions to complete this task

What Counter method returns the N most common elements?

💪 Exercises & Challenges

📝 MCQ Medium +20 XP

Regex & Log Parsing MCQ

MCQ: Regex & Log Parsing MCQ

Start →
⚙️ Practical Medium +30 XP

Access Log Analyzer

Access Log Analyzer

Start →
🚩 Challenge Hard +50 XP

Regex Flag Extraction

Regex Flag Extraction

Start →