Regular Expressions & Log Parsing

Master Python's re module to write patterns that extract IPs, timestamps, usernames, and anomalies from real-world log files.

Medium 70m 3 tasks

Prerequisites: File I/O & JSON

Learning Objectives

→ Write and compile regex patterns using the re module
→ Use re.search, re.match, re.findall, and re.sub
→ Extract named capture groups from log lines
→ Parse Apache, nginx, and auth.log format entries
→ Aggregate findings into structured reports

The re Module

import re

Key functions:
| Function | Description |
|----------------|--------------------------------------------|
| re.search() | Find first match anywhere in string |
| re.match() | Match only at start of string |
| re.findall() | Return all non-overlapping matches |
| re.sub() | Replace matches with a string |
| re.compile() | Compile a pattern for repeated use |

Basic Patterns

# IP address
ip_pat = re.compile(r'\b(\d{1,3}\.){3}\d{1,3}\b')

# Email
email_pat = re.compile(r'[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}')

# URL
url_pat = re.compile(r'https?://[^\s"]+')

search and match

line = "Failed password for root from 192.168.1.50 port 22 ssh2"

m = re.search(r'from (\d+\.\d+\.\d+\.\d+)', line)
if m:
    print(m.group(1))   # 192.168.1.50
    print(m.group(0))   # full match: 'from 192.168.1.50'

findall

log = "10.0.0.1 accessed /api, 10.0.0.2 accessed /login, 10.0.0.1 tried /admin"
ips = re.findall(r'\d{1,3}(?:\.\d{1,3}){3}', log)
print(ips)   # ['10.0.0.1', '10.0.0.2', '10.0.0.1']
print(set(ips))  # unique

Named Groups

pattern = re.compile(
    r'(?P<ip>\d+\.\d+\.\d+\.\d+) - - '
    r'\[(?P<time>[^\]]+)\] '
    r'"(?P<method>\w+) (?P<path>[^"]+) HTTP/[^"]+" '
    r'(?P<status>\d+)'
)

line = '192.168.1.1 - - [10/Jan/2024:10:23:45 +0000] "GET /admin HTTP/1.1" 403'
m = pattern.search(line)
if m:
    print(m.group("ip"))      # 192.168.1.1
    print(m.group("status"))  # 403
    print(m.groupdict())

re.sub — Redaction

# Redact credit card numbers
text = "Card: 4111 1111 1111 1111 charged"
redacted = re.sub(r'\b\d{4}[\s-]?\d{4}[\s-]?\d{4}[\s-]?\d{4}\b', '[REDACTED]', text)
print(redacted)   # Card: [REDACTED] charged

Parsing auth.log

import re
from collections import Counter

def parse_ssh_failures(path):
    pat = re.compile(r'Failed password for (?:invalid user )?(\S+) from (\S+)')
    ip_counts = Counter()
    user_counts = Counter()

    with open(path) as f:
        for line in f:
            m = pat.search(line)
            if m:
                user, ip = m.group(1), m.group(2)
                ip_counts[ip] += 1
                user_counts[user] += 1

    return ip_counts, user_counts

# ip_c, usr_c = parse_ssh_failures("/var/log/auth.log")
# print("Top attacker:", ip_c.most_common(1))

Parsing Apache Access Log

APACHE_PAT = re.compile(
    r'(?P<ip>[\d.]+) \S+ \S+ \[(?P<time>[^\]]+)\] '
    r'"(?P<method>\w+) (?P<path>\S+)[^"]*" '
    r'(?P<status>\d+) (?P<size>\d+|-)'
)

def parse_apache(path):
    hits = []
    with open(path) as f:
        for line in f:
            m = APACHE_PAT.match(line)
            if m:
                hits.append(m.groupdict())
    return hits

Mini Project – Log Anomaly Detector

import re, json
from collections import Counter, defaultdict

def analyze_log(path, threshold=10):
    pat = re.compile(r'(\d+\.\d+\.\d+\.\d+).*?(GET|POST|HEAD) (\S+).* (\d{3})')
    ip_4xx   = defaultdict(int)
    path_hit = Counter()

    with open(path) as f:
        for line in f:
            m = pat.search(line)
            if not m:
                continue
            ip, method, url_path, status = m.groups()
            path_hit[url_path] += 1
            if status.startswith("4"):
                ip_4xx[ip] += 1

    suspects = {ip: c for ip, c in ip_4xx.items() if c >= threshold}
    return {
        "suspects":  suspects,
        "top_paths": dict(path_hit.most_common(5)),
    }

Write regex patterns to extract: (1) all IPv4 addresses from a string, (2) email addresses, (3) HTTP status codes (3 digits at end of log line).

✦ Answer the questions to complete this task

What re function returns all non-overlapping matches as a list?

What does \\d match in a regex?

Write the Apache log parser from the lesson. Create a sample log file with 5 lines and parse them into a list of dicts. Print only lines where status is 403.

✦ Answer the questions to complete this task

How do you access a named group 'ip' from a match object m?