HTTP Requests & Web Scraping
Use the requests library and BeautifulSoup to make HTTP calls, scrape web content, probe APIs, and automate form submissions.
Learning Objectives
- → Install and use the requests library for HTTP GET/POST
- → Inspect response status codes, headers, and body
- → Parse HTML with BeautifulSoup to extract links and data
- → Submit forms and handle sessions/cookies
- → Understand rate limiting and ethical scraping
Almost every security tool talks HTTP. Vulnerability scanners, threat-intel lookups (VirusTotal, AbuseIPDB, Shodan), ticketing integrations, cloud APIs and SIEM connectors are all, underneath, a program sending HTTP requests and reading the answers. Once you can do that from Python, you can automate the repetitive parts of security work: enrich a list of 500 suspicious IPs, check which of your own sites are missing security headers, or inventory every form on an application before a test.
This lesson covers the requests library for sending requests, and BeautifulSoup for reading the HTML that comes back. It ends with two small tools, a crawler and a content-discovery script, built with the habits that keep automated traffic safe and polite.
Before you start: you should know how dictionaries and functions work (Control Flow & Functions, Data Structures). For what actually travels over the wire, see HTTP Deep Dive.
Ground Rules: Only Automate Against Targets You May Test
Sending requests to a website is legal and normal; your browser does it all day. But automated tools send many requests quickly, and some of the examples here look for hidden pages. Against a system you don't own or haven't been explicitly authorised to test, that can breach terms of service or computer misuse laws.
Use the safe targets in this lesson (httpbin.org exists for exactly this purpose), your own lab, or a platform that gives written permission. Whatever the target, apply three habits from the start:
- Always set a timeout. Without one,
requestscan wait forever on an unresponsive server and your script hangs. - Rate-limit yourself. A short
time.sleep()between requests avoids overloading small servers and tripping rate limits. - Identify your tool honestly in the
User-Agent, so the site owner can see what the traffic is and contact you.
The requests Library
requests is not part of the standard library. Install it, along with BeautifulSoup for HTML parsing, inside your virtual environment:
pip install requests beautifulsoup4
GET Requests
A GET request asks the server for a resource. requests.get() returns a Response object holding everything the server sent back:
import requests
r = requests.get("https://httpbin.org/get", timeout=5)
print(r.status_code) # 200
print(r.headers["Content-Type"])
print(r.text[:300])
print(r.json()) # if response is JSON
r.status_codetells you whether the request worked (see the table below).r.headersis a case-insensitive dictionary of response headers. Security headers such asContent-Security-PolicyorStrict-Transport-Securityappear here, so a few lines of code can audit them across many sites.r.textis the body as a string;r.contentis the raw bytes, which you need for images or binaries.r.json()parses a JSON body into Python objects. It raises an error if the body isn't JSON, so checkContent-Typefirst when you're not sure.
Query Parameters
Don't build query strings by hand. Pass a dictionary and requests handles the URL encoding, including characters like : and spaces that would otherwise break the URL:
params = {"q": "site:example.com", "limit": 10}
r = requests.get("https://httpbin.org/get", params=params, timeout=5)
print(r.url) # URL with ?q=site%3Aexample.com&limit=10
Custom Headers
Headers carry metadata about the request: who is asking, what format they accept, and credentials. APIs typically expect an Accept header and an API key or bearer token:
headers = {
"User-Agent": "Mozilla/5.0 (security-scanner/1.0)",
"Accept": "application/json",
}
r = requests.get("https://httpbin.org/headers", headers=headers, timeout=5)
Never hard-code API keys in a script you might share or commit. Read them from an environment variable with os.environ["VT_API_KEY"] instead.
POST Requests
POST sends data to the server: logging in, submitting a form or calling an API action. The key choice is how the body is encoded, because the server only understands the format it expects:
# Form data
data = {"username": "admin", "password": "test"}
r = requests.post("https://httpbin.org/post", data=data, timeout=5)
# JSON body
payload = {"action": "scan", "target": "192.168.1.1"}
r = requests.post("https://api.example.com/scan", json=payload, timeout=5)
data=sendsapplication/x-www-form-urlencoded, which is what an HTML form submits.json=serialises the dictionary and setsContent-Type: application/json, which is what most REST APIs expect.
If an API answers 400 Bad Request or 415 Unsupported Media Type, a wrong body encoding is the first thing to check.
Status Codes
The status code is the server's one-number summary of what happened. Your scripts should branch on it rather than assume success:
| Code | Meaning |
|---|---|
| 200 | OK |
| 301 | Redirect (permanent) |
| 401 | Unauthorized |
| 403 | Forbidden |
| 404 | Not Found |
| 429 | Too Many Requests |
| 500 | Server Error |
if r.status_code == 200:
print("Success")
elif r.status_code == 401:
print("Authentication required")
r.raise_for_status() # raises HTTPError for 4xx/5xx
401 means "who are you?" (no valid credentials), while 403 means "I know who you are, and the answer is no". When a response is 429, slow down: the server is telling you its rate limit, often with a Retry-After header saying how long to wait.
By default requests follows redirects automatically, so a 301 usually becomes the 200 of the final page. r.history lists the redirects that happened. Pass allow_redirects=False when you need to see the redirect itself, for example to check where a login form sends you.
Sessions (Cookies)
Each requests.get() call is independent: cookies from one response aren't sent with the next request. That breaks anything that needs a login. A Session keeps cookies between requests, reuses the underlying connection (which is much faster for many requests to the same host), and lets you set default headers once:
s = requests.Session()
s.headers.update({"User-Agent": "Tester/1.0"})
# Login
s.post("https://example.com/login", data={"user": "test", "pass": "test"}, timeout=5)
# Now subsequent requests carry the session cookie
r = s.get("https://example.com/dashboard", timeout=5)
Many real login forms also include a hidden CSRF token that must be sent back with the POST. You fetch the login page first, extract the token with BeautifulSoup (next section), then include it in the form data.
BeautifulSoup
r.text gives you HTML as one long string. BeautifulSoup parses it into a tree you can search by tag, attribute or CSS selector, which is far more reliable than matching HTML with regular expressions:
from bs4 import BeautifulSoup
r = requests.get("https://example.com", timeout=5)
soup = BeautifulSoup(r.text, "html.parser")
# Find elements
title = soup.find("title").text
links = [a["href"] for a in soup.find_all("a", href=True)]
forms = soup.find_all("form")
inputs = soup.find_all("input", {"type": "hidden"})
print(title)
print("Links found:", len(links))
From a security point of view, the interesting parts of a page are its attack surface: forms (where user input enters the application), hidden inputs (tokens, IDs and sometimes values that should never have been sent to the browser), links to other areas, and the JavaScript files it loads. soup.select("script[src]") lists those scripts with a CSS selector.
BeautifulSoup only sees the HTML the server sent. Pages that build their content with JavaScript in the browser will look nearly empty here. For those, look in the browser's developer tools for the API the page calls, and request that API directly.
Link Crawler (Simple)
A crawler maps a site by following its links: fetch a page, collect its links, queue the ones on the same site, repeat. It is the first step of most web scanners, because you can't test pages you don't know exist.
import time
from collections import deque
from urllib.parse import urljoin, urldefrag
import requests
from bs4 import BeautifulSoup
def crawl(base_url, max_pages=10, delay=0.5):
visited = set()
to_visit = deque([base_url])
while to_visit and len(visited) < max_pages:
url = to_visit.popleft()
if url in visited:
continue
visited.add(url)
try:
r = requests.get(url, timeout=3)
except requests.RequestException:
continue
if "text/html" not in r.headers.get("Content-Type", ""):
continue
soup = BeautifulSoup(r.text, "html.parser")
for a in soup.find_all("a", href=True):
link, _ = urldefrag(urljoin(url, a["href"]))
if link.startswith(base_url) and link not in visited:
to_visit.append(link)
time.sleep(delay) # be polite
return visited
# pages = crawl("https://example.com")
Details that matter:
urljoin(url, href)resolves a relative link like../aboutagainst the current page, not the start page. Using the start URL here is a common bug that produces wrong addresses as soon as the crawler goes one level deep.urldefragstrips#sectionanchors, so/pageand/page#toparen't crawled twice.deque.popleft()gives breadth-first order efficiently;list.pop(0)gets slow on large queues.- The
startswith(base_url)check keeps the crawler on the site you're allowed to test. Without it, it would wander off across the internet. max_pagesanddelaycap the load you put on the server.
Mini Project – Directory Bruteforcer
Sites often contain pages that aren't linked from anywhere: an old /backup folder, a forgotten /admin panel, an exposed .env file full of secrets. A crawler can't find them, because nothing links to them. Content discovery, also called directory brute-forcing, tries a list of common names and looks at the status code of each answer. Tools like gobuster and ffuf do this at scale. Writing a minimal version shows how they work and what they can miss.
import requests
def dir_brute(base_url, wordlist):
found = []
for word in wordlist:
url = f"{base_url.rstrip('/')}/{word}"
try:
r = requests.get(url, timeout=2, allow_redirects=False)
if r.status_code in (200, 301, 302, 403):
print(f" [{r.status_code}] {url}")
found.append(url)
except requests.RequestException:
pass
return found
words = ["admin", "login", "backup", "config", "api", "uploads", ".env"]
found = dir_brute("https://httpbin.org", words)
print(f"Found: {len(found)} paths")
This version sends requests as fast as it can. Making it polite is the task at the end of this lesson. Reusing one requests.Session() for all requests is another easy improvement: it keeps the connection open instead of opening a new one for every word.
Reading the results:
200: the path exists and is readable. An exposed.envorbackup.zipis a finding on its own.301/302: often a directory (redirecting to add a trailing/) or a page that redirects to a login.allow_redirects=Falseis what lets you see this.403: the path exists but access is denied. That still confirms it's there.- Soft 404s. Some applications answer
200with a "page not found" message for everything. Request a random path you know doesn't exist first; if it returns200, compare response lengths instead of status codes.
For defenders, this is also a useful self-audit: run it against your own sites with a list of sensitive names (.git/, .env, backup.sql, phpinfo.php). Anything that answers 200 should be removed or blocked. On the detection side, content discovery leaves a clear trace: a burst of 404s from one IP. The log-parsing lesson builds a detector for exactly that pattern.
FAQ
What's the difference between data= and json= in requests.post()?
data= sends form-encoded fields like an HTML form; json= sends a JSON document and sets the Content-Type: application/json header. Use whichever format the server expects. Most APIs want JSON.
Why does my script hang forever?
You probably didn't set a timeout. requests has no default timeout, so a server that accepts the connection but never answers blocks your script indefinitely. Always pass timeout=.
Why does BeautifulSoup find nothing on a page I can see in my browser?
The content is probably rendered by JavaScript after the page loads. requests only downloads the initial HTML. Find the API the page calls in your browser's Network tab and request it directly.
Should I disable certificate checks with verify=False?
Only in a lab with self-signed certificates, never against real services. Disabling verification lets anyone in the network path impersonate the server and read or change your traffic.
Next lesson: Regular Expressions & Log Parsing. You'll parse the server logs that the requests in this lesson leave behind.
Use requests.get on https://httpbin.org/get. Print the status code, Content-Type header, and the User-Agent field from the JSON response.
What attribute gives you the HTTP status code from a requests response?
What method parses a JSON response body into a Python dict?
Fetch https://example.com and use BeautifulSoup to extract: page title, all link hrefs, and count of paragraphs.
Which BeautifulSoup parser works with Python's standard library?
How do you find all <a> tags in a BeautifulSoup object?
Implement the dir_brute function. Add a delay of 0.1s between requests to be polite. Run it against https://httpbin.org with the built-in wordlist.
What requests argument prevents following 301/302 redirects?