Spider's Lair Scraper

Challenge project for bo7.online: eight pages block naive HTTP clients. Each door requires a different strategy (cookies, headers, geo-restricted proxies, browser automation, stealth, or timing). This scraper fetches a session cookie once, runs all eight challenges, validates success strings in the HTML, and saves results to output/.

Categories

#	Category	Challenge
1–3	Mysterious Passages	the_door_of_echoed_steps, the_clockwork_door, the_exiled_door
4–6	Curious Reflections	the_fractured_mirror, the_silver_veil, the_mirrored_gaze
7–8	Shattered Thresholds	the_sleeping_vault, the_verity_gate

Setup & run

python3 -m venv venv
./venv/bin/pip install -r requirements.txt
./venv/bin/playwright install chromium firefox

PYTHONPATH=. ./venv/bin/python main.py

main.py takes no CLI arguments. Configuration is via environment variables (prefix SPIDER_LAIR_) or a .env file:

Variable	Default	Purpose
BASE_URL	https://bo7.online	Target site
OUTPUT_DIR	output	Where HTML files are saved
TIMEOUT_SECONDS	15	Request / page-load timeout
RATE_LIMIT_DELAY	1.0	Seconds between challenges
LOG_LEVEL	INFO	Logging verbosity
MEXICAN_PROXY	http://45.174.56.21:999	Default proxy for exiled door

Per-challenge arguments

Each challenge is defined in crawlers/runner.py as (path, success_string, handler, kwargs).

Mysterious Passages

Challenge	Handler	Arguments	How it was solved
the_door_of_echoed_steps	requests	success_string: "The door slides open"	Homepage cookie + browser-like headers (User-Agent, Accept, Accept-Language).
the_clockwork_door	requests	success_string: "The door slides open"	Same cookie/header approach; page is static once authenticated.
the_exiled_door	requests	proxy_url, success_string: "The Exiled Door Resonates"	Requires a Mexican IP. Cookie is fetched through the same proxy as the request. RotatingProxyHandler retries up to 106 proxy sources until one works.

Curious Reflections

Challenge	Handler	Arguments	How it was solved
the_fractured_mirror	playwright	use_stealth: true, success_string: "The Mirror Aligns"	Headless Chromium with playwright-stealth to bypass bot detection.
the_silver_veil	playwright	success_string: "The Mirror Aligns"	Plain headless Chromium; cookie injected, networkidle wait.
the_mirrored_gaze	playwright	browser_type: "firefox", success_string: "The Mirror Aligns"	Firefox instead of Chromium — challenge checks browser fingerprint.

Shattered Thresholds

Challenge	Handler	Arguments	How it was solved
the_sleeping_vault	playwright	wait_after_load: 1000, success_string: "The door slides open"	1 s post-load wait for async JS to fetch and render content.
the_verity_gate	playwright	use_stealth: true, wait_after_load: 1000, success_string: "The door slides open"	Stealth + post-load wait — combines bot evasion and JS timing.

Playwright kwargs reference

Argument	Type	Effect
success_string	str	HTML must contain this text to count as solved
use_stealth	bool	Wraps Playwright with playwright-stealth
browser_type	"chromium" | "firefox"	Browser engine
wait_after_load	int (ms)	Extra delay after networkidle

Requests kwargs reference

Argument	Type	Effect
success_string	str	HTML must contain this text to count as solved
proxy_url	str	HTTP/HTTPS proxy for geo-restricted doors

Tests

# Full integration: hits bo7.online, saves output, compares to golden files
PYTHONPATH=. ./venv/bin/pytest tests/test_expected_output.py::test_output_matches_expected -v

# Offline: compare existing output/ against bucket/tests/expected_html_results/
PYTHONPATH=. ./venv/bin/pytest tests/test_expected_output.py::test_existing_output_matches_expected -v

# Rate-limit stress: 0.2 s delay (~5 req/s), expects ≥1 challenge to pass
PYTHONPATH=. ./venv/bin/pytest tests/test_rate_limit.py -v

# All tests
PYTHONPATH=. ./venv/bin/pytest tests/ -v

Skip network tests: SKIP_NETWORK_TESTS=1 pytest tests/ -v

Golden files live in bucket/tests/expected_html_results/. Each test checks that the success string is present and the normalized HTML matches the expected file.