An automated web scraping pipeline built with Scrapy and PostgreSQL that extracts tournament results, player ratings, and match pairings from Chess-Results.com.
- Tournaments: ~2,800
- Players: ~16,000 (with FIDE IDs, titles, and ratings)
- Matches: ~190,000 pairings and outcomes
- Language: Python
- Scraper: Scrapy
- Database: PostgreSQL (
psycopg2)
tournaments:tournament_id(PK),tourney_name,date,base_minutes,increment_seconds,locationplayers:player_id(PK / FIDE ID),name,title,standard_rating,blitz_rating,rapid_rating,gendermatches:match_id(PK),tournament_id,,white_id,black_id,result,white_rtg,black_rtg
pip install scrapy psycopg2-binarySet your PostgreSQL credentials in pipelines.py and run your schema creation script:
psql -U postgres -d chess_data -f schema.sqlRun the spiders in this sequence to satisfy relational foreign key constraints:
scrapy crawl tourneySpider
scrapy crawl playerSpider
scrapy crawl matchSpiderTo dump the dataset into CSV files for analysis:
\copy tournaments TO 'tournaments.csv' DELIMITER ',' CSV HEADER ENCODING 'UTF8';
\copy players TO 'players.csv' DELIMITER ',' CSV HEADER ENCODING 'UTF8';
\copy matches TO 'matches.csv' DELIMITER ',' CSV HEADER ENCODING 'UTF8';- Code: MIT
- Dataset: [CC0] Public Domain