S3 Log Extraction

Fast extraction of access summary data from S3 logs.

Originally developed for the DANDI Archive.

Read more about S3 logging on AWS.

⚠️ This package currently only supports processing of access data (GET-type requests); if you wish to use this package for other types of requests (PUT/DELETE/HEAD, etc.) please reach out by raising an issue. ⚠️

Installation

pip install s3-log-extraction

Note for Windows users: This package requires GAWK and is not natively supported on Windows. Windows users should use Windows Subsystem for Linux (WSL) to run this package.

Workflow

flowchart TD
    A[Configure cache<br/><br/>Initialize home and cache directories]
    B[Extract logs<br/><br/>Process raw S3 logs and store minimal extracted data]
    C[Update IP indexes<br/><br/>Generate anonymized indexes for each IP address]
    D[Update region codes<br/><br/>Map IPs to ISO 3166 region codes using the local GeoLite2 database]
    E[Update coordinates<br/><br/>Look up latitude/longitude for each region code in the bundled ISO 3166 tables]
    F[Generate summaries<br/><br/>Create per-dataset summaries for reporting]
    G[Generate totals<br/><br/>Aggregate statistics across datasets or archive]
    H[Share!<br/><br/>Post the summaries and totals in a public data repository]

    A --> B
    B --> C
    C --> D
    D --> E
    E --> F
    F --> G
    G --> H

Loading

Generic Usage

[Optional] Configure a non-default cache directory on a mounted disk that has sufficient space (the default is placed under ~/.cache). This will be the main location where extracted logs and other useful information will be stored.

s3logextraction config cache set <new cache directory>

[Optional] Configure a non-default base directory for temporary files. Each extraction run creates its own temporary directory inside this one, holding per-worker output until each batch is merged into the cache. Without this setting, runs use the system temporary directory, which on many systems is a small RAM-backed /tmp that a large extraction can exhaust.

s3logextraction config tmp set <temporary directory>

Use s3logextraction config tmp reset to go back to the system temporary directory.

To extract the logs:

s3logextraction extract <log directory>

To override the cache or temporary directory for a single extraction run (without changing global config):

s3logextraction extract <log directory> --cache <cache directory> --tmp <temporary directory>

[Optional] Exclude individually reviewed requesters. List their addresses in ~/.s3-log-extraction/excluded_ips.txt, which is the only place the excluded addresses are read from. Their requests are dropped at extraction, so they never enter the cache, and they are also left out of every summary.

192.0.2.1  # Reviewed mirror
192.0.2.2
2001:db8::1  # Same actor as 192.0.2.2

The file holds one IPv4 or IPv6 address per line. Anything after a # is a comment, so each address can carry a note on why it is excluded, and blank lines are ignored. Every entry must be a single exact address. Networks in CIDR notation, wildcards, and regular expressions are refused with an error. The file is edited by hand and never written by the package, and an absent or empty file excludes no one.

Excluding an address only affects logs extracted from then on. Requests from that address already in the cache are still left out of the summaries.

Exclusion only works in one direction over time. Adding an address can only drop its requests, from future extraction and, once removed with s3logextraction reset excluded, from the cache. Removing an address from the file does not bring its requests back, since they were never extracted or were deleted. To include an address again, rerun the extraction from scratch with s3logextraction reset extraction followed by a full s3logextraction extract.

To check whether the cache still holds requests from an excluded address, such as those extracted before it was listed, run

s3logextraction validate excluded

It prints how many object keys it scanned, then lists each object key that holds such requests with its count, without printing the addresses. It exits with an error when it finds any, or when a directory of the cache cannot be read, so a scheduled job running it fails. Pass --encryption false for a cache extracted without encryption.

To remove those requests from the cache, so that they leave the summaries generated from then on, run

s3logextraction reset excluded --confirm

Without --confirm, it only lists what it would remove. The matching lines are removed from every per-request file of an object key together, and an object key left without any request is deleted. Back up the cache first, and do not run an extraction at the same time. It refuses to start while the extraction records show an unfinished extraction. The removal is permanent, as described above.

NOTE: If you feel like this command is taking a long time on your system, DO NOT interrupt it via ctrl+C or pkill. Instead, you can safely interrupt it by running:

s3logextraction stop

This will allow it to finish processing the current batch of logs and then exit gracefully.

After your logs are extracted, ensure the environment variables for the geolocation database are set:

• MAXMIND_ACCOUNT_ID and MAXMIND_LICENSE_KEY
  • Credentials of a free MaxMind account, used to download the GeoLite2-City database.
  • The database is downloaded into the cache directory on first use and refreshed automatically once it is more than a week old, so the credentials only need to be set on machines that generate the summaries.

export MAXMIND_ACCOUNT_ID="your_account_id_here"
export MAXMIND_LICENSE_KEY="your_license_key_here"

To generate top-level summaries and totals (that is, per dataset):

s3logextraction update summaries
s3logextraction update totals

Requesters are geolocated while the summaries are generated. Each IP address is checked against the published ranges of GitHub, AWS, GCP, Azure, and known VPN or datacenter providers, and labeled by that service if it falls in one. GitHub is checked before Azure, since the ranges of its hosted Actions runners lie inside those of Azure. An address outside every service is looked up in the local GeoLite2 database and labeled with its ISO 3166-1 alpha-3 country code and ISO 3166-2 subdivision code (e.g. "USA/CA" for California). Fetching the published ranges needs network access. The GitHub ranges come from the GitHub API, which allows 60 unauthenticated requests an hour per IP address. Set a token in the GITHUB_TOKEN environment variable to raise that limit; it is sent to the GitHub API and nowhere else. The last listing obtained from each source is saved in the service_networks folder of the cache directory, and when a source cannot be reached, a saved copy up to 30 days old is used in its place, with a warning. No requester's location is written to disk; only the aggregated by_region.tsv summaries are.

To give every region of the published summaries a coordinate, for maps:

s3logextraction update ip coordinates

This step needs no credentials: each region code is looked up in ISO 3166 tables bundled with the package, whose coordinates come from the public-domain Natural Earth dataset. Cloud service regions such as AWS/us-east-1 are located with the GeoLite2 database.

To force a fresh download of the GeoLite2 database (this happens automatically when it is stale):

s3logextraction update ip database --force

This product includes GeoLite2 Data created by MaxMind, available from https://www.maxmind.com.

Finally, to generate archive-wide summaries and totals:

s3logextraction update summaries --mode archive
s3logextraction update totals --mode archive

Remote S3 Bucket Extraction

To extract logs from a remote S3 bucket, use the --mode remote flag. For large buckets, we strongly recommend setting up AWS S3 Inventory and downloading the inventory locally before running the extraction. Scanning the bucket directly via live network calls (the default when no --inventory path is given) can be extremely slow for buckets with millions of objects.

Using S3 Inventory (recommended)

AWS S3 Inventory generates periodic snapshots of all objects in your bucket as gzip-compressed CSV files. Once downloaded locally, the inventory lets s3logextraction enumerate all log files without making any live S3 listing calls, providing a significant performance improvement over direct bucket scanning.

Expected inventory directory layout:

<inventory_directory>/
├── <timestamp>/               # e.g. 2026-05-03T01-00Z/
│   ├── manifest.json
│   └── manifest.checksum
├── data/
│   └── <uuid>.csv.gz          # gzip-compressed CSV inventory files
└── hive/
    └── dt=<YYYY-MM-DD-HH-MM>/ # e.g. dt=2026-05-03-01-00/
        └── symlink.txt        # references to data/*.csv.gz

Pass the path to the downloaded inventory directory via the --inventory option:

s3logextraction extract s3://my-logs-bucket --mode remote --inventory /path/to/inventory

To check how many log files are in the inventory and the total size:

s3logextraction stats --inventory /path/to/inventory

To report what percentage of log files have already been processed:

s3logextraction completion --inventory /path/to/inventory

To check whether an extraction has anything to do, without extracting anything, add --check to the same command:

s3logextraction extract s3://my-logs-bucket --mode remote --inventory /path/to/inventory --check

It stops at the first log file that has not been extracted. It exits with status 3 when some are pending and 0 when none are, which keeps pending work apart from an error (status 1). A scheduled job can then start an extraction only when there is something to do.

s3logextraction extract s3://my-logs-bucket --mode remote --inventory /path/to/inventory --check
if [ $? -eq 3 ]; then
    s3logextraction extract s3://my-logs-bucket --mode remote --inventory /path/to/inventory
fi

Without an S3 Inventory (not recommended)

If you do not have an S3 Inventory available, do not use --mode remote without --inventory — the live bucket scan will be extremely slow for large buckets. Instead, use s5cmd to manually download the unprocessed log files to a local directory and then run the local extraction on those files:

s5cmd cp "s3://my-logs-bucket/*" /path/to/local/logs/
s3logextraction extract /path/to/local/logs/

If you're new to using AWS S3 buckets and haven't yet enabled the logging this project utilizes, you can follow these simple instructions to get started.

1. Log into your AWS console.
2. Create a new PRIVATE S3 bucket - typically the name of the new bucket is the name of the one you wish to enable logging on with -logs added to the end. For example, dandiarchive-logs.
   • NEVER share this bucket publicly as it contains sensitive information.
3. Navigate back to the S3 bucket you wish to enable logging on.
4. Under the Properties tab, scroll down to the section called Server access logging and select Edit.
5. Toggle the selection to Enable, then specify the destination where logs will be stored as the new S3 bucket you created in step (2).
6. Recommended:
   • Specify the Log object key format as the nested pattern shown below.
   • Ensure the Source of date used in log object key format is the S3 event time.

Developer Notes

Throughout the codebase, various processes are referred to in the following ways:

• parallelized: The process can be run in parallel across multiple workers, which increases throughput.
• interruptible: The process can be safely interrupted (ctrl+C or pkill) with only a very low chance of causing corruption. For parallelized interruption you may have to either pkill the main dispatch process or spam ctrl+C multiple times.
• updatable: The process can be resumed from the last checkpoint without losing any progress. It can also be run fresh at different times, such as on a CRON cycle, and it will only interact with unprocessed data.

Performance

By leveraging GAWK, this version of the S3 log handling is considerably more efficient than the previous attempts.

The previous attempt used a multistep process which took several days to run (even on multiple workers). It also required an additional ~200 GB cache to allow lazy updates of the per-object bins.

This version requires no intermediate cache, stores only the minimal amount of data to be shared, and takes less than a day to do a fresh run (and is also lazy regarding daily CRON updates).

Validation

In lieu of attempting fully validated parsing of each and every line from the log files (which is a hard, unsolved problem - see s3-log-parser), we instead validate the heuristics in a targeted manner through specific validation scripts.

These can also be used to verify the current state of the extraction process, such as warning about corrupt records or incomplete cache files.

Excluded IPs

The extraction heuristic pre-validator skips the lines of the same excluded IP addresses that extraction drops. See the generic usage above for how to list them.

Submission of line decoding errors

Should you discover any lines in your S3 log files that cause failures in the codebase, please email them to the core maintainer (cody.c.baker.phd@gmail.com) before raising issues or submitting PRs contributing them as examples, to more easily correct any aspects that might require anonymization.