Description

This app was created to learn and experiment with:

• Litestar
• Postgres without ORM
• Postgres as a job queue
• Simple and custom DB migrations script
• Asyncpg
• Msgspec instead of pydantic
• JWT (Json Web Token)
• HTMX
• Manipulating IP networks (here's some magic to exclude a network from another faster than the builtin python address_exclude)

There is a test instance running online, and a test account is available (testuser/This1sForTest!), API docs are open.

This app allows you to create lists of IP addresses in their CIDR notation.
Lists have two types:

• denylist
• safelist

Safelists exist exclusively to filter the denylists, when you add an address to an enabled safelist, that address is removed from all the existing denylists. The same thing happens when you enable a disabled safelist, all the denylists are filtered with the addresses of that safelist.

For example, if you have the following networks/addresses in a denylist (or spread in many denylists):

66.66.1.0/24
12.12.1.0/24

And you create a safelist with the following networks:

66.66.1.0/26
12.12.1.10/32
12.12.1.12/32

The denylists are filtered and end up with the following:

12.12.1.0/29
12.12.1.11/32
12.12.1.128/25
12.12.1.13/32
12.12.1.14/31
12.12.1.16/28
12.12.1.32/27
12.12.1.64/26
12.12.1.8/31
66.66.1.128/25
66.66.1.64/26

This also works if you try to add the original addresses to the denylist when they are present in a safelist, basically safelists act as a filter for denylists.

About address exclusion

Excluding a network from another is easy with the ipaddress python library:

import ipaddress

a = ipaddress.ip_network("1.1.0.0/16")
b = ipaddress.ip_network("1.1.1.0/24")

for i in a.address_exclude(b):
    print(i)

# Output:
1.1.128.0/17
1.1.64.0/18
1.1.32.0/19
1.1.16.0/20
1.1.8.0/21
1.1.4.0/22
1.1.2.0/23
1.1.0.0/24

That is when you have to exclude one-from-one but when you have thousands of addresses (safe addresses) that should exclude recursively another set of thousands of addresses (deny addresses) things start to complicate.
After you exclude address a with address b (checking if b is contained in a of course) you might end up with:

• Nothing, if b is a supernet of a (you cannot use address_exclude directly here and must check explicitly if b is a supernet of a)
• Nothing if b is equal to a
• One or many subnets (if b is a subnet of a)
• The original address a if b just doesn't exclude anything from a

When you end up with one or multiple extra subnets after an exclusion, you need to iterate over them again to ensure that they are fully excluded with your safe addresses. That can become tedious and slow, you also cannot be sure on how many times you need to iterate until everything is excluded correctly.

That is solved with the following functions that exclude addresses faster and correctly:

• This is were the background job that filters addresses starts - filter_safe_cidrs
• This takes an address and filters it using many addresses - address_exclude_many
• This is a faster implementation of python's ipaddress/address_exclude and is called by address_exclude_many - exclude_address_raw

Installation & development

A compose.yaml file is provided as an example on how to run the app.

To develop locally, run make install to create a venv with all the dependencies, make test always starts an empty database and runs all the tests.
The app can be started by running the litestar cli tool: cd src; litestar run --reload.

Usage

1. Create as many denylists as you need, either by using the web interface or with the API, tag them according to your needs.
2. Create one or many enabled safelists, and add all the addresses that should never be present in a denylist. If you add more in the future, denylists get filtered automatically.
3. Consume the denylists by tags using the endpoints /v1/cidr/.*, for example /v1/cidr/collapsed returns all the matched addresses collapsed into networks.

License

LICENSE (MIT)