This project provides a demo of Kafka capabilities including Connect, Kafka Streams, and ksqlDB using live flight data from OpenSky Network. Global flights are displayed by default. A dashboard of information built using Kafka Streams for aggregation is displayed at the top left. You can also draw a geofence using the polygon tool to display a filtered view of flights within the geofence region. Flights update every 10 seconds (occassional failures hitting the OpenSky end point may skip an update).
This project has several dependencies including:
- KSQLGeo - geospatial UDFs for ksqlDB
- KafkaGeoDemo - geo demo used for installing ksqlDB UDFs
- kafka-connect-opensky - OpenSky Network Kafka Connector
- Kafka Flight Demo Streams - Kafka Streams applications for transformation and dashboard
- Cesium - WebGL geospatial toolkit
- Confluent Platform 6.0
- Java 11 or higher
- Docker
docker-compose up -d
- Uncheck the Global Flights option
- Select the Filtered Flights option
- Draw a polygon around the region to filter flights - click for each point, double click to close the polygon
Add the following in the C3 ksql editor. This will create a new entry in the dashboard with the count of flights from Ireland.
CREATE STREAM FLIGHTS_RAW WITH (KAFKA_TOPIC='flights_raw', PARTITIONS=1, REPLICAS=1, VALUE_FORMAT='AVRO');
CREATE TABLE IRELAND_FLIGHTS WITH (KAFKA_TOPIC='dashboard-data', PARTITIONS=1, REPLICAS=1, VALUE_FORMAT='JSON') AS SELECT
'total-ireland-flights' as X,
'Total Ireland Flight' `dashboardTitle`,
COUNT(*) `dashboardValue`
FROM FLIGHTS_RAW
WHERE (originCountry = 'Ireland')
GROUP BY 'total-ireland-flights'
EMIT CHANGES;`
And set auto.offset.reset to earliest to get a count from the beginning.
If you would like to use test data instead of live data (or if there is a problem with OpenSky), the following command will publish 500 flight updates every second for a total of 100,000 updates.
kafka-producer-perf-test --topic flights_raw --payload-file test_data/flights.out --num-records 100000 --throughput 100 --producer-props bootstrap.servers=localhost:9092
If you only see a blue globe with no land, then you may need to update the Cesium version. You will need to update the references in the index.html file. For example, change the two lines in the head section:
<script src="https://cesium.com/downloads/cesiumjs/releases/<latest>/Build/Cesium/Cesium.js"></script>
<link href="https://cesium.com/downloads/cesiumjs/releases/<latest>/Build/Cesium/Widgets/widgets.css" rel="stylesheet">
You can find the latest version at https://cesium.com/downloads/