raphtory-akka-0.3.0

A large number of changes have occurred in dev causing it to diverge largely from master and current documentation. Before any larger changes are completed (notably snapshotting, breaking away from docker and the creation of a new analysis API) we are releasing 0.3.0. The changes for this are listed below:

Major Changes - Raphtory Management

• Raphtory have been upgraded to use Akka 2.6 and swapped from Netty to Artery
• All messaging is now handled through Kryo serialiser instead of the default Java serialiser
• The Watchdog SeedNode and Watermark Manager are now combined into an orchestration actor group that manages the whole cluster.
• All Raphtory Components are now managed by a Raphtory Component Connector which ensures cluster startup by reporting to the watchdog.
• Writer Workers can now martial and un-martial the state of their allocated entity storage to/from parquet.

Major Changes - Analysis Management

• Raphtory’s logic for handling analysis has been totally rewritten. The Analysis Manager now spawns a task that contains the full control logic for each submitted query. This task requests the Reader Workers to create a separate actor for the analysis (the AnalysisSubtaskWorker) which contains all vertex visitors. Once the analysis is completed (across all flattenings) both the Task and SubtaskWorkers can be killed, removing all analytical states and stopping the build-up of visitors/analysis properties over time.
• The above drastically simplified the VertexMultiQueue which now only needs an odd and an even mailbox instead of storing timestamp and windowsize as well.
• PubSub was removed as a communication method between Analysis Task and SubtaskWorkers in favour of direct actor messaging. Completed for performance and practical reasons (new actors spawning requires gossiping of their location which is slow and causes intermittent errors).
• VertexMessageHandler was created to track all vertex messages between SubtaskWorkers.
• ViewLens and WindowedLens compressed into one class (GraphLens)

Major Changes - Analysis API

• Analysers now require the user to return a map of results which can then be serialised in a variety of ways. This makes the analyser more general and removes the need to edit the code when the user wants to swap from saving to a text file to saving to mongo etc.
• Raphtory queries may now be submitted with a serialiser class which contains the logic to save the results in the users desired format.
• Raphtory Serialisers handle both windowed and unwindowed flattenings, therefore, ProcessResults and ProcessWindowResults have been replaced with extractResults, removing A LOT of duplicated code.
• The old serialiser type which extended Analyser is now removed as redundant.
• Analysers are now typed, specifying what each subtask worker is returning and, therefore, what the analysis task is aggregating. This removes the need for unpleasant casting inside of extractResults.
• The Query API has had the explicit window type arguments removed (true, false, batched). A user may now simply submit a window set (which can include one window) and raphtory will handle it internally.
• Vertex Visitor and Edge Visitor were renamed to Vertex and Edge for user clarity.
• Double args array submission is now no longer possible with the RaphtoryGraph removing confusion.
• Example algorithms have been updated with the new API
• Example Algorithms have been given named param alternatives to the args array.

Test Changes

• All Commands Test changed to use set generated file as all scala versions seem to do something different in utils.Random (see testUpdates.txt in dev/allcommands)
• Datablast Analyser added which throws large arrays of data from all subtask workers to see how well Raphtory handles it.
• All commands test converted to unit tests which are fully automated to make comparisons between versions a lot simpler.

Minor Changes

• build.sbt has been cleaned up and organised
• A large amount of package refactoring ahead of breaking the project into core + modules
• The router has been officially renamed GraphBuilder
• Initial SnapshotManager included, but currently stub
• Initial GraphAlgorithm included, but currently a stub
• AnalysisUtils created for misc runtime compilation code.
• Swapped from 32 bit murmur3 hash to 64 bit xxHash to remove chances of collisions.