FastNetMon Advanced 2.0.340

Changes:

• Upgraded partner integration
• Added logic to retrieve MAC address for IPFIX 315 and Inline Monitoring services modes
• Added logic to retrieve MAC address in port mirror, sFlow and Netflow IPFIX payload modes
• Added reliable http code check for Telegram API
• Added logic to print sampling rate when we learn it from device and added logic to log sampling rate changes
• Changed logging logic to print active and inactive flow timeouts when we receive them from routers
• We must not fail installation when sysctl wasn't applied correctly as it happens often on Docker
• Added logic to decode source and destination MAC addresses when sent via Netflow v9
• Added source_mac and destination_mac to simple packet structure and added them to JSON
• Extracted Netflow template logic into separate file for Netflow
• Added 2 byte encoding option for NETFLOW9_BYTES_FROM_DESTINATION_TO_SOURCE used by Huawei
• Added logic to log details about all cases when we increase netflow_v9_too_large_field
• Changed logic for netflow_mark_zero_next_hop_and_zero_output_as_dropped in IPFIX mode to mark traffic with zero output interface and zero ip_next_hop_ipv4 as dropped. Previously we marked as dropped only traffic with zero output interface and zero bgp_next_hop_ipv4
• Added logic to decode NETFLOW9_BGP_NEXT_HOP_IPV4_ADDRESS
• Added new meta field ip_next_hop_ipv4 which is different from bgp_next_hop_ipv4 in case of IPFIX
• Added logic to identify CPU model in ARM format when FastNetMon x86_64 runs on ARM based macOS
• Added logic to add exact values of flexible counters when we trigger attack via flexible thresholds
• Added logic to pass values of flexible counters to attack structure
• Added agent IP for attack reports in text format as it's very important metric
• Increased granularity of flow duration tracking for Netflow and added buckets in range 0-15 seconds
• Removed raw use of ntohs and ntohl in packet parser

FastNetMon Advanced 2.0.339

Changes:

• Switched sFlow plugin to new generation parser and removed af_packet_use_new_generation_parser flag
• Switched port mirror AF_PACKET mode to new generation parser and removed af_packet_use_new_generation_parser flag
• Reworked NG parser to avoid touching buffer
• Added full test for GRE logic
• Fixed HA fields for Partner integration

FastNetMon Advanced 2.0.338

Changes:

• Upgrade partner integration to latest version to add HA capabilities
• Reworked email attack alerts about attacks detected via flexible thresholds to print information about direction of flexible threshold
• Added logic to print loaded flexible rules
• Better logging for attack detection in flexible mode
• Better logging for configuration file management
• Reworked logic to retrieve ethertype from ethernet_vlan_header_t without altering field in original structure
• Added logic to load hostname for FastNetMon

FastNetMon Advanced 2.0.337

Changes:

• Removed obsoleted logic to write not parsed sFlow packets to pcap dump
• Better naming for fields keeping payload length in simple packet
• Added warning as we may have issues with IPv6 offset encoding
• Added option to populate offset length from NG parser for fragmented IPv4 traffic
• Added logic to stop producing random UDP ports for subsequent packets when fragmentation involved. This issue may have caused random UDP port numbers for fragmented traffic
• Added field ip_fragment_offset for simple packet
• Fixed filtering logic to get rid of empty entries in show host counters output
• Addressed race condition in sudo fcli show host_counters_ipv6 and unified logic with IPv4
• Made both sudo fcli show single_host_counters and single_host_counetrs_ipv6 less verbose when we have no IP.
• Improved IPv6 logic to return error when we have no known entries about IP in hash table
• Added logic to read IPv6 addresses from Flow Spec JSON
• Ported filter logic to new IPv6 enabled Flow Spec structures
• Integrated IPv4/IPv6 logic into BGP protocol logic
• Passed IPv6 lookup tree to flow spec validation function in API
• Improved flow spec detection logic error processing
• Added IPv6 source and prefix subnets for flow spec announces
• New flow spec method names from flow spec detection logic
• Improved function to retrieve CPU model to support Linux in Docker running on mac book with ARM64 CPUs
• Added better logic to propagate error from collect_hardware_data
• Added new installer option -docker to install only Docker Engine. Removed limitation only for CentOS 7 for Docker install and allowed it for all RedHats
• Switched from apt_package_manager_install_multiple_packages to apt_package_manager_install_multiple_packages_check_for_installed install batches of packages faster on Ubuntu and Debian
• Added package existence check for MongoDB on Ubuntu and Debian before installing as it will reduce issues with conflict between version if we have packages already
• Commit reworked advanced install logic to use apt_get_install_package_check_for_installed instead of apt_get_install_package
• Added function apt_get_install_package_check_for_installed to avoid installation when packet is already here
• Pretty list of InfluxDB databases
• Added logic to debug exit code and stdout with stderr for systemctl start
• Added better logging about license request process
• Removed msmtp configuration option tls_trust_file if email_notifications_disable_certificate_checks is set as it triggers error: cannot use tls_trust_file with tls_certcheck turned off
• Improved logic to implement email_notifications_disable_certificate_checks
• Added option to install ARM64 packages from installer for CentOS
• Added logic to bind on IPv6 for Grafana interface for ports 443 and 81
• Switched to native MongoDB builds for Ubuntu 22.04 without old manually installed libssl
• Integrated native IPv6 support for fastnetmon_client which can be switched online

FastNetMon Advanced 2.0.336

Changes:

• Switched XDP plugin to new generation of packet parser by default. Removed configuration option xdp_use_new_generation_parser
• Switched IPFIX inline monitoring services and Netflow v9 Lite parser to new parser as old parser had serious issues with parsing IPv6.
• Added logic to track successful parser calls for inline parser in IPFIX inline and Netflow Lite logic
• Moved syslog includes away from main common log4cpp include file to logic as we use it only from main logic file
• Unified rename_all_existing_configuration_collections with fcli and added strict error checking when we cannot make backup to stop processing
• Reworked create_configuration function to rename collections only when they exist and return error when rename collection fails. Addresses hard to find issue with FerretDB
• Added support for different local ASNs for different peers
• Better naming for Prometheus errors
• Implemented logic to reload API login and password without API daemon restart
• Added details about inlining experiment for build_speed_counters_from_packet_counters and build_average_speed_counters_from_speed_counters
• Switched from stdint.h to cstdint as more C++ friendly approach
• Added logic to cleanup install folder as it may have remains of licensing apps
• Added logic to test alternative hashes for speed calculation
• Unified abstract counters to allow easy switch to another hash easier

FastNetMon Advanced 2.0.335

Changes:

• Switched CentOS 7 to Clickhouse LTS as Stable crashes
• Disabled fork for MongoDB on CentOS distros to address crashes during installation
• Fixed example callback script
• Disabled telemetry for manually issued licenses
• Added capture for last X lines from Clickhouse log files to catch installation errors
• Added more context to debug MongoDB issues
• Added unmask logic for InfluxDB to address masked service
• Improved install atomic logic
• Added FastNetMon API command to get external IP from license server
• Added timestamp for JSON export format
• Added recommendation to install visual graphs after finishing install for community version
• Improved error message when we cannot bind on socket from sFlow plugin
• Added logic to print packets
• Added logic to run attack detection on input JSON
• Grafana binary was renamed in upstream and we made this adjustment in installer
• Increased default threshold per host from 20k to 100k as 20k is too low for modern traffic speeds
• Reworked BGP community logic to ignore only broken community and keep legitimates ones from configuration
• Added option to specify custom auth mechanism for MongoDB for FastNetMon daemon using mongodb_auth_mechanism configuration option in /etc/fastnetmon/fastnetmon.conf
• Improved gTEST configuration
• Added flat map to real traffic speed
• Added test for boost::unordered_flat_map
• Upgrade Boost to 1.81
• Added new hashes for Clickhouse
• Added CGNAT checking function for license server
• Switched to more secure permissions for /etc/fastnetmon/license.lic and set it to 755 instead of 777
• Upgrade Clickhouse to 2.3.0 to address description leaks when connection fails: ClickHouse/clickhouse-cpp#228
• Better description for netflow host option
• Added logic to retrieve IPv4 address when IPv4 and IPv6 used in same time
• Improved sockets code in Netflow plugin
• Added per protocol version counters for recv_from mode
• Added new metrics netflow_ipfix_total_ipv4_packets and netflow_ipfix_total_ipv6_packets to count number of Netflow UDP packets received using different protocols
• Fixed CentOS 7 build procedure

FastNetMon Advanced 2.0.334

Changes:

• Added logic to automatically enable interface in AF_PACKET mode
• Switched to new repository for Clickhouse
• Added new types 152 and 153 for flow start and end encoding for Netflow v9
• Added support for Cisco ASA bi-directional flows
• Added configuration option dump_internal_traffic to dump internal traffic
• Improved number of packets encoding logic for IPFIX
• Reworked little endian encoding logic for IPFIX flow length
• Unified little endian conversion fort source and destination ports for IPFIX protocol
• Removed old logic for big endian to little endian conversion and replaced it by in place conversion in wire reader
• Improved protocol retrieval logic in case of Netflow v9
• Added more log messages when we run Installer in legacy mode
• Re-enabled old global gpg keys for Ubuntu 16.04 as it does not support distro specific keys
• Switched APT repository key for Debian and Ubuntu to new one for Grafana and changed to repository specific GPG keys
• Added one more handler for Go installer logic
• InfluxDB changed their gpg key path
• Updated InfluxDB gpg key
• Added logic to uncompress binary dependencies in parallel and re-arranged build dependencies
• Added logic to track duration of install process
• We must use dnf instead of yum
• Added logic to validate SHA512 hash for all binary dependencies retrieved from S3
• Added script to simplify hash regeneration during full rebuild
• Updated key for Grafana

FastNetMon Advanced 2.0.333

Changes:

• Addressed uid conflict between dashboards fastnetmon-advanced-top-10-asns fastnetmon-advanced-top-10-hosts
• Addressed uid conflict between dashboards sflow-statistics and netflow-v5-statistics
• Switched sleep duration calculation to monotonic time source to address potential outages and traffic miscalculations during time adjustments
• Extracted logic to check installer version into function and improved it
• Added logic to check version of installer only if we got it from Fastly
• Added logic to handle connect: network is unreachable errors with download in a special way
• Added logic to print reasons of gateway detection issues for IPv4 and IPv6
• Regenerated Installer gRPC with new binary
• Upgraded logic to generate Go gRPC logic with new libraries
• Arranged cmake file better way
• Regenerated gRPC using new proto file for installer
• Added logic to retrieve latest version of unstable builds from S3
• Added logic to retrieve latest commit and version for unstable builds
• Added logic to set latest developer build version and commit for S3

FastNetMon Advanced 2.0.332

Changes:

• New logic to create new or update existing hostgroups via single API call
• Removed logic to unconditionally overwrite configuration when we do new installation. Instead we check for existence of configuration and create it only when it does not exist. You still can overwrite database using fcli command create_configuration
• Added logic to collect more details why MongoDB failed to connect
• Enabled RPATH logic for gRPC
• Moved get_af_packet_stats to ifdef to address compilation issues
• Removed legacy gRPC interface from traffic_db
• Improved gRPC library search logic and fixed Boost warning in cmake about new version
• Added time tracking to optimize builds
• Added target to build installer
• Reworked FastNetMon build logic to new version of Go schema generator
• Extracted Golang generation logic into separate file for clarity
• Verbose loggings
• Fixed Go dependencies build path
• Complete logic to build installer in separate folder
• Removed installer from main build path
• Added new warning message for installer

FastNetMon Advanced 2.0.331

Changes:

• Added workaround to fix Grafana installation on RHEL 9
• Switched to new Grafana repo
• Added logic to disable AVX downgrade for RHEL 9
• Added support for RHEL 9
• Unified CentOS build logic
• Added native Debian support
• Added error checking for make schema
• Moved common build dependencies to single function
• Added zlib as dependency to build gRPC
• Added missing NO_DEFAULT_PATH for protobuf
• Better logic to search for Protobuf
• Added libabseil path on RHEL systems to fix build issues
• Added logic to link with C-Ares to fix gRPC linking issues
• Regenerated Protobuf schemas with new version of Protobuf
• Ported gRPC installer to use official cmake build
• Improved RPATH management logic
• Upgrade Protobuf from 3.11.4 to 3.15.7
• Added missing signal.h dependency
• Added C-Ares to dependencies for gRPC and fixed gRPC build system
• Upgraded gRPC for new installer to align versions
• We need to specify LD_LIBRARY_PATH for all cmake calls
• Added logic to export per host IPv6 counters to Clickhouse metrics
• Unified host metrics creation for Clickhouse
• Unified logic to print IPv6 address
• Unified IPv4 per host counters
• Unified per host counters
• Added logic to export per network traffic to Clickhouse
• Added table for Clickhouse IPv6 per network counters
• Unified logic to push network counters to Clickhouse
• Added more debug about number of build threads for compilation
• Get rid of global variables in Clickhouse metrics export code
• Added logic to export total IPv6 counters to Clickhouse
• Unified Clickhouse export function for total traffic
• Unified Clickhouse metrics export function
• Fixed nasty bug with number of CPUs to build
• Added number of CPUs on machine for debugging
• Added Ubuntu 22.04 native build
• Extended timeout for build as I'm going to try full build for Ubuntu 20.04
• Added libmpfr dependency for Ubuntu builds
• Upgrade to Boost 1.80 and new Boost build
• Get rid of hard coded path to unify build process with Community edition
• Added detailed error about reasons of license extension process failure via sudo fcli show license
• Added field about reasons of license retrieval failure
• Added logic to store reasons of license retrieval process into global variable
• Extracted some logic from retrieve_and_apply_new_license into request_license_from_licensing_server
• Added logic to return sudo fcli show license in human friendly format instead of JSON
• Better logic to return API license status
• Reworked license validation process. Removed logic which refreshes license on daemon start as we have background license refresh process now
• Added logic to import all available dashboards from installer
• Added lots of new dashboards
• Completely eliminated logic which uses Grafana API. Switched to provision from files
• Switched to new provisioning logic for Grafana to address issues with data source creation
• Implemented logic threshold_specific_ban_details for IPv6 when traffic buffer is not in use
• Complete logic for threshold_specific_ban_details in case of IPv4 mode when we do not have traffic buffer logic engaged