Feature/param server integration

.github/workflows/create_dbc_and_proto.yml

@@ -3,10 +3,6 @@ on:
   push:
     branches:
       - master
-  pull_request:
-    branches:
-      - master
-  workflow_dispatch:
 
 jobs:
   build-packages:

.vscode/settings.json

@@ -1,3 +1,4 @@
 {
-  "python.pythonPath": "/nix/store/qp5zys77biz7imbk6yy85q5pdv7qk84j-python3-3.11.6/bin/python"
+  "python.pythonPath": "/nix/store/qp5zys77biz7imbk6yy85q5pdv7qk84j-python3-3.11.6/bin/python",
+  "python.analysis.typeCheckingMode": "off"
 }

README.md

@@ -97,23 +97,54 @@ output:
     - once both the data writer and the foxglove websocket have finished processing the data delete the data from the container
 - a desired workflow is that it all we need to do to add a new input that we will be seeing over the wire is to add a .proto to a specific folder. No code changes should be required.
 
+previous flow:
 ```mermaid
+
 flowchart TD
-    CAN[RPI CAN] --> py_async_q[encoded CAN data]
-    py_async_q --> des[DBC based CAN parser] 
-    des --> pb_pack[protobuf packet creation]
-
-    pb_pack --> data_q1[webserver protobuf packet queue]
-    pb_pack --> data_q2[MCAP file writer protobuf packet queue]
-    subgraph websocket thread
-        data_q1 --> enc[serialize into protobuf packet]
-        enc --> py_foxglove[foxglove server websocket]
-    end
-    subgraph file writer thread
-        data_q2 --> py_mcap[MCAP file writer]
+    CAN -- raw CAN message --> dec[CAN decode and pb msg encode]
+    UDP-s[udp socket recv port] --union oneof type msg--> packet_mux[oneof message de-mux]
+
+    subgraph web interface
+
+
+            web_interface[frontend website] -- param change ---> pb_msg[param pb msg]
+            web_interface -- some other command --> pb_msg_other[other command pb msg ...]
+            web_interface -- recorder commands --> mcap_rec_command_queue    
+            pb_msg_other ---> message_packer[pb message serialization handler]
+            pb_msg --> message_packer
+        end
+
+    subgraph data acquisition
+
+
+        mcap_rec_command_queue --> mcap_file_writer
+        packet_mux -- encoded raw pb received msg --> enqueue
+        enqueue -- decoded param pb message -----> web_interface
+        %% packet_mux -- decoded pb msg --> mcap_rec_q[mcap recorder queue]
+
+        mcap_rec_q --> mcap_file_writer
+        enqueue --all decoded pb msg--> mcap_rec_q[mcap recorder queue]
+        enqueue --all encoded pb msg--> live_qlive_q[live pb msg queue]
+
+
+        dec -- encoded pb received CAN msg  --> enqueue[enqueue handler]
+        %% dec --decoded pb msg--> mcap_rec_q
+        %% dec --encoded pb msg--> live_q
+
+
+        live_qlive_q --> foxglove-websocket
+
+        pb_msg -- encoded param pb msg --> enqueue
+        pb_msg_other -- encoded command pb msg --> enqueue
+
     end
+    message_packer -- oneof union message --> udp_send[udp send socket port]
+    mcap_file_writer--> mcap_file[mcap file on car]
+    foxglove-websocket --> fxglv[user foxglove studio viz]
 ```
 
+
+
 ### notes:
 - filter journalctl based on service: `journalctl -u nginx.service`
 

V2.md

@@ -0,0 +1,72 @@
+after struggling a lot with the big ass architecture of this boi using co-routines, im gonna try to switch to a threaded architecture as i know that it will give me what I want directly. I should be able to use pretty much the same architecture, just with different tools cuz ive been treating co-routines like they are threads and it hasnt really been working lol.
+
+multi-threaded handling:
+
+1 thread for all input data that handles receiving, parsing streaming and logging
+	inputs:
+		- CAN data
+		- UDP data
+	- will check for CAN messages and / or UDP messages, parse them into protobuf messages and put them into the output queue
+	- will append to a SINGLE queue the data
+1 thread for handling the streaming of the data into foxglove and the mcap file and output the current parameter / config that gets parsed from the data queue
+	- will consume the main data queue coming from the interface producers 
+
+    - will consume another queue coming from the web interface
+	- will maintain part of the asyncio infrastructure in that the foxglove webserver in that the mcap writer and the foxglove webserver will be in two different tasks
+	- will look to see if the incoming data is a parameter message (or some other msg that the frontend needs) and will output it to the frontend data queue
+
+1 thread for handling the website and the display state 
+	- will wait for the following to trigger a run of the processes:
+		- frontend data queue data receive
+		- user input from the website
+	- will consume the frontend data queue 
+	- will handle the user interaction
+	- will handle the serving of the website
+	- will append to the output data queue data created by the user (config change, etc.)
+
+
+tools I will use:
+
+`threading.Condition()`
+
+condition will be used in the website handling thread 
+```python
+import threading
+
+class OddEvenPrinter:
+    def __init__(self):
+        self.lock = threading.Lock()
+        self.condition = threading.Condition(self.lock)
+        self.number = 1
+
+    def print_odd(self):
+        for _ in range(10):
+            with self.condition:
+                while self.number % 2 == 0:
+                    self.condition.wait()
+                print("Odd:", self.number)
+                self.number += 1
+                self.condition.notify()
+
+    def print_even(self):
+        for _ in range(10):
+            with self.condition:
+                while self.number % 2 != 0:
+                    self.condition.wait()
+                print("Even:", self.number)
+                self.number += 1
+                self.condition.notify()
+
+printer = OddEvenPrinter()
+
+odd_thread = threading.Thread(target=printer.print_odd)
+even_thread = threading.Thread(target=printer.print_even)
+
+odd_thread.start()
+even_thread.start()
+
+odd_thread.join()
+even_thread.join()
+
+print("Printing finished")
+```

default.nix

@@ -4,20 +4,18 @@
 , py_mcap_pkg
 , py_foxglove_websocket_pkg
 , asyncudp_pkg
-, hytech_np_proto_py
 , proto_gen_pkg
-, py_foxglove_protobuf_schemas
-, vn_protos_np_proto_py
+, hytech_eth_np_proto_py
+, hytech_np_proto_py
 }:
 
 python311Packages.buildPythonApplication {
   pname = "py_data_acq";
   version = "1.0.1";
 
   propagatedBuildInputs = [
-    (python311Packages.cantools.overridePythonAttrs (_: { doCheck = false; }))
-    #python311Packages.cantools
-    #python311Packages.systemd   #commented out cuz linux only
+    python311Packages.cantools
+    python311Packages.systemd
     python311Packages.websockets
     python311Packages.pprintpp
     python311Packages.can
@@ -30,11 +28,13 @@ python311Packages.buildPythonApplication {
     py_mcap_pkg
     hytech_np_proto_py
     proto_gen_pkg
-    py_foxglove_protobuf_schemas
-    vn_protos_np_proto_py
     python311Packages.flask
     python311Packages.flask-cors
     python311Packages.hypercorn
+    python311Packages.quart
+
+    python311Packages.asgiref # required for flask async 
+    hytech_eth_np_proto_py
   ];
 
   src = ./py_data_acq;