From c941fe4a3829b867c786c838adf938b571ae26ee Mon Sep 17 00:00:00 2001 From: Tyler Gu Date: Tue, 16 Jan 2024 12:46:55 -0600 Subject: [PATCH] Update development doc Signed-off-by: Tyler Gu --- docs/development.md | 75 +++++++++++++++++++++++++-------------------- 1 file changed, 42 insertions(+), 33 deletions(-) diff --git a/docs/development.md b/docs/development.md index 673f619077..ead89c3363 100644 --- a/docs/development.md +++ b/docs/development.md @@ -1,37 +1,46 @@ -For example, to run Acto to test the cass-operator, just run +# Setting up development environment + +## Creating the Python development environment +First install all the prerequisites described in the README.md. + +Acto requires Python version >=3.10. +If your system Python does not satisfy this requirement (Ubuntu20.04 depends on Python3.8), + you would need to manually install another Python and create a venv for developing and using Acto. + +To create a venv for Acto with the right Python installation, +run `python3.10 -m venv /path/to/new/virtual/environment`. +Then activate the venv by running `source .venv/bin/activate`. + +While the virtual environment is active, `python3`/`python` commands would point to the Python in the virtual environment, +and `pip` would install packages inside the virtual environment. +To install the development requirements, run `python3 -m pip install -r requirements-dev.txt`. + +## Coding style +We use several tools to help the development of Acto. +We use `black` to format the code to adhere to the pep8 style, +`isort` to clean the import statements, +`pylint` to lint the source code. +Optionally, we use `mypy` to do type checking. + +Check out the [pyproject.toml](../pyproject.toml) for their configurations. + +We follow [Google's Python Style Guide](https://google.github.io/styleguide/pyguide.html). + +Acto current is going through the process of transitioning from research prototype to a mature + open-source project, so you will find some part of the code not adhering the coding style. +We are updating the codebase progressively, and the coding style is only enforced on the changed files + in the commit. + +## Install the pre-commit hooks +Acto comes with pre-commit hooks to enforce some coding style. +They are run at the time you run `git commit ...`, and would report errors or format code. +To install the pre-commit hooks, run ```sh -python3 -m acto --config data/cass-operator/config.json --num-workers 4 --workdir testrun-cass +pip install pre-commit +pre-commit install ``` -## Interpreting the results produced by Acto - -Acto will first generate a test plan using the operator's CRD and the semantic information. The test plan is serialized at `testrun-cass/testplan.json`. Note that Acto does not run the tests according to the order in the `testplan.json`, the tests are run in a random order at runtime. - -Acto then constructs the number of Kubernetes clusters according to the `--num-workers` argument, - and start to run tests. -Tests are run in parallel in separate Kubernetes clusters. -Under the `testrun-cass` directory, Acto creates directories `trial-XX-YYYY`. `XX` corresponds to the worker id, i.e. `XX` ranges from `0` to `3` if there are 4 workers. -`YYYY` starts from `0000`, and Acto increments `YYYY` every time it has to restart the cluster. This means every step inside the same `trial-xx-yyyy` directory run in the same instance of Kubernetes cluster. - -Inside each `trial-*` directory, you can find the following files: -- `mutated-*.yaml`: These files are the inputs Acto submitted to Kubernetes to run the state transitions. Concretely, Acto first applies `mutated-0.yaml`, and wait for the system to converge, and then applies `mutated-1.yaml`, and so on. -- `system-state-*.json`: After each step submitting `mutated-*.yaml`, Acto collects the system state and store it as `system-state-*.json`. This file contains the serialized state objects from Kubernetes. -- `cli-output-*.log` and `operator-*.log`: These two files contain the command line result and operator log after submitting the input. -- `delta-*.log`: This file contains two parts. This file is for convenient debug purposes, it can be computed from `mutated-*.yaml` and `system-state-*.log`: - - Input delta: The delta between current input and last input - - System delta: The delta between current system state and last system state -- `events-*.log`: This file contains the list of detailed Kubernetes event objects happened after each step. -- `not-ready-pod-*.log`: Acto collects the log from pods which are in `unready` state. This information is helpful for debugging the reason the pod crashed or is unhealthy. -- `result.json`: The result for this trial. It contains results for each oracle Acto runs. If an oracle fails, the corresponding field in the `result.json` would contain an error message. Otherwise, the corresponding field in the `result.json` would be `Pass` or `None`. Note that Acto could write `result.json` even if there is no error (e.g. when the test campaign is finished), in that case every oracle field in the `result.json` will be `Pass` or `None`. Legend for relevant fields in the `result.json`: - - `duration`: amount of time taken for this trial - - `error`: - - `crash_result`: if any container crashed or not - - `health_result`: if any StatefulSet or Deployment is unhealthy, by comparing the ready replicas in status and desired replicas in spec - - `state_result`: consistency oracle, checking if the desired system state matches the actual system state - - `log_result`: if the log indicates invalid input - - `custom_result`: result of custom oracles, defined by users - - `recovery_result`: if the recovery step is successful after the error state - -## Schema Matching and Pruning for Testcase Generation + +# Schema Matching and Pruning for Testcase Generation ![Input Model Diagram](./input_model.jpg) -The diagram illustrates the sequence of operation applied to the `DeterministicInputModel` class from within the `Acto.__init__`, detailing the steps for automatic and manual CRD schema matching and pruning for generating test cases. Furthermore, it provides a visual representation of the class inheritance hierarchy for both schemas and value generator classes. \ No newline at end of file +The diagram illustrates the sequence of operation applied to the `DeterministicInputModel` class from within the `Acto.__init__`, detailing the steps for automatic and manual CRD schema matching and pruning for generating test cases. Furthermore, it provides a visual representation of the class inheritance hierarchy for both schemas and value generator classes.