Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move config documentation into docstrings #920

Merged
merged 1 commit into from
May 10, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 1 addition & 31 deletions docs/Configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,37 +41,7 @@ of configurations of `class<BrowserParams>`.
- [`save_content`](#save_content)


## Platform Configuration Options

- `data_directory`
- The directory into which screenshots and page dumps will be saved
- [Intended to be removed by #232](https://github.com/mozilla/OpenWPM/issues/232)
- `log_path` -> supported file extensions are `.log`
- The path to the file in which OpenWPM will log. The
directory given will be created if it does not exist.
- `failure_limit` -> has to be either of type `int` or `None`
- The number of command failures the platform will tolerate before raising a
`CommandExecutionError` exception. Otherwise the default is set to 2 x the
number of browsers plus 10. The failure counter is reset at the end of each
successfully completed command sequence.
- For non-blocking command sequences that cause the number of failures to
exceed `failure_limit` the `CommandExecutionError` is raised when
attempting to execute the next command sequence.
- `testing`
- A platform wide flag that can be used to only run certain functionality
while testing. For example, the Javascript instrumentation
[exposes its instrumentation function](https://github.com/mozilla/OpenWPM/blob/91751831647c37b769f0039d99d0a164384c76ae/automation/Extension/firefox/data/content.js#L447-L449)
on the page script global to allow test scripts to instrument objects
on-the-fly. Depending on where you would like to add test functionality,
you may need to propagate the flag.
- This is not something you should enable during normal crawls.
- `process_watchdog`
- It is part of default manager_params. It is set to false by default which can manually be set to true.
- It is used to create another thread that kills off `GeckoDriver` (or `Xvfb`) instances that haven't been spawned by OpenWPM. (GeckoDriver is used by Selenium to control Firefox and Xvfb a "virtual display" so we simulate having graphics when running on a server).
- `memory_watchdog`
- It is part of default manager_params. It is set to false by default which can manually be set to true.
- A watchdog that tries to ensure that no Firefox instance takes up too much memory. It is set to false by default
- It is mostly useful for long running cloud crawls
<!--- ## Platform Configuration Options -->

## Browser Configuration Options

Expand Down
16 changes: 16 additions & 0 deletions openwpm/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -120,15 +120,31 @@ class ManagerParams(DataClassJsonMixin):
default=Path.home() / "openwpm",
metadata=DCJConfig(encoder=path_to_str, decoder=str_to_path),
)
"""The directory into which screenshots and page dumps will be saved"""
log_path: Path = field(
default=Path.home() / "openwpm" / "openwpm.log",
metadata=DCJConfig(encoder=path_to_str, decoder=str_to_path),
)
"""The path to the file in which OpenWPM will log. The
directory given will be created if it does not exist."""
testing: bool = False
"""A platform wide flag that can be used to only run certain functionality
while testing. For example, the Javascript instrumentation"""
memory_watchdog: bool = False
"""A watchdog that tries to ensure that no Firefox instance takes up too much memory.
It is mostly useful for long running cloud crawls"""
process_watchdog: bool = False
"""- It is used to create another thread that kills off `GeckoDriver` (or `Xvfb`) instances that haven't been spawned by OpenWPM. (GeckoDriver is used by
Selenium to control Firefox and Xvfb a "virtual display" so we simulate having graphics when running on a server)."""
num_browsers: int = 1
_failure_limit: Optional[int] = None
"""- The number of command failures the platform will tolerate before raising a
`CommandExecutionError` exception. Otherwise the default is set to 2 x the
number of browsers plus 10. The failure counter is reset at the end of each
successfully completed command sequence.
- For non-blocking command sequences that cause the number of failures to
exceed `failure_limit` the `CommandExecutionError` is raised when
attempting to execute the next command sequence."""

@property
def failure_limit(self) -> int:
Expand Down