diff --git a/.github/workflows/sphinx.yml b/.github/workflows/sphinx.yml new file mode 100644 index 0000000..1212193 --- /dev/null +++ b/.github/workflows/sphinx.yml @@ -0,0 +1,33 @@ +name: Sphinx build + +on: + push: + branches: [ "main", "development" ] + pull_request: + branches: [ "main" ] + + +jobs: + build: + runs-on: ubuntu-latest + steps: + # Checkout and build the docs with sphinx + - uses: actions/checkout@v2 + - name: Build HTML + uses: ammaraskar/sphinx-action@master + with: + docs-folder: "docs/user_guide" + # pre-build-command: "mkdir /tmp/sphinx-log" + - name: Upload artifacts + uses: actions/upload-artifact@v1 + with: + name: html-docs + path: docs/user_guide/build/html/ + # Deploys to the gh-pages branch if the commit was made to main, the + # gh-pages then takes over serving the html + - name: Deploy + uses: peaceiris/actions-gh-pages@v3 + if: github.ref == 'refs/heads/main' + with: + github_token: ${{ secrets.GITHUB_TOKEN }} + publish_dir: docs/build/html \ No newline at end of file diff --git a/.gitignore b/.gitignore index 8932e95..806b931 100644 --- a/.gitignore +++ b/.gitignore @@ -2,4 +2,8 @@ .pytest_cache/ *egg-info .venv/ -tests/test_dir* \ No newline at end of file +tests/test_dir* +**/.DS_Store + +build/ +docs/user_guide/build/ \ No newline at end of file diff --git a/README.md b/README.md index d52bc6f..43761c4 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,8 @@ CEDA Near-Line Data Store Client ================================ +For more information please see the [documentation](https://cedadev.github.io/nlds-client/index.html). + This is the client used for interacting with the CEDA Near-Line Data Store ([NLDS](https://github.com/cedadev/nlds)). @@ -9,32 +11,67 @@ It consists of a library part (clientlib) and a command line client NLDS client is built upon [Requests](https://docs.python-requests.org/en/master/index.html) and [Click](https://click.palletsprojects.com/en/8.0.x/) -NLDS client requires Python 3. It has been tested with Python 3.8 and Python 3.9. +NLDS client requires Python 3. It has been tested with Python 3.8, 3.9, 3.10 +and 3.11. Installation ------------ -1. Create a Python virtual environment: - `python3 -m venv ~/nlds-client` - -2. Activate the nlds-client: - `source ~/nlds-client/bin/activate` +1. Create a python virtual environment: + ``` bash + python3 -m venv ~/nlds-client + ``` -3. Install the nlds-client package with editing capability: - `pip install -e ~/Coding/nlds-client` +2. Activate your new virtual environment: + ``` bash + source ~/nlds-client/bin/activate + ``` -Using ------ +3. Install the nlds-client package directly from this github repo: + ``` bash + pip install git+https://github.com/cedadev/nlds-client.git + ``` -Using the pip install method, an alias for the NLDS client is created: `nlds`. -This can be used to `PUT` and `GET` files and filelists to the NLDS server. +If installing this repository for development we recommend you install with the +editable flag (`-e`). Config ------ -NLDS client requires a config file in the user's home directory: `~/.nlds-config`. This file contains information about the JASMIN infrastructure and so is not included in the GitHub repository. A Jinja-2 template is included in the `nlds_client/templates/nlds-config.j2` file. +NLDS client requires a config file in the user's home directory: `~/.nlds-config`. This file contains information about the JASMIN infrastructure and so is not included in the GitHub repository. A Jinja-2 template is included in the `nlds_client/templates/nlds-config.j2` file. + +This config file requires information about the JASMIN OAuth2 authentication server, including the `oauth_client_id`, `oauth_client_secret` and various URLs. +These can be populated by using the command -This config file requires information about the JASMIN OAuth2 authentication server, including the `oauth_client_id`, `oauth_client_secret` and various URLs. +``` +nlds init +``` -When the NLDS client is first used, the user will be prompted to enter their username and password. This is the username and password for the JASMIN accounts portal. +This will need to be pointed at an appropriate NLDS server url but will default to the one hosted on JASMIN - see the [relevant section](https://cedadev.github.io/nlds-client/configuration.html) of the docs for more details. +When the NLDS client is first used, the user will be prompted to enter their username and password. This is the username and password for the JASMIN accounts portal. The token is stored at a location defined in the `~/.nlds-config` file with the key `oauth_token_file_location`. + +To use the NLDS you will also need to provide a `token` and `secret_key` to access the object store you would like to use as an NLDS cache. +Details for how to do this are in the [documentation](https://cedadev.github.io/nlds-client/configuration.html). + +Usage +----- + +Upon installation, an alias for the NLDS client is created: `nlds`. This can be used to `PUT` and `GET` files and filelists to and from an NLDS server, as well as a whole host of other commands. See the [documentation](https://cedadev.github.io/nlds-client/tutorial.html) for an extensive tutorial. +A [reference](https://cedadev.github.io/nlds-client/command_ref.html) of all available commands is also available. + + +Tests +----- + +Automatic unit-testing is run with pytest, to manually run the tests for a local development environment, first ensure the appropriate version of pytest is installed in your venv. From the root of the repository, run: + +``` bash +pip install -r tests/requirements.txt +``` + +and then, similarly from the root of the repo, run the tests with: + +```bash +pytest +``` diff --git a/docs/.nojekyll b/docs/.nojekyll new file mode 100644 index 0000000..e69de29 diff --git a/docs/index.html b/docs/index.html new file mode 100644 index 0000000..0368041 --- /dev/null +++ b/docs/index.html @@ -0,0 +1,8 @@ + + + + + + + + \ No newline at end of file diff --git a/docs/user_guide/requirements.txt b/docs/user_guide/requirements.txt new file mode 100644 index 0000000..e3627be --- /dev/null +++ b/docs/user_guide/requirements.txt @@ -0,0 +1,4 @@ +Sphinx==7.1.2 +sphinx-click==5.1.0 +sphinx-rtd-theme==2.0.0 +-r ../../requirements.txt \ No newline at end of file diff --git a/docs/user_guide/source/catalog_organisation.rst b/docs/user_guide/source/catalog_organisation.rst new file mode 100644 index 0000000..d63a73c --- /dev/null +++ b/docs/user_guide/source/catalog_organisation.rst @@ -0,0 +1,175 @@ +.. _catalog_organisation: + +Catalog Organisation +==================== + +When a user PUTs files to the NLDS, the files are recorded in a catalog on +behalf of the user. The user can then list which files they have in the catalog +and also search for files based on a regular expression. Additionally, users +can associate a label and tags, in the form of *key:value* pairs with a file or +collection of files. + +*Figure 1* shows a simplified version of the structure of the catalog, with just +the information relevant to the user remaining. + +.. figure:: ./simple_catalog.png + + Figure 1: Simplified view of the NLDS data-catalog + +The terms in figure 1 are explained below: + +#. :ref:`Holdings` +#. :ref:`Transactions` +#. :ref:`Tags` +#. :ref:`File` +#. :ref:`Location` + +.. _holding: + +Holdings +-------- + +**Holdings** are collections of files, that the user has chosen to collect +together and assign a label to the collection. A reason to collect files in a +holding might be that they are from the same experiment, or climate model run, +or measuring campaign. Users can give the holding a **label**, but if they do +not then a seemingly random **label** will be assigned to the holding. This is +actually the id of the first **transaction** that created the holding. Users +can change the **label** that a holding has at any time. + +**Holdings** are created when a user PUTs a file into the NLDS, using either the +``nlds put`` or ``nlds putlist`` command. These commands take a **label** +argument with the ``-l`` or ``--label`` option. The first time a user PUTs a +file, or list of files, into a **holding**, the **holding** will be created. +If a **label** is specified then the **holding** will be assigned that **label**. +If a **label** is not specified then the seemingly random **label** will be +assigned. + +After this, if a user PUTs a file into the NLDS and specifies a **label** for a +**holding** that already exists, then the file will be added to that **holding**. +If the **holding** with the specified **label** does not exist then the file +will be added to a new **holding**. This leads to the behaviour that, if a +**label** is not specified when PUTting a file (or list of files) into the NLDS, +a new **holding** will be created for each file (or list of files). + +Reading this, you may ask the question "What happens if I add a file that +already exists in the NLDS?". This is a good question, and a number of rules +cover it: + +1. The ``original_path`` of a file must be unique within a **holding**. An +error is given if a user PUTs a file into a **holding** that already exists and +the file with ``original_path`` already exists in the **holding**. + +2. The ``original_path`` does not have to be unique across **holdings**. +Multiple files with the same ``original_path`` can exist in the NLDS, providing +that they belong to different **holdings**, with different **labels**. + +3. Users can GET files without specifying which **holding** to get them from, +i.e. the ``-l`` or ``--label`` option is not given when ``nlds get`` or ``nlds +getlist`` commands are invoked. In this case, the newest file is returned. + +Organising the catalog in this way means that users can use the NLDS as an +iterative backup solution, by PUTting files into differently labelled +**holdings** at different times. GETting the files will returned the latest +files, while leaving the older files still accessible by specifying the +**holding** **label**. + +.. _transaction: + +Transactions +------------ + +**Transactions** record the user's action when PUTting a file into the NLDS. +As alluded to above, in the :ref:`_holding` section, each **holding** can contain +numerous **transactions**. A **transaction** is created every time a user PUTs +a single file, or list of files, into the NLDS. This **transaction** is assigned +to a holding based on the **label** supplied by the user. If a **label** is +specified for a number of PUT actions, then the **holding** with that label will +contain all the **transactions** arising from the PUT actions. + +A **transaction** contains very little information itself, but its place in the +catalog hierarchy is important. As can be seen in figure 2, it contains a list +of **files** and it belongs to a **holding**. This is the mapping that allows +users to add files to **holdings** iteratively and at different times. For +example, a user may PUT the files ``file_1``, ``file_2`` and ``file_3`` into the +**holding** with ``backup_1`` **label** on the 23rd Dec 2023. The user may then +PUT ``file_4``, ``file_5`` and ``file_6`` into the same **holding** on the 4th +Jan 2024, by specifying the label ``backup_1``. This will have the effect of +creating two **transactions** - one containing ``file_1``, ``file_2`` and ``file_3`` +and the other containing ``file_4``, ``file_5`` and ``file_6``, with the +``backup_1`` **holding** containing both **transactions**. Therefore, all **files** +(``file_1`` through to ``file_6``) are associated with the ``backup_1`` +**holding** at particular ``ingest_times``. + +If, at a later time, the user puts ``file_1`` to ``file_6`` into +another **holding** with a **label** of ``backup_2`` then another +**transaction** will be created with a later ``ingest_time`` and the **files** +will be associated with the **transaction** and the ``backup_2`` **holding**. +The **files** may have changed in the interim and, therefore, the **files** +with the same filenames may be different in ``backup_2`` than they are in +``backup_1``. This is the mechanism by which NLDS allows users to perform +iterative backups and how users can get the latest files, via the ``ingest_time``. + +.. _tags: + +Tags +---- + +NLDS allows the user to associate **tags** with a **holding**, in a +``key:value`` format. For example, a series of **holding** could have **tags** +with the ``key`` as ``experiment`` and ``value`` as the experiment name or +number. + +A **holding** can contain numerous **tags** and these are in addition to the +**holdings** **label**. **Tags** can be used for searching for files in the +``list`` and ``find`` commands. + +.. _file: + +File +---- + +The very purpose of NLDS is the long term storage of **files**, recording their +details in a data catalog and then accessing (GETting) them when they are +required. The **file** object in the data catalog records the details of a +single **file**, including the original path of the file, its size and the +ownership and permissions of the file. Users can GET files in a number of ways, +including by using just the ``original_path`` where the NLDS will return the +most recent file with that path. + +Also associated with **files** is the checksum of the file. NLDS supports +different methods of calculating checksums, and so more than one checksum can +be associated with a single file. + +.. _location: + +Location +-------- + +The user interacts with the NLDS by PUTting and GETting **files**, without knowing +(or caring) where those **files** are stored. From a user view, the **files** are +stored in the NLDS. In reality the NLDS first writes the **files** to *object +storage*. Later the **files** are backed up to *tape storage*. When the NLDS +*object storage* approaches capacity, **files** will be removed from the +*object storage* depending on a policy which takes into account several variables, +including when they were last accessed. If a user subsequently GETs a **file** +that has removed from the *object storage* then the NLDS will first retrive +the **file** from the *tape storage* to the *object storage* before copying it +to the user specified target. + +The **location** object in the Catalog database is associated to a file, and +can have one of three states: + +1. The **file** is held on the *object storage* only. It will be backed up +to the *tape storage* later. + +2. The **file** is held on both the *object storage* and *tape storage*. Users +can access the file without any staging required by the NLDS. + +3. The **file** is held on the *tape storage* only. If a user accesses the +**file** then the NLDS will *stage* it to the *tape storage*, before completing +the GET on behalf of the user. The user does not need to concern themselves +with the details of this. However, accessing a file that is stored only on +*tape* will take longer than if it was held on *object storage*. + + diff --git a/docs/user_guide/source/command_ref.rst b/docs/user_guide/source/command_ref.rst new file mode 100644 index 0000000..3e349e8 --- /dev/null +++ b/docs/user_guide/source/command_ref.rst @@ -0,0 +1,33 @@ +.. _command-ref: + +Command Line Reference +====================== + +The primary method of interacting with the Near-Line Data Store is through a +command line client, which can be installed using the instructions. + +Users must specify a command to the ``nlds`` and options and arguments for that +command. + +``nlds [OPTIONS] COMMAND [ARGS]...`` + +As an overview the commands are: + +Commands: + | ``find Find and list files.`` + | ``get Get a single file.`` + | ``getlist Get a number of files specified in a list.`` + | ``init Set up the nlds client with a config file on first use.`` + | ``list List holdings.`` + | ``meta Alter metadata for a holding.`` + | ``put Put a single file.`` + | ``putlist Put a number of files specified in a list.`` + | ``stat List transactions.`` + +Each command has its own specific options. The argument is generally the file +or filelist that the user wishes to operate on. The full command listing is +given below. + +.. click:: nlds_client.nlds_client:nlds_client + :prog: nlds + :nested: full \ No newline at end of file diff --git a/docs/user_guide/source/conf.py b/docs/user_guide/source/conf.py index a432b94..5dbce95 100644 --- a/docs/user_guide/source/conf.py +++ b/docs/user_guide/source/conf.py @@ -5,11 +5,11 @@ # -- Project information ----------------------------------------------------- # https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information - -project = 'Near-Line Data Store client' -copyright = '2023, Neil Massey and Jack Leland' +project = 'Near-Line Data Store' +copyright = '2023, Centre for Environmental Data Analysis, Science and Technologies Facilities Council, UK Research and Innovation' author = 'Neil Massey and Jack Leland' -release = '0.1.0' +version = '0.1.1' +release = '0.1.1-RC1' # -- General configuration --------------------------------------------------- # https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration @@ -24,5 +24,5 @@ # -- Options for HTML output ------------------------------------------------- # https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output -html_theme = 'alabaster' +html_theme = 'sphinx_rtd_theme' html_static_path = ['_static'] diff --git a/docs/user_guide/source/configuration.rst b/docs/user_guide/source/configuration.rst new file mode 100644 index 0000000..59e296d --- /dev/null +++ b/docs/user_guide/source/configuration.rst @@ -0,0 +1,95 @@ +.. _configuration: + +Configuration File +================== + +When the user first invokes ``nlds`` from the command line or issues a command +from the ``nlds_client.clientlib`` API, a configuration file is required in the +user's home directory with the path: + +``~/.nlds-config`` + +This configuration file is JSON formatted and contains the authentication +credentials required by: + + * The OAuth server + * The Object Storage + +It also contains the default user and group to use when issuing a request to the +NLDS. These can be overriden by the ``-u|--user`` and ``-g|--group`` command +line options. + +Finally, it contains the URL of the server and the API version, and the location +of the OAuth token file that is also created the first time the ``nlds`` command +is invoked. + +An example configuration file is shown below. + +:: + + { + "server" : { + "url" : "{{ nlds_api_url }}", + "api" : "{{ nlds_api_version }}" + }, + "user" : { + "default_user" : "{{ user_name }}", + "default_group" : "{{ user_gws }}" + }, + "authentication" : { + "oauth_client_id" : "{{ oauth_client_id }}", + "oauth_client_secret" : "{{ oauth_client_secret }}", + "oauth_token_url" : "{{ oauth_token_url }}", + "oauth_scopes" : "{{ oauth_scopes }}"", + "oauth_token_file_location" : "~/.nlds-token" + }, + "object_storage" : { + "tenancy" : "{{ object_store_tenancy }}", + "access_key" : "{{ object_store_access_key }}", + "secret_key" : "{{ object_store_secret_key }}" + + }, + "option" : { + "resolve_filenames" : "false" + } + } + +The values for the authentication section have been redacted for this +documentation, we recommend you use the ``init`` command to populate them. + +.. _init: + +The init command +---------------- + +The required values for the ``authentication`` section of the config file can be +obtained via the ``nlds init`` command, which will get values for each of the +the ``oauth_`` keys in the ``authentication`` section of the config and insert +them either (a) into an existing config, or (b) into a blank template config +file if one doesn't already exist at the expected path in the user's home +directory. This will get you started if you don't already have a fully filled-in +config file. Note however that the oauth_token_file_location will not be +altered. For more information about this command please see the relevant part +of the :ref:`API reference` section. + +.. note:: + If you are taking part in the ``nlds`` beta testing program, you will also + need to specify a url with the ``-u`` switch and turn off ssl verification + with the ``-k`` flag, so your init command will look a bit like this: + + .. code-block:: bash + + > nlds init -u {{ nlds-testing-url }} -k + + but with the ``{{ nlds-testing-url }}`` replaced with the url provided to + you by the JASMIN staff. + + +The init command will not populate the ``default_user`` or ``default_group``, so +these will have to be filled in with the relevant user and group-workspace for +your use case. The ``access_key`` and ``secret_key`` will also need to be +populated with a ``token`` and ``secret_key`` generated on the +`object-store portal `_ for the +appropriate ``tenancy`` you wish to use. In most cases this will be the default +``nlds-cache-02-o``, so we recommend that, unless you know what you are doing, +the ``tenancy`` field should just be omitted. diff --git a/docs/user_guide/source/index.rst b/docs/user_guide/source/index.rst new file mode 100644 index 0000000..30a1e81 --- /dev/null +++ b/docs/user_guide/source/index.rst @@ -0,0 +1,41 @@ +.. Near-Line Data Store client documentation master file, created by + sphinx-quickstart on Thu Feb 2 16:17:53 2023. + You can adapt this file completely to your liking, but it should at least + contain the root `toctree` directive. + +Near-Line Data Store documentation +================================== + +The Near-Line Data Store (NLDS) is a multi-tiered storage solution that uses +Object Storage as a front end to a tape library. It catalogs the data as it is +ingested and permits multiple versions of files. It has a microservice +architecture using a message broker to communicate between the parts. +Interaction with NLDS is via a HTTP API, with a Python library and command-line +client provided to users for programmatic or interactive use. + +.. toctree:: + :maxdepth: 1 + :caption: Contents: + + installation.rst + configuration.rst + catalog_organisation.rst + tutorial.rst + status_codes.rst + command_ref.rst + license.rst + +Indices and tables +================== + +* :ref:`genindex` +* :ref:`modindex` +* :ref:`search` + +NLDS was developed at the `Centre for Environmental Data Analysis `_ +with support from the ESiWACE2 project. The project ESiWACE2 has received +funding from the European Union's Horizon 2020 research and innovation programme +under grant agreement No 823988. + +NLDS is Open-Source software with a BSD-2 Clause License. The license can be +read :ref:`here `. \ No newline at end of file diff --git a/docs/user_guide/source/installation.rst b/docs/user_guide/source/installation.rst new file mode 100644 index 0000000..56063e3 --- /dev/null +++ b/docs/user_guide/source/installation.rst @@ -0,0 +1,23 @@ +.. |br| raw:: html + +
+ +.. _installation: + +Installation +============ +To use the NLDS, first you must install the client software. This guide will show +you how to install it into a Python virtual-environment (virtualenv) in your +user space or home directory. + +#. Log onto the machine where you wish to install the NLDS client into your + user space or home directory. + +#. Create a Python virtual environment: |br| + ``python3 -m venv ~/nlds-client`` + +#. Activate your new virtual environment: |br| + ``source ~/nlds-client/bin/activate`` + +#. Install the nlds-client package from github: |br| + ``pip install git+https://github.com/cedadev/nlds-client.git`` diff --git a/docs/user_guide/source/license.rst b/docs/user_guide/source/license.rst new file mode 100644 index 0000000..c819e08 --- /dev/null +++ b/docs/user_guide/source/license.rst @@ -0,0 +1,35 @@ +.. _license: + +NLDS License +------------ + +NLDS is Open Source software made available under a BSD 2-Clause License. + +BSD 2-Clause License +==================== + +Copyright (c) 2019-2023, Centre of Environmental Data Analysis Developers, +Scientific and Technical Facilities Council (STFC), +UK Research and Innovation (UKRI). +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. \ No newline at end of file diff --git a/docs/user_guide/source/simple_catalog.png b/docs/user_guide/source/simple_catalog.png new file mode 100644 index 0000000..c225493 Binary files /dev/null and b/docs/user_guide/source/simple_catalog.png differ diff --git a/docs/user_guide/source/simple_catalog.puml b/docs/user_guide/source/simple_catalog.puml new file mode 100644 index 0000000..36bcf30 --- /dev/null +++ b/docs/user_guide/source/simple_catalog.puml @@ -0,0 +1,37 @@ +@startuml simple_catalog + +object "**Holding**" as holding { + label [STRING](unique with user) +} + +object "**Transaction**" as transaction { + ingest_time [DATETIME] +} + +object "**Tag**" as tag { + key [STRING] + value [STRING] +} + +object "**File**" as file { + original_path [STRING] + size [INT] + user [STRING] + group [STRING] + file_permissions [INT] +} + +object "**Location**" as location { + storage_type [OBJECT_STORAGE|TAPE] + access_time [DATETIME] +} + +object "**Checksum**" as checksum { +} + +transaction "1" *-- "many" file +holding "1" *-- "many" transaction +holding "1" *-- "many" tag +file "1" *-- "many" location +file "1" *-- "many" checksum +@enduml diff --git a/docs/user_guide/source/status_codes.csv b/docs/user_guide/source/status_codes.csv new file mode 100644 index 0000000..fb4a55e --- /dev/null +++ b/docs/user_guide/source/status_codes.csv @@ -0,0 +1,20 @@ +Value,State,Description +``-1``,``'INITIALISING'``,Transaction is starting. +``0``,``'ROUTING'``,Transaction is in the message queue. +``1``,``'SPLITTING'``,Transaction is being split into smaller transactions. The user doesn't have to worry about this. +``2``,``'INDEXING'``,Scanning the files and directories in the PUT transaction. +``3``,``'CATALOG_PUTTING'``,Recording the scanned files into the catalog entry for the transaction. +``4``,``'TRANSFER_PUTTING'``,"Putting the files to the NLDS, actually transferring the data. " +``5``,``'CATALOG_ROLLBACK'``,"Remove, from the catalog, any inaccessible files in the transaction." +``10``,``'CATALOG_GETTING'``,Get a catalog entry. +``11``,``'ARCHIVE_GETTING'``,Getting the files from the NLDS tape store. +``12``,``'TRANSFER_GETTING'``,Getting the files from the NLDS cache. +``20``,``'ARCHIVE_INIT'``,Archive transaction is starting. +``21``,``'CATALOG_ARCHIVE_AGGREGATING'``,Aggregating files for a tape write. +``22``,``'ARCHIVE_PUTTING'``,Putting the aggregated files into the tape store from the NLDS cache. +``23``,``'CATALOG_ARCHIVE_UPDATING'``,Updating the catalog with post-tape-write information. +``40``,``'CATALOG_ARCHIVE_ROLLBACK'``,Remove tape entries from the catalog if something went wrong during tape-write. +``100``,``'COMPLETE'``,Transaction has completed successfully. +``101``,``'FAILED'``,Transaction has failed completely. +``102``,``'COMPLETE_WITH_ERRORS'``,"Transaction has completed, but with some errors." +``103``,``'COMPLETE_WITH_WARNINGS'``,"Transaction has completed, but with some warnings." \ No newline at end of file diff --git a/docs/user_guide/source/status_codes.rst b/docs/user_guide/source/status_codes.rst new file mode 100644 index 0000000..94d4514 --- /dev/null +++ b/docs/user_guide/source/status_codes.rst @@ -0,0 +1,17 @@ +.. _status_codes: + +Status Codes +============ + +When the NLDS is asked to PUT or GET some data, the transaction goes through a +number of states from initialisation to completion. +The state of a transaction can be queried by using the ``nlds stat`` command +in the ``nlds_client``. + +Transaction states +------------------ + +.. csv-table :: + :file: status_codes.csv + :widths: 10 30 60 + :header-rows: 1 \ No newline at end of file diff --git a/docs/user_guide/source/tutorial.rst b/docs/user_guide/source/tutorial.rst new file mode 100644 index 0000000..5c5de2c --- /dev/null +++ b/docs/user_guide/source/tutorial.rst @@ -0,0 +1,947 @@ +Tutorial +======== + +This page is a tutorial on NLDS covering: + +* :ref:`Introduction to the NLDS ` +* :ref:`Setting up the NLDS client ` +* :ref:`Running the NLDS client for the first time ` +* :ref:`How the NLDS data catalog is organised ` +* :ref:`Getting help on the NLDS commands ` +* :ref:`Copying a single file (PUT) to the NLDS ` +* :ref:`Copying a list of files (PUTLIST) to the NLDS ` +* :ref:`Querying the status of a transaction (STAT) ` +* :ref:`Querying the file collections the user holds on the NLDS (LIST) ` +* :ref:`Querying the files the user holds on the NLDS (FIND) ` +* :ref:`Changing the label of a file collection (META)