Mongodb DAO Update (#112)

* Updating all versions to the highest possible w/o conflicts.
The goal is to have the mongodb driver to something > 6.0.

* Replacing the 'del' package by the 'rimraf' package to avoid
deprication warnings.

* Updating the production/.env with some reasonable values.
Most important for us is adding the 'COLLECTION_ID=' because
sometimes the test data doesn't have the 'doi' as a key and
we change it to the '_id' value, which is always available
in mongodb. In general we don't understand where the .env
is really used in production. When using docker, docker-compose
or kubernetes, we would specify all environment variable
specifically anyway. Like in '-e' or the environment tag
in docker compose.

* Updating the dao driver code to comply with mongdb >= 6.
 - Using then/catch with mongdb.connect.
 - Using the Promise returned by library calls instead of creating
   our own Promise.
 - It seems that MongoClient.filter is no longer used in the 6.0
   mongodb library. So we change to 'any'. Is of course less
   type save.

* Using the 'date' substructure of releaseDate. Error otherwise.

* TS now requires to use 'const pino ..' instead of 'import pino ..'
for the 'pino' package.

* Adding an upgrade of npm to [email protected].

* Adding a description of the required variables. Can to to README.md
if necessary.
Adding an example on how to start the docker container.

* Formatting fixed.

* a) mongo 'Filter' type replacing 'any' b) Formatting improved

* Mongodb already returns Promises, which we forward to the calling functions.

* This is to get the test running again.
a) Using tsx instead of tsc. Didn't get the mocha to work with tsc.
b) Fixing the the expected return from the oai-pmh API calles.
   - Lower case to the proper answer
   - Removing date structures before the comparison.
c) Reading the dotenv part first before loading the test. Mostly
   to get the proper LOG_LEVEL value.
d) The PUT of the 'test' record into the DB is not defined by
   the V2 oai-pmh protocol and the return of the insertOne mongodb
   call changed. That is reflected in the fixtures ... record-doi.ts
   file.

* Run the TEST

* Deployment taken from BE

* Describing necessary ENV variables

* Was moved to README.md

* Adding the real update time

* Fixing format with XML namespace url

* Applying workflow to master instead of the development branch.

* Formal Fix: Merging MongoClient, Filter import from mongodb.

* Changed default LOG_LEVEL to 'error' in .env and README.md

* Putting { id:1 } projection back to identifiersQuery.

* Switching back to the node:18-slim version as apline seems to produce errors during the githup workflow

* Making the mongoFilter more typesafe with Filter<StatusFilter>), avoiding error message during compilation.

* Only pushing output image to ghcr if we push code to master

* Changing trigger back to master

* Pushing to the GitHub Container Registry only on 'push' not on 'pr'

---------

Co-authored-by: Patrick Fuhrmann <[email protected]>

.github/workflows/ci.yml

@@ -1,6 +1,10 @@
 name: CI
 
-on: [pull_request]
+on:
+  push:
+    branches: [ "master" ]
+  pull_request: 
+    branches: [ "master" ]
 
 jobs:
   test_build:

.github/workflows/deploy.yml

@@ -0,0 +1,42 @@
+name: Deploy
+
+on:
+  push:
+    branches: [ "master" ]
+  pull_request:
+    branches: [ "master" ]
+
+
+jobs:
+  deploy:
+    name: Build and deploy
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
+
+      - name: Login to GHCR
+        uses: docker/login-action@v3
+        with:
+          registry: ghcr.io
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Create image tags
+        id: meta
+        uses: docker/metadata-action@v5
+        with:
+          images: ghcr.io/${{ github.repository }}
+          flavor: latest=true # adds :latest tag to outputs.tags
+          tags: type=sha,format=long,prefix= # adds :<sha> tag to outputs.tags
+
+      - name: Build and push
+        uses: docker/build-push-action@v6
+        with:
+          context: .
+          platforms: linux/amd64,linux/arm64/v8
+          push: ${{github.event_name == 'push' }}
+          tags: ${{ steps.meta.outputs.tags }}

.mocharc.json

@@ -0,0 +1,5 @@
+{
+  "require": "tsx",
+  "extension": ["ts"],
+  "spec": [ "src/server/env.ts" , "test/**/*.ts"]
+}

Dockerfile

@@ -1,10 +1,12 @@
-FROM node:18-alpine
+FROM node:18-slim
 
 WORKDIR /home/node/app
 
 COPY package*.json /home/node/app/
 COPY . /home/node/app/
 
+RUN npm install -g [email protected]
+
 RUN npm ci
 RUN npm run compile
 

README.md

@@ -14,9 +14,10 @@ The National Library of Finland.
 
 ## Dependenices
 
-* Node 8.9.4+
-* Typescript 2.7.2+
-* npm 5.6.0+
+* Node v18.20.38+
+* Typescript 5.5.3+
+* npm 10.8.1+
+* tsx v4.16.2 (for mocha tests only)
 
 ## Capabilities
 
@@ -56,6 +57,69 @@ Add new records to your mongodb instance by HTTP PUT using the following route:
 * `http://localhost:3000/scicat/Publication`
 
 
+## ENVIRONMENT Variables
+
+*System variables**
+ Key | Description | Default
+ --------:|-------------| --------
+DAPP_ID | No Idea | oai-pmh-service
+CONNECTOR | (don't change) | mongodb
+ADMIN_USER_EMAIL | E-Mail address of the admin user | none
+LOG_LEVEL | default/error/warning | error  
+
+**Mongo DB variables**
+ Key | Description | Default
+ --------:|-------------| --------
+CONNECTOR | (don't change) | mongodb
+DB_HOST | Database Hostname | none
+DB_PORT | Database Port | none
+DB_USER | Database Username | none
+DB_PASS | Database Password | none
+DB_URL |  [<user>:<password>@]<host>:<port>/<dbName>| none
+DATABASE | Publication Database | dacat-next
+COLLECTION | Collection to storage Publation Documents| PublishedData
+COLLECTION_ID | Unique Identifier of records | 'doi' 
+BASE_URL | Prefix to link back to this server | http://localhost 
+
+**Note**: When DB_URL is specified, DB_HOST/DB_PORT/DB_USER/DB_PASS and DATABASE are ignored.
+
+**OAI_PMH Listen Port**
+
+ Key | Description | Default
+ --------:|-------------| --------
+HOST_CONFIGURATION | web server configuration | production/host_config.json
+
+The content of the *host_config.json* file are the json encoded variables to steer the web server itself. 
+As far as I can see, the 'host' variable is ignored in the code, but the 'port' variable is honored.
+
+**Example** for a *host_config.json* file.
+```
+{
+   "host": "localhost",
+    "port": 3000
+}
+```
+
+### Docker start example
+Docker start example, assuming there is a network called
+'scicatlive_default'.
+```
+docker run \
+   --network=scicatlive_default  \
+   -e CONNECTOR=mongodb \
+   -e DB_HOST=mongodb \
+   -e DB_PORT=27017   \
+   -e DB_USER=""      \
+   -e DB_PASSWORD=""  \
+   -e DATABASE=dacat-next \
+   -e COLLECTION=PublishedData \
+   -e COLLECTION_ID="_id" \
+   -e HOST_CONFIG=server/host_config.json \
+   -p 7002:3001 \
+   --name=oai-pmh-provider \
+   oai-pmh-provider
+```
+
 ## Run in *production* mode:
 
 At the simplest level:

gulpfile.js

@@ -23,7 +23,6 @@
  */
 
 var gulp = require("gulp");
-var del = require("del");
 var ts = require("gulp-typescript");
 require('source-map-support').install();
 
@@ -47,8 +46,16 @@ gulp.task("copy-production", function() {
     return gulp.src(["production/.env", "production/host_config.json"])
         .pipe(gulp.dest("dist/server"));
 });
-gulp.task("dist-clean", function() {
-    return del(["dist/**/*", "dist/server/.env"]);
-});
 
 
+const rimraf = require('rimraf');
+
+gulp.task("dist-clean", function(done) {
+  try {
+    rimraf.sync('dist');
+  } catch (err) {
+    console.error('Error deleting files:', err);
+  }
+  done();
+});
+