From 63dadda40b954d5ae5c635da4487dd9b16338711 Mon Sep 17 00:00:00 2001 From: Adrian Coveney Date: Wed, 24 Mar 2021 16:24:17 +0000 Subject: [PATCH 01/11] Add AMS prereqs to migration doc --- migrating_to_ams.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/migrating_to_ams.md b/migrating_to_ams.md index 9d0ba131..6c5ecb79 100644 --- a/migrating_to_ams.md +++ b/migrating_to_ams.md @@ -1,5 +1,18 @@ # Migrating from using EGI ActiveMQ Message Brokers to using EGI ARGO Messaging Service +## Prerequisites for using AMS + +- A valid host certificate from an IGTF Accredited CA. +- A GOCDB 'Site' entry flagged as 'Production'. +- A GOCDB 'Service' entry of the correct service type flagged as 'Production'. The following service types are used: + - For Grid accounting use 'gLite-APEL'. + - For Cloud accounting use 'eu.egi.cloud.accounting'. + - For Storage accounting use 'eu.egi.storage.accounting'. +- The 'Host DN' listed in the GOCDB 'Service' entry must exactly match the certificate DN of the host used for accounting. Make sure there are no leading or trailing spaces in the 'Host DN' field. +- Messages sent via AMS must be below 1 Megabyte in size, and the messaging service is optimised around this limit. If your messages start hitting this limit when using SSM, see the advice at the bottom of this document. + +## Software requirements + Migration requires upgrading APEL SSM to at least version 2.4.0, installing the ARGO AMS Library, and adding new values to your configuration. The ARGO AMS Library is available in UMD as `python-argo-ams-library`. Versions above 0.5.0 are recommended. From a6fdc07d69034083b37c7d0cc15679fd60ec0f79 Mon Sep 17 00:00:00 2001 From: Adrian Coveney Date: Wed, 24 Mar 2021 16:25:21 +0000 Subject: [PATCH 02/11] Tweak migration doc hierarchy --- migrating_to_ams.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/migrating_to_ams.md b/migrating_to_ams.md index 6c5ecb79..18f412b8 100644 --- a/migrating_to_ams.md +++ b/migrating_to_ams.md @@ -17,7 +17,9 @@ Migration requires upgrading APEL SSM to at least version 2.4.0, installing the The ARGO AMS Library is available in UMD as `python-argo-ams-library`. Versions above 0.5.0 are recommended. -## Sender +## Configuration changes + +### Sender The sender configuration is usually found under `/etc/apel/sender.cfg`. Follow the steps below to migrate. @@ -55,7 +57,9 @@ The next time `ssmsend` runs it should be using the AMS. You can check this by l 2018-09-19 14:18:07,862 - ssmsend - INFO - ======================================== ``` -## Receiver +### Receiver + +This is only used for the central Accounting Repository, Accounting Portal, and regional accounting servers. 1. Follow the steps 1 to 4 as per the [Sender documentation](#Sender) but editing your receiver configuration instead, usually found under `/etc/apel/receiver.cfg`, naming the section `[receiver]` rather than `[sender]`. 1. Change `destination` to be the subscription you are using to pull messages down. From c5cf8a92e45df50cf1b730dadee60ae1f00bc897 Mon Sep 17 00:00:00 2001 From: Adrian Coveney Date: Wed, 24 Mar 2021 16:26:12 +0000 Subject: [PATCH 03/11] Add guidance when messages are too large --- migrating_to_ams.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/migrating_to_ams.md b/migrating_to_ams.md index 18f412b8..1c47187f 100644 --- a/migrating_to_ams.md +++ b/migrating_to_ams.md @@ -67,3 +67,20 @@ This is only used for the central Accounting Repository, Accounting Portal, and ``` token: your_token_here ``` + +## Issues + +### Messages too large + +- Cloud sites using cASO should ensure they are using at least version 1.4.0 of cASO as this version limits the number of records in a message. +- Grid sites using the APEL accounting client need to be using APEL 1.9.0 and SSM 3.2.0. They can then modify their APEL client script, usually located at `/usr/bin/apelclient`. At the moment, this requires a manual change, but will become a configuration option in the next version of APEL. For example, to halve the number of records per message from the default of 1000, add the line `unloader.records_per_message = 500` after the call to `DbUnloader`: + ``` + @@ -233,6 +233,7 @@ def run_client(ccp): + + unloader = DbUnloader(db, unload_dir, include_vos, exclude_vos, + local_jobs, withhold_dns) + + unloader.records_per_message = 500 + try: + if interval == 'latest': + msgs, recs = unloader.unload_latest(table_name, send_ur) + ``` From cdb28cb38dbceaa23729626e55b3eb3b04efffe6 Mon Sep 17 00:00:00 2001 From: Adrian Coveney Date: Thu, 25 Mar 2021 16:23:14 +0000 Subject: [PATCH 04/11] Add openssl as dependency for rpm and deb builds openssl is a dependency of SSM, but it is called using suprocess so it's not the most obvious failure if it's not installed. --- apel-ssm.spec | 2 +- scripts/ssm-build-deb.sh | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/apel-ssm.spec b/apel-ssm.spec index b8ea0e86..e2fcaa2e 100644 --- a/apel-ssm.spec +++ b/apel-ssm.spec @@ -21,7 +21,7 @@ BuildArch: noarch BuildRequires: python-devel %endif -Requires: stomppy < 5.0.0, python-daemon, python-ldap +Requires: stomppy < 5.0.0, python-daemon, python-ldap, openssl Requires(pre): shadow-utils %define ssmconf %_sysconfdir/apel diff --git a/scripts/ssm-build-deb.sh b/scripts/ssm-build-deb.sh index 7e996616..1a283067 100755 --- a/scripts/ssm-build-deb.sh +++ b/scripts/ssm-build-deb.sh @@ -58,6 +58,7 @@ fpm -s python -t deb \ --depends python-ldap \ --depends libssl-dev \ --depends libsasl2-dev \ +--depends openssl \ --deb-changelog $SOURCE_DIR/ssm-$TAG/CHANGELOG \ --python-install-bin /usr/bin \ --python-install-lib $PYTHON_INSTALL_LIB \ From a233c7db6170f44a6084502625fa92a54bebb837 Mon Sep 17 00:00:00 2001 From: Adrian Coveney Date: Fri, 26 Mar 2021 16:53:30 +0000 Subject: [PATCH 05/11] Add additional tests for check_cert_key - Add tests to complete check of ordering of cert and key. - Add checks of invalid cert and key files. --- test/test_crypto.py | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/test/test_crypto.py b/test/test_crypto.py index 1f8adbc0..da85b4ad 100644 --- a/test/test_crypto.py +++ b/test/test_crypto.py @@ -4,6 +4,7 @@ import logging import os from subprocess import call, Popen, PIPE +import tempfile import quopri from ssm.crypto import check_cert_key, \ @@ -75,6 +76,19 @@ def test_check_cert_key(self): if check_cert_key(TEST_CERT_FILE, TEST_CERT_FILE): self.fail('Accepted certificate as key.') + # Check incorrect ordering of cert and key path arguments. + self.assertFalse(check_cert_key(TEST_KEY_FILE, TEST_KEY_FILE), + 'Accepted key as cert.') + self.assertFalse(check_cert_key(TEST_KEY_FILE, TEST_CERT_FILE), + 'Accepted key and cert wrong way round.') + + # Check behaviour with an invalid cert or key file. + with tempfile.NamedTemporaryFile() as tmp: + self.assertFalse(check_cert_key(tmp.name, TEST_KEY_FILE), + 'Accepted invalid cert file.') + self.assertFalse(check_cert_key(TEST_CERT_FILE, tmp.name), + 'Accepted invalid key file.') + if not check_cert_key(TEST_CERT_FILE, TEST_KEY_FILE): self.fail('Cert and key match but function failed.') From 21da9db5da697293939ca56337a712768bbdbcb4 Mon Sep 17 00:00:00 2001 From: Adrian Coveney Date: Fri, 26 Mar 2021 17:55:02 +0000 Subject: [PATCH 06/11] Refactor check_cert_key tests to modern Python --- test/test_crypto.py | 40 +++++++++++++++++++--------------------- 1 file changed, 19 insertions(+), 21 deletions(-) diff --git a/test/test_crypto.py b/test/test_crypto.py index da85b4ad..2f613ca7 100644 --- a/test/test_crypto.py +++ b/test/test_crypto.py @@ -60,38 +60,36 @@ def tearDown(self): os.remove(self.ca_certpath) def test_check_cert_key(self): - ''' - This will print an error log message for the tests that are - supposed to fail; you can ignore it. - ''' - - # One version of the method would have passed this, because of the - # way it checked for validity. - try: - if check_cert_key('hello', 'hello'): - self.fail('Accepted non-existent cert and key.') - except CryptoException: - pass - - if check_cert_key(TEST_CERT_FILE, TEST_CERT_FILE): - self.fail('Accepted certificate as key.') - - # Check incorrect ordering of cert and key path arguments. + """Check that valid cert and key works.""" + self.assertTrue(check_cert_key(TEST_CERT_FILE, TEST_KEY_FILE), + 'Cert and key match but function failed.') + + def test_check_cert_key_invalid_paths(self): + """Check invalid file paths don't return True.""" + self.assertFalse(check_cert_key('hello', 'hello'), + 'Accepted invalid file paths.') + self.assertFalse(check_cert_key(TEST_CERT_FILE, 'k'), + 'Accepted invalid key path.') + self.assertFalse(check_cert_key('c', TEST_KEY_FILE), + 'Accepted invalid cert path.') + + def test_check_cert_key_arg_order(self): + """Check incorrect order of cert and key path args doesn't succeed.""" + self.assertFalse(check_cert_key(TEST_CERT_FILE, TEST_CERT_FILE), + 'Accepted certificate as key.') self.assertFalse(check_cert_key(TEST_KEY_FILE, TEST_KEY_FILE), 'Accepted key as cert.') self.assertFalse(check_cert_key(TEST_KEY_FILE, TEST_CERT_FILE), 'Accepted key and cert wrong way round.') - # Check behaviour with an invalid cert or key file. + def test_check_cert_key_invalid_files(self): + """Check behaviour with an invalid cert or key file.""" with tempfile.NamedTemporaryFile() as tmp: self.assertFalse(check_cert_key(tmp.name, TEST_KEY_FILE), 'Accepted invalid cert file.') self.assertFalse(check_cert_key(TEST_CERT_FILE, tmp.name), 'Accepted invalid key file.') - if not check_cert_key(TEST_CERT_FILE, TEST_KEY_FILE): - self.fail('Cert and key match but function failed.') - def test_sign(self): ''' I haven't found a good way to test this yet. Each time you sign a From 014fc47f640d2709ea16f39ebf945acae8f2d3f6 Mon Sep 17 00:00:00 2001 From: Carlos A Date: Sat, 4 Apr 2020 15:43:29 +0200 Subject: [PATCH 07/11] Generalize certificate and a key matching Change to comparing the public keys of the cert and key as this allows for both RSA and EC keys. --- ssm/crypto.py | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/ssm/crypto.py b/ssm/crypto.py index 4000e6d7..e9380396 100644 --- a/ssm/crypto.py +++ b/ssm/crypto.py @@ -63,24 +63,23 @@ def check_cert_key(certpath, keypath): if cert == key: return False - p1 = Popen(['openssl', 'x509', '-noout', '-modulus'], + p1 = Popen(['openssl', 'x509', '-pubkey', '-noout'], stdin=PIPE, stdout=PIPE, stderr=PIPE, universal_newlines=True) - modulus1, error = p1.communicate(cert) + pubkey1, error = p1.communicate(cert) if error != '': log.error(error) return False - p2 = Popen(['openssl', 'rsa', '-noout', '-modulus'], + p2 = Popen(['openssl', 'pkey', '-pubout'], stdin=PIPE, stdout=PIPE, stderr=PIPE, universal_newlines=True) - modulus2, error = p2.communicate(key) + pubkey2, error = p2.communicate(key) if error != '': log.error(error) return False - return modulus1.strip() == modulus2.strip() - + return pubkey1.strip() == pubkey2.strip() def sign(text, certpath, keypath): """Sign the message using the certificate and key in the files specified. From 9779dc0b0e3e66d37f954260cf11aa0e053676a4 Mon Sep 17 00:00:00 2001 From: Adrian Coveney Date: Tue, 30 Mar 2021 11:16:44 +0100 Subject: [PATCH 08/11] Update version numbers for rc1 --- apel-ssm.spec | 4 ++-- scripts/ssm-build-deb.sh | 2 +- scripts/ssm-build-rpm.sh | 2 +- ssm/__init__.py | 2 +- 4 files changed, 5 insertions(+), 5 deletions(-) diff --git a/apel-ssm.spec b/apel-ssm.spec index e2fcaa2e..bfabf03f 100644 --- a/apel-ssm.spec +++ b/apel-ssm.spec @@ -4,8 +4,8 @@ %endif Name: apel-ssm -Version: 3.2.0 -%define releasenumber 1 +Version: 3.2.1 +%define releasenumber 0.1.rc1 Release: %{releasenumber}%{?dist} Summary: Secure stomp messenger diff --git a/scripts/ssm-build-deb.sh b/scripts/ssm-build-deb.sh index 1a283067..f51fbcdb 100755 --- a/scripts/ssm-build-deb.sh +++ b/scripts/ssm-build-deb.sh @@ -16,7 +16,7 @@ set -eu -TAG=3.2.0-1 +TAG=3.2.1-0.1.rc1 SOURCE_DIR=~/debbuild/source BUILD_DIR=~/debbuild/build diff --git a/scripts/ssm-build-rpm.sh b/scripts/ssm-build-rpm.sh index d4e3aa8f..7b9ca6e0 100644 --- a/scripts/ssm-build-rpm.sh +++ b/scripts/ssm-build-rpm.sh @@ -10,7 +10,7 @@ rpmdev-setuptree RPMDIR=/home/rpmb/rpmbuild -VERSION=3.2.0-1 +VERSION=3.2.1-0.1.rc1 SSMDIR=apel-ssm-$VERSION # Remove old sources and RPMS diff --git a/ssm/__init__.py b/ssm/__init__.py index 0dfbaae1..d5a80aac 100644 --- a/ssm/__init__.py +++ b/ssm/__init__.py @@ -19,7 +19,7 @@ import logging import sys -__version__ = (3, 2, 0) +__version__ = (3, 2, 1) LOG_BREAK = '========================================' From 2af50fdc184e284d7bccc2732c8d80b8967168ec Mon Sep 17 00:00:00 2001 From: Adrian Coveney Date: Fri, 23 Apr 2021 12:23:55 +0100 Subject: [PATCH 09/11] Update version numbers for release --- apel-ssm.spec | 2 +- scripts/ssm-build-deb.sh | 2 +- scripts/ssm-build-rpm.sh | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/apel-ssm.spec b/apel-ssm.spec index bfabf03f..3ac4b52c 100644 --- a/apel-ssm.spec +++ b/apel-ssm.spec @@ -5,7 +5,7 @@ Name: apel-ssm Version: 3.2.1 -%define releasenumber 0.1.rc1 +%define releasenumber 1 Release: %{releasenumber}%{?dist} Summary: Secure stomp messenger diff --git a/scripts/ssm-build-deb.sh b/scripts/ssm-build-deb.sh index f51fbcdb..f5a070d6 100755 --- a/scripts/ssm-build-deb.sh +++ b/scripts/ssm-build-deb.sh @@ -16,7 +16,7 @@ set -eu -TAG=3.2.1-0.1.rc1 +TAG=3.2.1-1 SOURCE_DIR=~/debbuild/source BUILD_DIR=~/debbuild/build diff --git a/scripts/ssm-build-rpm.sh b/scripts/ssm-build-rpm.sh index 7b9ca6e0..3e652bac 100644 --- a/scripts/ssm-build-rpm.sh +++ b/scripts/ssm-build-rpm.sh @@ -10,7 +10,7 @@ rpmdev-setuptree RPMDIR=/home/rpmb/rpmbuild -VERSION=3.2.1-0.1.rc1 +VERSION=3.2.1-1 SSMDIR=apel-ssm-$VERSION # Remove old sources and RPMS From dfc59ebe8d4491a7898b60a1bb02d39013401440 Mon Sep 17 00:00:00 2001 From: Adrian Coveney Date: Wed, 28 Apr 2021 12:08:52 +0100 Subject: [PATCH 10/11] Update changelogs for 3.2.1 --- CHANGELOG | 4 ++++ apel-ssm.spec | 4 ++++ 2 files changed, 8 insertions(+) diff --git a/CHANGELOG b/CHANGELOG index 80e5eeff..60e3edc0 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -1,5 +1,9 @@ Changelog for ssm ================= +* Wed Apr 28 2021 Adrian Coveney - 3.2.1-1 + - Changed certificate and key comparison to allow both RSA and EC keys. + - Corrected dependencies to include OpenSSL. + * Thu Mar 18 2021 Adrian Coveney - 3.2.0-1 - Added logging of what certificate DNs/subjects are being used to facilitate troubleshooting. diff --git a/apel-ssm.spec b/apel-ssm.spec index 3ac4b52c..9a78c28e 100644 --- a/apel-ssm.spec +++ b/apel-ssm.spec @@ -100,6 +100,10 @@ rm -rf $RPM_BUILD_ROOT %doc %_defaultdocdir/%{name} %changelog +* Wed Apr 28 2021 Adrian Coveney - 3.2.1-1 + - Changed certificate and key comparison to allow both RSA and EC keys. + - Corrected dependencies to include OpenSSL. + * Thu Mar 18 2021 Adrian Coveney - 3.2.0-1 - Added logging of what certificate DNs/subjects are being used to facilitate troubleshooting. From 35fbccfa0ff446d3de6c3016ab72e7136ca494e0 Mon Sep 17 00:00:00 2001 From: Adrian Coveney Date: Wed, 28 Apr 2021 12:20:47 +0100 Subject: [PATCH 11/11] Add OpenSSL as prereq in README --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 94c9bea5..d818e358 100644 --- a/README.md +++ b/README.md @@ -32,6 +32,7 @@ SSM is provided by [STFC](https://stfc.ukri.org/), a part of [UK Research and In The EPEL repository must be enabled. This can be done by installing the RPM for your version of SL, which is available on this page: http://fedoraproject.org/wiki/EPEL +You will also need to have the OpenSSL library installed. Other prerequisites are listed below. The Python STOMP library (N.B. versions between 3.1.1 (inclusive) and 5.0.0 (exclusive) are currently supported)