Role rework, update to latest Roundcube 1.3 #24

ganto · 2017-08-07T05:27:43Z

As @drybjed has started refreshing the mail stack, I thought it would finally be time to also update the roles that I contributed. Starting with the (hopefully) easier one, I reworked the Roundcube role which is currently stuck at upstream release 1.1.x so that it can finally also install the latest 1.3.0 release (Fixes #19).

I also updated the DebOps dependencies and ported the role to debops.php (Fixes #9).

I still have to check how (if) I want to support older distribution releases. At the moment the PR depends on PHP composer which is only available in stretch/xenial and newer.

Fixes "Uncaught Error: Class 'PEAR' not found in /srv/www/roundcube/sites/roundcube.example.com/public/program/lib/Roundcube/bootstrap.php:101"

Add new variables 'roundcube__database_password_path' and 'roundcube__database_name'.

Fixes "Call to undefined function mb_strtolower()"

drybjed · 2017-08-07T06:07:08Z

Very nice work! 👍 I'll look at it in a few days while working on Dovecot support.

ganto · 2017-08-08T06:08:38Z

@cultcom If you like, can you also have a look at the proposed changes? I didn't had time yet, to add an installation method via archive download therefore I couldn't really implement a fix for #17

I know, the PHP package installation is still kind of a mess. Unfortunately it seems that there is now way that we can still support Jessie or comparably old Ubuntu releases as their PHP dependencies are too old and they miss PHP composer. Not sure yet how to handle that. If you want to test, just make sure, that you have Debian Stretch or something comparable new.

drybjed

Very nice work! There are some changes I would like to see, check the other comments.

I'm fine with sticking to the Debian Stable release and dropping support for Oldstable due to composer support. It will happen sooner or later, and since the role is not yet part of the DebOps core, I think that's fine.

drybjed · 2017-08-09T06:40:48Z

defaults/main.yml

+  # - 'openssl'
+  # - 'session'
+  # - 'sockets'
+  # - 'xml'


I'm not sure how that will be parsed by yaml2rst. Perhaps a better way would be to move the list of included packages to the comment above, and make it an inline list instead of one entry for a line, which should make the comment smaller. Ditto for other instances.

I copied that from debops.owncloud. Will have a look at it.

drybjed · 2017-08-09T06:42:53Z

defaults/main.yml

-# -----------------------------------
+# List of additional Debian packages (e. g. language dictionaries) that should
+# be installed with Roundcube.
+roundcube__extra_packages: []


The convention I'm using is to have the list of packages in a variable named with a prefix, ie.:

roundcube__base_packages: [ 'curl', 'file', 'unzip' ]

Then the users are free to use the "normal" variation in an inventory if they wish:

roundcube__packages: []

drybjed · 2017-08-09T06:46:18Z

defaults/main.yml

+#
+# -----------------------------
+#   Packages and installation
+# -----------------------------


You could convert the defaults/main.yml to the current standard by using the section headers with just one line underneath, and include vim fold markers. The whole file should also be enclosed in vim folds with a file header. See for example debops.apt role.

drybjed · 2017-08-09T06:49:46Z

defaults/main.yml

-
-# .. envvar:: roundcube__git_checkout
+                                                                   # ]]]
+# .. envvar:: roundcube__git_checkout [[[
 #
 # Default path where Roundcube source files will be deployed
 roundcube__git_checkout: '{{ roundcube__www + "/sites/" +
                             (roundcube__domain if roundcube__domain is string else roundcube__domain[0]) +
                             "/public" }}'


Perhaps using a domain directly here should be dropped, since the role maintains just one Roundcube install. This would also avoid issues if the roundcube__domain value is changed. Multiple Roundcube installations on one host could be handled via internal LXC containers if necessary.

drybjed · 2017-08-09T06:51:19Z

defaults/main.yml

-roundcube__database_user: 'roundcube'
-
+# Database definition to use from the :envvar:`roundcube__database_map`.
+roundcube__database: 'sqlite-default'


Not necessarily in this PR, but role could check if a MariaDB database is installed on the host via Ansible facts and switch to MariaDB database automatically - one less thing to handle via inventory. You can see how it's done in debops.gitlab role.

drybjed · 2017-08-09T06:54:08Z

defaults/main.yml

+roundcube__database_password_path: '{{ secret + "/credentials/" + ansible_fqdn
+                                       + "/roundcube/" + roundcube__database
+                                       + "/" + roundcube__database_user + "/password" }}'
+                                                                   # ]]]


If the database user account is handled by debops.mariadb role, the password stored in the secret/ directory is in a different directory. Did you test this PR with a MariaDB database?

I don't fully understand. This would give the debops.mariadb role the path to store the password. Works perfectly fine.

See:

https://github.com/debops-contrib/ansible-roundcube/pull/24/files#diff-7eeda618087b49ae876084ab6c73fdbbR232

https://github.com/ganto/ansible-roundcube/blob/5e8c63943219bf8fe298389b18ccb8ce60344a7b/defaults/main.yml#L255

https://github.com/debops-contrib/ansible-roundcube/pull/24/files#diff-5c059070706cb70613b3010f4f0e679eR51

drybjed · 2017-08-09T06:55:49Z

defaults/main.yml

+      fastcgi_busy_buffers_size       256k;
+      fastcgi_temp_file_write_size    256k;
+
+                                                                   # ]]]


With the YAML text block above, the vim fold marker will be included in the configuration file. It won't prevent nginx from working but looks wonky. Put the php_upstream key below the YAML text block to avoid this.

drybjed · 2017-08-09T06:58:35Z

templates/srv/www/sites/composer.json.j2

@@ -0,0 +1,2 @@
+{% set _composer_json = (roundcube__register_composer_json.stdout | from_json) %}
+{{ _composer_json | to_nice_json }}


Why is this needed? It seems to be a roundabout way to copy the contents of the dist file to another file right next to it. Won't simple cp or even fancy copy module with remote_src: True do the trick?

Actually I'm not fully sure 😉 No, of course at the moment not. But I wanted to have a way to influence the composer.json so that I can remove/add some packages according to our needs. E.g. add the LDAP driver. The idea was, to install as much as possible via package manager and only the remaining via composer. But I didn't test yet, if that works.

ganto · 2017-08-14T05:11:35Z

Hmn, after updating the issue in the debops.php role (debops/ansible-php#41), there is the next issue with composer...

Loading composer repositories with package information
Reading composer.json of kolab/net_ldap3 (pear-Net-LDAP3-1.0.0)
Reading composer.json of kolab/net_ldap3 (pear-Net-LDAP3-1.0.1)
Reading composer.json of kolab/net_ldap3 (pear-Net-LDAP3-1.0.2)
Reading composer.json of kolab/net_ldap3 (pear-Net-LDAP3-1.0.3)
Reading composer.json of kolab/net_ldap3 (pear-Net-LDAP3-1.0.4)
Reading composer.json of kolab/net_ldap3 (pear-Net-LDAP3-1.0.5)
Reading composer.json of kolab/net_ldap3 (pear-Net-LDAP3-1.0.6)
Reading composer.json of kolab/net_ldap3 (master)
                                                 
Updating dependencies (including require-dev)
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - Installation request for pear/crypt_gpg ~1.6.0 -> satisfiable by pear/crypt_gpg[v1.6.0].
    - pear/crypt_gpg v1.6.0 requires ext-mbstring * -> the requested PHP extension mbstring is missing from your system.
  Problem 2
    - phpunit/phpunit 5.7.21 requires ext-dom * -> the requested PHP extension dom is missing from your system.
    - phpunit/phpunit 5.7.20 requires ext-dom * -> the requested PHP extension dom is missing from your system.
    - phpunit/phpunit 5.7.19 requires ext-dom * -> the requested PHP extension dom is missing from your system.
    - phpunit/phpunit 5.7.18 requires ext-dom * -> the requested PHP extension dom is missing from your system.
    - phpunit/phpunit 5.7.17 requires ext-dom * -> the requested PHP extension dom is missing from your system.
    - phpunit/phpunit 5.7.16 requires ext-dom * -> the requested PHP extension dom is missing from your system.
    - phpunit/phpunit 5.7.15 requires ext-dom * -> the requested PHP extension dom is missing from your system.
    - Installation request for phpunit/phpunit ~5.7.15 -> satisfiable by phpunit/phpunit[5.7.15, 5.7.16, 5.7.17, 5.7.18, 5.7.19, 5.7.20, 5.7.21].

  To enable extensions, verify that they are enabled in your .ini files:
    - /etc/php/7.0/cli/php.ini
    - /etc/php/7.0/cli/conf.d/00-ansible.ini
    - /etc/php/7.0/cli/conf.d/10-opcache.ini
    - /etc/php/7.0/cli/conf.d/10-pdo.ini
    - /etc/php/7.0/cli/conf.d/20-calendar.ini
    - /etc/php/7.0/cli/conf.d/20-ctype.ini
    - /etc/php/7.0/cli/conf.d/20-curl.ini
    - /etc/php/7.0/cli/conf.d/20-exif.ini
    - /etc/php/7.0/cli/conf.d/20-fileinfo.ini
    - /etc/php/7.0/cli/conf.d/20-ftp.ini
    - /etc/php/7.0/cli/conf.d/20-gd.ini
    - /etc/php/7.0/cli/conf.d/20-gettext.ini
    - /etc/php/7.0/cli/conf.d/20-iconv.ini
    - /etc/php/7.0/cli/conf.d/20-json.ini
    - /etc/php/7.0/cli/conf.d/20-mcrypt.ini
    - /etc/php/7.0/cli/conf.d/20-pdo_sqlite.ini
    - /etc/php/7.0/cli/conf.d/20-phar.ini
    - /etc/php/7.0/cli/conf.d/20-posix.ini
    - /etc/php/7.0/cli/conf.d/20-pspell.ini
    - /etc/php/7.0/cli/conf.d/20-readline.ini
    - /etc/php/7.0/cli/conf.d/20-shmop.ini
    - /etc/php/7.0/cli/conf.d/20-sockets.ini
    - /etc/php/7.0/cli/conf.d/20-sqlite3.ini
    - /etc/php/7.0/cli/conf.d/20-sysvmsg.ini
    - /etc/php/7.0/cli/conf.d/20-sysvsem.ini
    - /etc/php/7.0/cli/conf.d/20-sysvshm.ini
    - /etc/php/7.0/cli/conf.d/20-tokenizer.ini
    - /etc/php/7.0/cli/conf.d/30-memory_limit.ini
  You can also run `php --ini` inside terminal to see which files are used by PHP in CLI mode.

Depends on debops/ansible-php#41

This fixes debops-contrib#15

ganto · 2017-08-21T05:27:08Z

The role should be in an acceptable shape now. It is able to successfully setup Roundcube. Some tasks that are still giving me some headache are the PHP composer and JS downloads. They are not especially safe especially regarding man-in-the-middle and I don't know yet, ho well they can handle a Roundcube update. Any suggestions are welcome.

drybjed

Very nice work!

ganto · 2017-08-25T06:20:27Z

There was one more issue popping up. When upgrading the release, Roundcube needs to run a updatedb.sh script. I added this, but when testing the upgrade on a setup with a remote MySQL database, I get the error:

TASK [debops-contrib.roundcube : Update database schema] *****************************************************************************************************************
fatal: [webmail]: FAILED! => {
  "changed": false,
  "cmd": ["php", "bin/updatedb.sh", "--package=roundcube", "--dir=/srv/www/sites/webmail.example.com/public/SQL"],
  "delta": "0:00:00.057101",
  "end": "2017-08-24 12:57:40.265906",
  "failed": true,
  "rc": 255,
  "start": "2017-08-24 12:57:40.208805",
  "stderr": "ERROR: SQLSTATE[HY000] [2002] Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)Error 500: Error connecting to database: SQLSTATE[HY000] [2002] Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)", "stderr_lines": ["ERROR: SQLSTATE[HY000] [2002] Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)Error 500: Error connecting to database: SQLSTATE[HY000] [2002] Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)"],
  "stdout": "",
  "stdout_lines": []
}

When this is fixed, I'm confident, that this PR is in a state where it can be merged.

ganto · 2017-08-28T16:07:50Z

@drybjed or @ypid: Could you please trigger the documentation rebuild of https://debops-contrib.readthedocs.io/en/latest/? Thanks

ypid · 2017-08-28T17:33:29Z

Very well done @ganto ! Done and up and running. Enjoy :)

ganto · 2017-08-29T06:01:40Z

Thanks, glad you like it 👍

ganto added 23 commits July 26, 2017 18:54

Fix license

e42ae47

Update mariadb users variable

851eb08

Update keepachangelog.com version

52f753a

Make system account more configurable

be3f1d0

Migrate to 'nginx__dependent_upstreams'

abec951

Migrate to 'nginx__dependent_servers'

2e291e1

Add dependency to debops.ferm to allow http/https access

58ee54e

Migrate from 'debops.php5' to 'debops.php' role

194ed45

Define forgotten variable 'roundcube__custom_php_packages'

8c7a4a9

PHP iconv is already included by default

751f885

Add 'debops.apt_preferences' role dependency

195a73f

Set 'roundcube__git_version' to 1.3.0, fix 'roundcube__home'

244e5ff

Cleanup hard-coded php package dependencies

516f9bc

Fix 'roundcube__src' if 'ansible_local.root' is not defined

f87ff35

Re-define php upstream used by nginx

bba4626

Add 'pear' to required PHP packages

e7517dd

Fixes "Uncaught Error: Class 'PEAR' not found in /srv/www/roundcube/sites/roundcube.example.com/public/program/lib/Roundcube/bootstrap.php:101"

Cleanup database definition, fix sqlite driver installation

09c8b31

Add new variables 'roundcube__database_password_path' and 'roundcube__database_name'.

Add 'mbstring' to required PHP packages

8b604f6

Fixes "Call to undefined function mb_strtolower()"

Install some PHP packages via package manager depending on release

709dbc9

Run PHP composer to install missing packages

20037b0

PHP 'zip' is not included in common, add it to optional packages

e758d76

Run post-installation script for javascripts, install deps

fe812a3

Remove support for packaged Roundcube in wheezy

c49c6e7

ganto added 2 commits August 7, 2017 18:22

Update changelog

8d0212f

Only run install-jsdeps.sh when jquery is not found yet

2dc43a3

ganto requested a review from drybjed August 8, 2017 06:03

Re-organize PHP package installation, add composer.phar

c16623c

drybjed requested changes Aug 9, 2017

View reviewed changes

ganto mentioned this pull request Aug 14, 2017

Add filter for commonly bundled PHP packages debops/ansible-php#41

Merged

ganto added 6 commits August 14, 2017 18:22

List all PHP packages also the one's bundled with php-common

0c62680

Depends on debops/ansible-php#41

Fix condition for composer.phar, update docs to be more explicit

2670406

Conditionally set changed state on composer.phar run

cf28e89

Fix composer package installation condition

70ef1c5

Remove roundcube user name from 'roundcube__www'

1f7c642

This fixes debops-contrib#15

Rework meta and update README.md

5e8c639

drybjed approved these changes Aug 21, 2017

View reviewed changes

ganto added 6 commits August 23, 2017 07:32

Add upgrade guide

d4fade3

Fix missing include, add hint about manual SQL update

a1dd00e

Run database upgrade if version update is detected

6a3c299

Adjust SQL upgrade command, fix typo

e2e108d

Fix database upgrade condition for fresh installation

1b8328f

Improve condition for version query on role upgrade

9e2bbf0

ganto mentioned this pull request Aug 23, 2017

Shrink CSS and JS on deployment #17

Open

ganto added 8 commits August 25, 2017 08:31

Make sure upgradedb.sh is run after configuration task

f04e18b

Re-group tasks and add Ansible tags

b7b63b6

Update documentation, add doc for Ansible tags

9be2f6a

Fix links to version diffs

2a554cd

Fix build failure due to copy/paste error

9a8def1

Remove erroneously committed lookup template

6e0dd11

Add hint about future use of the composer.json.j2

98ac47b

Switch rolespec test repository to debops/test-suite

ab029c0

ganto merged commit 0987063 into debops-contrib:master Aug 28, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Role rework, update to latest Roundcube 1.3 #24

Role rework, update to latest Roundcube 1.3 #24

ganto commented Aug 7, 2017

drybjed commented Aug 7, 2017

ganto commented Aug 8, 2017

drybjed left a comment

drybjed Aug 9, 2017

ganto Aug 10, 2017

drybjed Aug 9, 2017

drybjed Aug 9, 2017

drybjed Aug 9, 2017

drybjed Aug 9, 2017

drybjed Aug 9, 2017

ganto Aug 21, 2017 •

edited

Loading

drybjed Aug 9, 2017

drybjed Aug 9, 2017

ganto Aug 10, 2017

ganto commented Aug 14, 2017

ganto commented Aug 21, 2017

drybjed left a comment

ganto commented Aug 25, 2017 •

edited

Loading

ganto commented Aug 28, 2017

ypid commented Aug 28, 2017

ganto commented Aug 29, 2017

		@@ -0,0 +1,2 @@
		{% set _composer_json = (roundcube__register_composer_json.stdout \| from_json) %}
		{{ _composer_json \| to_nice_json }}

Role rework, update to latest Roundcube 1.3 #24

Role rework, update to latest Roundcube 1.3 #24

Conversation

ganto commented Aug 7, 2017

drybjed commented Aug 7, 2017

ganto commented Aug 8, 2017

drybjed left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ganto Aug 21, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ganto commented Aug 14, 2017

ganto commented Aug 21, 2017

drybjed left a comment

Choose a reason for hiding this comment

ganto commented Aug 25, 2017 • edited Loading

ganto commented Aug 28, 2017

ypid commented Aug 28, 2017

ganto commented Aug 29, 2017

ganto Aug 21, 2017 •

edited

Loading

ganto commented Aug 25, 2017 •

edited

Loading