When developing:
python2.7 ./bootstrap.py -c development.cfg
bin/buildout -c development.cfg
bin/supervisord
bin/client1 fg
The supervisor controls only the zeoserver by default.
Updating the theme (this happens automatically when running buildout):
bin/compass compile -c src/dssweb.theme.magic/config.rb --app-dir src/dssweb.theme.magic
Deployments are done via AWS OpsWorks. The current default stack is called
DSS Stack and resides in the us-west-1 (Oregon) region. All configuration
for the servers should be done via the OpsWorks web console. Configuration
changes made via SSH or the EC2 console are likely to be overwritten or lost.
A few important concepts from OpsWorks:
-
Stack: a container for application and server configuration
-
Layer: a set of recipes and dependencies to be run on a server throughout its lifecycle
-
Instance: a description of a server (not the same as an EC2 instance), can be assigned multiple Layers of functionality, and started and stopped. Instance store instances are destroyed when stopped and created when started, EBS instances have persistent storage which is retained across stops and starts.
-
Application: a description of an application, including its code repositories, credentials, and database connections.
-
Recipe: a set of resources to be created/managed and commands to run, defined using the Chef configuration management system.
There are some specific stages of the instance lifecycle which trigger recipes to be run:
-
Setup: The first set of recipes run after an instance boots. These install dependencies and setup services.
-
Configure: Recipes run automatically on all instances whenever a new instance is added or removed from the stack.
-
Deploy: Recipes run when an application is deployed (code is checked out and installed)
-
Shutdown: Recipes run when in an instance is stopped.
The setup and configure stages can be triggered manually at any time using the
Run Command button from the Stack or Deployments panels. Applications
can be deployed from the App or Deployments panels. Specific recipes can
also be run manually using the Run Command button. Similarly, you can
install the dependencies for layers, or update all packages or even the OS
(not recommended) using the Run Command button. The default stack
configuration automatically updates OS packages nightly.
When new code is ready for production, you'll want to update any dependency versions and
tag a new release for deployment. The revisions of released add-ons are set in
cfg/versions.cfg, and the versions of unreleased add-ons are set in cfg/sources.cfg
(for stable unreleased addons) and cfg/production_sources.cfg (for add-ons that are
under active development). Once the version updates and code has been tested, you should
tag a release. First list the existing tags:
git tag
Then create a new tag and push it to the origin:
git tag 3.3.1
git push origin 3.3.1
To deploy the new tag in OpsWorks, edit the plone instances App, and update the
Branch/Revision to your new tag number. Then deploy the app to any running
application instances.
To deploy code changes, navigate to the Deployments section of the OpsWorks
console for the stack, click the Deploy an App button and deploy the Plone Instances app. This will create a fresh clone of the buildout from Github,
and if the configuration has changed, re-run buildout and restart the
instances (with an optional rolling delay) on all selected servers.
The stack is configured from the Stack panel in the OpsWorks console
using the Stack Settings button. From there you can see and edit the JSON
configuration for the stack. For details on specific settings see the
opsworks plone buildout documentation.
To add a new hostname mapping to a specific subsite, simply update the
plone_instances[subsites] mapping in the Custom JSON. For example:
...
"subsites": {".mycustomdomain.ucdavis.edu": "customdomain_path"}
...
That configuration would add a virtual host for *.mycustomdomain.ucdavis.edu
to the site path /Plone/customdomain_path. Once that change is made you can
use the Run Command button from either the Stack or Deployments panel, to
initiate a Configure on all of the servers in the HAProxy layer (which
manages the front end configuration).
There is another mapping in the Custom JSON subsite_config, which allows you
to set further custom nginx configuration declarations for a given subsite.
You can set things like the following:
...
"subsite_config": {
".mycustomdomain.ucdavis.edu":
"\nif ($host ~ '^mycustomdomain.ucdavis.edu$') {\n rewrite ^(.*)$ http://www.mycustomdomain.ucdavis.edu$1 permanent;\n break;\n}\n"
}
...
Which will redirect all requests to mycustomdomain.ucdavis.edu to
corresponding urls on www.mycustomdomain.ucdavis.edu. Once again, you would
use the Configure action to update the nginx configuration and reload the
server.
If including configuration directly in the Custom JSON is not desirable, you can include more complex nginx configuration files which live in the buildout:
...
"subsite_config": {
".mycustomdomain.ucdavis.edu":
"\ninclude /srv/www/plone_instances/current/src/mycustomdomain_config/config\n"
}
...
Here we reference a file included in the src/ directory by mr.developer,
which allows us to pull updates without forcing a deploy. When the included
file is updated, you would need to run an update using
Stack -> Run Command -> Execute Recipes. Enter
plone_buildout::instances-develop-up, plone_buildout::nginx, then under Advanced
enter {"nginx_plone": {"force_reload": true}} into the Custom Chef JSON
and select the servers in the HAProxy layer before exceuting the recipes. This will
update the repositories in src/ and then update the nginx configuration.
Similar to the subsite configuration above, you can set custom nginx
configuration using the nginx_plone["additional_config"] attribute of the
Custom JSON. The nginx server can be updated and reloaded as described above.
Info on writing Nginx configuration directives can be found in the Nginx documentation. Specific info on converting Apache rewrite Rules can be found here. More info on URL rewriting can be found here. When working with rewrites, it's important to keep in mind that If is Evil.
From the Instances panel, click the + Instance button under the Plone Instances layer. This will allow you to add a new server to run Plone
clients. Choose an instance size, availability zone, optionally an SSH key,
and select 24/7 under Advanced -> Scaling type. I recommend choosing
Instance store for a full time instance, for lower costs and better
performing disk caches. Click Add Instance, and then start the new
instance. After a short while, a new instance should be up and running with
an appropriate number of Plone clients automatically registered with the load
balancer.
If you have predictable periods of high load and would like to run additional
servers during those periods, you can use Time-based instances. Follow the
steps above for adding a full time instance, but select Time-based under
Advanced -> Scaling type. Now you can click the instance name, and follow
the link under Time-based (configuration) to set the times and days of the
week during which the instance will run (all times are set in UTC). You will
only be charged for these instances when they are running.
If you want to automatically create a new set of Plone instances when the
server load reaches certain thresholds, you can use Load-based instances.
Follow the instructions above, but select Load-based under Advanced -> Scaling type. For load-based instances I recommend using EBS backed
instances with SSD storage, in order to ensure faster instance creation during
periods of high load. Now you can click the instance name, and follow the
link under Load-based (configuration). There you can enable the instance
and set thresholds for stopping and starting instances based on CPU, RAM, or
Load usage.
You can also use the sub-panels under Instances to manage load-based and
time-based instances.
If you wish to have a high-availability stack, you will want to have servers
with all layers (except DB Packing) in multiple Availability Zones. You may
wish to have two servers with all layers assigned, or two smaller servers
assigned to e.g. HAProxy and Memcached layers, and additional larger
servers assigned to Plone Instances. The DB Packing layer should only be
run on a single server assigned to the Plone Instances layer, regardless of
your instances' configuration. The RDS database should also be setup for
Multi- AZ operation, as it is by default for production use.
By default the production stack is setup with an Elastic Load Balancer layer
which is assigned to the HAProxy layer, and routes traffic among servers in
that layer.
The first step (because it takes a little time to complete) is to clone the
database. Navigate to the RDS console in the appropriate region, select the
desired RDS instance and use the Instance Actions menu to Take Snapshot.
Choose a snapshot name and click Take Snapshot. You should see your named
snapshot in the list under the Snapshots section, and when the snapshot is
completed, you can select it and click Restore Snapshot. This will allow
you to create a new RDS instance from the snapshot. You should need only to
give it a unique identifier and select whether or not it will be Multi-AZ
before clicking "Restore DB Instance". If you want to clone and RDS to
another region, you will first need to "Copy Snapshot" into that region before
restoring it.
If you want to create additional server stacks with their own databases (e.g.
for staging or development), you can use the stack cloning feature. Navigate
to the OpsWorks Dashboard, and use the Actions menu next to the desired
stack and choose clone. By default all attributes of the stack except for
the instances and DB assignments will be cloned, but you can make
modifications to the settings, including the region in which the cloned stack
will be located.
You'll need to add an RDS layer for your new RDS database. Navigate to the
Layers panel and click + Layer at the bottom of the screen. Choose the
RDS tab, select your newly created RDS instance, set the User (usually
root), and Password for the DB (these can be reset in the RDS console for
the DB instance), and click Register with Stack (this will only work once
the RDS clone is complete and in the available state).
You will then need to edit the Plone Instance App from the Apps panel, and
set the Data source type to RDS, the Database instance to your newly
registered RDS server, and Database name to the same value it had in the App
in the original stack (probably plone_rds_primary).
Once that's done, you can assign instances to the layers. Under the
Instances panel, you can add a new instance to a layer (as described above),
and then in later layers, you can choose to add an Existing OpsWorks
instance to add the instance to additional layers. Once your instances are
assigned to all the desired layers, you can start them.
Once you have e.g. a staging stack setup, you should be able to add new RDS
layers for newly cloned RDS database in order to update the data. You'll just
need to update the database section in the Apprun a Configure command on
the stack to deploy the configuration changes.
The initial RDS DB has 200 GB of allocated storage, but is easily upgradeable.
From the RDS console, you can select the RDS instance and choose Instance Actions -> Modify to modify the parameters of the DB. From there you can
upgrade the DB instance class (CPU and RAM), and the allocated storage size
and type (I do not recommend changing the storage type for General Purpose
SSD). Additionally, you can update the password, backup settings, and DB
parameters. Unless there is an urgent need to upgrade immediately, it is best
to allow the modifications to occur during the weekly maintenance window. If
the modifications are needed urgently, then you can check the Apply Immediately box to apply the changes immediately, which may result in some
performance degradation, depending on the actions being taken.
I recommend performing rolling server upgrades. You can create a new instance and assign it to all the layers of the instance it is intended to replace, and start it. Once it is available, you can stop the older instance.
If donwtime isn't a problem, you can simply stop the instance, edit it to change the instance type, and then start it again. There will be some limitations on instance type, based on the originally selected instance type (e.g. whether it is EBS or instance store backed)
You can create a staging stack by cloning the existing stack (minus the application and instances).
Once that's done, you can create a clone of the production DB by navigating to the snapshots section of the RDS control panel, selecting the latest nightly snapshot and using 'Restore Snapshot' to create a new RDS instance from that snapshot. Generally speaking you'll want to make staging a non Multi-AZ RDS instance in the same Availability Zone as your stack instance with maintenance and backups disabled. This process can take up to 15 minutes, depending on the size of the DB.
Once that's done you can create an RDS Layer in you Stack pointing to your new RDS instance, and an App pointing to the RDS DB and the Repo/Branch for the buildout you want to use for staging. Finally, you can add an instance to the stack layers (no need to include the packing/backup layers for staging). You'll probably also want to edit to nginx configuration in the Stack configuration, since most of the virtualhosting config is not relevant to a staging server.
Once the stack is configured, you can stop and start the instance as needed. You can also delete the RDS instance, if you don't want it running. When starting up the staging stack again, you'll want to first create a new RDS instance from a snapshot, and update the RDS layer in the stack to point to your new RDS instance before starting the stack instances.