Upgrades #

We plan two comprehensive upgrades per year for all of our clients. We ask them to budget for between 2 and 4 upgrades as security releases may be required in between planned upgrades.

By comprehensive upgrade, we mean upgrading all of the following components to the latest compatible version:

• CiviCRM
• CiviCRM extensions
• the CMS (typically WordPress)
• any CMS plugins, themes and modules
• PHP

In some instances we may choose to hold back one of more of the above components, due to known bugs or incompatibilities.

Whenever possible we upgrade all clients at the same time. This is to ensure that, wherever possible, all clients are using the same versions of software, which makes it easier to support them. But there may be exceptions to this rule, for example, if a particular client needs a newly released feature, we may choose to upgrade them earlier than others.

We may sometimes choose a different month than the default month. This could happen, for example if we have recently upgraded a site due to a security release.

Security releases #

Important security upgrades for various parts of our technology stack are frequently released outside of our preferred upgrade schedule. In many cases, these need to be applied as soon as possible.

For all sites, we should limit the amount of plugins and extensions that we use to reduce the likelihood that we need need to carry out a security upgrade outside of our planned release.

Once we are aware of a security release of a component, we need to establish which clients are vulnerable to the issue that the security release fixes.

Client are not necessarily vulnerable - it depends on how they are using that component. We can assess whether they are vulnerable by understanding the security release and considering how the client is using the system. If it isn't practical to do this, we should just upgrade them anyway.

For each client that IS vulnerable we:

1. Decide on a upgrade strategy (see below)
2. Do the upgrade
3. Document the upgrade

For each client that is NOT vulnerable

1. Document why the client isn't vulnerable
2. Let the client know why they aren't vulnerable
3. Silence any security warnings if possible

CMS security releases #

CMS security releases are usually more straightforward to apply. It is often a case of applying the latest patch release.

The aim is to apply the update as quickly as possible by making minimal code changes to reduce the risk of breakages. In most cases we will apply the update directly to the site without getting client QA and only testing internally.

The process is as follows:

• Perform the update on dev and commit to git
  • Update the civicrm-docker image
  • Update the CMS
  • Check for security releases for plugins/modules (but don't update if no security release)
• Apply update to test
• Minimal internal QA
• Apply update to prod

You can check for WordPress plugin vulnerabilities using Wordfence. For Drupal modules, drush ups will indicate if there are security releases (or look at Available Updates in the UI).

CiviCRM security releases #

Because of CiviCRM's release cycle, updates are likely to be more than a simple patch and will involve changing minor version of CiviCRM (e.g. from 5.43 to 5.47). This is more likely to break things. So the upgrade will be similar to a comprehensive upgrade and require client QA.

If possible, we should try to keep the code changes to the minimum necessary. In particular, if the patch is simple, we may be able to patch the affected files rather than performing a full upgrade.

The process is as follows:

• Perform update on dev and commit to git
  • Update the civicrm-docker image
  • Update CiviCRM core
  • Check release notes for CiviCRM extensions - apply if necessary (e.g. older version isn't compatible with latest CiviCRM version)
  • Check for CMS security releases
  • Check for plugin/module security releases
• Apply update to test
• More extensive internal QA
• Client QA
• Apply update to prod

PHP security release #

These should be pretty straightforward to apply. A new docker image is created periodically with the latest PHP patch version. This should be rolled out to all clients when a security patch is released.

The process is as follows:

• Update the civicrm-docker image on dev and run a quick check
• Update the image on test
• Minimal internal QA
• Apply update to prod

MySQL security release #

As above a new docker image is created periodically with the latest MySQL patch version. There is only one MySQL instance so this only needs to be done once for all clients.

The process is as follows:

• Update the civicrm-mysql image on dev and run a quick check
• Update the image on test
• Minimal internal QA
• Apply update to prod

cd src/civicrm-mysql
docker compose pull
docker compose up -d

Upgrade process #

Below are the stages of the upgrade process. Each client is likely to have different steps in this process so it should be customised per client.

Preparation #

• Inform client and agree timetable for upgrade
• Read the client wiki
• Read the release notes
  • CiviCRM
  • Version specific upgrade steps
  • CiviCRM extensions
  • WordPress
  • Drupal
  • WordPress plugins / Drupal modules
• Sync from prod to dev ./sync

CiviCRM #

• Check for changes to core overrides with com.klangsoft.overrides
• Review message template customisations (if any)
• Download CiviCRM and localisation files from
  • CiviCRM Releases
  • ESR Releases
• Delete civicrm folder and replace with latest version
• Copy l10n folder from localisation files into civicrm folder
• Upgrade the CiviCRM database docker compose exec -u civicrm app cv updb
• Re-apply patches to core files (if necessary)
• Replace the phpcs.xml file in WordPress sites at /src/wordpress/wp-content/plugins/civicrm/civicrm/phpcs.xml - see Coding standards
• Merge changes into extensions that override core files (if necessary)
• Merge changes into customised message templates (if necessary)
• Check for and fix missing indices - see dev/core#4544

  docker compose exec -u civicrm app cv api System.getmissingindices 
  docker compose exec -u civicrm app cv api System.updateindexes
  

• Update the wiki as required

CiviCRM extensions #

• Apply updates via the UI
• Re-apply patches to extensions (if required)
• Merge core changes into overrides (if not done above)

$civicrm_setting['domain']['ext_repo_url'] = 'https://civicrm.org/extdir/ver={ver}|cms={uf}|ready=';

Drupal #

• Check for updates docker compose exec -u civicrm app drush ups
• Update Drupal core docker compose exec -u civicrm app drush up drupal
• Update modules docker compose exec -u civicrm app drush up [module]

WordPress #

• Check for updates docker compose exec -u civicrm app wp core check-update
• Update WordPress (minor) docker compose exec -u civicrm app wp core update --minor
• Update WordPress (major) docker compose exec -u civicrm app wp core update
• Update plugins docker compose exec -u civicrm app wp plugin update --all
• Update themes docker compose exec -u civicrm app wp theme update --all
• Update translations - via UI

Docker #

• Update PHP version in docker-compose.*.yml to highest supported by CiviCRM
• Update docker image

  docker compose pull
  docker compose up -d
  

Reset test branch to master #

Over time the test branch may diverge from master. There may be MRs that have been deployed to test that haven't been deployed to master. This can cause problems when testing, particularly if there are CiviCRM core overrides that don't match the upgrade branch.

To avoid this it is best to reset the test branch to master so that there is a clean test environment and we are only testing what is in the upgrade.

Resetting the branch means that any code in the test branch will be lost so you may want to compare the two branches (test and master) before doing this to see what the differences are. You can also create a branch (e.g. test-old) so that you don't lose these changes. If needed other MRs can be re-applied to test after the upgrade testing is complete.

The steps in GitLab are:

• (optional) create a new branch test-old from test
• delete the test branch
• create a new test branch from master

You will then need to update the test branch on the test server:

git fetch --all
git checkout master
git branch -D test
git checkout test

Deploy to test #

• Sync from prod ./sync
• Clear caches

  docker compose exec -u civicrm app wp cache flush
  docker compose exec -u civicrm app drush cc all
  docker compose exec -u civicrm app cv flush
  

• Pre-upgrade steps - e.g. uninstalling extensions/plugins/modules
• Merge branch into test
• Update code git pull test
• Update CiviCRM database docker compose exec -u civicrm app cv updb - best to update CiviCRM before CMS
• Update WordPress database docker compose exec -u civicrm app wp core update-db
• Update Drupal database docker compose exec -u civicrm app drush updb
• Update docker image

  docker compose pull
  docker compose up -d
  

• Post-upgrade steps - e.g. installing new extensions/plugins/modules

Deploy to production #

• Finalise date with client - preferably in the moring at the start of the week - avoid deployment on a Friday or immediately before holiday
• Inform client of expected downtime and whether staff/visitors can access the site during the upgrade
• Take a backup docker compose exec -u civicrm app civicrm-docker-dump
• Pre-upgrade steps - e.g. uninstalling extensions/plugins/modules
• Merge branch into master
• Update code git pull master
• Update CiviCRM database docker compose exec -u civicrm app cv updb - best to update CiviCRM before CMS
• Update WordPress database docker compose exec -u civicrm app wp core update-db
• Update Drupal database docker compose exec -u civicrm app drush updb
• Update docker image

  docker compose pull
  docker compose up -d
  

• Post-upgrade steps - e.g. installing new extensions/plugins/modules

GitLab issue template #

This checklist can be copied and pasted into the GitLab issue. More details about specific steps are found above.

## Release notes

_Put a summary of the key changes here. With links for further information if relevant._

## Versions

CiviCRM is being upgraded from _Old Version_ to _New version_
WordPress is being upgraded from _Old Version_ to _New version_

## Task list

Preparation

- [ ] Inform the client of the planned upgrade and timeline
- [ ] Sync dev instance from production
- [ ] Create a branch in GitLab for the upgrade
- [ ] Read the client wiki for information about any overrides or specific upgrade steps
- [ ] Read the release notes
  - [ ] CiviCRM
  - [ ] Version specific upgrade steps
  - [ ] CiviCRM extensions
  - [ ] CMS
  - [ ] CMS plugins/modules

CiviCRM

- [ ] Review core overrides
- [ ] Review message template customisations
- [ ] Upgrade CiviCRM core and localisation files
- [ ] Re-apply patches to core files (if any)
- [ ] Review the CiviCRM extensions in use
  - [ ] remove any that are no longer needed
- [ ] Update CiviCRM extensions
- [ ] Re-apply patches to CiviCRM extensions (if any)
- [ ] Replace `/src/wordpress/wp-content/plugins/civicrm/civicrm/phpcs.xml` (WordPress only)
- [ ] Merge changes to core files into extension overrides
- [ ] Merge core changes into message templates
- [ ] Check for and fix missing indices (if needed)
  ```sh
  docker compose exec -u civicrm app cv api System.getmissingindices 
  docker compose exec -u civicrm app cv api System.updateindexes
  ```
- [ ] Update the client wiki

CMS

- [ ] Update the CMS
- [ ] Re-apply any patches to the CMS (if any)
- [ ] Review the CMS plugins
  - [ ] Remove any that are no longer needed
- [ ] Update CMS plugins
- [ ] Re-apply patches to CMS plugins (if any)
- [ ] Update the documentation about CMS and plugin overrides

Docker

- [ ] Update PHP to highest version supported by CiviCRM
- [ ] Pull latest `civicrm-docker` image

Testing / QA

- [ ] Reset `test` branch to `master`
- [ ] Sync test server from production
- [ ] Pre-upgrade steps - e.g. uninstalling extensions/plugins/modules (if any)
- [ ] Merge MR into `test` branch and deploy to test server
- [ ] Apply database updates
  ```shell
  # CiviCRM and extensions
  docker compose exec -u civicrm app cv updb
  # WordPress and plugins
  docker compose exec -u civicrm app wp core update-db
  # Drupal and modules
  docker compose exec -u civicrm app drush updb
  ```
- [ ] Review update messages and add to issue for client to review (if relevant)
- [ ] Update docker image
  ```shell
  docker compose pull
  docker compose up -d
  ```
- [ ] Check for and fix missing indices (if needed)
  ```sh
  docker compose exec -u civicrm app cv api System.getmissingindices 
  docker compose exec -u civicrm app cv api System.updateindexes
  ```
- [ ] Post-upgrade steps - e.g. installing new extensions/plugins/modules (if any)
- [ ] Run internal QA
- [ ] Fix any issues arising from internal QA
- [ ] Ask client to run QA
- [ ] Fix any issues arising from client QA
- [ ] Perform additional QA and fixing as required

Schedule production upgrade

- [ ] Finalise date with client
- [ ] Inform client of expected downtime and whether staff/visitors can access the site during upgrade

Production upgrade

- [ ] Take backup
- [ ] Pre-upgrade steps - e.g. uninstalling extensions/plugins/modules (if any)
- [ ] Merge MR into `master` branch and deploy to production server
- [ ] Apply database updates
  ```shell
  # CiviCRM and extensions
  docker compose exec -u civicrm app cv updb
  # WordPress and plugins
  docker compose exec -u civicrm app wp core update-db
  # Drupal and modules
  docker compose exec -u civicrm app drush updb
  ```
- [ ] Update docker image
  ```shell
  docker compose pull
  docker compose up -d
  ```
- [ ] Check for and fix missing indices (if needed)
  ```sh
  docker compose exec -u civicrm app cv api System.getmissingindices 
  docker compose exec -u civicrm app cv api System.updateindexes
  ```
- [ ] Post-upgrade steps - e.g. installing new extensions/plugins/modules (if any)
- [ ] Run internal QA
- [ ] Ask client to run QA
- [ ] Update history page in client wiki

Post upgrade

- [ ] Debrief internally and with the client
- [ ] Update the testing checklist as appropriate
- [ ] Review any issues in GitLab marked `~after-upgrade`