Distributed Upgrade

Preupgrade advice

Upgrading a distributed infrastructure is more complicated than upgrading a single node installation. Even so, the basic principles stay the same.

When to upgrade

VoipNow Product Management notifies customers when a new version is available using the VoipNowUpdates account on Twitter. It is strongly recommended that you follow this account on Twitter.

At the same time, if you log in to your VoipNow system as Administrator and open the Unified Communications menu, you will see an alert at the top of the interface informing you that an update is available.

Planning

Planning is one of the most important steps of a successful upgrade. Without proper planning, deploying a new version in production might become a very frustrating experience.

Review the release notes

The Release Notes cover the most important things that have been changed in the new version. They include bug fixes as well as major and minor improvements.

They might also contain important advice. For instance, we might tell you that, once you have completed the upgrade, you need to take some extra steps (usually due to changes in behavior).

While we try to provide an accurate description for these changes, sometimes it might be hard to understand them, especially if you do not have any background information. Therefore, if you have any doubt about any item in the Release Notes, simply open a ticket in 4PSA Help Zone and ask 4PSA engineers for help.

Perform a test upgrade

Maintaining a testing environment is absolutely necessary, especially with large installations.

The testing deployment should be used whenever you need to execute tasks that would otherwise be dangerous in the production environment. Such tasks include, but are not necessary limited to:

• testing administration procedures
• testing Apps
• testing various integration
• testing new customer packages and charging strategies

It is essential to keep the configuration of the testing environment very similar to the configuration of the production environment. Otherwise, it might not be very useful.

It is required to start with an upgrade of the testing environment, followed by some smoke tests to detect any issue that might have occurred on account of the upgrade. If you detect any issue on your testing environment, do not hesitate to contact 4PSA Help Zone.

Plan the downtime

The VoipNow distributed installation involves downtime for your customers. While technically it is possible to perform upgrades without any downtime, the procedure is much more complicated and should only be followed in mission critical installations.

In most situations, this downtime takes less than 3 minutes. Therefore, it is strongly recommended to notify your customers about it.

Rollback

Rolling back a production installation might be tricky. Even though there are sufficient technical solutions for fast rollbacks (i.e. snapshots), you are unlikely to rollback an installation when information has been committed to database. Simply because this would mean that the entire information between the snapshot date and the actual time of the rollback will have been lost.

Therefore, do not rely on these facilities, even if they are available. They should only be used as a last resort.

Backup

Maintaining backups is a basic responsibility for any system administrator. 4PSA recommends to conduct backups using CDP (continuous data protection) technologies every several hours.

However, as it happens with rollbacks, you should never rely on backups when fixing upgrade issues. The same loss of information might occur. Therefore, backups too should be used as a last resort.

Upgrade procedure

4PSA recommends that you upgrade the VoipNow distributed installation following the procedure detailed below.

Update the MySQL master node

To update the MySQL Master node, you muse use the provided script and take the steps below:

1. Run the following command.

   curl "http://www.4psa.com/software/voipnowinstaller.sh" -o voipnowinstaller.sh -L

2. Ensure that the MySQL node has the following environment variables.

   export ALT_DB_HOST=<hostname>
   export ALT_DB_PORT=<port>
   export ALT_DB_USER=<user>
   export ALT_DB_PASSWD=<password>
   export ALT_DB_NAME=<database>
   Credentials can be found in voipnowcredentials.conf

3. Then run the following command.

   sh voipnowinstaller.sh mysql

Update the Infrastructure Controller node

The Infrastructure Controller is the first node in the system that must be upgraded after the SQL node (that in case there are MySQL updates). This upgrade does not cause any downtime. However, depending on the database schema upgrades, it might cause inconveniences to the other roles of the system.

The update requires taking the following steps:

1. VoipNow must know where MySQL is installed. For this, you must set the following shell environment variables.

   export ALT_DB_HOST=<hostname>
   export ALT_DB_PORT=<port>
   export ALT_DB_USER=<user>
   export ALT_DB_PASSWD=<password>
   export ALT_DB_NAME=<database>

2. Download the installer.

   curl "http://www.4psa.com/software/voipnowinstaller.sh" -o voipnowinstaller.sh -L

3. Run the installer.

   sh voipnowinstaller.sh

4PSA tries to minimize these inconveniences by reducing the number of schema upgrades and migrating to schema less databases. Check the Release Notes as we will be indicating the impact of the changes on the distributed upgrade. If the upgrade fails or if any problem should occur, do not continue the process. Contact 4PSA Help Zone instead.

Update the remaining roles

We recommend that you upgrade the other nodes of the system at the same time. This can be done if you simultaneously execute the Command Line Installer on all the other nodes.

Various strategies may be used to execute the CLI on multiple nodes:

• execute the CLI directly, using a simple shell script that loops through all your nodes. You can use Capistrano.
• schedule the upgrade on each node by adding the CLI to cron
• use an automation tool such as Puppet to schedule the task

No matter which method you use, make sure that you collect the output of the CLI in order to detect if any problem occurred.

To update any other role, take the steps below:

1. Download the installer script.

   curl "http://www.4psa.com/software/voipnowinstaller.sh" -o voipnowinstaller.sh -L

2. Make sure to set the installer to run in Standby mode.

   export VNSTANDBY=1

3. Finally, run the installer.

   sh voipnowinstaller.sh

Perform configuration changes

If the Release Notes advise you to perform configuration changes, please do accordingly. Usually, this is necessary when upgrading to major new versions.

Test the upgrade

Just like you did when you performed the upgrade on your testing environment, execute the smoke tests that allow you to detect any possible issues.