Update/Upgrade and Rollback procedures

Why staying current matters

Fewer breaking changes to absorb at once. Several releases changed things that older deployments depend on:

Rel-1.5 moved configuration from files to environment variables and docker-compose, and refactored the database field names.
Rel-1.12 added the Customer (Network) ID. LTI_customer_id is now a required agent parameter, and the FAQ lists “customerID is required” as an API error.
Rel-2.5 changed the MCP data layout, so certs and data folders now live under mcp/. It also introduced stacked licenses and an optional random analyzer password.

A site that jumps several releases takes all of these in one maintenance window and has more to troubleshoot if something fails.

Security and licensing.

Rel-2.1 ran vulnerability tests, updated the open-source components, and added secured discovery APIs.
Rel-2.5 added randomized analyzer passwords.
Rel-2.5 added the agent license-expiry mechanism, with warnings in Grafana and a grace period before data stops.
The stack now ships Grafana 12.4.2, InfluxDB 2.8.0, Kafka 8.2.0 and Telegraf 1.38.

Measurement accuracy and reliability. Recent fixes affect data quality directly:

Rel-2.4.1 fixed lingering TCP sockets that stopped LIFBE measurements for about an hour.
Rel-2.4.2 fixed iPerf3 multi-profile timing.
Rel-2.5 fixed iPerf recording only the first agent’s throughput when two agents send at once.
Rel-2.5 fixed delayed iPerf UDP measurements.
Rel-2.5 sends PCI values for 4G, not just 5G.

Capabilities that arrive incrementally.

Rel-2.3: MCP server, new OpenAPI endpoints, IPv6 support for agent-to-reflector connections.
Rel-2.4: enhanced heatmaps, troubleshooting tools, XR-90 support.
Rel-2.5: interruptions dashboard, percentiles in the detailed latency view.

Each release brings new dashboards, APIs and hardware support, and a larger gap means more adoption work in one go.

Support. Customers on current releases are the easiest to diagnose and fix, and the documentation always describes the latest behavior.

How can I update/upgrade my version of the Analyzer, Reflector and QoS Agents?

When a new release is out (see the Release Notes), upgrade the components in this order:

Analyzer
Reflector
QoS Agents

Verify each component before moving on to the next one. If a check fails, follow the rollback procedure for that component.

If you have customized dashboards, contact your LatenceTech vendor before updating the Analyzer.

Before you start

Plan a maintenance window. Measurements are not stored while the Analyzer is being backed up or upgraded.
Have all your current license keys at hand.
Run the Analyzer update as the user who installed the Analyzer, from that user's home directory, which must contain the lti_analyzer/ folder. InfluxDB and Grafana run with that user's ID and their data folders belong to it, so another user would cause permission errors.
If you use a random analyzer password, it must be at least 8 characters long. It is saved in ~/.saved_lti_password on the Analyzer host and is set as LTI_SERVICE_KEY on your QoS Agents.
Note the following values from lti_analyzer/docker-compose.yml: ADMIN_KEY, MCP_TOKEN, LTI_license_key, LTI_password, and LTI_PERCENTILE_FILTER or LTI_percentile_filter. The online update script restores them automatically, but you need them for the offline procedure and to verify the result.
Make sure each host has enough free disk space for the backup. The image archives can be several GB.

1) Back up your system

The backup is stored in ~/lti-rollback/. Do not use ~/lti-backup/: the Analyzer update script writes to that folder and overwrites its content.

A virtual machine snapshot taken before the upgrade is also a valid backup and allows a faster rollback.

Analyzer host

The Analyzer keeps its data in folders under ~/lti_analyzer/ (influxdb/data, grafana/grafana, mcp/certs and mcp/data), so the backup below covers all of it. It also saves two files from your home directory: ~/.cached_lti_license_key, which is mounted into the API container, and ~/.saved_lti_password, which only exists if you use a random password.

Create the backup:

mkdir -p ~/lti-rollback/config
cp ~/.saved_lti_password ~/lti-rollback/config/ 2>/dev/null
cp ~/.cached_lti_license_key ~/lti-rollback/config/ 2>/dev/null

# Stop the whole Analyzer so that the InfluxDB and Grafana files are consistent
docker compose -f ~/lti_analyzer/docker-compose.yml stop

# Save the InfluxDB data
sudo tar -czf ~/lti-rollback/influxdb-data.tar.gz -C ~ lti_analyzer/influxdb/data

# Save the rest of the Analyzer folder (docker-compose.yml, Grafana data, mcp, certs, data, ...)
sudo tar -cf ~/lti-rollback/lti_analyzer-config.tar --exclude='lti_analyzer/influxdb/data' -C ~ lti_analyzer

# Start the Analyzer again
docker compose -f ~/lti_analyzer/docker-compose.yml start

# Save the current container images, to be able to roll back without downloading
docker images --format '{{.Repository}}:{{.Tag}}' | grep -v '<none>' | xargs docker save -o ~/lti-rollback/images-pre-upgrade.tar

Check the backup before going further:

ls -lhA ~/lti-rollback ~/lti-rollback/config
sudo tar -tzf ~/lti-rollback/influxdb-data.tar.gz | head -5
sudo tar -tf ~/lti-rollback/lti_analyzer-config.tar | grep -m3 'lti_analyzer/grafana/grafana'

Each command must print something, and config/ must contain .cached_lti_license_key (and .saved_lti_password if you use a random password).

# The backup contains passwords and keys: restrict access to it
sudo chmod -R go-rwx ~/lti-rollback

Reflector host

Run from the directory that contains lti_reflector.yml:

mkdir -p ~/lti-rollback
cp lti_reflector.yml ~/lti-rollback/lti_reflector.yml.pre-upgrade
docker save -o ~/lti-rollback/reflector-image.tar registry.latence.ca/software/reflector

QoS Agent hosts

Run from the directory that contains lti_qos-agent.yml:

mkdir -p ~/lti-rollback
cp lti_qos-agent.yml ~/lti-rollback/lti_qos-agent.yml.pre-upgrade
docker save -o ~/lti-rollback/qos-agent-image.tar registry.latence.ca/software/qos-agent

2) Upgrade the Analyzer

You do not have customized dashboards (online installation):

Go to the directory that contains the lti_analyzer/ folder (your home directory), download the script and run it with sudo:

cd ~
wget https://api.latence.ca/software/update.sh

sudo bash update.sh

The script asks whether you want to back up your InfluxDB data. Answer y to keep your historical data. If you answer n, the Analyzer is reinstalled with an empty database.

The script then stops and removes your current Analyzer containers and images, reinstalls the latest Analyzer, puts back your InfluxDB data, and restores your license keys, percentile filter, analyzer password, ADMIN_KEY, MCP_TOKEN and MCP data automatically.

Because the script runs the standard installer, you are also asked two questions:

95 percentile filter (true/false): answer with your current setting.
Random password (true/false): this question is only asked if you do not already use a random password (that is, if ~/.saved_lti_password does not exist). Answer false to keep your current password. If you answer true, a new random password is generated and applied, and you must set it as LTI_SERVICE_KEY on every QoS Agent, otherwise they stop sending data.

The installer does not ask for a license key during an update: it reuses the key saved in ~/.cached_lti_license_key. At the end it waits for the license check, which can take a few minutes.

Notes:

Do not interrupt the script. If it is interrupted or fails partway, do not run it again. Follow the rollback procedure.
If the script stays on "Waiting for License logs" for more than a few minutes, check docker logs lti_analyzer-influx-writer-1. If the license reports 0 allowed agents, the script asks for a valid license key.
The backup made by the script (in ~/lti-backup/) only contains the InfluxDB data and two configuration files. It is not a rollback backup: the backup from step 1 is still required.
The script prints "Update completed successfully" even if a step failed. Always run the checks below.

You have customized dashboards:

Contact your LatenceTech vendor before updating.

Offline installation:

Prerequisites:

Having the new Analyzer_offline.zip. It can be downloaded with wget https://api.latence.ca/software/Analyzer_offline.zip or obtained on a USB stick from LatenceTech.
Run all commands from the directory that contains both lti_analyzer/ and Analyzer_offline.zip. In this example, lti_analyzer/ is in your home directory (~/); adapt the commands to your location.
Step 1 (backup) completed.

2a) Delete containers

Warning: this command removes all containers and images of the host. Only run it on a host dedicated to the Analyzer.

# Delete containers and images
docker stop $(docker ps -q) && docker rm $(docker ps -a -q) && docker rmi $(docker images -q) && docker system prune -af

# Cleaning up
sudo rm -rf ~/lti_analyzer/ ~/lti_analyzer.launch.log

2b) Re-installation

# Unzip
unzip Analyzer_offline.zip 2>&1

# Go in the folder
cd Analyzer_offline/

# Execute script
bash install-run-analyzer-offline.sh

# Go back to the previous directory
cd ..

2c) Put back your data

# Stop containers influxdb and influx-writer
docker stop lti_analyzer-influxdb-1
docker stop lti_analyzer-influx-writer-1

# Extract influxdb data
sudo tar -xf ~/lti-rollback/influxdb-data.tar.gz -C ~

# Restarting influxdb and influx-writer
docker start lti_analyzer-influxdb-1
docker start lti_analyzer-influx-writer-1

# Place back the MCP server data, certs and data folders (the ones that exist in your backup)
for d in mcp certs data; do sudo tar -xf ~/lti-rollback/lti_analyzer-config.tar -C ~ lti_analyzer/$d 2>/dev/null; done

If certs/ or data/ exist at the root of lti_analyzer/, move them into the new mcp/ folder structure:

mkdir -p ~/lti_analyzer/mcp
sudo mv ~/lti_analyzer/certs ~/lti_analyzer/mcp/ 2>/dev/null
sudo mv ~/lti_analyzer/data ~/lti_analyzer/mcp/ 2>/dev/null

2d) Restore configuration

# Restore home config files (if backed up)
cp ~/lti-rollback/config/.saved_lti_password ~/.saved_lti_password 2>/dev/null
chmod 600 ~/.saved_lti_password 2>/dev/null
cp ~/lti-rollback/config/.cached_lti_license_key ~/.cached_lti_license_key 2>/dev/null

Edit lti_analyzer/docker-compose.yml and put back the values you noted before the upgrade (ADMIN_KEY, MCP_TOKEN, LTI_license_key, LTI_password, percentile filter).

If you had an analyzer password, re-apply it to Grafana and InfluxDB (replace YOUR_PASSWORD):

GRAFANA_ID=$(docker ps --format "{{.ID}} {{.Image}}" | grep grafana | awk '{print $1}')
INFLUX_ID=$(docker ps --format "{{.ID}} {{.Image}}" | grep influxdb | awk '{print $1}')
docker exec -u root $GRAFANA_ID grafana-cli admin reset-admin-password YOUR_PASSWORD
docker exec $INFLUX_ID influx user password --name LatenceTech --password YOUR_PASSWORD --host http://localhost:8086

2e) Restart services

# Restart influx-writer to apply license key
docker stop lti_analyzer-influx-writer-1
docker start lti_analyzer-influx-writer-1

If you restored a password, relaunch all services:

docker compose -f lti_analyzer/docker-compose.yml up -d

If you only restored ADMIN_KEY or MCP_TOKEN, restart MCP services:

docker compose -f lti_analyzer/docker-compose.yml restart latencetech_mcp chatbot_api

Check the Analyzer before continuing

All Analyzer containers are running (docker ps).
The dashboards load at http://YOUR_ANALYZER_HOST_IP:12021.
Your historical data is displayed.
The license status and expiry date are displayed correctly.

If any of these checks fail, follow the rollback procedure.

3) Upgrade the Reflector

For the reflector you can either use the update script here:

wget https://api.latence.ca/software/update_reflector.sh

bash update_reflector.sh

OR you can do it manually following the steps below:

docker stop lti_reflector && docker rm lti_reflector && docker rmi registry.latence.ca/software/reflector

Delete the old version of the installation file:

rm lti_reflector.yml

You can then re-download the yaml file here and modify it with your License key and ID:

wget https://api.latence.ca/software/lti_reflector.yml

Re-launch the Reflector:

docker compose -f lti_reflector.yml up -d

Offline installation: download Reflector_offline.zip (wget https://api.latence.ca/software/Reflector_offline.zip) or get it from the USB stick, stop and remove the old container as above, unzip the file and run the script inside the folder with bash. The script creates the new image. Then deploy the container by filling lti_reflector.yml as explained in the documentation.

Check the Reflector before continuing

docker compose -f lti_reflector.yml logs

The logs show no errors.

4) Upgrade the QoS Agents

We recommend upgrading one agent first, checking it, and then upgrading the others.

To update the QoS Agent to the latest version, you have to kill and remove the images of the running dockers by the following command:

docker stop lti_qos-agent && docker rm lti_qos-agent && docker rmi registry.latence.ca/software/qos-agent

Delete the old version of the installation file:

rm lti_qos-agent.yml

You can then re-download the yaml file here and modify it with your values:

wget https://api.latence.ca/software/lti_qos-agent.yml

If you use a random analyzer password, set LTI_SERVICE_KEY to the analyzer password. To avoid forgetting a value, compare the new file with your backup:

diff ~/lti-rollback/lti_qos-agent.yml.pre-upgrade lti_qos-agent.yml

And then run the following to download the new images.

docker compose -f lti_qos-agent.yml up -d

Other ways to upgrade an agent:

QoS Agent Management Dashboard: if it is installed on the agent host, use Docker Management > Update to pull the latest container images and restart the services.
Offline installation: download Agent_offline.zip (wget https://api.latence.ca/software/Agent_offline.zip) or get it from the USB stick, stop and remove the old container as above, unzip the file and run the script inside the folder with bash. The script creates the new image. Then deploy the container by filling lti_qos-agent.yml as explained in the documentation.
Cradlepoint and SDK agents: follow the Cradlepoint Deployment and QoS Agent SDK Deployment pages.

Check the QoS Agent

docker compose -f lti_qos-agent.yml logs

The logs show no errors, and the agent's data appears in the dashboards under its agent ID.

5) Final verification

All agents appear in the dashboards and send data.
The Analyzer reports the expected license status.
Keep ~/lti-rollback/ on each host until the system has run normally for a full cycle.

What is the rollback procedure in case of upgrade failure?

You are in charge of maintaining a rollback procedure. We strongly advise to back up the system before upgrading, using step 1 of the upgrade procedure. The procedure below restores the system from that backup and must be performed in the reverse order of the upgrade: QoS Agents, then Reflector, then Analyzer.

Each component can be rolled back on its own:

If the Analyzer upgrade fails, roll back the Analyzer only (nothing else has been upgraded yet).
If the Reflector upgrade fails, roll back the Reflector only.
If a QoS Agent upgrade fails, roll back the affected agents only.
If you decide to abandon the whole upgrade, roll back in this order: QoS Agents, Reflector, Analyzer.

Do not run docker compose pull, the update scripts, or the Update button of the QoS Agent Management Dashboard during a rollback. They would download the new version again.

If the Analyzer update script failed partway, do not run it a second time. Roll back first.

Prerequisites

The ~/lti-rollback/ folder created in step 1 of the upgrade procedure, on each host. This folder is different from ~/lti-backup/, which is used by the Analyzer update script and is not used for rollback.
The saved image archives (images-pre-upgrade.tar, reflector-image.tar, qos-agent-image.tar).

1) Roll back the QoS Agents

On each agent host, from the directory that contains lti_qos-agent.yml:

# Remove the new version
docker stop lti_qos-agent && docker rm lti_qos-agent
docker rmi registry.latence.ca/software/qos-agent

# Restore the previous image and configuration file
docker load -i ~/lti-rollback/qos-agent-image.tar
cp ~/lti-rollback/lti_qos-agent.yml.pre-upgrade lti_qos-agent.yml

# Start the previous version without downloading anything
docker compose -f lti_qos-agent.yml up -d --pull never

If the QoS Agent Management Dashboard is installed, you can also restore a previous configuration version from its Agent Configuration Editor. The container image itself must still be restored with the commands above.

2) Roll back the Reflector

On the reflector host, from the directory that contains lti_reflector.yml:

# Remove the new version
docker stop lti_reflector && docker rm lti_reflector
docker rmi registry.latence.ca/software/reflector

# Restore the previous image and configuration file
docker load -i ~/lti-rollback/reflector-image.tar
cp ~/lti-rollback/lti_reflector.yml.pre-upgrade lti_reflector.yml

# Start the previous version without downloading anything
docker compose -f lti_reflector.yml up -d --pull never

3) Roll back the Analyzer

The backup contains the whole lti_analyzer/ folder (configuration, Grafana data, MCP data, certificates) and the InfluxDB data. If the Analyzer update script failed partway, it may have left copies of the mcp, certs and data folders in your home directory. They are already part of the backup, and you can delete them once the rollback is verified.

Run the docker compose commands below as the user who installed the Analyzer, without sudo. The compose file mounts ${HOME}/.cached_lti_license_key, and with a different home directory Docker would create a folder in place of that file.

# Stop and remove the new version (skip the first command if the folder no longer exists)
docker compose -f ~/lti_analyzer/docker-compose.yml down
docker ps -a --filter name=lti_analyzer -q | xargs -r docker rm -f
sudo rm -rf ~/lti_analyzer ~/lti_analyzer.launch.log

# Restore the previous Analyzer folder (docker-compose.yml, Grafana data, mcp, certs, ...) and images
sudo tar -xf ~/lti-rollback/lti_analyzer-config.tar -C ~
docker load -i ~/lti-rollback/images-pre-upgrade.tar

# Restore the InfluxDB data saved before the upgrade
sudo tar -xf ~/lti-rollback/influxdb-data.tar.gz -C ~

# Restore home config files (before starting the Analyzer: the license key file is mounted into the API container)
cp ~/lti-rollback/config/.saved_lti_password ~/.saved_lti_password 2>/dev/null
chmod 600 ~/.saved_lti_password 2>/dev/null
cp ~/lti-rollback/config/.cached_lti_license_key ~/.cached_lti_license_key 2>/dev/null

# Start the previous version without downloading anything
docker compose -f ~/lti_analyzer/docker-compose.yml up -d --pull never

# Restart influx-writer to apply the license key
docker stop lti_analyzer-influx-writer-1
docker start lti_analyzer-influx-writer-1

If you cannot log in with your previous analyzer password, re-apply it to Grafana and InfluxDB (replace YOUR_PASSWORD):

GRAFANA_ID=$(docker ps --format "{{.ID}} {{.Image}}" | grep grafana | awk '{print $1}')
INFLUX_ID=$(docker ps --format "{{.ID}} {{.Image}}" | grep influxdb | awk '{print $1}')
docker exec -u root $GRAFANA_ID grafana-cli admin reset-admin-password YOUR_PASSWORD
docker exec $INFLUX_ID influx user password --name LatenceTech --password YOUR_PASSWORD --host http://localhost:8086

If you did not save the container images

Reinstall the previous version using the same method you used to install it, then restore your data and configuration from ~/lti-rollback/:

Offline installations: use the previous Analyzer_offline.zip, Reflector_offline.zip and Agent_offline.zip, and follow steps 2c to 2e of the offline Analyzer upgrade.
Online installations: the online scripts always install the latest release. Contact your LatenceTech vendor to obtain the previous version.

After the rollback

Open the dashboards at http://YOUR_ANALYZER_HOST_IP:12021 and confirm that historical data is displayed and that your dashboards and users are present.
Confirm that every agent is sending data, using the dashboards and docker compose -f lti_qos-agent.yml logs.
If you use a random analyzer password, confirm that LTI_SERVICE_KEY on each agent matches the restored password.
Measurements collected between the upgrade and the rollback are not part of the restored database.
Keep ~/lti-rollback/ until the system has run normally for a full cycle.

If the rollback does not restore a working system, send data to support and open a ticket with the LatenceTech support team.