Administrations, Logs, Backups, Updates

Application Logs

Our backend image follows the usual logging best practices by writing to stdout, so you can follow logs with docker logs or any other appropriate tool. Apart from some service activity logs, all application logs are JSON formatted.

It is up to you to set up log management (collection, aggregation, storage, retention) the way you want, according to your own log management policy. On our cloud, we use ELK stack for this task.

Backup and Restore

You need to backup:

  • the content of the MongoDB database
  • the folder mounted as a volume in the backend container at the /app/storage location

If one day you need to restore from scratch the Toucan Toco app, you will only need to restore the Mongo database and to recreate a container with the last backed up storage folder mounted in /app/storage.

Choose backup frequency, storage place and retention duration according to your own backup policy.

Toucan Toco does provide a toolbox image to backup and restore your stack.

How to backup a stack or small app using the toolbox image:

To full backup your stack, you only need to run the following command:

$> docker run --rm \
  --network=STACK_NETWORK \
  -v /path/where/to/store/your/backup/:/tmp \
  -v /path/to/stack/shared/volumes/storage:/storage \
  -e "MONGO_URI=mongodb://MONGO_USER:MONGO_PASSWORD@MONGO_HOST:MONGO_PORT" \
  -e "BACKEND_URL_SPECIFIC=https://BACKEND_URL" \
  quay.io/toucantoco/backup:toucan_backend_toolbox_100.5.1-latest \
  toucan_backup

To only backup a specific small app, run the following command:

$> docker run --rm \
  --network=STACK_NETWORK \
  -v /path/where/to/store/your/backup/:/tmp \
  -v /path/to/stack/shared/volumes/storage:/storage \
  -e "MONGO_URI=mongodb://MONGO_USER:MONGO_PASSWORD@MONGO_HOST:MONGO_PORT" \
  -e "BACKEND_URL_SPECIFIC=https://BACKEND_URL" \
  -e "SMALL_APP_SPECIFIC=THE_SMALL_APP_TO_BACKUP" \
  quay.io/toucantoco/backup:toucan_backend_toolbox_100.5.1-latest \
  toucan_backup

After running one of these commands, the backup will be stored locally in /path/where/to/store/your/backup/ under a folder named on-premise_backup-XXXXXXXX where XXXXXXXX is a random string.

The backup will contain the following files:

on-premise_backup-XXXXXXXX
├── laputa_version
├── mongodump.tar.gz # Contains all the related MongoDB data of your stack or small app
└── storage.tar.gz # Contains all the related storage data of your stack or small app

How to restore a stack using the toolbox image:

To restore a full backup made by the toolbox image, you only need to mount the backup as a volume to the docker image and run the following command:

$> docker run --rm \
  --network=STACK_NETWORK \
  -v /path/to/your/backup/on-premise_backup-XXXXXXXX:/tmp/on-premise_backup-XXXXXXXX \
  -v /path/to/stack/shared/volumes/storage:/storage \
  -e "MONGO_URI=mongodb://MONGO_USER:MONGO_PASSWORD@MONGO_HOST:MONGO_PORT" \
  -e "BACKEND_URL_SPECIFIC=https://BACKEND_URL" \
  -e "BACKEND_USER_SPECIFIC=USER" \
  -e "BACKEND_PASSWORD_SPECIFIC=PASSWORD" \
  quay.io/toucantoco/backup:toucan_backend_toolbox_100.5.1-latest \
  toucan_restore /tmp/on-premise_backup-XXXXXXXX /storage

After running the command, the instance will be fully restored to the state of the backup.

To restore a specific small app, run the following command:

$> docker run --rm \
  --network=STACK_NETWORK \
  -v /path/to/your/backup/on-premise_backup-XXXXXXXX:/tmp/on-premise_backup-XXXXXXXX \
  -v /path/to/stack/shared/volumes/storage:/storage \
  -e "MONGO_URI=mongodb://MONGO_USER:MONGO_PASSWORD@MONGO_HOST:MONGO_PORT" \
  -e "BACKEND_URL_SPECIFIC=https://BACKEND_URL" \
  -e "BACKEND_USER_SPECIFIC=USER" \
  -e "BACKEND_PASSWORD_SPECIFIC=PASSWORD" \
  -e "SMALL_APPS_SPECIFIC=THE_SMALL_APP_TO_RESTORE" \
  quay.io/toucantoco/backup:toucan_backend_toolbox_100.5.1-latest \
  toucan_restore /tmp/on-premise_backup-XXXXXXXX /storage

ℹ️ You can change restore the small app with another name by specifying the new name in the SMALL_APPS_NEW_NAME_SPECIFIC environment variable.

After running the command, the small app will be fully restored to the state of the backup.

Information about the Toucan Toco Toolbox

This image is currently used by us to backup and restore stacks of SaaS clients. However, it’s a beta for on-premise users.

Upgrading the Stack

I have a Toucan Toco stack installed. How do I upgrade it?

Toucan Toco releases every month a new version of the Toucan Toco backend and frontend. The packages are automatically available on Quay.io when they are ready.

Usually upgrading the stack is simple as:

  • changing the version of the backend and the frontend container images
  • and restart them

All potential migrations are automatically run at the boot of the backend container.

Please always use a pinned version in your production environment instead of using the latest-monthly tags for idempotent concerns.

Warning

Review the “Breaking changes” section (below) carefully and please run a full backup before starting the upgrade process.

About Toucan Toco Versions

Please note each brick of the Toucan Toco stack (backend and frontend) has its own version.

To be compatible, the major version number should be the same, for example:

  • Backend in v26.0.2 and Frontend in v26.0.7 are compatible, as they are both v26
  • Backend in v25.2.10 and Frontend in v25.1.5 are compatible, as they are both v25
  • Backend in v25.2.11 and Frontend in v26.0.7 are not compatible, as one is v25 and the other is v26

Breaking Changes

v94.0.0

Warning

The MongoDB backend migrates from 4.4 to 5.0

Please refer to the official documentation your Toucan Toco stack.

If you a use docker-compose stack, you only need to update the version of the MongoDB image and as an admin of the service, just launch:

db.adminCommand({
    setFeatureCompatibilityVersion: '5.0'
});

v90.0.0

Headers
We added the header X-JSON-Flavor. It acts as a temporary feature flag to allow passing dates in a specific format through the JSON API. It’ll be removed in the next release.

v89.0.0

Warning

The MongoDB backend migrates from 4.2 to 4.4

Please refer to the official documentation your Toucan Toco stack.

If you a use docker-compose stack, you only need to update the version of the MongoDB image and as an admin of the service, just launch:

db.adminCommand({
    setFeatureCompatibilityVersion: '4.4'
});

v77.0.0

Warning

The MongoDB backend migrates from 4.0 to 4.2

Please refer to the official documentation your Toucan Toco stack.

If you a use docker-compose stack, you only need to update the version of the MongoDB image and as an admin of the service, just launch:

db.adminCommand({
    setFeatureCompatibilityVersion: '4.2'
});

v76.0.0

Following the release of the refresh token, we added encryption in the applicative database for sensitive fields like tokens and client_secrets. As an administrator, running Toucan Toco self-hosted, you now must provide a TOUCAN_DB_ENCRYPTION_SECRET before running the app. An error will show if not provided, preventing it to start.

Choose a strong one! and keep it secret ;)

v60.0.0 Acid Citra Chinook

We completed a series of improvements on our backend Docker image.

  • It is much smaller and we strive to make it pass static security scans.
  • All the configuration states can be passed via environment variables (no need to mount a config file anymore).
  • The features requiring taking screenshots of Toucan pages are supported by a new separate image.
  • All this documentation was updated accordingly, please read before you deploy.

v56.1.0 Chou Bruxellensis

Warning

The MongoDB backend migrates from 3.6 to 4.0

I manage my own MongoDB

Please refer to the official documentation to upgrade your backend.

I don’t manage my own MongoDB

If you use the Toucan Toco shipping script to manage the MongoDB installation and configuration (when "mongo_flag: enable"), everything will be upgraded auto-magically.

Warning

Please ensure that your current version is strictly superior to 44.4.0. otherwise, upgrade first to the 44.4.0 version

v44.4.0 Helpless Supernova

Warning

The MongoDB backend migrates from 3.4 to 3.6

I manage my own MongoDB

Please refer to the official documentation to upgrade your backend.

I don’t manage my own MongoDB

If you use the Toucan Toco shipping script to manage the MongoDB installation and configuration (when "mongo_flag: enable"), everything will be upgraded auto-magically.

Monitoring

When you are in a self-hosted mode, we recommend you to monitor the following parts:

Toucan Toco App

You should monitor:

  • https://api-toucan.domain.com/ returns a 200 HTTP status
  • https://api-toucan.domain.com/status with User Agent Plop 4rQD3KzCxWzTYaRyp0NSEfd6 returns a 200 HTTP status

The status page checks that:

  • MongoDB is up and reachable by the Toucan Toco app
  • Redis is up and reachable by the Toucan Toco app
  • All Toucan Toco app processes are ok

System

Be sure the system is ok by monitoring:

  • free disk space on all partitions
  • load average of the server
  • disk IO
  • CPU, memory and SWAP usage
  • RAID state
  • hardware health

Of course, thresholds directly depend on your own monitoring policy.

Logs

All logs should be monitored (via ELK stack, Logentries,…) to prevent segfault, OOM killer, unexpected behaviors.

MongoDB

You should monitor that:

  • the mongoDB process is up and running
  • the mongoDB port (usually MONGO_PORT) is available and opened from the node where the Toucan Toco app is installed
  • a basic user auth can be completed

Redis

You should monitor that:

  • the redis process is up and running
  • the redis port (usually 6379) is available and opened from the node where the Toucan Toco app is installed

How To Monitor

All we described above can be monitored by the usual Nagios, Zabbix, Shinken, Munin, Collectd plugins.