Administrations, Logs, Backups, Updates¶

Application Logs¶

Our backend image follows the usual logging best practices by writing to stdout, so you can follow logs with docker logs or any other appropriate tool. Apart from some service activity logs, all application logs are JSON formatted.

It is up to you to set up log management (collection, aggregation, storage, retention) the way you want, according to your own log management policy. On our cloud, we use ELK stack for this task.

Backup and Restore¶

You need to backup:

the content of the MongoDB database
the folder mounted as a volume in the backend container at the /app/storage location

If one day you need to restore from scratch the Toucan Toco app, you will only need to restore the Mongo database and to recreate a container with the last backed up storage folder mounted in /app/storage.

Choose backup frequency, storage place and retention duration according to your own backup policy.

Toucan Toco does provide a toolbox image to backup and restore your stack.

How to backup a stack or small app using the toolbox image:¶

To full backup your stack, you only need to run the following command:

$> docker run --rm \
  --network=STACK_NETWORK \
  -v /path/where/to/store/your/backup/:/tmp \
  -v /path/to/stack/shared/volumes/storage:/storage \
  -e "MONGO_URI=mongodb://MONGO_USER:MONGO_PASSWORD@MONGO_HOST:MONGO_PORT" \
  -e "BACKEND_URL_SPECIFIC=https://BACKEND_URL" \
  quay.io/toucantoco/backup:toucan_backend_toolbox_100.5.1-latest \
  toucan_backup

To only backup a specific small app, run the following command:

$> docker run --rm \
  --network=STACK_NETWORK \
  -v /path/where/to/store/your/backup/:/tmp \
  -v /path/to/stack/shared/volumes/storage:/storage \
  -e "MONGO_URI=mongodb://MONGO_USER:MONGO_PASSWORD@MONGO_HOST:MONGO_PORT" \
  -e "BACKEND_URL_SPECIFIC=https://BACKEND_URL" \
  -e "SMALL_APP_SPECIFIC=THE_SMALL_APP_TO_BACKUP" \
  quay.io/toucantoco/backup:toucan_backend_toolbox_100.5.1-latest \
  toucan_backup

After running one of these commands, the backup will be stored locally in /path/where/to/store/your/backup/ under a folder named on-premise_backup-XXXXXXXX where XXXXXXXX is a random string.

The backup will contain the following files:

on-premise_backup-XXXXXXXX
├── laputa_version
├── mongodump.tar.gz # Contains all the related MongoDB data of your stack or small app
└── storage.tar.gz # Contains all the related storage data of your stack or small app

How to restore a stack using the toolbox image:¶

To restore a full backup made by the toolbox image, you only need to mount the backup as a volume to the docker image and run the following command:

$> docker run --rm \
  --network=STACK_NETWORK \
  -v /path/to/your/backup/on-premise_backup-XXXXXXXX:/tmp/on-premise_backup-XXXXXXXX \
  -v /path/to/stack/shared/volumes/storage:/storage \
  -e "MONGO_URI=mongodb://MONGO_USER:MONGO_PASSWORD@MONGO_HOST:MONGO_PORT" \
  -e "BACKEND_URL_SPECIFIC=https://BACKEND_URL" \
  -e "BACKEND_USER_SPECIFIC=USER" \
  -e "BACKEND_PASSWORD_SPECIFIC=PASSWORD" \
  quay.io/toucantoco/backup:toucan_backend_toolbox_100.5.1-latest \
  toucan_restore /tmp/on-premise_backup-XXXXXXXX /storage

After running the command, the instance will be fully restored to the state of the backup.

To restore a specific small app, run the following command:

$> docker run --rm \
  --network=STACK_NETWORK \
  -v /path/to/your/backup/on-premise_backup-XXXXXXXX:/tmp/on-premise_backup-XXXXXXXX \
  -v /path/to/stack/shared/volumes/storage:/storage \
  -e "MONGO_URI=mongodb://MONGO_USER:MONGO_PASSWORD@MONGO_HOST:MONGO_PORT" \
  -e "BACKEND_URL_SPECIFIC=https://BACKEND_URL" \
  -e "BACKEND_USER_SPECIFIC=USER" \
  -e "BACKEND_PASSWORD_SPECIFIC=PASSWORD" \
  -e "SMALL_APPS_SPECIFIC=THE_SMALL_APP_TO_RESTORE" \
  quay.io/toucantoco/backup:toucan_backend_toolbox_100.5.1-latest \
  toucan_restore /tmp/on-premise_backup-XXXXXXXX /storage

ℹ️ You can change restore the small app with another name by specifying the new name in the SMALL_APPS_NEW_NAME_SPECIFIC environment variable.

After running the command, the small app will be fully restored to the state of the backup.

Information about the Toucan Toco Toolbox¶

This image is currently used by us to backup and restore stacks of SaaS clients. However, it’s a beta for on-premise users.

Upgrading the Stack¶

I have a Toucan Toco stack installed. How do I upgrade it?

Toucan Toco releases every month a new version of the Toucan Toco backend and frontend. The packages are automatically available on Quay.io when they are ready.

Step 1: Update the deployment configuration file

In the docker-compose.yaml, hosted on your deployment machine, you’ll find a field called image. This is where you can replace the existing image with a more recent version.

All potential migrations are automatically run at the boot of the backend container.

Warning

Review the “Breaking changes” section (below) carefully and please run a full backup before starting the upgrade process.

Here’s a quick list of link to find the most recent versions:

For redis: https://hub.docker.com/_/redis/tags
For mongodb: https://hub.docker.com/_/mongo
For laputa (backend): https://quay.io/repository/toucantoco/backend?tab=tags
For tucana (frontend): https://quay.io/repository/toucantoco/frontend?tab=tags
For the screenshot service (pdf renderer): https://quay.io/repository/toucantoco/screenshot?tab=tags

For example, I’m updating the backend from v111.0.0 to v111.0.137:

 version: '3'

 services:
   mongo:
     image: mongo:5.0.4
     expose:
       - "27017"
   redis:
     image: redis:6.2.1-alpine
     expose:
       - "6379"
   backend:
-   image: quay.io/toucantoco/backend:v111.0.0
+   image: quay.io/toucantoco/backend:v111.0.137
     environment:
       TOUCAN_PUPPETEER_URL: http://screenshot:3000/
       TOUCAN_DB_ENCRYPTION_SECRET: "ChangeItForProductionContext"
     depends_on:
       - mongo
       - redis
       - screenshot
     ports:
       - "5000:80"
   frontend:
     image: quay.io/toucantoco/frontend:v111.0.186
     environment:
       API_BASEROUTE: http://localhost:5000
     ports:
       - "8000:80"
   screenshot:
     image: quay.io/toucantoco/screenshot:latest
     expose:
       - "3000"

As v2 has entered in the Long Term Support (LTS) phase, it is safe to update the frontend, the backend and screenshot service to the latest version.

Step 2: Restart the stack

In the directory, alongside the docker-compose.yaml file. Run the following command:

# Downloads the docker images
docker-compose pull # optional, but recommended to minimize the upgrade time

# Stops the stack
docker-compose down

# Starts the stack
# -d: run in the background
docker-compose up -d

This will upgrade the stack to the new version.

About Toucan Toco Versions¶

Please note each brick of the Toucan Toco stack (backend and frontend) has its own version.

To be compatible, the major version number should be the same, for example:

Backend in v26.0.2 and Frontend in v26.0.7 are compatible, as they are both v26
Backend in v25.2.10 and Frontend in v25.1.5 are compatible, as they are both v25
Backend in v25.2.11 and Frontend in v26.0.7 are not compatible, as one is v25 and the other is v26

Breaking Changes¶

Please take a look to the release note for the self-hosted breaking changes

I manage my own MongoDB

Please refer to the official documentation to upgrade your backend.

I don’t manage my own MongoDB

If you use the Toucan Toco shipping script to manage the MongoDB installation and configuration (when "mongo_flag: enable"), everything will be upgraded auto-magically.

Monitoring¶

When you are in a self-hosted mode, we recommend you to monitor the following parts:

Toucan Toco App¶

You should monitor:

https://api-toucan.domain.com/ returns a 200 HTTP status
https://api-toucan.domain.com/status with User Agent Plop 4rQD3KzCxWzTYaRyp0NSEfd6 returns a 200 HTTP status

The status page checks that:

MongoDB is up and reachable by the Toucan Toco app
Redis is up and reachable by the Toucan Toco app
All Toucan Toco app processes are ok

System¶

Be sure the system is ok by monitoring:

free disk space on all partitions
load average of the server
disk IO
CPU, memory and SWAP usage
RAID state
hardware health

Of course, thresholds directly depend on your own monitoring policy.

Logs¶

All logs should be monitored (via ELK stack, Logentries,…) to prevent segfault, OOM killer, unexpected behaviors.

MongoDB¶

You should monitor that:

the mongoDB process is up and running
the mongoDB port (usually MONGO_PORT) is available and opened from the node where the Toucan Toco app is installed
a basic user auth can be completed

Redis¶

You should monitor that:

the redis process is up and running
the redis port (usually 6379) is available and opened from the node where the Toucan Toco app is installed

How To Monitor¶

All we described above can be monitored by the usual Nagios, Zabbix, Shinken, Munin, Collectd plugins.