Speed up your application

The following sections will give you some advice about optimizing rendering times in your stories.

Limit the size of your datasets with Requesters

Larger datasets will take longer to be rendered into charts. To limit your datasets, use requesters instead of filters.

See the documentation on requesters.

Accelerate your queries with Mongo Indexes

Toucan DataStore uses MongoDB and each domain is a MongoDB collection. In order to select the data matching a query, MongoDB scans each documents in a collection. Indexes can accelerate your queries by limiting the number of documents to scan. See the official doc for more information.

To choose the indexes for your app, you will need to have a good knowledge of its data structure.

Usage

MongoDB indexes are described in the etl_config file. You can create indexes for each domain. Each domain is a key in the MONGO_INDEXES configuration block.

MONGO_INDEXES:
  domain_a: [
    ‘year’
    [‘city’, ‘kpi_code’, ‘version’]
    [‘city’, ‘entity’, ‘version’]
  ]
  domain_b: [
    [‘date’, ‘filter’]
  ]

The domain domain_a have 3 indexes :

  • the 1st is an index on a single field year
  • the 2nd and the 3rd are compound indexes.

For compound indexes, the order of the fields is important in the index but not in the query. In addition to supporting queries that match on all the index fields, compound indexes can support queries that match on the prefix (a subset at the beginning of the set) of the index fields.

Query Example

  • Success with the index [‘city’, ‘kpi_code’, ‘version’]

    query:
      kpi_code: “CA”
      city: “Paris”
      version: 7
    
  • Success with the index [‘city’, ‘kpi_code’, ‘version’]

    query:
      city: “Paris”
      kpi_code: “CA”
    
  • Success with the index [‘city’, ‘kpi_code’, ‘version’] but only for city

    query:
      city: “Paris”
      version: 7
    
  • Failure with the index [‘city’, ‘kpi_code’, ‘version’] (city is missing)

    query:
      version: 7
      kpi_code: “CA”
    

Tips

Creating indexes everywhere for everything is not a magical solution. It is time and memory consuming.

Our Data Request Sandbox displays performance reports for your queries, with information such as the usage and the pertinence of your indexes.

Use your own database

You may have the necessary data infrastructure custom-built to query your data in milliseconds. In this case, we recommend to leverage on your own capabilities and to bypass our DataStore.

See the documentation.