Add data to my small app

In order to add new data to your small app, we have several methods. The simplest way is to upload files. You can also use remote files - on a ftp server for example - or connect your small app to an external data provider like MongoDB.

The description of your data are the datasources. They are all declared in a configuration file called the etl_config.cson. Datasources must have a type - Excel, CSV, MongoDB… - and a domain name. The domain is used to query data in your charts.

Tutorial : Product Corporation You can download the CSV file for our tutorial.

data-product-corporation.csv

How to add a new file as a data source

To add a file, use the DATA > DATA SOURCES menu. You can visualize all the datasources that you can update with a file. CSV, Excel and XML files are accepted.

Add your datasource block

There are two ways to add a datasource :

  • using the studio, only for files (beta)
  • editing etl_config.cson file

Tutorial : Product Corporation Solution 1 (only for files) :

  • Drop your file on Add datasource. The fields are already filled :
  • domain is the name used to query data in your chart : data-product-pro
  • file is the name of your file on our server (it does not have to be the same as your file name) : data-product-corporation.csv
  • Save changes. You have created a new datasource block.

Solution 2 :

  • Click on the FILES menu option.
  • Download the etl_config.cson as Staging file.
  • Add your datasource in the block DATA_SOURCES of the etl_config.cson
DATA_SOURCES: [
 {
   domain: 'data-product-corpo'
   file: 'data-product-corporation.csv'
   skip_rows: 0
   separator: ','
   encoding: 'utf-8'
   type: 'csv'
 }
]
  • Drag and drop your new etl_config.cson in the CONFIG FILES page
  • Go to your ‘DATA SOURCES’ page. You have created a new datasource block.

For more options, see the - File-based datasources Section.

Upload your file

To upload your file for the first time or update it, you can upload or drag and drop the file in the datasource block created. They will be stored on Toucan Toco’s servers.

Tutorial : Product Corporation

  • Click or drag your file data-product-corporation.csv and wait for the validation to finish.
  • Your file is now on the server and ready to be loaded as a domain.

Data ready for charts

Even if your datasources are declared and your file have been uploaded, you can’t yet use it in a chart. As a last step, you need to launch an OPERATION named Preprocess. It will load your data in the declared domain.

To see available domains to be queried in your charts, see the section Available domains for the staging environment in the DATA SOURCES page.

Tutorial : Product Corporation

  • See Available domains for the staging environment at the bottom of the page. This is the domains that can be queried in your app. Your domain data-product-corpo is not yet in it. One last effort.
  • Click on the button OPERATIONS on the bottom
  • Choose the Preprocess and click on START NOW
  • Wait for the end the operation : Last operations report updated. You can have more details in the black window on the bottom of the page.
  • See Show domain samples on the bottom of the page.
  • Congratulations, your new domain is here! You can see a sample of it, download it and use it in a chart.

File-based datasources

CSV files

Declare a .csv data source with:

  • the type : csv
  • the file name (file: 'my_filename.csv')
  • the separator used in the csv file (separator: ',', usually ',', ';' or tabulations \t)
  • a domain field to identify and use the datasource (domain: 'my_csv_domain')

Other optionnal options exist:

  • encoding(by default utf-8) to choose encoding of a file. For other common encoding, see https://en.wikipedia.org/wiki/Character_encoding#Common_character_encodings.
  • dtype to change the type of columns, by example from int to sring.
  • skiprows : skip n rows at the begining of the file. Useful if your files comes with blank lines
  • decimal : character to recognize as the decimal separator (for example “,” in France)
  • na_values : [] : values to be interpreted as “na” (null values in your file that are not written as 0)
  • all options available on pandas documentation.
DATA_SOURCES = [
  domain: 'my_csv_domain1'
  type: 'csv'
  file: 'my_filename1.csv'
  separator: ';'
  encoding: 'latin-1'
  dtype:
    date: 'str'
,
  domain: 'my_csv_domain2'
  type: 'csv'
  file: 'my_filename2.csv'
  separator: ','
  skiprows: 1
]

Excel files

Declare a .xls or .xlsx file with:

  • the type excel,
  • the file name (file: 'my_filename.xls' or file: 'my_filename.xlsx')
  • a domain field to identify and use the datasource (domain: 'my_excel_domain'):

Other optionnal options exist:

  • encoding(by default utf-8) to choose encoding of a file. For other common encoding, see https://en.wikipedia.org/wiki/Character_encoding#Common_character_encodings.
  • dtype to change the type of columns, by example from int to sring.
  • skiprows : skip n rows at the begining of the file. Useful if your files comes with blank lines
  • na_values : [] : values to be interpreted as “na” (null values in your file that are not written as 0)
  • all options available on pandas documentation.
DATA_SOURCES = [
  domain: 'my_csv_domain1'
  type: 'excel'
  file: 'my_filename1.xls'
  encoding: 'latin-1'
  dtype:
    date: 'str'
  skip_rows: 0
]

Sheetname option

To deal with multiple sheets, several solutions exist :

  • Associate one sheet to one domaine

    domain: 'my_excel_domain1'
    type: 'excel'
    file: 'my_filename.xls'
    sheetname: 'sheetname1'
    
  • Concatenate all sheets in one domain with null value

    domain: 'my_excel_domain'
    type: 'excel'
    file: 'my_filename.xls'
    sheetname: null
    
  • Concatenate some sheets in one domain by using a list

    domain: 'my_excel_domain'
    type: 'excel'
    file: 'my_filename.xls'
    sheetname: ['sheetname1', 'sheetname2']
    

The two last solutions create a column __sheet__ in the domain containing the sheet name.

XML files

Declare .xml files with:

  • the type (type: 'xml')
  • a file name (file: 'my_filename.xml')
  • a domain field to identify and use the datasource (domain: 'my_xml_data').

If your data is nested in the dom and you need to access it or modify it before it is imported you can use the filter field to write a jq filter (for ex.: filter: 'records.record'). jq is the stantard we use to preprocess nested data souces into tabular data. See this part of the Toucan Toco documentation for a detailed introduction to jq. In the specific case of xml data we use the following spec to transform it, and then apply your jq filter.

Example data:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<records>
  <record id="1">
    <title>Black album</title>
  </record>
  <record id="2">
    <title>Graffiti bridge</title>
  </record>
</records>

Datasource declaration:

DATA_SOURCES = [
   domain': 'XML Fixture'
   type': 'xml'
   file': 'fixture.xml'
   filter': '.records.record'
]

Resulting dataframe:

    @id    title
0   1       Black album
1   2       Graffiti bridge

Remote files

Sometimes your data files are uploaded on remote servers, and it is simpler to keep them there. In this case just put your url as the file path ! We will read the file over a network connection according to the given url scheme.

For example this is how you can read a CSV file directly on Dropbox or your FTP server.

DATA_SOURCES: [
  domain: 'db_test'
  type: 'csv'
  file: 'sftp://<username>:<password>@ftp.example.com/my_folder/my_data.csv'
,
  domain: 'my_remote_data'
  type: 'csv'
  file: 'https://www.dropbox.com/s/9yu9ekfjk8kmjlm/fake_data.csv?dl=1'
]

We support ftp (as well as sftp or ftps), http (and https), S3 and a long list of other schemes (‘mms’, ‘hdl’, ‘telnet’, ‘rsync’, ‘gopher’, ‘prospero’, ‘shttp’, ‘ws’, ‘https’, ‘http’, ‘sftp’, ‘rtsp’, ‘nfs’, ‘rtspu’, ‘svn’, ‘git’, ‘sip’, ‘snews’, ‘tel’, ‘nntp’, ‘wais’, ‘svn+ssh’, ‘ftp’, ‘ftps’, ‘file’, ‘sips’, ‘git+ssh’, ‘imap’, ‘wss’).

FTP Server

When your data files are too big to be transfered via the studio data upload interface, you can store them in a FTP server. The FTP server can either be in the Toucan Toco side (ask for support to set it up) or in your side.

Tutorial : Product Corporation

  • You need an access to your FTP server.
  • Open Filezilla
  • Connect to your FTP and look at what files are available.
  • Right-click on the one you want to use
  • Select “Copy URL(s) to clipboard”
  • Add your password to the url generated

Important

💡 If you don’t want to write the password in your etl_config file! Contact us via discourse or your Delivery contact to set up a hidden password.

  • Paste the url to your datasource block
  • Your datasource block is now ready
DATA_SOURCES: [
   domain: 'db_test'
   type: 'csv'
   file: 'ftps://<login>:<password>@ftps.toucantoco.com:990/my_db.csv'
   separator: ";"
]

Toucan Toco FTP Server

You can send data to Toucan Toco FTP Server with the following credentials:

  • Host: ftps.toucantoco.com
  • Port: 990 (for the connection) and range 64000-64321 (for data transfert)
  • Protocol: FTPS (if you use FileZilla it’s implicit FTP over TLS)
  • Mode: Passive Mode
  • User : Given by the Toucan Toco Team
  • Password: Given by the Toucan Toco Team

It’s possible to send us your data encrypted with our following public GPG key:

Public GPG key:
-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1

mQINBFoF0doBEACutAVSzl00YBVhspH6CSCP6KqLvg48PjnMLvcFCe9uLnRGpYE/
6we0mpnU0DFrpc6aISEMaBcOUuQt+fjFYc1A2XVSITjGAegMYMqknBJruaiXF39r
c4jkt0TnY6cSGAgUNfvTnbTfHElO4IWmLSgM2p55Fz2vX/0BcrgXwugu0FKN0y45
spcYH5hQoMSI/S5IO+n6LaaKFjGSANaVgUb6tINn6z5E2aAsKlNEjofm+hf/yst0
Z9FsIbQxFWht5yzlquETTKCWm0Zdn9dqb/WTBhHvI7NZTgolOi6Acj9Pmg+emZV0
CLxCySp5v5eJuSn3ITDl89MiT2yVPuav2CWYPT5KP+Ic4KZIsNvyiIiIcSXiAVvP
A1G+bn33+5LEqEDT/gEi5udV9YSsLUUZ8c2ZFtvkbtxjfV2+6cpSc4CY3vCVJ04N
A3tSOZdJOJMAkIqO8CkzsDOJj5Xsah2z6ZNIs8vGZ1fC0Um0Mh59PMpBGDkn0hYD
vpr5SynpMqoYUhczGHA6VM+YsqoVpfmwv7K4kC53frxipjG/Pk/+Hr2nseRMQUFc
fWcVrrhC0it+7DUIX2BMqP62J/cUALm6kznShnZxzvNWszx4cpeObm7vQULiVV61
J29E0yTE6PBy/MVRL6k8mXQxfOTWKYrgfLduPfc2LZbaVhZaVnAniDiQKQARAQAB
tCRUb3VjYW5fVG9jb19EZXYgPGRldkB0b3VjYW50b2NvLmNvbT6JAjgEEwECACIF
AloF0doCGwMGCwkIBwMCBhUIAgkKCwQWAgMBAh4BAheAAAoJEH4bz1mt3TJK4YwP
/jg0e2Iq8bvo+du8DOe587KTUJKHS1Hqca1XJzxHK1iXIa2psRpFLBPBAwM7js5X
Xec2voEEXXUiLVYlvdFjvg11/dslEhU6Z0ZXXLYkNRQw8kOE/E1+2pWtRgO4lp5l
kvsPyTa28uX/hAsI5br11bGW9Aky8md+9ruR4OwbHd8+nBFDncH21eYSM/s0Yhgd
vsKkwDJzbpkZRF2DvkZNOvunPI2VPtoWNMvDvrTnrJn0FHjiiXO+X0c8cC9FugaB
IPbjPy5nYDA8KQaQfdCz+FSFccxJqKdNur8KC8VR3AKvvfC3D0nugke9Tptraokx
rJbyLXNK82SPxP83roE4/RwhX3wVg0nkFwP4oYT6jx/uNWISkE7Qpr/hIJXK7AOC
jy6Dsk10olLoldq9SyBK0sc9udDjR90vYudcZ0Tq1q1JeJRKpctJ2IozliX0Rk4A
77tOWf+IpG6o5YbEw5pE/I9VFTAcL+PVyv+s/xr7uXsYo/7eWAaHz+/ECh8N9+Bg
bA5gYQ4dEZgS8UZCxh5A18mSECS1AtUhqOWFmtvpALtRaYrvSHVHtf1P5owhCqou
7utPGPyxCUKBGSQtfDiYrZuHflwP+fsI0/o88pHYiWwHEaJ7iKfOQNqMegxlYrgZ
FJNmu06mTTT7fZR1z6NFktUJHmYepKsrnwklMucl3FaVuQINBFoF0doBEACVlPBY
ey3PI6g3tMEqSYBbezLjF3OZ5FezARIt3ItTrvu5mr73ixgOeK3jCZhm+/RwJWxu
e7u7AXQ5ztU/WjgtAvjxhVTModnA8SI83dziH9A7ECINStjPoLdKEQQ9HYM1INFs
YqSWfW7ZI/8sDue8/w0MCbDljubLZaeTjcIOawUemIu+eLsYTTjDaGvGFV30kS8V
brYErb5Amf/BcvmDC72GLOfQy15Mcs8m6SOknjmQgPR7At5cRTLPJz0r/w/ie3Pa
zrlH5/PR9vAqFdejXSjeCuRAu7rDUM3FXVuCeiz/5b2Y2n7eC9MyX982pZ2Y4vGW
RYmeCJ1YapgRBq/NOQ2lq3peFz+O+ID/R5SQFunnotC6CdhxydOruYE/Xl/G6bl2
JkzwBpPgampmtQfO+kTVb9mXsVZtK3ItqDHyY2J7z6U0pGa8BeDaZ3NkK9tB1+my
ZiJ2Ic4eE3UzsuiFQGJjKL2O+leK1yLFnbUYwbDg0O7IUdUB2BfS2yQkS1EtM88n
hnZj7FR/oGLFbnnH3j3FFk0b1FGQVAQqSbNGcABJSMbCbERDaIKmGiPrZL54TPMZ
15B5ghfhPT52qEwAJOVwU1EES9yKYlXGF8yalygaGgkdI3JshrmKd156cgOceKDD
Bt4E/bnxInkCFAEkfxAI/ycCggt1zI/wsdjG2QARAQABiQIfBBgBAgAJBQJaBdHa
AhsMAAoJEH4bz1mt3TJKgvEP/Rj47oaj8Zymw+lSTU7o7p16dPfAnUrNy4fpl866
nMf2SSEuST5tsGv5lJuLkalwZcIOpnZSgyyLp1a3ips3ZMp+myX9NxwWqWAut8Lb
+NlcmHCLfbW+TrsqVV3RztAvJc0QGEdhcvs0PrvIORU8yglOkehSW+imxAu6I6BU
x6HdiOq9bV1M9VgUUZRLtg3VinqYmtdxCL54b3uF+L2CCSgSpFQtUWDQEVJJgSm3
8oATxBjDDC1eOThKGzt8ZdN5PbcGLsad5FGkcE0y2vgphxmld86MrBQaOIMk2rPT
AcGchztNbpV/Fkc1oamuCXESVCC5+Xu2EQo9WRVyrBLpY9xlxLkwyCsEG7DouIa8
eUX2YAbldXhbPrX3whY/E3NEEbUj0bR7mJoXiaAkN5/phRubUfb0oNXc/hyDxyCq
wy5i57zbV84KoidkO5djYQKlsyxFVOVYYtjd+UXix8y8Vb4Hldvh4uxIupZCgrwS
4NtjK/Vs+hD3jd0FYZB+HRKbdsXlpaMS6xjJ+iMOVtkHH4Z42kINyDe7vBJ80PBT
iAjwA8CFO2mFjmaZQ/Hr1sCWLt0fUBqWKWlHs4jr0pnlv+ICDxlW/FuxwnCM6BsP
lyoNBcwQA+JNwTaR0QmHBKLmsspa7L80Q2DQk9bq0YIvC85SnWLs9LLfIZqXek9g
hZ/f
=tSbW
-----END PGP PUBLIC KEY BLOCK-----

sha1sum of the key: f9dd1369378877068048398f3e7529f0d7d819d6

S3 Bucket

The access key and secret key for your data files hosted on S3 buckets can be configured this way:

s3://<access key>:<secret key>@mybucket/filename'

For example:

DATA_SOURCES: [
    domain: 'my_data'
    type: 'csv'
    file: 's3://<access key>:<secret key>@mybucket/my_data.csv'
    separator: ";"
]

Note

If your access key or secret key contains special characters such as “/”, “@” or “:” you have to encode them. URL encoding converts special characters into a format that can be transmitted over the Internet. You will find more infos about this topic here (as well as an automatic encoder).

Matching multiple files

You can load multiple files - uploaded on our server or on a FTP server - in a unique domain with the option match: true. Your domain after Preprocess will contain a column __filename__ with the name of your files.

Tutorial Product Corporation Your corporation has now a new file of data each month : data-product-corporation-201801.csv, data-product-corporation-201802.csv … You want them to be loaded in a single domain called data-product-corpo

  • Find the regular expression (regex) that matches your files with regex101.com.

data-product-corporation-\d{6}\.csv

  • Don’t forget to use ‘^’ and ‘$’ to be more restrictive.

^data-product-corporation-\d{6}\.csv$

  • Add a backslach to escape backslaches.

^data-product-corporation-\\d{6}\\.csv$

  • Copy your regular expression in the file option of your datasource block

  • Add the option match: true

    DATA_SOURCES: [
     {
       domain: 'data-product-corpo'
       file: '^data-product-corporation-\\d{6}\\.csv$'
       skip_rows: 0
       separator: ','
       encoding: 'utf-8'
       type: 'csv'
       match: true
     }
    ]
    
  • Drop your etl_config.cson in the interface with your new block.

  • Now you can drop multiple files in the new block in the DATA SOURCE page and they will all be loaded in the domain data-product-corpo after a Preprocess

  • Your domain data-product-corpo will contain a column __filename__ with the name of your files.

External data providers

Using our connectors, you can query use multiple external data providers in Toucan Toco apps. Data providers are server-side, which means they require a connection from Toucan Toco’s server to the data source.

Step 1 - Declaration

Each external data provider must be declared in the DATA_PROVIDERS block of etl_config.cson. The required parameters are :

  • the type of the connector
  • the name which is a unique identifier. Multiple options fields must be added depending on the type (e.g. username and password).
DATA_PROVIDERS: [
  name: "My provider"
  type: "providerType"
  options:
    username: "username"
    password: "password"
]

Step 2 - Usage

When creating a DATA_SOURCE block, you can use a data provider with:

  • name: name of the data provider (same as the one inside the DATA_PROVIDERS list)
  • query options
DATA_SOURCES: [
  domain: 'db_test'
  name: 'My provider'
  [... other options ...]
]

The table or the query result will be loaded after a Preprocess step like csv or excel files.

Available data providers

The list of available data provider can be find in toucan-connectors. Make a support request to deploy them on your instance.

AdobeAnalytics connector

Adobe Analytics Connector using Adobe Analytics’ REST API v1.4.
It provides a high-level interfaces for reporting queries (including Data Warehouse requests).

Data provider configuration

DATA_PROVIDERS: [
  type:    'AdobeAnalytics'
  name:    '<name>'
  username:    '<username>'
  password:    '<password>'
  endpoint:    '<endpoint>'
,
  ...
]

Data source configuration

  • domain: str, required
  • name: str, required
  • suite_id: str, required
  • dimensions: str, list of str or list of dict
  • metrics: str or list of str, required
  • date_from: str, required
  • date_to: str, required
  • segments: str or list of str, default to None
  • last_days: int
  • granularity: hour, day, week, month, quarter, year, default to None
  • source: str
DATA_SOURCES: [
  domain:    '<domain>'
  name:    '<name>'
  suite_id:    '<suite_id>'
  dimensions:    '<dimensions>'
  metrics:    '<metrics>'
  date_from:    '<date_from>'
  date_to:    '<date_to>'
  segments:    '<segments>'
  last_days:    '<last_days>'
  granularity:    '<granularity>'
  source:    '<source>'
,
  ...
]

AzureMSSQL connector

Import data from Microsoft Azure SQL Server.

Data provider configuration

  • type: "AzureMSSQL"
  • name: str, required
  • host: str, required
  • user: str, required
  • password: str, required
  • connect_timeout: int
DATA_PROVIDERS: [
  type:    'AzureMSSQL'
  name:    '<name>'
  host:    '<host>'
  user:    '<user>'
  password:    '<password>'
  connect_timeout:    <connect_timeout>
,
  ...
]

Data source configuration

  • domain: str, required
  • name: str, required
  • database: str, required
  • query: str (not empty), required
DATA_SOURCES: [
  domain:    '<domain>'
  name:    '<name>'
  database:    '<database>'
  query:    '<query>'
,
  ...
]

Dataiku connector

This is a basic connector for Dataiku using their

Data provider configuration

  • type: "Dataiku"
  • name: str, required
  • host: str, required
  • apiKey: str, required
  • project: str, required
DATA_PROVIDERS: [
  type:    'Dataiku'
  name:    '<name>'
  host:    '<host>'
  apiKey:    '<apiKey>'
  project:    '<project>'
,
  ...
]

Data source configuration

  • domain: str, required
  • name: str, required
  • dataset: str, required
DATA_SOURCES: [
  domain:    '<domain>'
  name:    '<name>'
  dataset:    '<dataset>'
,
  ...
]

Elasticsearch connector

Data provider configuration

  • type: "elasticsearch"
  • name: str, required
  • hosts: list of Host, required
  • send_get_body_as: str, (if not GET method)
Host configuration
  • url: str, required
  • port: int
  • username: str
  • password: str
  • headers: dict
DATA_PROVIDERS: [
  type:    'elasticsearch'
  name:    '<name>'
  hosts:    '<hosts>'
  send_get_body_as:    '<send_get_body_as>'
,
  ...
]

Data source configuration

  • domain: str, required
  • name: str, required
  • search_method: search or msearch, required
  • body: dict for search or list for msearch, required
  • index: str (required for search)
  • parameters: dict
DATA_SOURCES: [
  domain:    '<domain>'
  name:    '<name>'
  search_method:    '<search_method>'
  body:    '<body>'
  index:    '<index>'
  parameters:    '<parameters>'
,
  ...
]

FacebookInsights connector

Import data from facebook insights API.

Data provider configuration

  • type: "facebook_insights"
  • name: str, required
DATA_PROVIDERS: [
  type:    'facebook_insights',
  name:    '<name>'
]

Data source configuration

  • domain: str, required
  • name: str, required
  • metrics: list of str, required, the list of metrics that need to be fetched like page_total_actions,
  • period: str, optional, the aggregation period, defaults to week,
  • date_preset: str, optional, a date range preset like last_week or yesterday, defaults to last_30d.
DATA_SOURCES: [
  domain:    '<domain>',
  name:      '<name>',
  pages:  {
    // page_id: page_token
    '292074147300': 'EAAFXzXZAi46xkkOxuJtfOciUQbC8u8A1aU0M2CvRGJACGkSzPRoyQCtEB5Yo9dwsdASOfDzGhonJl49oG1SZct1LJsjfJIhmdnT9dH3x8QyYy4nJnmLd7LH8yOytdn8nbeN3F29yXDxKnOkkwZzGJLYyxuEbbrQAewhqxtz1ki1dxowDCRFcqMTq32CLrY2IFQoYj',
  },
  metrics: ['page_total_actions', 'page_impressions'],
  period: 'week',
  date_preset: 'last_year',
]

For more details on expected values and semantics, check https://developers.facebook.com/docs/graph-api/reference/v2.8/insights

GoogleAnalytics connector

Data provider configuration

GoogleCredentials
For authentication, download an authentication file from console.developper.com
and use the values here. This is an oauth2 credential file. For more information
  • type: str
  • project_id: str
  • private_key_id: str
  • private_key: str
  • client_email: str
  • client_id: str
  • auth_uri: str
  • token_uri: str
  • auth_provider_x509_cert_url: str
  • client_x509_cert_url: str
DATA_PROVIDERS: [
  type:    'GoogleAnalytics'
  name:    '<name>'
  credentials:  {
    type: '<type>'
    project_id: '<project_id>'
    ...  # see documentation below
  }
  scope:    '<scope>'
,
  ...
]

Data source configuration

DATA_SOURCES: [
  domain:    '<domain>'
  name:    '<name>'
  report_request:    '<report_request>'
,
  ...
]

GoogleBigQuery connector

Data provider configuration

  • type: "GoogleBigQuery"
  • name: str, required
  • credentials: GoogleCredentials, required
  • dialect: Dialect, default to legacy
  • scopes: list(str), default to [“https://www.googleapis.com/auth/bigquery”]
GoogleCredentials
For authentication, download an authentication file from console.developper.com
and use the values here. This is an oauth2 credential file. For more information
  • type: str
  • project_id: str
  • private_key_id: str
  • private_key: str
  • client_email: str
  • client_id: str
  • auth_uri: str
  • token_uri: str
  • auth_provider_x509_cert_url: str
  • client_x509_cert_url: str
DATA_PROVIDERS: [
  type:    'GoogleBigQuery'
  name:    '<name>'
  credentials:    '<credentials>'
  dialect:    '<dialect>'
  scopes:    '<scopes>'
,
  ...
]

Data source configuration

  • domain: str, required
  • name: str, required
  • query: str, required
DATA_SOURCES: [
  domain:    '<domain>'
  name:    '<name>'
  query:    '<query>'
,
  ...
]

GoogleCloudMySQL connector

Import data from Google Cloud MySQL database.

Data provider configuration

  • type: "GoogleCloudMySQL"
  • name: str, required
  • host: str, required
  • user: str, required
  • password: str, required
  • port: int
  • charset: str, default to utf8mb4
  • connect_timeout: int
DATA_PROVIDERS: [
  type:    'GoogleCloudMySQL'
  name:    '<name>'
  host:    '<host>'
  user:    '<user>'
  password:    '<password>'
  port:    <port>
  charset:    '<charset>'
  connect_timeout:    <connect_timeout>
,
  ...
]

Data source configuration

  • domain: str, required
  • name: str, required
  • database: str, required
  • query: str (not empty), required
DATA_SOURCES: [
  domain:    '<domain>'
  name:    '<name>'
  database:    '<database>'
  query:    '<query>'
,
  ...
]

GoogleMyBusiness connector

Data provider configuration

  • type: "GoogleMyBusiness"
  • name: str, required
  • credentials: required (see “get credentials” section below)
  • token: str
  • refresh_token: str
  • token_uri: str
  • client_id: str
  • client_secret: str
  • scopes: list of str, default to [‘https://www.googleapis.com/auth/business.manage’]
DATA_PROVIDERS: [
  type:    'GoogleMyBusiness'
  name:    '<name>'
  credentials:    '<credentials>'
  scopes:    ['<scope>']
,
  ...
]

Data source configuration

  • domain: str, required
  • name: str, required. Should match the data provider name
  • metric_requests: list of Metric (see below)
  • time_range: required
  • start_time: str
  • end_time: str
  • location_ids: list of str, optional. Defaults to all locations available.

Metric

  • metric: str, required
  • options, list of str, optional.
DATA_SOURCES: [
  domain:    '<domain>'
  name:    '<name>'
  metric_requests: [
    metric: 'QUERIES_DIRECT'
  ,
    metric: 'QUERIES_INDIRECT'
  ]
  time_range:
    start_time: '2019-01-27T00:00:00.045123456Z'
    end_time: '2019-02-27T23:59:59.045123456Z'
,
  ...
]

Get credentials

First, you will need a valid client_secret.json file. You can download it from your Google Cloud Platform Console
in API & Services > Credentials > OAuth 2.0 client IDs.

Then, in a virtualenv with google_auth_oauthlib and google-api-python-client package, you can use this python code to get your credentials:

import json
from google_auth_oauthlib.flow import InstalledAppFlow
from googleapiclient.discovery import build

CLIENT_SECRETS_FILE = "client_secret.json"
SCOPES = ["https://www.googleapis.com/auth/business.manage"]

flow = InstalledAppFlow.from_client_secrets_file(CLIENT_SECRETS_FILE, SCOPES)
credentials = flow.run_console()
json_credentials = json.dumps({
    "token": credentials.token,
    "refresh_token": credentials.refresh_token,
    "token_uri": credentials.token_uri,
    "client_id": credentials.client_id,
    "client_secret": credentials.client_secret,
}, indent=2)
print(json_credentials)

GoogleSpreadsheet connector

Share the spreadsheet

Unless the spreadsheet is public, you will have to manually share it.

Open the google spreadsheet inside your web browser. Inside the File menu, there a
Share option. Click on it and enter the email address of your service account.
If you are on Toucan Toco’s cloud, it is:
toucanserviceaccount@testproj-204816.iam.gserviceaccount.com
Data provider configuration
GoogleCredentials
For authentication, download an authentication file from console.developper.com
and use the values here. This is an oauth2 credential file. For more information
  • type: str
  • project_id: str
  • private_key_id: str
  • private_key: str
  • client_email: str
  • client_id: str
  • auth_uri: str
  • token_uri: str
  • auth_provider_x509_cert_url: str
  • client_x509_cert_url: str
If you’re on Toucan Toco’s cloud and your application can be automatically deployed via package,
credentials are already set up for service account toucanserviceaccount@testproj-204816.iam.gserviceaccount.com.
In order to use these credentials in your ETL config file, you can use the syntax secrets.google_spreadsheet_credentials.xxx.
⚠️ Note : Be sure to use the option escape_newlines for the private_key field.
DATA_PROVIDERS: [
  type:    'GoogleSpreadsheet'
  name:    '<name>'
  credentials:
    type: '{{ secrets.google_credentials.type }}'
    project_id: '{{ secrets.google_credentials.project_id }}'
    private_key_id: '{{ secrets.google_credentials.private_key_id }}'
    private_key: '{{ secrets.google_credentials.private_key | escape_newlines }}'
    client_email: '{{ secrets.google_credentials.client_email }}'
    client_id: '{{ secrets.google_credentials.client_id }}'
    auth_uri: '{{ secrets.google_credentials.auth_uri }}'
    token_uri: '{{ secrets.google_credentials.token_uri }}'
    auth_provider_x509_cert_url: '{{ secrets.google_credentials.auth_provider_x509_cert_url }}'
    client_x509_cert_url: '{{ secrets.google_credentials.client_x509_cert_url }}'
  scope:    '<scope>'
,
  ...
]
Data source configuration
  • domain: str, required
  • name: str, required. Should match the data provider name
  • spreadsheet_id: str, required. Id of the spreadsheet which can be found inside the url: https://docs.google.com/spreadsheets/d//edit?pref=2&pli=1#gid=0,
  • sheetname: str. By default, the extractor return the first sheet.
  • skip_rows: int, default to 0
DATA_SOURCES: [
  domain:    '<domain>'
  name:    '<name>'
  spreadsheet_id:    '<spreadsheet_id>'
  sheetname:    '<sheetname>'
  skip_rows:    <skip_rows>
,
  ...
]

Hive connector

Data provider configuration

  • type: "Hive"
  • name: str, required
  • host: str, required
  • port: int, default to 10000
  • auth: str, default to 'NONE', available values: 'LDAP', 'NOSASL', 'KERBEROS'
  • configuration: dict, a dictionary of Hive settings.
  • kerberos_service_name: str, use with auth: 'KERBEROS' only
  • username: str
  • password: str, use with auth: 'LDAP' only
DATA_PROVIDERS: [
  type:    'Hive'
  name:    '<name>'
  host:    '<host>'
  port:    '<port>'
  auth:    '<auth>'
  configuration:    '<configuration>'
  kerberos_service_name:    '<kerberos_service_name>'
  username:    '<username>'
  password:    '<password>'
,
  ...
]

Data source configuration

  • domain: str, required
  • name: str, required
  • database: str, default to 'default'
  • query: str, required
DATA_SOURCES: [
  domain:    '<domain>'
  name:    '<name>'
  database:    '<database>'
  query:    '<query>'
,
  ...
]

HttpAPI connector

This is a generic connector to get data from any HTTP APIs (REST style APIs).

This type of data source combines the features of Python’s requests
library to get data from any API with the filtering langage jq for
flexbile transformations of the responses.
Please see our complete tutorial for
an example of advanced use of this connector.

Data provider configuration

  • type: "HttpAPI"
  • name: str, required
  • baseroute: str, required
  • auth: {type: "basic|digest|oauth1|oauth2_backend|custom_token_server", args: [...], kwargs: {...}} cf. requests auth and requests oauthlib doc.
  • template: dict. See below.
DATA_PROVIDERS: [
  type:    'HttpAPI'
  name:    '<name>'
  baseroute:    '<baseroute>'
  auth:    '<auth>'
  template:
    <headers|json|params>:
        <header key>: '<header value>'
,
  ...
]
Template
You can use this object to avoid repetition in data sources.
The values of the three attributes will be used or overridden by
all data sources using this provider.
  • json: dict
  • headers: dict
  • params: dict
  • proxies: dict

Data source configuration

DATA_SOURCES: [
  domain:    '<domain>'
  name:    '<name>'
  url:    '<url>'
  method:    '<method>'
  headers:    '<headers>'
  params:    '<params>'
  data:    '<data>'
  filter:    '<filter>'
  auth:    '<auth>'
  parameters:    '<parameters>'
,
  ...
]

Complete example:

The complete spec of an HttpAPI entry in DATA_SOURCES is as follows:

DATA_PROVIDERS: [
    name: "open-data-paris"
    type: "HttpAPI"
    baseroute: 'https://opendata.paris.fr/api/'
    template:
        headers:
            requested-by: 'toucantoco'
]
DATA_SOURCES: [
  domain: "books"
  type: "HttpAPI"
  name: "open-data-paris"
  method: "GET"
  url: "records/1.0/search/"
  params:
    dataset: 'les-1000-titres-les-plus-reserves-dans-les-bibliotheques-de-pret'
    facet: 'auteur'
  filter: ".records[].fields"
]

The JSON response looks like this:

{
  "nhits": 1000,
  "parameters": { ... },
  "records": [
    {
      "datasetid": "les-1000-titres-les-plus-reserves-dans-les-bibliotheques-de-pret",
      "recordid": "4b950c1ac5459379633d74ed2ef7f1c7f5cc3a10",
      "fields": {
        "nombre_de_reservations": 1094,
        "url_de_la_fiche_de_l_oeuvre": "https://bibliotheques.paris.fr/Default/doc/SYRACUSE/1009613",
        "url_de_la_fiche_de_l_auteur": "https://bibliotheques.paris.fr/Default/doc/SYRACUSE/1009613",
        "support": "indéterminé",
        "auteur": "Enders, Giulia",
        "titre": "Le charme discret de l'intestin [Texte imprimé] : tout sur un organe mal aimé"
      },
      "record_timestamp": "2017-01-26T11:17:33+00:00"
    },
    {
      "datasetid":"les-1000-titres-les-plus-reserves-dans-les-bibliotheques-de-pret",
      "recordid":"3df76bd20ab5dc902d0c8e5219dbefe9319c5eef",
      "fields":{
        "nombre_de_reservations":746,
        "url_de_la_fiche_de_l_oeuvre":"https://bibliotheques.paris.fr/Default/doc/SYRACUSE/1016593",
        "url_de_la_fiche_de_l_auteur":"https://bibliotheques.paris.fr/Default/doc/SYRACUSE/1016593",
        "support":"Bande dessinée pour adulte",
        "auteur":"Sattouf, Riad",
        "titre":"L'Arabe du futur [Texte imprimé]. 2. Une jeunesse au Moyen-Orient, 1984-1985"
      },
      "record_timestamp":"2017-01-26T11:17:33+00:00"
    },
    ...
  ]
}
We apply the filter .records[].fields which means that for
every entry in the records properity it will extract all the
properties of the fields object. So we end up with a table of
results looking like this (I’m skipping columns in this example but you
see the point):
nombre_de_reservations auteur skipped columns…
1094 Enders, Giulia
746 Sattouf, Riad
Note: the reason to have a filter option is to allow you to take any
API response and transfom it into something that fits into a column
based data frame. jq is designed to be concise and easy to use for simple
tasks, but if you dig a little deeper you’ll find a featureful
functional programming language hiding underneath.

MicroStrategy connector

Import data from MicroStrategy using the JSON Data API for cubes and
reports.

Data provider configuration

  • type: "MicroStrategy"
  • name: str, required
  • base_url: str, required
  • username: str, required
  • password: str, required
  • project_id: str, required
DATA_PROVIDERS: [
  type:    'MicroStrategy'
  name:    '<name>'
  base_url:    '<base_url>'
  username:    '<username>'
  password:    '<password>'
  project_id:    '<project_id>'
,
  ...
]

Data source configuration

dataset parameter lets you specify whether you want to use the cube or reports endpoints, with the specified id. viewfilter allows you to refine the data returned from a report or cube (see usage).

limit and offset parameters allow you to “paginate” the data returned. If you just want the full result, set limit to -1.

  • domain: str, required
  • name: str, required
  • dataset: str, cube or report or search, required
  • id: str
  • viewfilter: dict
  • offset: int, default to 0
  • limit: int, default to 100 (set it to -1 if you want no limit)
  • parameters dict, optional. Allow to parameterize the viewfilter.
DATA_SOURCES: [
  domain:    '<domain>'
  name:    '<name>'
  dataset:    '<dataset>'
  id:    '<id>'
  viewfilter:    '<viewfilter>'
  offset:    '<offset>'
  limit:    '<limit>'
,
  ...
]
Search amongst available cubes and reports

Set dataset to search:

DATA_SOURCES: [
  domain:    '<domain>'
  name:      '<name>'
  dataset:   'search'
  id: "revenue analysis"  # id is optional here, and allows to filter by pattern
]
enhanced viewFilter

Microstrategy’s viewFilter does not natively support filtering by attribute or metric name, but we provide “syntaxic sugar” for them.

Example: instead of writing :

viewfilter:
  operator: "Equals"
  operands: [
    type: "form"
    attribute:
      id: "8D679D3511D3E4981000E787EC6DE8A4"
    form:
      id: "CCFBE2A5EADB4F50941FB879CCF1721C"
  ,
    type: "constant"
    dataType: "Char"
    value: "Miami"
  ]

you can write the equivalent:

viewfilter:
  operator": "Equals"
  operands: [
    attribute: "Call Center@DESC"
  ,
    constant: "Miami"
  ]

If attribute, metric or constant keys are found in the viewfilter, they will be expanded (to achieve this, a first api call is done to fetch the dataset fields definition).

attribute
{
  attribute: 'Call Center'
}

is expanded to

{
  'type': 'attribute',
  'id': '8D679D3511D3E4981000E787EC6DE8A4'
}
form (“attribute@form”)
{
  attribute: 'Call Center@DESC'
}

is expanded to

{
  'type': 'form',
  'attribute': {'id': '8D679D3511D3E4981000E787EC6DE8A4'},
  'form': {'id': 'CCFBE2A5EADB4F50941FB879CCF1721C'}
}
metric
{
  metric: '% Change to Profit'
}

is expanded to

{
  'type': 'metric',
  'attribute': {'id': '8D679D3511D3E4981000E787EC6DE8A4'},
}
constant

(limited to Real and Char datatypes)

{
  constant: 42
}

is expanded to

{
  'type': 'constant',
  'dataType': 'Real',
  'value': '42'
}

MongoDB connector

Retrieve data from a MongoDB database.
This is the connector used to retrieve loaded data in Toucan Toco interface.

Data provider configuration

  • type: "MongoDB"
  • name: str, required
  • host: str, required
  • port: int, required
  • username: str
  • password: str
DATA_PROVIDERS: [
  type:    'MongoDB'
  name:    '<name>'
  host:    '<host>'
  port:    <port>
  username:    '<username>'
  password:    '<password>'
,
  ...
]

Data source configuration

Supports simple, multiples and aggregation queries as desribed in
  • domain: str, required
  • name: str, required
  • database: str, required
  • collection: str, required
  • query: str (translated to a query {domain: <value>}), dict or list, required
DATA_SOURCES: [
  domain:    '<domain>'
  name:    '<name>'
  database:    '<database>'
  collection:    '<collection>'
  query:    '<query>'
,
  ...
]

Note

The Mongo connector can be used as a context manager to avoid opening
and closing a connection to a same database.
For example:
from toucan_connectors.mongo.mongo_connector import MongoConnector, MongoDataSource

queries = [
    {'domain': 'domain1', 'country': 'France'},
    {'domain': 'domain1', 'country': 'England'},
]

with MongoConnector(name='mycon', host='myhost', port=27017) as con:
    for query in queries:
        datasource = MongoDataSource(collection='test_col', query=query)
        con.get_df(datasource)

MSSQL connector

Import data from Microsoft SQL Server.

Data provider configuration

  • type: "MSSQL"
  • name: str, required
  • host: str, required
  • user: str, required
  • password: str
  • port: int
  • connect_timeout: int
DATA_PROVIDERS: [
  type:    'MSSQL'
  name:    '<name>'
  host:    '<host>'
  user:    '<user>'
  password:    '<password>'
  port:    <port>
  connect_timeout:    <connect_timeout>
,
  ...
]

Data source configuration

  • domain: str, required
  • name: str, required
  • database: str
  • query: str (not empty), required
DATA_SOURCES: [
  domain:    '<domain>'
  name:    '<name>'
  database:    '<database>'
  query:    '<query>'
,
  ...
]

MySQL connector

Import data from MySQL database.

Data provider configuration

  • type: "MySQL"
  • name: str, required
  • host: str, required
  • user: str, required
  • password: str
  • port: int
  • charset: str, default to utf8mb4
  • connect_timeout: int
DATA_PROVIDERS: [
  type:    'MySQL'
  name:    '<name>'
  host:    '<host>'
  user:    '<user>'
  password:    '<password>'
  port:    <port>
  charset:    '<charset>'
  connect_timeout:    <connect_timeout>
,
  ...
]

Data source configuration

Either query or table are required, both at the same time are not supported.

  • domain: str, required
  • name: str, required
  • database: str, required
  • query: str (not empty), required if table is not provided.
  • table: str (not empty), required if query is not provided, will read the whole table.
  • follow_relations: bool, default to false. Merges data from foreign key relations.
  • parameters dict, optional. Allow to parameterize the query.
DATA_SOURCES: [
  domain:    '<domain>'
  name:    '<name>'
  database:    '<database>'
  query:    'SELECT * FROM city WHERE country = %(country)s'
  parameters:
    country: '<value>'
,
  ...
]

OData connector

Data provider configuration

  • type: "OData"
  • name: str, required
  • baseroute: str, required
  • auth: {type: "basic|digest|oauth1|oauth2_backend|custom_token_server", args: [...], kwargs: {...}} cf. requests auth and requests oauthlib doc.
DATA_PROVIDERS: [
  type:    'OData'
  name:    '<name>'
  baseroute:    '<baseroute>'
  auth:    '<auth>'
,
  ...
]

Data source configuration

  • domain: str, required
  • name: str, required
  • entity: str, required
  • query: dict, required
DATA_SOURCES: [
  domain:    '<domain>'
  name:    '<name>'
  entity:    '<entity>'
  query:    '<query>'
,
  ...
]

OracleSQL connector

⚠️ Using this connector requires the installation of Oracle Instant client library. Please refer to Oracle installation instructions as it probably won’t be available in your server current package manager.

Data provider configuration

  • type: "OracleSQL"
  • name: str, required
  • dsn: str following the DSN pattern, required. The host, port and service name part of the dsn are required. For example: localhost:80/service
  • user: str
  • password: str
  • encoding: str
DATA_PROVIDERS: [
  type:    'OracleSQL'
  name:    '<name>'
  dsn:    <dsn>
  user:    '<user>'
  password:    '<password>'
  encoding:    '<encoding>'
,
  ...
]

Data source configuration

  • domain: str, required
  • name: str, required
  • query: str, required
DATA_SOURCES: [
  domain:    '<domain>'
  name:    '<name>'
  query:    '<query>'
,
  ...
]

Postgres connector

Import data from PostgreSQL.

Data provider configuration

  • type: "Postgres"
  • name: str, required
  • user: str, required
  • host or hostname : str, required
  • If you have a host name (eg aeaze.toucan.com), use the hostname parameter
  • If you have an IP address (e.g. 1.2.3.4), use the host parameter
  • charset: str
  • password: str
  • port: int
  • connect_timeout: int
DATA_PROVIDERS: [
  type:    'Postgres'
  name:    '<name>'
  user:    '<user>'
  host:    '<host>'
  hostname:    '<hostname>'
  charset:    '<charset>'
  password:    '<password>'
  port:    <port>
  connect_timeout:    <connect_timeout>
,
  ...
]

Data source configuration

  • domain: str, required
  • name: str, required
  • database: str, required
  • query: str (not empty), required
DATA_SOURCES: [
  domain:    '<domain>'
  name:    '<name>'
  database:    '<database>'
  query:    '<query>'
,
  ...
]

SapHana connector

Import data from Sap Hana.

Data provider configuration

  • type: "SapHana"
  • name: str, required
  • host: str, required
  • port: str, required
  • user: str, required
  • password: str, required
DATA_PROVIDERS: [
  type:    'SapHana'
  name:    '<name>'
  host:    '<host>'
  port:    '<port>'
  user:    '<user>'
  password:    '<password>'
,
  ...
]

Data source configuration

  • domain: str, required
  • name: str, required
  • query: str (not empty), required
DATA_SOURCES: [
  domain:    '<domain>'
  name:    '<name>'
  query:    '<query>'
,
  ...
]

Snowflake connector

Import data from Snowflake data warehouse.

Data provider configuration

DATA_PROVIDERS: [
  type:    'Snowflake'
  name:    '<name>'
  user:    '<user>'
  password:    '<password>'
  account:    '<account>'
  ocsp_response_cache_filename:    <ocsp_response_cache_filename>
,
  ...
]

Data source configuration

  • domain: str, required
  • name: str, required
  • query: str (not empty), required
  • database: str
  • warehouse: str
DATA_SOURCES: [
  domain:    '<domain>'
  name:    '<name>'
  query:    '<query>'
  database:    '<database>'
  warehouse:    '<warehouse>'
,
  ...
]

ToucanToco connector

Get data from a Toucan Toco instance, usefull to build analytics applications.

Data provider configuration

  • type: "ToucanToco"
  • name: str, required
  • host: str, required
  • username: str, required
  • password: str, required
DATA_PROVIDERS: [
  type:    'ToucanToco'
  name:    '<name>'
  host:    '<host>'
  username:    '<username>'
  password:    '<password>'
,
  ...
]

Data source configuration

Use the all_small_apps parameter to get results from an endpoint on all small apps.

  • domain: str, required
  • name: str, required
  • endpoint: Endpoints, required
  • all_small_apps: bool, default to False
DATA_SOURCES: [
  domain:    '<domain>'
  name:    '<name>'
  endpoint:    '<endpoint>'
  all_small_apps:    '<all_small_apps>'
,
  ...
]

Wootric connector

Import data from Wootric API.

Data provider configuration

  • type: "Wootric"
  • name: str, required
  • client_id: str, required, represents the OAUTH client_id
  • client_secret: str, required, represents the OAUTH client_secret
  • api_version: str, optional, defaults to "v1"
DATA_PROVIDERS: [
  type:    'Wootric'
  name:    '<name>'
  client_id:    'cc84asl8p980a9sjdlkxcz'
  client_secret: '5477aslk8clkaxcjlakscaas'
,
  ...
]

Data source configuration

  • domain: str, required
  • name: str, required
  • query: str (not empty), required, the API endpoint (e.g. responses)
  • batch_size: int, default to 5, how many HTTP queries should be batched together for a better optimization of network latency,
  • max_pages: int, default to 30, maximum number of pages to fetch on the wootric API endpoint. For instance, wootric limit to 30 the number of pages for the responses endpoint so there would no point in putting a higher limit.
  • properties: list of str, the response properties to filter out. Default is to fetch everything
DATA_SOURCES: [
  domain:    '<domain>'
  name:    '<name>'
  query:    'responses'
  batch_size: 10
,
  ...
]

For more information, check the api documentation

Bypass datastore querying directly your live data

If you have the necessary infrastructure, you can query your databases directly without the need of the in-memory layer of Toucan Toco. The steps are :

  1. setup your data provider in etl.cson file
  2. query the data provider to create live domains in DATA_SOURCES in etl.cson file (with the option live_data: true, and that’s it !)
  3. use the domains in your views in front_config.cson

Note

Keep in mind that in Toucan Toco, you can combine both the batch processing mode (where you store data views in the Toucan datastore) and the live data mode, where you query directly your system in real time. You does not need to choose between both, just use the most appropriate for your needs. Maybe you want some aggregations to be updated weekly, while you need to see detailed views in real time…

1 - setup your data provider

As there is no data store, you will directly query your data from a data provider. The data provider is setup in the DATA_PROVIDERS array in etl.cson file. Here what it should look like :

DATA_PROVIDERS: [
  {
    type:    'my_type'
    name:    'my_data_provider'
    host:    '142.0.0.3'
    port:    93748
    database:    'api-tctc-toctoc'
    username:    'my_username'
    password:    'secure_password'
  }
]

The type available are :

  • MongoDB
  • GoogleAnalytics
  • MySQL
  • PostgreSQL
  • HttpAPI

2 - query the data provider to create domains

Creating data sources using this new data provider will expose live data domains to be used in your stories. You MUST set parameter live_data: true on each of these data source. The DATA_SOURCES array should look like this in the etl.cson:

DATA_SOURCES: [
  {
    domain:    'my_domain'
    name:    'name_of_data_provider'

    # ...
    # use here the syntax of your data provider
    # ...


    live_data: true
  }
,
  {
    domain:    'an_other_domain'
    name:    'name_of_data_provider'

    # ...
    # use here the syntax of your connector
    # ...

    live_data: true
  }
,
...
]

NB : these domains won’t appear in our app’s data explorer.

3 - use the domains in views

In order to use live data sources in your stories, simply declare the domain you want to use in your query.

chartOptions:
  data:
    query:
      domain: 'my_domain'

You can use multiple queries :

chartOptions:
  data:
    query:[
      domain: 'my_domain1'
    ,
      domain: 'my_domain2'
    ]
    multiple_queries: true

You can also adjust your data using postprocess and multiple query.

4 - Templating

In the front_config you might want to use templating :

chartOptions:
  data:
    query:
      domain: 'my_domain'
      column_name: '<%= requestersManager.requesterOnMyColumn %>'

In this case you have to report this template in the etl :

DATA_SOURCES: [
  {
    domain:    'my_domain'
    name:    'name_of_data_provider'
    query:[
      domain :
      column_name: '%(column_name)s' # you report here your template
    ...
    ]
    live_data: true # mandatory
  }