Attention! The Google BigQuery integration has to be enabled to use this feature.

Data import

SegmentStream can download feeds in NDJSON format and export their content into Google Big Query.

The feeds should be generated by the customer’s back-end in accordance with the requirements:

Our system sends the date GET parameter when requesting a feed. You can use it to generate data for a specific date. The date parameter should be of the YYYYMMDD format.

Connecting and configuring

The process of connecting data sources is described in detail in the overview.

In order to enable this data source, go to “DATA IMPORT → Automatic”, select the data source, enter login, enter password and save.

Configuring data source

Enter a title for the data source.

Feed URL - the address of the file you generated (the requirements for the file contents are listed below)

Example record - copy an example of a record in your feed. Below is a table with the mapping of the example record property types to BigQuery types:

Record type BigQuery Type Example
boolean BOOLEAN “isFirst”: false,
Date TIMESTAMP “created”: “2020-10-17T16:47:05.0Z”,
object RECORD “product”: {…},
string in date format DATE “bday”: “2020-12-31”,
integer INTEGER “total”: 900,
number FLOAT “unitSalePrice”: 10.9,
string STRING “name”: “John”,
null STRING “surname”: null,
  • Please note, if a parameter can have a value with decimals in an instance of your record, use a value with a decimal in the example, so that the BigQuery schema type will be correctly set to FLOAT.

Destination table name - the name of the table in BigQuery to which the data from the file will be written.

Partition table by date - if this option is enabled, a new table with new content will be created every day. Otherwise, the data will be overwritten every day.

To save and enable the data source, click “Save”.

The “Disconnect” button is used to revoke the authorization data. The settings are saved.

You can enable or disable the data source at any time.