JSON Feed
Attention! The Google BigQuery integration has to be enabled to use this feature.
Data import
SegmentStream can download feeds in NDJSON format and export their content into Google Big Query.
The feeds should be generated by the customer’s back-end in accordance with the requirements:
- Newline Delimited JSON file format (http://ndjson.org/)
- Feed must be available at a specific URL, for example: https://website/feed.txt
- Access by URL must be closed using the basic auth method.: https://en.wikipedia.org/wiki/Basic_access_authentication
Our system sends the date GET parameter when requesting a feed. You can use it to generate data for a specific date. The date parameter should be of the YYYYMMDD format.
Connecting and configuring
The process of connecting data sources is described in detail in the overview.
In order to enable this data source, go to “DATA IMPORT → Automatic”, select the data source, enter login, enter password and save.
Configuring data source
Enter a title for the data source.
Feed URL - the address of the file you generated (the requirements for the file contents are listed below)
Example record - copy an example of a record in your feed. Below is a table with the mapping of the example record property types to BigQuery types:
| Record type | BigQuery Type | Example |
|---|---|---|
| boolean | BOOLEAN | “isFirst”: false, |
| Date | TIMESTAMP | “created”: “2020-10-17T16:47:05.0Z”, |
| object | RECORD | “product”: {…}, |
| string in date format | DATE | “bday”: “2020-12-31”, |
| integer | INTEGER | “total”: 900, |
| number | FLOAT | “unitSalePrice”: 10.9, |
| string | STRING | “name”: “John”, |
| null | STRING | “surname”: null, |
- Please note, if a parameter can have a value with decimals in an instance of your record, use a value with a decimal in the example, so that the BigQuery schema type will be correctly set to FLOAT.
Destination table name - the name of the table in BigQuery to which the data from the file will be written.
Partition table by date - if this option is enabled, a new table with new content will be created every day. Otherwise, the data will be overwritten every day.
To save and enable the data source, click “Save”.
The “Disconnect” button is used to revoke the authorization data. The settings are saved.
You can enable or disable the data source at any time.