Data Exporter

Introduction

If you are reading this article it is because as a Bryj customer, you are interested in ways to retrieve raw data collected by Bryj in your mobile apps in your ecosystem.

The purpose of this article is to :

Provide guide on how to configure your Data Exporter
Acknowledge which data can be exported
Provide clear examples of data samples you could get from such exports.

For the time being Data exporter is compatible with AWS S3 only. Other storage solutions may be covered based on the roadmap

Configuration

It is possible to configure one Data Exporter per Project in this version. This Data Exporter will point to a specific AWS bucket.

Storage configuration

You will be requested to fill in information on the target location and the credentials to use to feed the bucket with the data.

Encryption settings are disabled by default. It is an optional security layer which can be activated based on the project’s Data Exporter requirements. Activating this option will have an impact on the exported files and will require decryption while processing the exported files in your ecosystem.

Data Exporter configuration

Once your storage configuration is done, you will be requested to provide more information on the data to export:

• Apps: select the apps the Data Exporter must export data for.
• Timeframe: select wether the Data Exporter should export all historical data stored for this app OR if you only want data since a specific date.
• Data: Select tables to be exported and choose if you would like to remove the header from the exported files.

More detailed information on the Data is provided later in this document. But you will be able to retrieve raw data on: Attributes, Campaigns, Details, Devices, Events and Sessions. When a table is selected, all the columns available in this table will be exported.

Once your configuration is ready, you will be able to activate the Data Exporter upon click on Activate at the bottom right.

Upon activation your data will not be available right away in your storage. You will have to be a little patient as Data Exporters runs are scheduled through the night between 12AM and 2AM CET.

Data Exporter management

Edition

Once a Data Exporter is configured it is possible to edit:

• Storage configuration, if a typo was made in these settings preventing the Data Exporter to run properly, you will be able to fix it without having to redo the whole setup.
• Data set for the Data Exporter, more specifically the list of tables selected for the Data Exporter configuration.

It is not possible to edit Apps, timeframe and frequency once an export is activated.

Deletion

Either to pursue testing purposes or because you unsubscribed to the service, it is possible to delete a Data Exporter. If so, the sync with the storage will be reset completely as well as the whole Data Exporter configuration.

Deleting a Data Exporter will not have an impact on the files previously exported in the storage. If required, you will need to clean the files from the storage manually.

De-activation

It is possible to de-activate a Data Exporter to stop the export of data. It can be useful if you notice an issue in your storage configuration which takes a bit of time to be resolved for example.

Monitoring

A console also called log book is provided by Bryj so that you can make sure your data is exported smoothly. Check the number of files exported and their status per type (Campaign, sessions…).

You can autonomously copy an ID related to an export for debugging purposes in case you need to contact support.

Exported files

The purpose of this section is to provide more details on how the Data Exporter will feed the storage regarding the folders & file created, their naming, path and what is means!

Generated folders & Path handling

The Data Exporter will either create or override a file named “data-exports” at the location configured in Storage configuration. Within this data-exports folder, a sub-folder per data type configured is created:

data-exports

• app_devices
• attributes
• campaigns
• details
• events
• sessions

For each data type, you will then find a folder per exporter, meaning one folder for an initial export, and other folders if a recurrence was configured (one per daily run).

A technical way to phrase it would be:

{s3bucket}/{subFolder}/data-exports/{exportType}/[{exportSince}_]{exportDate}_{initial or daily}_
{exportIdentifier}

Where the export identifier is taken from the logbook (check Monitoring).

Example:

Let’s say I have the following configuration

• bryj-exports-example is clients bucket
• sub is clients subfolder

And I am checking those exports

• events is a customer export type
• BRYJDATAEXP_a012705db1924916ba1d399c05c2ce8e is the customer’s export identifier taken from UI log book
• export date is 2023-02-08

The paths in the bucket would be the following:

• S3 path for initial export since 2022-01-01 will be: s3a://bryj-exports-example/sub/data-export/events/2022-01-01_2023- 02-08_initial_BRYJDATAEXP_a012705db1924916ba1d399c05c2ce8e/…
• S3 path for all time initial export will be: s3a://bryj-exports-example/sub/data-export/events/2023-02- 08_initial_BRYJDATAEXP_a012705db1924916ba1d399c05c2ce8e/…

Files names

Let’s say I have the following configuration bryj-exports-example is clients bucket

sub is clients subfolder
And I am checking those exports:

events is a customer export type
BRYJDATAEXP_a012705db1924916ba1d399c05c2ce8e is the customer’s export identifier taken from UI log book export date is 2023-02-08

The paths in the bucket would be the following:

S3 path for initial export since 2022-01-01 will be: s3a://bryj-exports-example/sub/data-export/events/2022-01-01_2023- 02-08_initial_BRYJDATAEXP_a012705db1924916ba1d399c05c2ce8e/…
S3 path for all time initial export will be: s3a://bryj-exports-example/sub/data-export/events/2023-02- 08_initial_BRYJDATAEXP_a012705db1924916ba1d399c05c2ce8e/…

Now you understand how the exported data is structured thanks to the previous chapter, let’s focus on the files themselves.

Based on the volume data to export, a specific folder may contain several files (it will contain only one file if there is little data to export). For example if an initial export is set on Devices table, I may have a lot of data to export, it will be partitioned in several files.

The file naming is part_{index}.csv where index is valued from 0 to [amount of files – 1].

Examples:

• For an export with 3 files naming will be part_0.csv, part_1.csv and part_2.csv
• For an export with 1 file you would only have part_0.csv

Encryption impacts

If encryption option is enabled in the Data Exporter, then file extensions will be .enc instead of .csv To decrypt files and transform your .enc in .csv you can use openssl tool:

1. Download file and run command:

     openssl enc -d -aes-256-cbc -pbkdf2 -iter 200000 -in ./part_0.enc -out ./part_0.csv

2. Provide your secret key as requested by the console

3. Collect your files. In this example, your part_0.enc will be transformed to part_0.csv .

4. Exploit your .csv data!

Table mapping

Events

Variables	data_type	Business Value
month	string	Technical field created by bryj for partitioning data
date_utc	date	Event timestamp happened on device (from user device)
app_id	string	Unique identifier for an application
device_id	string	Unique identifier for a device
id	string	Unique identifier for an event
log_type	int	Type of event:1 = Custom event (public)
name	string	Technical name of the event
string_value	string	“simple detail” associated to an event
details	map<string,string>	Map containing event key-value pairs
value_type	int	This value is taken from event_definition tablestring: code=1,number: code=2boolean: code=3date: code=4
number_value	double	Value of a detail if the value_type is a number
user_id	string	Unique identifier of the user who triggered the event
created_at	timestamp	Date and time when the log was generated

Details

Variables	data_type	Business Value
month	string	Technical field created by bryj for partitioning data
date_utc	date	Event timestamp happened on device (from user device)
app_id	string	Unique identifier for an application
device_id	string	Unique identifier for a device
log_id	string	Unique identifier for an event
log_name	string	Technical name of the event
log_type	int	Type of event:1 = Custom event (public)
key	string	Technical name of the event detail
value_type	int	This value is taken from event_definition tablestring: code=1,number: code=2boolean: code=3date: code=4
string_value	string	Value of a detail if the value_type is a string
number_value	double	Value of a detail if the value_type is a number
user_id	string	Unique identifier of the user who triggered the event
created_at	timestamp	Date and time when the log was generated

Attributes

Variables	data_type	Business Value
app_id	string	Unique identifier for an application
user_id	string	Unique identifier for a user
key	string	Technical attribute name, which should be presented in attribute_definitions
device_id	string	Unique identifier for a device
type	int	Represents the format of the values associated with this attributestring: code=1,number: code=2boolean: code=3date: code=4
string_value	string	Attribute value if the attribute is STRING format
number_value	double	Attribute value if the attribute is NUMBER format
boolean_value	boolean	Attribute value if the attribute is BOOLEAN format
date_value	timestamp	Attribute value if the attribute is DATE_VALUE format
date	timestamp	Timestamp, when attribute was set on device
created_at	timestamp	Technical field, created by bryj as soon as request received from user device. Last time when device sent an “attribute” type event
updated_at	timestamp	Technical field, created by bryj as soon as request received from user device. First time when device sent an “attribute” type event

Devices

Variables	data_type	Business Value
app_id	string	Unique identifier for an application
device_id	string	Unique identifier for a device
updated_at	timestamp	Technical field, created by bryj as soon as request received from user device. Last time device sent a “device” type event
created_at	timestamp	Technical field, created by bryj as soon as request received from user device. First time device sent a “device” type event
last_user_id	string	The last user_ID that was used for this device
last_user_is_logged_in	boolean	Will be true if last event was send by logged in user
opt_in_analytics	boolean	Is SDK opt-in analytics?
opt_in_push	boolean	Is device opt-in for push notification for this app?
uninstalled	boolean	Is the application still installed on the device?
device_model	string	Technical name given by the device manufacturer
device_type	string	Type of device corresponding to the device (phone or tablet)
manufacturer	string	Phone brand name
country_code	string	Last position of the phone based on its last IP address
city	string	Last position of the phone based on its last IP address
language_code	string	ISO code representing the language in which the device is configured
timezone_offset	int	Timezone offset in seconds (from user device)
timezone_name	string	Timezone name coming (from user device)
app_version	string	Current version of the application installed on the device
sdk_version	string	Current version of the SDK installed on the installed application
os_version	string	Current version of the OS installed on the device
push_token	string	Token allowing the push service (APNS, Firebase Cloud Messaging, etc.) to identify a device and its eligibility to receive a push notification
push_token_h	string	Same as push_token field, but for huawei devices
last_token_type	string	Indicate what token type (Android, Huawei or Apple) was sent in last event
last_seen_at	date	Calculated from updated_at field, without time
tracking_allowed	boolean	Boolean field from user device

Sessions

Variable	data_type	Business Value
month	string	Technical field created by bryj for partitioning data
app_id	string	Unique identifier for an application
device_id	string	Unique identifier for a device
user_id	string	Unique identifier for a user
start_date	timestamp	Datetime the session was started
start_day	date	Date the session was done
end_date	timestamp	Datetime the session was ended
nb_starts	int	Number of time the application is launched during a session
nb_crashes	int	Number of time the application has crashed during a session
opt_in_analytics	boolean	Is the device performing the session opt-in analytics?
opt_in_push	boolean	Is the device performing the session opt-in for push notification?
device_model	string	Technical name given by the device manufacturer
device_type	string	Type of device corresponding to the device (phone or tablet)
country_code	string	Last position of the phone based on its last IP address
city	string	Last position of the phone based on its last IP address
lang	string	Language in which the device is configured
os	string	Current version of the OS installed on the device
sdk_version	string	Current version of the SDK installed on the installed application
app_version	string	Current version of the application installed on the device
session_tz	int	Timezone offset in seconds (from user device)
last_user_is_logged_in	boolean	Will be true if last event was send by logged in user
created_at	timestamp	Technical field, created by bryj as soon as request received from user device. First time when device sent an event
updated_at	timestamp	Technical field, created by bryj as soon as request received from user device. Last time when device sent an event

Campaign

ariables	data_type	Business Value
month	string	Technical field created by bryj for partitioning data
date	timestamp	Event timestamp happened on device (from user device)
app_id	string	Unique identifier for an application
device_id	string	Unique identifier for a device
campaign_id	string	Unique identifier for a campaign
campaign_grouping_id	string	Unified identifier for cross platform campaigns (iOS & Android)
campaign_name	string	Name of the campaign (given in the Bryj UI)
campaign_delivery_type	string	Campaign delivery type possible values: one-time, reccuring, api-trigger, local-trigger
campaign_type	string	Campaign format possible values: push, inapp, push+lp
name	string	Name of the campaign eventActionCancelledDisplayedDismissedErrorNotSentSentTriggered
action_id	string	Name of the campaign action (button clicked)when name=Action, from parameters[id]when name=NotSent from parameters[errorCode]
action_level	int	Identify on which In App page the action took place
user_id	string	Unique identifier for a user
details	string	From user devicewhen name=Error, from parameters[description]when name=Sent, from details
request_id	string	Presented only for push results. Its technical field to distinguish campaign push batches.
created_at	timestamp	Technical field, created by bryj as soon as request received from user device