S3 Data Export Reports

Often the easiest way to integrate Gamesight's reporting data into your existing BI and ETL workflows is through directly ingesting the row-level reporting data. To support this use case you can configure Gamesight to automatically export raw row-level reporting data into your storage bucket daily.

Raw Data Exports support multiple different cloud providers' S3-compatible object stores. You can find specific guides depending on your cloud provider below:

• AWS S3
• GCP Cloud Storage

The data covers five data structures:

• Touchpoint - Record for each Impression and Click that was directly recorded by your Trackers.
• Event - Event logs for all events recorded for your Game across your various data sources.
• Goal - Records each of the Goals triggered for your Game. This includes both attributed and unattributed (organic) goals.
• External Network Performance - Normalized data synchronized from your integrated at platforms (Facebook Ads, Google Ads, etc) providing Impression, Click, and Cost data aligned into your Gamesight reporting structure.
• Steam UTM Analytics - Reporting data from the Steam UTM analytics feature aligned into your Gamesight reporting structure.

Specification

To assist with parsing and ingesting the data exports we provide complete data dictionaries for each of the data structures below.

docs.google.com

Gamesight S3 Export Data Dictionary

Default Structure

The data exported into S3 is formatted in the following file structure by default.

OBJECT_TYPE/y=XXXX/m=XX/d=XX/000

• Each file contains an uncompressed CSV with a single header row or optionally can be provided in Parquet format.
• The export process begins at 00:10 UTC daily and all data is available by 01:00 UTC at the latest
• The export process re-exports the last 3 days of data on each run. Please ensure that your ELT processes are designed to reprocess the last 3 days of data at each run. This allows for data to be updated in case there was a delay in reporting.
• The numbers in the partition structure (year, month, day) are not zero-padded.
• The dates in the partition structure represent the date that the actions in the reports were recorded, not the date that they were exported. So pulling records from touchpoint/y=2020/m=1/d=1 will return all conversions that were recorded on 2020-01-01 UTC.

An example of the structure you might see in S3 after the first report runs:

touchpoint/y=2020/m=1/d=1/000
event/y=2020/m=1/d=1/000
goal/y=2020/m=1/d=1/000
external_network_performance/y=2020/m=1/d=1/000

Options

In addition to the default delivery structure, there are a number of options which can be configured to customize your delivery.

• Delivery Frequency - By default data is delivered once per day, this can be customized to allow for multiple unloads per day down to hourly frequency.
• Delivery Time - By default, delivery runs at 00:10 UTC but this can be adjusted to any time of day
• Time Resolution - By default each table delivers one file per day, this can be adjusted to hourly partitions.
• Partitioning Structure - Data Export can support custom pathing templates for S3 unloads. This can enable you to add new partitions and date formats to ensure it complies with your existing Data Lake partitioning structure. Please let us know if you would prefer a customized unload directory structure.

Please let us know if you would like to configure any of these options on your data delivery and we'd be happy to confirm.

AWS Getting Started

To get started with Raw S3 Data Export Reporting you'll need to set up an S3 bucket in your AWS account and provide Gamesight with access.

Create the Bucket

Feel free to use your favorite IaC tool (Cloudformation, Terraform, etc), but if you want to do things in the AWS console here is a quick overview of how to get that setup.

First, click "Create Bucket" on the AWS S3 Console and fill out the form. Take note of the Bucket name and Region as we will need those values later.

Also, please leave the "Block all public access" option checked to prevent accidental data exposure.

Create IAM Role

The next step is to create an IAM role with write access into the bucket. Gamesight will use this role when running exports. Unfortunately due to some restrictions with Redshift we cannot use a bucket attached policy for access.

Head over to the IAM Console -> Roles and click "Create Role".

Select "Custom trust policy" as the Trusted entity type and paste the following trust policy.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::796380527832:role/GamesightS3ExportProd"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

📘

Hybrid SaaS Customers

If your Game is running in a Hybrid SaaS environment, please contact us before setting the Trust Policy on your export Role. The permissions for your unload will be slightly different that the ones documented above.

Continue through the Role creation flow skipping the Policy attachment step and filling in whatever tags you need to for your org.

You should reach this screen where you can provide a nice descriptive name and description for your role. Click "Create Role" and navigate to your new role.

Attach the Policy

Now we need to attach a policy to our new role enabling it to write into the S3 bucket we created. Head over to your role and click "Add inline policy"

When prompted fill in the following Policy to the JSON tab

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject"
            ],
            "Resource": "arn:aws:s3:::YOUR-BUCKET-NAME-HERE/*"
        }
    ]
}

Be sure to replace YOUR-BUCKET-NAME-HERE with the actual name of your bucket. Give the policy a name and save it to attach it to your role.

That's it! The last step is to just send some details over to us so we can enable the export for your account.

Finish Up

Once you have completed the above steps, please contact us through live chat or email us at [email protected] and we can finish off the process required to get your reporting live. Please include the following pieces of information with your request: bucket name, bucket region, and role ARN.

Here is an example of each of those values:

• Bucket name: gamesight-export-demo-us-west-2
• Bucket region: us-west-2
• Role ARN: arn:aws:iam::123456789012:role/GamesightS3Export

GCP Getting Started

To get started with Data Export Reporting in Google Cloud Storage you'll need to set up an GCS bucket in your GCP account and provide Gamesight with access.

Create the Bucket

Feel free to use your favorite IaC tool (Terraform, etc), but if you want to do things in the GCP console here is a quick overview of how to get that setup.

First, click the "Create" button on the Cloud Storage Console and fill out the form. Take note of the Bucket name as we will need this value later.

Also, please leave the "Enforce public access prevention on this bucket" option checked to prevent accidental data exposure.

Create Service Account

Next, you will create a Service Account that Gamesight can use to write objects into your new bucket. Detailed instructions for this process can be found in Google's documentation.

Assign Permissions

Once your service account is created, we will want to assign IAM permissions that allow it to write data into your bucket.

On the Permissions tab for your bucket, press the "Grant Access" button to begin assigning a role to your new service account.

Type in the full name for the service account you created in the previous step and grant it the "Storage Object Admin" role which permits this service account to write and overwrite objects in this bucket.

Generate HMAC Key

The final step in to generate HMAC keys for our service account so Gamesight can assume this role when writing data to your bucket.

Head over to the Cloud Storage Console > Settings > Interoperability and press the "Create a key for a service account" button. Select your service account and press new key to generate a new key.

Copy down both the Access key and Secret

Finish Up

Once you have completed the above steps, please contact your account rep and we can finish off the process required to get your reporting live.

You will need to include the following pieces of information with your request: bucket name, access key, secret. Due to the sensitive nature of these credentials we ask that you use a secure transform mechanism such as a one time link to share these details (such as 1Password or onetimesecret).

Here is an example of each of those values:

• Bucket name: gamesight-export-demo-us-west-2
• Access key: GOOGXXXXX
• Secret: XXXXXXXXX