Raw S3 Data Export Reports

Often the easiest way to integrate Gamesight's reporting data into your existing BI and ETL workflows is through directly ingesting the row-level reporting data. To support this use case you can configure Gamesight to automatically export raw row-level reporting data into your S3 bucket daily.

📘

V2 and V3 Data Exports

S3 Raw Data Export Reports currently supports 2 different data formats - V2 and V3. The V2 formats are based on the V2 Reporting API which contains aggregated reports (*_summary in the Data Dictionary). The V3 formats are based on unaggregated data used to power the V3 Reporting API such as Touchpoints, Events, and Goals.

If you are starting a new integration please plan on using the V3 reporting data as the V2 reporting is deprecated and is being scheduled for removal. If you have an existing V2 reporting integration please plan on migrating to V3. We have not yet set a EOL date for the V2 API but we will provide at least 6 months notice before removal.

The V3 data covers five data structures:

  • Touchpoint - Record for each Impression and Click that was directly recorded by your Trackers.
  • Event - Event logs for all events recorded for your Game across your various data sources.
  • Goal - Records each of the GoalsGoals - Specific Events or combinations of Events that you are trying to drive users to complete. This may include buying your game, first match completed, user reactivation, or any other combination of events. triggered for your Game. This includes both attributed and unattributed (organic) goals.
  • External Network Performance - Normalized data synchronized from your integrated at platforms (Facebook Ads, Google Ads, etc) providing Impression, Click, and Cost data aligned into your Gamesight reporting structure.
  • Steam UTM Analytics - Reporting data from the Steam UTM analytics feature aligned into your Gamesight reporting structure.

The V2 data covers four data structures:

  • Campaign Summary - Clicks and impression traffic volumes broken down by the maximum granularity of our reports.
  • User Goal Summary - Records each attributed Goal Trigger and associates it with a row in Campaign Summary data.
  • User Event Summary - Summarized lifetime value and retention information for each user that has been attributed to a Goal. This data can be joined to the User Goal Summary data to create reports on LTV or Retention by acquisition source.
  • Goal Trigger Summary - Includes a row for each goal that was triggered, whether it was attributed to a Goal or not. This data can be useful for calculating the volume of unattributed ("organic") users that are installing your game.

Specification

To assist with parsing and ingesting the data exports we provide complete data dictionaries for each of the data structures below.

The data exported into S3 is formatted in the following file structure by default.

OBJECT_TYPE/year=XXXX/month=XX/day=XX/000
  • Each file contains an uncompressed CSV with a single header row or optionally can be provided in Parquet format
  • For V2 reporting, each exported file is at most 100 MB in size. If the total size of the exported data exceeds 100 MB for a single day then the data is split into multiple files (000, 001, 002, etc). Each file has the same format.
  • The export process begins at 00:10 UTC daily and all data is available by 01:00 UTC at the latest
  • The export process re-exports the last 3 days of data on each run. Please ensure that your ELT processes are designed to reprocess the last 3 days of data at each run. This allows for data to be updated in case there was a delay in reporting.
  • The numbers in the partition structure (year, month, day) are not zero-padded.
  • The dates in the partition structure represent the date that the actions in the reports were recorded, not the date that they were exported. So pulling records from user_goal_summary/year=2020/month=1/day=1 will return all conversions that were recorded on 2020-01-01 UTC.
  • The User Event Summary rows are mutable. They are exported and partitioned based on the date of their last record. Please be aware of this when writing your ingest pipelines.

An example of the structure you might see in S3 after the first report runs:

campaign_summary/year=2020/month=1/day=1/000
campaign_summary/year=2020/month=1/day=1/001
user_goal_summary/year=2020/month=1/day=1/000
user_event_summary/year=2020/month=1/day=1/000
goal_trigger_summary/year=2020/month=1/day=1/000

Note that on the first date the campaign summary data is split into 2 files (000 and 001) since the total export for that day exceeded 100 MB.

📘

Custom Unload Paths

V3 Data Export can support custom pathing templates for S3 unloads. This can enable you to add new partitions and date formats to ensure it complies with your existing Data Lake partitioning structure. Please let us know if you would prefer a customized unload directory structure.


Getting Started

To get started with Raw S3 Data Export Reporting you'll need to set up an S3 bucket in your AWS account and provide Gamesight with access.

Create the Bucket

Feel free to use your favorite IaC tool (Cloudformation, Terraform, etc), but if you want to do things in the AWS console here is a quick overview of how to get that setup.

First, click "Create Bucket" on the AWS S3 Console and fill out the form. Take note of the Bucket name and Region as we will need those values later.

Also, please leave the "Block all public access" option checked to prevent accidental data exposure.

Create IAM Role

The next step is to create an IAM role with write access into the bucket. Gamesight will use this role when running exports. Unfortunately due to some restrictions with Redshift we cannot use a bucket attached policy for access.

Head over to the IAM Console -> Roles and click "Create Role".

You can just select "Lambda" as your trusted entity for now, we will replace the Trust Relationship in a minute. Continue through the Role creation flow skipping the Policy attachment step and filling in whatever tags you need to for your org.

You should reach this screen where you can provide a nice descriptive name and description for your role. Click "Create Role" and navigate to your new role.

Attach the Policy

Now we need to attach a policy to our new role enabling it to write into the S3 bucket we created. Head over to your role and click "Add inline policy"

When prompted fill in the following Policy to the JSON tab

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject"
            ],
            "Resource": "arn:aws:s3:::YOUR-BUCKET-NAME-HERE/*"
        }
    ]
}

Be sure to replace YOUR-BUCKET-NAME-HERE with the actual name of your bucket. Give the policy a name and save it to attach it to your role.

Update Trust Relationship

The last step is to update the trust relationship for this role, giving Gamesight permission to assume it when exporting reports.

Fill in the following Policy when prompted and click "Update Trust Policy"

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::796380527832:role/GamesightS3ExportProd"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

📘

Hybrid SaaS Customers

If your Game is running in a Hybrid SaaS environment, please contact us before setting the Trust Policy on your export Role. The permissions for your unload will be slightly different that the ones documented above.

That's it! The last step is to just send some details over to us so we can enable the export for your account.

Finish Up

Once you have completed the above steps, please contact us through live chat or email us at [email protected] and we can finish off the process required to get your reporting live. Please include the following pieces of information with your request: bucket name, bucket region, and role ARN.

Here is an example of each of those values:

  • Bucket name: gamesight-export-demo-us-west-2
  • Bucket region: us-west-2
  • Role ARN: arn:aws:iam::123456789012:role/GamesightS3Export

Did this page help you?