Raw S3 Data Export Reports

Often the easiest way to integrate Gamesight's reporting data into your existing BI and ETL workflows is through directly ingesting the row-level reporting data. To support this use case you can configure Gamesight to automatically export raw row-level reporting data into your S3 bucket daily.

The data covers four data structures:

  • Campaign Summary - Clicks and impression traffic volumes broken down by the maximum granularity of our reports.
  • User Goal Summary - Records each attributed Goal Trigger and associates it with a row in Campaign Summary data.
  • User Event Summary - Summarized lifetime value and retention information for each user that has been attributed to a Goal. This data can be joined to the User Goal Summary data to create reports on LTV or Retention by acquisition source.
  • Goal Trigger Summary - Includes a row for each goal that was triggered, whether it was attributed to a Goal or not. This data can be useful for calculating the volume of unattributed ("organic") users that are installing your game.

Specification

To help with parsing and ingesting the data exports we provide complete data dictionaries for each of the data structures below.

The data exported into S3 is formatted in the following file structure.

OBJECT_TYPE/year=XXXX/month=XX/day=XX/000
  • Each file contains an uncompressed CSV with a single header row or optionally can be provided in Parquet format
  • Each exported file is at most 100 MB in size. If the total size of the exported data exceeds 100 MB for a single day then the data is split into multiple files (000, 001, 002, etc). Each file has the same format.
  • The export process begins at 00:20 UTC daily and all data is available by 01:00 UTC at the latest
  • The export process re-exports the last 3 days of data on each run. Please ensure that your ELT processes are designed to reprocess the last 3 days of data at each run. This allows for data to be updated in case there was a delay in reporting.
  • The numbers in the partition structure (year, month, day) are not zero-padded.
  • The dates in the partition structure represent the date that the actions in the reports were recorded, not the date that they were exported. So pulling records from user_goal_summary/year=2020/month=1/day=1 will return all conversions that were recorded on 2020-01-01 UTC.
  • The User Event Summary rows are mutable. They are exported and partitioned based on the date of their last record. Please be aware of this when writing your ingest pipelines.

An example of the structure you might see in S3 after the first report runs:

campaign_summary/year=2020/month=1/day=1/000
campaign_summary/year=2020/month=1/day=1/001
user_goal_summary/year=2020/month=1/day=1/000
user_event_summary/year=2020/month=1/day=1/000
goal_trigger_summary/year=2020/month=1/day=1/000

Note that on the first date the campaign summary data is split into 2 files (000 and 001) since the total export for that day exceeded 100 MB.


Getting Started

To get started with Raw S3 Data Export Reporting you'll need to set up an S3 bucket in your AWS account and provide Gamesight with access.

Create the Bucket

Feel free to use your favorite IaC tool (Cloudformation, Terraform, etc), but if you want to do things in the AWS console here is a quick overview of how to get that setup.

First, click "Create Bucket" on the AWS S3 Console and fill out the form. Take note of the Bucket name and Region as we will need those values later.

Also, please leave the "Block all public access" option checked to prevent accidental data exposure.

Create IAM Role

The next step is to create an IAM role with write access into the bucket. Gamesight will use this role when running exports. Unfortunately due to some restrictions with Redshift we cannot use a bucket attached policy for access.

Head over to the IAM Console -> Roles and click "Create Role".

You can just select "Lambda" as your trusted entity for now, we will replace the Trust Relationship in a minute. Continue through the Role creation flow skipping the Policy attachment step and filling in whatever tags you need to for your org.

You should reach this screen where you can provide a nice descriptive name and description for your role. Click "Create Role" and navigate to your new role.

Attach the Policy

Now we need to attach a policy to our new role enabling it to write into the S3 bucket we created. Head over to your role and click "Add inline policy"

When prompted fill in the following Policy to the JSON tab

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject"
            ],
            "Resource": "arn:aws:s3:::YOUR-BUCKET-NAME-HERE/*"
        }
    ]
}

Be sure to replace YOUR-BUCKET-NAME-HERE with the actual name of your bucket. Give the policy a name and save it to attach it to your role.

Update Trust Relationship

The last step is to update the trust relationship for this role, giving Gamesight permission to assume it when exporting reports.

Fill in the following Policy when prompted and click "Update Trust Policy"

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::796380527832:role/GamesightS3ExportProd"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

That's it! The last step is to just send some details over to us so we can enable the export for your account.

Finish Up

Once you have completed the above steps, please contact us through live chat or email us at [email protected] and we can finish off the process required to get your reporting live. Please include the following pieces of information with your request: bucket name, bucket region, and role ARN.

Here is an example of each of those values:

  • Bucket name: gamesight-export-demo-us-west-2
  • Bucket region: us-west-2
  • Role ARN: arn:aws:iam::123456789012:role/GamesightS3Export

Did this page help you?