Import events from Mixpanel to AWS S3

In this post, we demonstrate how to use CloudFormation and AWS services to import data from Mixpanel and store it in S3. Specifically, we show you how to:

• Create a Lambda function that polls Mixpanel for data via their import API
• Store the data in S3
• Use a CloudWatch event rule and a Lambda function target to trigger the Lambda function on a daily basis

We provide a complete CloudFormation template in YAML format, along with a JavaScript implementation of the Lambda function. This is a quick and easy way to get started with importing data from Mixpanel into your AWS environment.

AWSTemplateFormatVersion: 2010-09-09
Parameters:
  MixpanelAPISecretParameter:
    Type: String
    Description: Your Mixpanel API secret for your project
    MinLength: 1
    NoEcho: true
  S3BucketNameParameter:
    Type: String
    Description: Your Mixpanel API Secret from the Mixpanel UI
    MinLength: 3
    MaxLength: 63
Resources:
   ScheduledEventRule:
    Type: AWS::Events::Rule
    Properties:
      ScheduleExpression: cron(0 12 * * ? *)
      State: ENABLED
      Targets:
        - Arn: !GetAtt MixpanelImportLambdaFunction.Arn
          Id: MixpanelImportLambdaFunction
  DestinationS3Bucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Ref S3BucketNameParameter
      AccessControl: Private
      VersioningConfiguration:
        Status: Enabled
  MixpanelImportLambdaFunction:
    Type: AWS::Lambda::Function
    Properties:
      Environment:
        Variables:
          MIXPANEL_API_SECRET: !Ref MixpanelAPISecretParameter
          S3_BUCKET_NAME: !Ref S3BucketNameParameter
      Code:
        ZipFile: |
          'use strict';
          
          const https = require('https');
          const AWS = require('aws-sdk');
          exports.handler = async (event) => {
            // Replace these values with your own Mixpanel API key and secret
            const mixpanelApiSecret =  process.env.MIXPANEL_API_SECRET;
            const s3Bucket = process.env.S3_BUCKET_NAME;
            const today = new Date();
            const yesterday = new Date(new Date(today) - 1);
            const fromDate = yesterday.toISOString().slice(0, 10);
            const toDate = today.toISOString().slice(0, 10);

            const options = {
              hostname: 'data.mixpanel.com',
              port: 443,
              path: `/api/2.0/export?from_date=${fromDate}&to_date=${toDate}`,
              method: 'GET',
              headers: {
                'Authorization': 'Basic ' + 
                    new Buffer(mixpanelApiSecret + ":").toString("base64")
              }
            };

            // Make the request to the Mixpanel export API
            const importedDataAsString = await new Promise((resolve, reject) => {
              const req = https.request(options, (res) => {
                res.setEncoding('utf8');
                 let responseBody = '';
                res.on('data', (chunk) => {
                  responseBody += chunk;
                });
                res.on('end', () => {
                  console.error(responseBody)
                  resolve(responseBody);
                });
              });
              req.on('error', (error) => {
                reject(error);
              });
              req.end();
            });
    
            // Store the imported data in S3
            const s3 = new AWS.S3();
            await s3.putObject({
              Bucket: s3Bucket,
              Key: `${toDate}-data.json`,
              Body: importedDataAsString,
            }).promise();
          };
      Handler: index.handler
      Role: !GetAtt MixpanelImportLambdaExecutionRole.Arn
      Runtime: nodejs14.x
      Timeout: 60
  MixpanelImportLambdaExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      Policies:
        - PolicyName: mixpanel_import_lambda_policy
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - logs:CreateLogGroup
                  - logs:CreateLogStream
                  - logs:PutLogEvents
                Resource: 'arn:aws:logs:*:*:*'
              - Effect: Allow
                Action:
                  - s3:PutObject
                Resource: !Join
                  - ''
                  - - 'arn:aws:s3:::'
                    - !Ref DestinationS3Bucket
                    - /*