Sending Charts to Twitter

Tutorial on how to use Zapier to integrate with Stencil

Stencil's Zapier app is coming soon. For now, we can use Stencil's webhook to do the integration.

Automating Charts Generation Workflow

In this demo we will demonstrate a real use case of workflow automation using Zapier, AWS Lambda and Twitter.

The workflow goal is to pull up data from external source, populate our template with the data, generate the image and post it to Twitter.

Despite having no Zapier app (yet, WIP), integration with Zapier is still possible. In fact, this illustrates how integration with other services can be achieved just by utilizing webhook functionality.

Requirements

If you're following along, you will need

1. Premium Zapier account

2. AWS Account

3. Twitter account

4. and of course Stencil

Premium Zapier account is required due to the Zaps that we are going to use

Overview

Here's how everything connects to each other.

1. We write Lambda function to pull the data we need and build modifications to send to Stencil.

2. We use Zapier to schedule a Zap that calls AWS Lambda function once a day.

3. Our lambda function then fetches the data, transforms it into modifications that can be applied to the template and then they are sent to Stencil to generate image.

4. We also specify Zapier webhook URL that Stencil will call once the image is generated.

5. The webhook is linked to Zapier Webhook Zap that in turns run another zap to parse the payload and tweet to Twitter.

Guides

1. Designing our template

The final image that will be tweeted

Our template consists of two bar graphs, three texts, one image and one rectangle that acts as a background.

We will leave the designing part as an exercise to the reader 😄

For both bar graphs, you can use the following JSON to create a default chart that we will override from the API.

{
  "labels": [
    "Johor",
    "Kedah",
    "Kelantan",
    "Melaka",
    "Negeri Sembilan",
    "Pahang",
    "Perak",
    "Perlis",
    "Pulau Pinang",
    "Sabah",
    "Sarawak",
    "Selangor",
    "Terengganu",
    "W.P. Kuala Lumpur",
    "W.P. Labuan",
    "W.P. Putrajaya"
  ],
  "datasets": [
    {
      "label": "Dose #1",
      "data": [
        100,
        5,
        15,
        100,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0
      ],
      "backgroundColor": "#ffcc00"
    },
    {
      "label": "Dose #2",
      "data": [
        10,
        100,
        150,
        200,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0
      ],
      "backgroundColor": "rgba(187, 153, 224, 1)"
    }
  ]
}

Paste the above JSON into your Dataset Editor (switch to JSON View).

If you need help with Charts, please see Chart's documentation

2. Preparing the data with AWS Lambda

We are going to use Malaysia's National Covid-​19 Immunisation Programme Open Data that can be found here, https://github.com/CITF-Malaysia/citf-public.

We are mostly interested in vaccination count by state that is referenced here, https://github.com/CITF-Malaysia/citf-public/blob/main/vaccination/vax_state.csv.

Lambda Function

from datetime import datetime, timedelta
import pytz
import pandas as pd
import requests as req
import json

def make_modification(name, states, dose1, dose2):
    datasets = [ { "data": dose1, "label": "Dose #1" }, { "data": dose2, "label": "Dose #2" } ]

    modification = {
        "name": name,
        "labels": states,
        "datasets": datasets
    }

    return modification

def run():
    url = "https://raw.githubusercontent.com/CITF-Malaysia/citf-public/main/vaccination/vax_state.csv"
    
    # Get yesterday's date because data is only available up to yesterday
    kl_timezone = pytz.timezone('Asia/Kuala_Lumpur')
    kl_time_now = datetime.now(kl_timezone)
    kl_time_yesterday = kl_time_now - timedelta(days=1)
    yesterday = kl_time_yesterday.strftime("%Y-%m-%d")

    # read the CSV from the URL
    df = pd.read_csv(url)
    
    # filter the data only for the date we are interested in
    rows = df.loc[df["date"] == yesterday]

    if not rows.empty:
        # Only get the data that we want to trend
        states = rows["state"].tolist()
        daily_dose1 = rows["dose1_daily"].tolist()
        daily_dose2 = rows["dose2_daily"].tolist()

        cumul_dose1 = rows["dose1_cumul"].tolist()
        cumul_dose2 = rows["dose2_cumul"].tolist()

        # Generate modificatiosn that fit Stencil's API format
        daily_graph = make_modification("bar_graph_daily", states, daily_dose1, daily_dose2)
        cumul_graph = make_modification("bar_graph_cumul", states, cumul_dose1, cumul_dose2)

        text_date = {
            "name": "text_date",
            "text": yesterday
        }
        modifications = [text_date, daily_graph, cumul_graph]
        
        # Set our Zapier webhook to call once image is generatd
        payload = {
            "template": "3bc487ef-33ad-46b3-a18c-0d57c3825723",
            "modifications": modifications,
            "webhook_url": "your-zapier-webhook"
        }

        headers = {
            "Authorization": "Bearer your-project-api-key",
            "Content-Type": "application/json"
        }

        # Call Stencil's API to generate the image asynchronously
        req.post("https://api.usestencil.com/v1/images", json=payload, headers=headers)


def lambda_handler(event, context):
    run()
    
    return {
        'statusCode': 200,
        'body': json.dumps('Request sent')
    }

Although the code may look daunting for some users, it is actually pretty straight forward. You could also use any other no-code or low-code tool out there to do this, but in this case a simple Python script should handle the job just fine.

We can now deploy our lambda function and set HTTP trigger for the lambda function. We will need this trigger URL in the next step.

3. Schedule Zap to run lambda function

The rest of the steps are straight forward and mostly filling in the details with proper information.

We are going to trigger the lambda function everyday at 7AM including weekends.

We provide the Lambda function trigger URL from previous step

4. Create Zap to listen for webhook callback and send a tweet

Our lambda function will call Stencil's API to generate the image. We also setup our payload to include Zapier webhook below. Once image is generated, it will call this webhoook.

This is the Zapier's Webhook that we set in our payload in step 2 (Lambda function)

We add another step to parse our JSON payload and assign our output variable to include the link to the image and the date of the data.

Zapier's output variable must be a dictionary

We parse the response we get from Stencil's API to send it to the next step (Twitter)

We parse the response we get from Stencil's API to send it to the next step (Twitter)

We add date to our message so the message is always unique.

You're done! Here is sample of the tweet.

Now do something productive with the extra time you've gained. This workflow is fully automated.

Debugging issues

Sometimes, problems can arise and you need to know where to look. You can view image requests by either clicking on Requests tab or by accessing your template's Console page.

You should be able to see your generated image in the list of requests. If you don't, that means Stencil does not receive your request. Please check your Zap or AWS Lambda function.

You can also see the webhook response in the log.

Twitter Zap can be disabled automatically by Zapier if they think you're sending a duplicate tweet. It is against Twitter's ToC.

Now try do the same workflow but with Integromat.