Using the ListenFirst API Python SDK

This tutorial introduces lfapi, the ListenFirst Python SDK. This library streamlines many of the processes mentioned in previous articles, handling most of the boilerplate code and allowing the user to interact more fluidly with the API.

Getting Started

To begin, you will need the following credentials, retrievable from https://app.listenfirstmedia.com/#api once provisioned for API access:

• client_id
• client_secret
• api_key

Whereas previously access token generation was the user's responsibility, token management in lfapi is encapsulated in the lfapi.Auth class:

import lfapi

auth = lfapi.Auth(<client_id>, <client_secret>)
access_token = auth.access_token

Similarly, all HTTP requests to the ListenFirst API are handled by the lfapi.Client class, which takes an API key and lfapi.Auth object as arguments:

client = lfapi.Client(<api_key>, auth)

Note that the lfapi.Auth object refreshes expired access tokens, abstracting this task away from the user. In addition, a lfapi.Client object can be loaded directly from a JSON file containing all three credential values, handling the creation of the lfapi.Auth object:

from lfapi import Client

client = Client.load(<credential_file_or_filename>)
"""The credential file will look something like the following:
{
  "client_id": <client_id>,
  "client_secret": <client_secret>,
  "api_key": <api_key>
}
"""

Making API Requests

Once the API client is built, the user can use it to fetch data via the API. The lfapi.Client class comes equipped with two methods for making HTTP requests to the API, namely secure_get() and secure_post(). They each take an API endpoint, as well as a params argument similar to in the requests library. secure_post() additionally takes a json argument, again as in the requests library. Here are some example cases:

# Example 1: retrieve and iterate through datasets
ds_res = client.secure_get('dictionary/datasets')
for dataset in ds_res.json()["records"]:
    print(dataset["id"])

# Example 2: make a synchronous API request and print the ID column of the results
request = {
    "dataset_id": "dataset_brand_listenfirst",
    "metrics": [
        "lfm.engagement_score.overall",
    ],
    "start_date": "{{nDaysAgo 7}}",
    "end_date": "{{maxDateAvailable}}",
    "filters": [
        {
            "field": "lfm.brand_view.id",
	    "operator": "IN",
	    "values": [1234]
        }
    ],
    "group_by": [
        "lfm.fact.date_str",
       "lfm.brand_view.id"
    ]
}
fetch_res = client.secure_post('analytics/fetch', json=request)
for column in fetch_res.json()["columns"]:
    print(column["id"])

Specialized Request Methods and Model Classes

In addition to token handling and request building, lfapi also handles response and data parsing via the lfapi.Client class's specialized request methods. For example, the GET request to https://listenfirst.io/v20200626/dictionary/datasets crafted above can be streamlined further using the list_datasets() method:

datasets = client.list_datasets()

With the exception of get_field_values(), these specialized methods wrap the API response with an appropriate subclass of lfapi.Model, for easier handling and manipulation:

for dataset in datasets:  # lfapi.ListModel containing lfapi.Dataset items
    print(dataset.id)

datasets.to_csv('example.csv')

These model classes are especially useful for analytic data queries.

Making Analytic Queries

Using the specialized fetch() method, we can perform the analytic query we wrote previously with the much more streamlined code below:

query_data = client.fetch(request)  # lfapi.AnalyticResponse model

With the response wrapped in a model, we can export it to a number of data storage formats, similar to the to_csv() utility we used above with the /dictionary/datasets response:

query_data.to_csv('example.csv')             # CSV file
query_data.to_json('example.json')           # JSON file
df = query_data.to_pandas()                  # Pandas DataFrame; raises NotImplementedError unless user has installed pandas
tbl = query_data.to_pyarrow()                # PyArrow Table; raises NotImplementedError unless user has installed pyarrow
query_data.write_parquet('example.parquet')  # Parquet file; raises NotImplementedError unless user has installed pyarrow.parquet

Note: pandas, pyarrow, and pyarrow.parquet are optional dependencies. The associated methods are available if their respective libraries are installed, but lfapi does not require them to perform properly.

Asynchronous fetch jobs have additionally been encapsulated in a specialized method:

fetch_job = client.create_fetch_job({"fetch_params": request})  # lfapi.FetchJob model
print(client.show_fetch_job(fetch_job.id).state)

Job polling has been implemented as its own convenience method, as has async page download:

fetch_job = client.poll_fetch_job(fetch_job.id)
for i, page in enumerate(fetch_job.download_pages()):  # lfapi.AnalyticResponse model
    page.to_csv(f'example_{i}.csv')

Scheduled configuration has also been given its own set of methods:

schedule_configs = client.list_schedule_configs()  # lfapi.ScheduleConfig model
new_config = client.create_schedule_config({
    "fetch_params": request,
    "cron_expression": '0 9 * * *'
})

Finally, both synchronous and asynchronous querying have been given their own convenience methods which take as input a query parameters object (along with some additional endpoint-specific arguments), and return a generator containing response pages:

sync_page_gen = client.sync_analytic_query(request, per_page=100, max_pages=5)
for i, page in enumerate(sync_page_gen):
    page.to_csv(f'sync_example_{i}.csv')

async_page_gen = client.async_analytic_query(request,
                                             client_context='test query',
                                             max_rows=1000)
for i, page in enumerate(async_page_gen):
    page.to_csv(f'async_example_{i}.csv')