Getting data

Estimated time: 20 minutes

Overview

Questions

• How can I read the dataset into Python or R
• How can I access data via an API using Python or R

Objectives

• Read a dataset from file or fetch data via API

Before looking at the LiteLLM API, we want to make sure that we have a dataset to work with. We will be using publications data from OpenAlex. You can either load a prepared dataset (see Setup instructions) or use the API programmatically and obtain the data yourself.

Callout

You do this part on your own. Follow the instructions for your language.

Method 1: read data from file

---

Python

R

Read csv data

There are many ways to read tabular data in Python. We’ll be using pandas. If you’re not familiar with the library, you can follow this Carpentries episode.

PYTHON

import pandas as pd

Let’s read the dataset:

PYTHON

data = pd.read_csv("publications2024.csv")

Read json data

Inverted abstracts are stored in the JSON format which we already encountered in the previous episode. In Python, you can read and write JSON using the json library.

PYTHON

import json

We can use the with statement to open and read the file, then use json to load the JSON as a Python object, a dictionary.

PYTHON

with open("abstracts_inverted.json", "r") as f:
    abstracts_inverted = json.loads(f.read())

Read csv data

There are many ways to read tabular data in R, for instance the built-in read.csv(), which you might have encountered in this Carpentries episode.

Here, we will use the readr package from the tidyverse. First you need to install it:

R

install.packages("readr")

Then load it:

R

library(tidyverse)

Let’s read the dataset:

R

data <- read_csv("publications2024.csv")

Read json data

Inverted abstracts are stored in the JSON format which we already encountered in the previous chapter. In R, you can read and write JSON using the jsonlite library.

R

install.packages("jsonlite")
library(jsonlite)

Let’s read the inverted abstracts:

R

abstracts_inverted <- read_json("abstracts_inverted.json")

To obtain the original abstracts, you need to process the data further. More examples on how to work with JSON data in R, can be found here.

Method 2: get data via API

---

Callout

For this part, we will need HTTP libraries for either Python or R. We will use the most common libraries, but if you are already familiar with another, feel free to use those.

GET vs. POST requests

The two most relevant HTTP request types for our purposes are GET and POST.

In simple words, GET requests retrieve data from the server, without sending any data, while POST requests send data and retrieve data in response.

When interacting with a language model, we are most likely going to send input data (prompts) to the server, which means we will be making POST requests. To retrieve data from OpenAlex, we will be using GET.

Python

R

First, we need to install the requests library:

BASH

pip install requests

Now, we can import both requests and the built-in library json to parse json data.

PYTHON

import requests
import json

Retrieving the data is as simple as calling the get() method using our URL from before:

PYTHON

response = requests.get("https://api.openalex.org/works?filter=institutions.ror:04dkp9463,publication_year:2024")

data = response.json()

There is also a Python library that simplifies using the OpenAlex API: pyalex. The library is well-documented and its usage won’t be covered in this workshop.

First, we need to install and load the httr2 package:

R

install.packages("httr2")
library(httr2)

httr2 uses objects and you can use the pipe to build a request step-by-step. We retrieve the data using our URL from before:

R

# Prepare the request
req <- request("https://api.openalex.org/works?filter=institutions.ror:04dkp9463,publication_year:2024")

# Submit the request
resp <- req %>% req_perform()

# Retrieve data from the response
data <- resp %>% resp_body_json() %>% str()

There is also an R library that simplifies using the OpenAlex API: openalexR. The library is well-documented and its usage won’t be covered in this workshop.