Building a RESTful Web API with Python

In this article, the goal is to build a web API or service that adheres to the RESTful principles. But before the design phase, let’s explain some concepts.

What is an API?

An API is a part of a computer program designed to be used or manipulated by another computer program. As user Interfaces exist for humans to interact with computer programs, APIs exist for computer programs to interact with other computer programs.

What Is a Web API?

A web API or service provides functionality or data to be manipulated by other programs via the Internet. Intuitively, we can already assume that there are other types of APIs apart from a web API. Desktop applications like spreadsheets use VBA APIs which doesn’t require a web service. A web service is an API (wrapped in HTTP) but an API is not necessarily a web service. Recently, the standard architecture for designing web services or APIs has taken a shift towards REST (REpresentational State Transfer).

What Is REST?

The REST architecture was originally designed to fit the HTTP (HyperText Transfer Protocol) that the World Wide Web uses. REST is based on the four methods defined by the HTTP protocol: POST, GET, PUT, and DELETE corresponding to the four traditional actions performed on data in a database: CREATE, READ, UPDATE, and DELETE. In this article, we’ll only be inserting into a database, reading from a database and updating a database. This means that we would only be concerned with the POST, GET and PUT requests respectively. The REST design principles has six major characteristics:

  • Client-Server: This means that the difference between a client and a server should be clear.
  • Stateless: The server is not allowed to use information provided in a request for another request. This means that the client should include authentication for every request sent to the server.
  • Uniform Interface: The method of communication between a client and a server must be uniform. This means that every entity in the application needs to be uniquely identified. The web API in this article will use URLs to uniquely identify our resources. Also, the format for the data to be exchanged must be agreed between the client and the server. The JSON (JavaScript Object Notation) format will be used to represent the resource. The relationship between different resources need to be specified clearly in the resource representation so that the client can take advantage of them to discover new resources.
  • Cacheable: For performance reasons, the server, client or the intermediate systems should be able to cache data. The server must indicate and provide instructions to the client or the intermediate system if requests can be cached or not and how they can be cached.
  • Layered System: The client does not need to be necessarily connected directly to the server instead it can be connected through an intermediate system that relays the request to the server.
  • Code on Demand: This indicates that the server can provide code that the client can execute in their context. This property is the only optional one and it’s not practical because there is a great variety of clients with different capabilities in terms of executing code.

At the end of this article, we would have built a web service or API that adheres to the RESTful principles while pulling from and pushing to a database.

Let’s Create a Virtual Environment for Our Project

A virtual environment is a tool that helps to keep project dependencies separate by creating an isolated Python virtual environment for our dependencies. My assumption is that we have Python installed already on our operating system and our chosen operating system is Windows. Before continuing, let’s create a project folder where we can save all the files we will be working with.

  • NOTE: Only few of the commands will change if we are working from another OS, but it’s the same procedure and so we can just google (if necessary) to follow along.

With that, let us open a terminal session in the project folder and run the following command (without the path):

The above command creates a virtual environment. myenv is the name of the virtual environment, we can also use this same name or choose a different one. Next, we need to activate the virtual environment and to do this we run the following command (without the path):

This activates your virtual environment and it’s obvious with the name of the virtual environment shown in brackets before the folder path as below:

Next, let’s install two packages that don’t come out of the box with Python, but will be needed for our project. To install these two packages, we need to run the following command (without the path):

The above command installs Pandas and Flask.

  • Pandas is a data structures and data analysis tool. This will be used to view the content of our database tables in a DataFrame.
  • Flask is a micro web framework that is easily extensible. This will be used for our API development.

Next, let’s create a script in our project folder to view the two tables we would be working with and name the script view_db.py (also, we can choose any name we want).

  • NOTE: In order to follow along without having to code just yet, we can clone from my GitHub repository.
An Overview of Our Database

The database used is SQLite, a lightweight database engine that can be handled by a Python library (that comes with the standard library). The source of the database to be used is SQLite Tutorial. We can download the database into our project folder. This is a screenshot of the database diagram below: 

Image Source: https://www.sqlitetutorial.net

In this database, we can see so many tables, but we will be using only two for our API: artists and albums. The artists and albums have a one to many relationship, meaning an artist can have several albums and not vice versa. Now, let us take a look at the content of these two tables using Python.

view_db.py

Basically, what the code does is that it first gets the information (for example, data type) of each column and saves it in two different variables. Then, queries the two database tables and saves the result into two other variables. Next, it creates two dataframes using the two variables that have the two database table queries. Finally, it outputs to the screen; the information of each column and the dataframes. The output below also contains the command to execute the code.

    • Output

    • SUMMARY OF OUTPUT

Our artists table has two columns (ArtistId and Name) and 275 rows while our albums table has three columns (AlbumId, Title and ArtistId) and 347 rows. The ArtistId in the albums table is a Foreign Key from the artists table and this helps to map albums to artists. Also, ArtistId and AlbumId columns take in only integers while Name and Title taken in only text with maximum of 120 and 160 characters respectively. Now that we know what our tables contain, we can start playing with the data and so it’s time to start designing our web service.

Design Phase

The final structure of our project is shown in the picture above. Already, we have view_db.py, myenv and chinook.db.

  • NOTE: We adapted this structure to make our program easy to maintain and the choice is not necessarily the best, its relative.

For the sake of quick wins, let’s run the following command on the terminal:

The command creates a requirements.txt file which contains all the installed dependencies of this project and the installed versions. This file is used to ensure that if the project needs to be moved to a new machine, with pip install –r requirements.txt on the new machine, all the packages/dependencies (with the same versions) gets installed on the new machine. So, open it up to check and what we should have is what is in the picture below (if we have been following diligently);

High Level View of the Other Files
  • api.py: This is the module that we will need to run to get our API started.
  • art_alb: This is a sub-directory to the project folder.
    • __init__.py: In Python, a folder that includes a __init__.py file is considered a package and this __init__.py automatically runs when the package is imported. Hence, art_alb is the package in this project. The __init__.py module will be used for creating an instance of our web API.
    • albums.py: This defines a route/resource to access the albums table and it implements the POST and PUT methods to insert into and update values in the albums table respectively and eventually returns the data in JSON format.
    • artists_albums.py: This defines a route/resource to access the artists and albums tables and it implements the GET method which queries the database. Then, it returns data in JSON format. The data returned contains the ArtistID, Name (of the artist) and Albums (a list of all the albums mapped to the particular artist).
    • artists.py: This defines a route to access the artists table and it implements the POST and PUT methods to insert into and update the artists table respectively and return the data in JSON format.
      NOTE: POST method appends to the end of the collection. That is, every time we implement the POST method, the new data becomes the last in the collection.
    • errors.py: This defines error handlers in order to return the errors in json format, thereby adhering to resource representation.

NOTE: Before continuing, I assume we have a little experience working with the Flask framework.

__init__.py

Firstly, we created an instance of our application after all the necessary imports. Then we defined the dict_factory function which is used to return values from our database with name-based columns. That is, it returns items from the database as dictionaries rather than lists. We prefer dictionaries returned because this work better when we output them to JSON. Also, it can be noticed that we imported our modules (our other resources) at the bottom of our file and not at the top (as usual). This has been done to avoid what we call circular dependency errors in Python. In other to understand what circular dependency errors are, you can watch this video by Miguel Grinberg.

artists_albums.py

Firstly, we import all the packages and modules needed for this resource. Then we create the resource itself by using a concept in Python called Decorators which is used by Flask to define routes.

  • NOTE: The route defined is based on choice but we should always ensure we define clear routes.

Decorators add new functionality to an existing object without modifying its structure. So the function defined under the decorator is a view function which contains the code written to respond to requests sent to an application. The function queries both tables which returns a list of dictionaries. Each dictionary represents a key-value pair of the column name and the value on each row. Next, we create a new key AlbumTitle in each of the artists dictionary whose value is a list of albums mapped to the artist. Then, we create a filter that can be obtained from the query parameters to filter using the ArtistId or Name. Finally, we checked for some conditions in order to ensure that the request sent to the server is valid (adhering to the UNIFORM INTERFACE characteristics of REST). For an invalid request, the proper error to be returned has been mapped. If it’s a valid request, it either returns all the artists or an artist (if there are query parameters).

errors.py

Notice that the decorators here are different from the one of the resources, these ones are error handlers. That is, we can write the logic to be implemented when the application fails on a particular HTTP status code. In this application, the HTTP status code for the errors we are handling are 400, 404 and 500. The logic in these error handlers just ensures that the error is returned in JSON format (and not HTML), thereby adhering to the UNIFORM INTERFACE characteristic of the REST principle.

  • NOTE: We can view the remaining files when we clone or download from GitHub (but we won’t view the content here). But we would run them to see what they do as defined in the HIGH LEVEL VIEW above.
Let’s Get Our Web Service Running

We will use curl (and not web browsers) to test our web service because web browsers cannot easily generate all types of HTTP requests.

  • NOTE: curl is used in command lines or scripts to transfer data. Install curl now if you don’t have it. Start the web service by running (from a console window):

and then open a new terminal and run the following command:

Because we have about 275 rows in our data, the returned data may look clumsy on the console window. So, let’s open JSON FORMATTER & VALIDATOR in any browser. Next, we copy the returned data from the console window and on the web page in our browser, paste the copied data into the JSON Data/URL textbox, and finally click the process button. The processed data is displayed in a Formatted JSON Data textbox. All the output can’t be captured in a view because we have about 275 rows of data. So, I took screenshots of the top and bottom and displayed below for us to see what it looks like:

  • FIRST 2 (ArtistId 1 and 2)

  • LAST 2 (Artist Id 274 and 375)

The above corresponds to the data we viewed using the DataFrame. Next, let’s filter using ArtistId by running the following command:

This command has successfully fetched us our artist with the ArtistId of 275. Next, let’s query with the Name and to confirm that the resource is working as expected, let’s query using the Name of the artist in our last result (Philip Glass Ensemble). To do this, let’s run the following command.

NOTE: We added %20 to encode the space between the name because URLs should not have spaces between them. We could have also used + instead of %20.

The returned value is the same as what we have above.

Finally, let’s query for an artist using an ArtistId that does not exist in our database (say 1000) to see what happens. Hence, let’s run the following command:

As expected, NOT FOUND is the returned value because the ArtistId does not exit. We can continue playing with this resource (if we choose to). Moving on to the artists.py containing the resource that gives us the capability to insert a new artist and update an existing artist in artists table. Starting with the POST method, let’s insert two new artists into our database. First, J Cole and next, Naira Marley. In order to do this, let’s run the following command:

Both commands return the whole content (with the new content) in our artists table in JSON format. However, since we have about 275 rows in our database and for the sake of visibility, we can as well use JSON FORMATTER & VALIDATOR to inspect as before. However, let’s run the following commands to query our database to confirm that these two new artists have been added:

Both artists are found as shown in the output. Next let’s update our artists database by implementing the PUT method. So, let’s change, J Cole to Kanye West and Naira Marley to Zlatan. This will help us show that we can either use the ArtistId or Name or both (and that means the ArtistId must match the Name) to determine the row to be updated. To do this, run the following command:

NOTE: The ArtistId for Naira Marley is 277.

Both commands return the whole content in our artists database in JSON format. To confirm our update has taken place, let’s query for Kanye West and Zlatan. To do this, run the following commands:

The output shows that Kanye West and Zlatan now exist in our database and that they don’t have any albums mapped to them yet. Let’s confirm that J Cole and Naira Marley don’t exist again. To do this, let’s run the following command to query for them:

The output shows NOT FOUND meaning that they don’t exist again as expected. Also, let’s confirm what happens when we try to update our database and the request the client sends contains ArtistId and Name (without them matching). Using Zlatan as the use case, Zlatan has an ArtistId of 277. So, we would use an ArtistId of 266 in our request. Hence, let’s try changing Zlatan to Kcee.

Yes, it shows a BAD REQUEST because the ArtistId and Name don’t match and I hope this makes sense because ArtistId is unique for each artist in our artists table. There’s more we can try with this resource but for the sake of time, let’s move to the albums.py. This is very similar to the artists.py and so we won’t dwell on this, I will just show that as soon as we create an album, it automatically maps to our artist. Next, Zlatan will be assigned an album with Jogor as the Title (since it is known to us that Zlatan has an ArtistId of 277 and ArtistId is part of what is required to create an album). So, let’s run the command below:

This returns the content of the album database (with the new album) in JSON format. Also, the following command should be run to show that this album has been actually mapped to Zlatan.

GREAT, mapping has been done correctly.

Believe me you, the resource provides more functionality than I have shown, so it would be nice if we tried more things on our own. This brings us to the END.

Thank you for reading. But before leaving, I have some ideas on how the project can be extended (if interested).

Ideas to Extend Project
  • Implementing the DELETE method for the resources.
  • Securing the web service by sending authentication with every request to the server.
  • Rather than the ArtistId in the returned data when we send a request to the resource in albums.py, we can have the path (URL) to the artist instead.
  • Add more tables from the database.

Related Posts