In the past two posts within our Pandas series, we analyzed data from Chipotle restaurant and Flipkart online store. Today, we're going to look at Spotify dataset from a perspective of a recording studio start-up.

Btw if you are looking for roles in AI do check out https://www.deeplearning.careers

Imagine wanting to get started in the music industry. You believe you have a skill for spotting great talent and promoting them to become stars. Yet, apart from your ingenious ability, you need to be familiar with facts.

You want to know:

1. What tracks are most popular amongst Spotify users?
2. How many tracks gained popularity over 90 out of 100.
3. Which tracks released in March 2020 gained popularity among over 80.
4. Is there a correlation between popularity and a track's traits?
5. How long should an average track last according to today's standards?
6. What is the correlation between tracks' different features?
7. Who is currently most popular and what genres do they represent?

Data to download: https://www.kaggle.com/lehaknarnauli/spotify-datasets

For a detailed explanation of the dataset: https://developer.spotify.com/documentation/web-api/reference/#category-tracks

Before you continue reading, check out our new ChatGPT Tool for Job Seekers

Job hunting can be a demanding and time-consuming process. We believe our tool can take some of the stress out of it by leveraging AI to help you write compelling, personalized cover letters. We're excited to share this tool with you and look forward to hearing about your experiences and successes.

Experience the future of job application preparation. Give our ChatGPT-powered tool a try today!

https://jobseekerai.netlify.app/

Pre-processing the data

Today we'll only start with the Pandas library.

import pandas as pd

The Spotify dataset is quite huge and there are several files containing slightly different data. Today we'll use tracks and artists' datasets. We'll start with the tracks dataset.

# Loading the datset
df_tracks = pd.read_csv('/content/drive/MyDrive/tracks.csv')
df_tracks

You'll see that this dataset consists of 122860 rows and 20 columns. To be sure if we can trust this dataset, it's important to check if any values are missing.

pd.isnull(df_tracks).sum().sum()
71

Pandas pd.isnull()returns a dataset with booleans True and False saying if the value is missing. Calling sum() twice on this gives us a total number of all the missing values in the dataset. If we only used it once, we'd get a sum of missing values for each column.

There are just 71 missing values in a dataset consisting of over 500k rows. It means it not bad, and our conclusions will be valid. Such a dataset is a Pythonista's delight!

To make things easier, we should change the release_date to a date type and then put months and years into separate columns. We did a similar operation last time, so you might already be familiar with this method. Let's practice.

year = df_tracks['release_date'].apply(lambda x : x.year)
month = df_tracks['release_date'].apply(lambda x : x.month)
df_tracks.insert(loc=8, column='year', value=year) 
df_tracks.insert(loc=9, column='month', value=month)

This time instead of assigning a new column by defining df_tracks['year'] we used the function insert(). This method allows us to choose the exact position of the new column ( loc ). If we had done it the old-fashioned way, the new columns would have ended up at the last possible index.

Exploring the dataset

The dataset we're using right now consists of 20 columns. We'll find here information such as name, popularity, duration, explicit, artist, release_date, and tracks traits like danceability, speechiness, loudness , etc.

Popularity is measured on a scale between 0 and 100, where 100 is the best. Given our knowledge of the music industry, let's check if what we feel is true.

• What are the most popular songs right now?

To check this, let's use a great pandas function query(). This is a filtering function that enables the selection and filters the columns of a dataFrame with a boolean expression.

It is important to note that Dataframe.query() the method only works if the column name doesn’t have any empty spaces. So before applying the method, spaces in column names should be replaced with ‘_’

Let's learn how this function is used.

most_popular = df_tracks.query('popularity>90', inplace=False).sort_values('popularity', ascending=False)
most_popular[:10]

It's important to note that the whole function's expression is passed in quotation marks.

At the first sight, we can see that the first 10 most popular songs were released either in 2020 or 2021 and that almost half of them contain some explicit content indicated by the binary 1 in the explicit column.

To see if our conclusions are right, let's sort the filtered values and show the columns of interest.

pop_date = most_popular.sort_values('release_date', ascending=False)
pop_date[['name', 'popularity', 'explicit','release_date']][:20]

We know which songs are the most popular in general, but as a good producer, you need to understand human emotions and how they shape the market.

In times of crisis, both artists and the audience have different tastes. Last year in March, the world went under a complete lockdown. It's natural to wonder what were some of the top songs released back then.

We're going to use query() again, but this time going one step further and dealing with a bit more complex problem since we are defining two conditions.

The conditions are as follows:-

1. Songs with popularity greater than or equal to 80.
2. Songs that released in March 2020

Let's see how we'll approach this problem.

most_popular_march_20 = df_tracks.query('(popularity > 80) and (year in ["2020"]) and (month in ["3"])')
most_popular_march_20

The title of the songs seems to be mirroring the world's mood back then: 'Save Your Tears', 'Heartbreak Anniversary', 'Maniac', 'Break My Heart', and more.

Features and Popularity

Let's hop on to our next problem. We know that different features of a song can impact its popularity in different ways, however, we want to dig deeper and see how. This is one of the most important questions that we should ask.

• How do different features of a song impact its popularity?

We get an intuition that large audiences like songs that are compatible with dancing. Let's see if we can back this up with data as well.

df_1=df_tracks.groupby('popularity')['danceability'].mean().sort_values(ascending=[False]).reset_index()
df_1.head()

We have created a different dataset df_1. This dataframe will have the popularity for different songs grouped by the mean of the danceability score.

Now, this makes it easier for us to analyze the correlation between these two features.

We will use the basics of a plotting library in python called Plotly. It is an interactive, open-source plotting library that supports various charts.

It's important to import the library before moving ahead.

import plotly.express as px   #importing plotly
fig2 = px.scatter(df_tracks, x="popularity", y="danceability", color="danceability",size='popularity')
fig2.show()

Bingo! We have plotted a scatter plot. In this case, since the area of the circles corresponds to the danceability score, we can call this chart a Bubble Chart as well. This implies that the more the popularity score the greater will be the area corresponding to that particular bubble and vice versa.

The graph in itself is interactive and with one quick look at it, we release that 'popularity' and 'danceability' are positively correlated, which implies that, as the popularity of the song increase, the danceability score for that song also increases.

We don't always need to plot a graph to check the correlation between two features. The same can also be achieved with a few simple codes given below.

We will use a module called Scipy.Stats for this. This module contains a large number of probability distributions as well as a growing library of statistical functions.

Next, we import pearsonr function from this module, which helps us calculate Pearson's Correlation Constant 'r' for two different features.

Let's learn how.

from scipy.stats import pearsonr    #importing the library
data1 = df_1['popularity']
data2 = df_1['danceability']

# calculate Pearson's correlation
corr, _ = pearsonr(data1, data2)
print('Pearsons correlation: %.3f' % corr)

Pearsons correlation: 0.888

Doing a quick revision. The following are the three conditions for the Pearson's Correlation Coefficient 'r':-

1. r>0, implies, positive correlation
2. r=0, implies, no correlation
3. r<0, implies, negative correlation.

We see that since the value for the correlation coefficient r =0.88 (>0), the two features are positively correlated, or we can say, an increase in one feature will have an increase in the other feature and vice versa.

Let's visualize the same for some other features as well.

How about "instrumentalness"? Instrumentalness of value 1 indicates that there are no words at all, the lower the value the more words the song contains.

Following a similar procedure and plotting a Bubble Chart for Popularity v/s Instrumentalness.

Looking at the graph, it is easily noticeable that the two features are negatively correlated, which implies that, an increase in one leads to a decrease in another and vice versa.

So, do we need to check the correlation for each feature one by one? Not at all.

Thanks to pandas and plotly, we can easily check the correlation between any two given features. How? Let's find out.

We will use the combination of a function corr() in pandas, as well as, heatmap in plotly, so what exactly do these two do?

1. corr() - This function of pandas helps us compute pairwise correlation between different columns of the dataset (excluding NA/null values).
2. Heatmap - This function under plotly shows the magnitude of a phenomenon as colour in two dimensions.

To see how the combination of these two can help us achieve the desired result, we need to first import another plotly library called Graph Objects.

import plotly.graph_objects as go    #importing the library

matrix=df_tracks.corr()  #returns a matrix with correlation of all features
x_list=['popularity','duration_ms','explicit',
        'danceability','energy','key','loudness',
        'mode','speechiness','acousticness','instrumentalness',
        'liveness','valence','tempo','time_signature']

fig_heatmap = go.Figure(data=go.Heatmap(
                   z=matrix,
                   x=x_list,
                   y=x_list,
                   hoverongaps = False))
fig_heatmap.update_layout(margin = dict(t=200,r=200,b=200,l=200),
    width = 800, height = 650,
    autosize = False )

fig_heatmap.show()

Here, the legend of the graph shows us how the color gets lighter as the correlation increases. We observe that there is no significant positive correlation between popularity and a song's feature. The most positive correlation occurs between popularity, danceability, loudness, and energy.

Let's just quickly check one more question.

• How long does a song lasts on average today? Has it always been like that?

Before we visualize our result, it's important to change the unit of duration from milliseconds to minutes.

df['duration']=df['duration_ms']//1000 #Floor division to get only the quotient
df.drop(['duration_ms'], axis = 1)

We observe that the average duration of the song has increased since 1969, but has remained more or less the same ever since. More information can be observed through the above graph.

Most popular artists

To check the most popular artists we'll use the artists' dataset.

We will start by importing the dataset into our python environment.

df_artists = pd.read_csv('/content/drive/MyDrive/artists.csv')
df_artists.head()

Our dataset has 1104349 rows and 5 columns.

• Analysis was done on Artists:-

artists_popular = df_artists.sort_values(by=['popularity'], ascending=False).reset_index()
artists_popular[:10]

We notice that the top ten songs and artists differ. Justin Bieber is an unquestionable king of pop, and although their songs are not the most popular right now, The Weeknd, Taylor Swift, and Drake are the listeners' favorites too.

• Analyzing the Genres:-

Looking at the head of the dataframe, we also observe that for many rows, the column 'genres' is an empty list, these can be seen as NA values as well. To handle this type of situation it's important to see the proportion of such rows to the overall shape of the dataset.

df_artists[df_artists["genres"]=='[]']

We see that there are 59742 rows with empty lists passed out of the total 89336 rows of the dataset.

For this particular case, we will create and perform our analysis on a new dataframe with only those rows that contain some value under the column 'genres'.

df_genre=df_artists[df_artists["genres"]!='[]']
df_genre.head()

We have successfully segregated the rows we require for our analysis.

We observe that the column 'genres' has a list passed as value. Let's split these lists into individual values. For this task, we will use a very special function of pandas, called explode().

• explode() function splits the list by each element and create a new row for each of the element.

df_sort_genres=pd.DataFrame(df_genre.assign(genres=df_genre.genres.str.split(",")).explode('genres'))
df_sort_genres.tail()

Replacing the square brackets with " " blank spaces to get only the keywords.

df_sort_genres['genres']=df_sort_genres.genres.str.replace('[',' ')
df_sort_genres['genres']=df_sort_genres.genres.str.replace(']',' ')

Let's analyze the top 30 genres now.

# get top 30 most commom genres
n = 30
top_30=pd.DataFrame(df_sort_genres['genres'].value_counts()[:n]).reset_index()
top_30.rename(columns = {'index':'Genres','genres':'Total_Count'}, inplace = True)
top_30

Should we visualize this output using a pie chart? Or wait, let's make it more interesting and visualize it using a Donut Chart

We observe that the top 30 genres have more or less the same count with the 'Dance Pop' at the top. More information can be seen using the graph above.

What we learned

• For tracks.csv file:-

1. We found the first 10 most popular songs released in 2020 & 2021 using query() function of pandas.
2. We found the titles of the most popular songs of March 2020.
3. We saw how different features are correlated with each other using the corr() and the heatmap functions of pandas and plotly respectively
4. We learned to visualize our result using Bubble Charts.

• For artists.csv:-

1. We found the top 10 artists
2. We learned the use of the explode() function and found the top 30 genres.
3. We visualized our result using a Donut Chart.

Now you know what to do to be a successful producer and use data and analytics to your advantage to pick the next big hit.

Go to https://datascience.fm/tag/pandas/ for more tutorials where we take popular datasets and analyze them with Pandas.

If you are looking for jobs in AI and DS check out Deep Learning Careers

Feedback is important to us. Write us at hello@datascience.fm

Tell us what articles you want to see more of and the kinds of YouTube videos we should create. for you.