Introduction to Siuba

Resources

• Workshop homepage
• Workshop repo (machow/siuba-workshop)
• Live notes (Notion doc)
• siuba docs
• plotnine docs

Introduction to siuba

Hi, I’m Michael–the creator of siuba, a library for data analysis in Python.

In this course, you’ll use Python to explore and visualize data.

The course is designed for people who have never programmed before, and anyone interested in siuba!

Data Analysis

Data analysis is the act of using data to produce effective communication. Communication that leads to meaningful action.

In this course, we’ll focus on one area of data analysis where siuba shines: transforming data.

We’ll also combine siuba with a library called plotnine to visualize data. Together, these libraries make a powerful combo.

Meet the data: Spotify top 200

We’ll use data that Spotify publishes every week on the 200 most streamed songs, in 62 different countries.

You can check it out on spotifycharts.com

Meet the data: Spotify top 200

music_top200

	country	position	track_name	artist	streams	duration	continent
0	Argentina	1	Tusa	KAROL G	1858666	200.960	Americas
1	Argentina	2	Tattoo	Rauw Alejandro	1344382	202.887	Americas
2	Argentina	3	Hola - Remix	Dalex	1330011	249.520	Americas
...	...	...	...	...	...	...	...
12397	South Africa	198	Black And White	Niall Horan	11771	193.090	Africa
12398	South Africa	199	When I See U	Fantasia	11752	217.347	Africa
12399	South Africa	200	Psycho!	MASN	11743	197.217	Africa

12400 rows × 7 columns

The data we’ll use is held in a DataFrame. A DataFrame is a big table of data, made up of rows and columns. In the example below, the variable called music_top200 lets us refer to and work on the data.

Notice that in the bottom-left of the table, it shows the number of rows and columns. This data has 12,000 rows and 6 columns.

Meet the data: Spotify top 200

music_top200

	country	position	track_name	artist	streams	duration	continent
0	Argentina	1	Tusa	KAROL G	1858666	200.960	Americas
1	Argentina	2	Tattoo	Rauw Alejandro	1344382	202.887	Americas
2	Argentina	3	Hola - Remix	Dalex	1330011	249.520	Americas
...	...	...	...	...	...	...	...
12397	South Africa	198	Black And White	Niall Horan	11771	193.090	Africa
12398	South Africa	199	When I See U	Fantasia	11752	217.347	Africa
12399	South Africa	200	Psycho!	MASN	11743	197.217	Africa

Every observation–or row–in the DataFrame is a track in the top 200 for a country.

For example, in the highlighted code is the track in position 2 (second most streamed) in Argentina. The track name is Tattoo, and has been streamed 1,344,382 times, and is 202 seconds long!

Meet the data: Spotify song features

Later in the course, we’ll look at measures Spotify calculates for each song.

Their measures for a song include:

• energy
• valence (how happy or positive)
• danceability
• speechiness
• acousticness

Data Analysis

The skills you’ll build doing transformation and visualization in this course will allow you to start analyzing your own data.

The course is interactive: between short lessons you’ll complete interactive exercises by typing your own code.

How code is structured

(track_features
  >> filter(_.artist == "The Weeknd")
  >> ggplot(aes("energy", "valence"))
   + geom_point()
)

Here’s an example of plotting the energy and valence of tracks by The Weeknd. Each point is a single track (for example, the song “Blinding Lights”).

The top line of the code is the data. Every additional line is an action applied to the data.

Click down to see what each line of code does.

How code is structured

(track_features
  >> filter(_.artist == "The Weeknd")

 
)

	artist	album	track_name	energy	valence	danceability	speechiness	acousticness	popularity	duration
568	The Weeknd	My Dear Melancholy,	Call Out My Name	0.593	0.175	0.461	0.0356	0.17000	82	228.373
2753	The Weeknd	Blinding Lights	Blinding Lights	0.796	0.345	0.513	0.0629	0.00147	75	201.573
3004	The Weeknd	In Your Eyes (Remix)	In Your Eyes (with Doja Cat) - Remix	0.731	0.727	0.679	0.0319	0.00518	81	237.912
...	...	...	...	...	...	...	...	...	...	...
23966	The Weeknd	Beauty Behind The Madness	The Hills	0.564	0.137	0.585	0.0515	0.06710	83	242.253
24688	The Weeknd	Starboy	Starboy	0.587	0.486	0.679	0.2760	0.14100	84	230.453
24982	The Weeknd	After Hours	In Your Eyes	0.719	0.717	0.667	0.0346	0.00285	91	237.520

23 rows × 10 columns

The first action is filtering the data, to keep only observations (rows) where artist is The Weeknd.

Don’t worry to much about the details for now. Filter is the first thing you’ll learn, once you start the first chapter!

How code is structured

(track_features
  >> filter(_.artist == "The Weeknd")
  >> ggplot(aes("energy", "valence"))
 
)

The next action, ggplot(...), gets ready to make a plot, based on the data in the previous step.

How code is structured

(track_features
  >> filter(_.artist == "The Weeknd")
  >> ggplot(aes("energy", "valence"))
   + geom_point()
)

Finally, geom_point() adds points to the plot. Each point comes from a row of data.

So in this case each point is a track by The Weeknd.

Let’s practice!

• Workshop setup instructions

For the practice below, you’ll get to explore the two datasets by testing out different options.

This is to make sure you get the big picture, before we dive into the specifics of how each piece of code works!

The filter verb

Now that you’ve seen some code in action on the music data, we’ll focus on writing your own code.

In this chapter, you’ll learn about the “verbs” in the siuba package. The first verb you’ll use is filter.

The filter verb

You use filter when you want to keep only a subset of your observations, based on a particular condition.

Filtering data is a common first step in an analysis.

Filter for top songs

(music_top200
  >> filter(_.position == 1)
)

Every time you apply a verb, you’ll use a pipe. A pipe is a block of commands, surrounded by parentheses.

For example, let’s say we want to keep only songs in the first position on the music charts. This is done by using the pipe shown on the slide.

Click the down arrow on the slideshow to see broken into 3 steps:

1. start the block
2. write the pipe operator and verb name
3. write the operation

Filter step 1: start the block

(music_top200

)

A block is written with opening and closing parentheses, and the name of your data in the middle.

Press enter twice after the name of your data, to make an empty line.

Filter step 2: pipe operator and verb name

(music_top200
  >> filter()
)

Next, write a pipe operator using two greater than signs. It says “take whatever is before the pipe operator, and feed it into the next step.”

In this case, the next step will be filter.

Filter step 3: write the operation

(music_top200
  >> filter(_.position == 1)
)

Finally, we can complete our first verb.

We have all 200 hit songs on the charts, but just want to get the first.

The “position equals equals 1” is the condition we are using to filter observations. The “equals equals” may be surprising: it’s what we call a “logical equals”- an operation to compare two values: each position, and the number 1.

A single equals here would mean something different in python, which you’ll see later.

Let’s see what this code outputs.

Filter for top songs

(music_top200
  >> filter(_.position == 1)
)

	country	position	track_name	artist	streams	duration	continent
0	Argentina	1	Tusa	KAROL G	1858666	200.960	Americas
200	Austria	1	Blinding Lights	The Weeknd	229576	201.573	Europe
400	Australia	1	Blinding Lights	The Weeknd	1757343	201.573	Oceania
...	...	...	...	...	...	...	...
11800	Uruguay	1	Tusa	KAROL G	120175	200.960	Americas
12000	Viet Nam	1	Sweet Night	V	189261	214.259	Asia
12200	South Africa	1	The Box	Roddy Ricch	94422	196.653	Africa

62 rows × 7 columns

Notice that now, we have only 62 rows: that’s how many countries are in the dataset.

It’s important to note that you’re not removing any rows from the original music data. You can still use the music object for other analyses, and it won’t be any different than it was before.

Instead, filter is returning a new dataset, one with fewer rows, that then gets printed to the screen.

Filter for country

(music_top200
  >> filter(_.country == "United States")
)

	country	position	track_name	artist	streams	duration	continent
7800	United States	1	The Box	Roddy Ricch	12987027	196.653	Americas
7801	United States	2	Myron	Lil Uzi Vert	9163134	224.955	Americas
7802	United States	3	Blueberry Faygo	Lil Mosey	8043475	162.547	Americas
...	...	...	...	...	...	...	...
7997	United States	198	Lights Up	Harry Styles	1606234	172.227	Americas
7998	United States	199	Without Me	Halsey	1606153	201.661	Americas
7999	United States	200	Enemies (feat. DaBaby)	Post Malone	1597824	196.760	Americas

200 rows × 7 columns

You could choose another condition to filter on, besides the position. For example, suppose we wanted to get only the observations from the United States.

We would write this as “filter country equals equals quote United States endquote”, resulting in only the 200 observations from that country.

The quotes around United States are important: otherwise Python won’t understand that the words “Hong” and “Kong” are the content of a text variable, as opposed to variable names. You didn’t need quotes around a number like 1, but you do around text.

Filter with two variables

(music_top200
  >> filter(_.position == 1, _.country == "United States")
)

	country	position	track_name	artist	streams	duration	continent
7800	United States	1	The Box	Roddy Ricch	12987027	196.653	Americas

1 rows × 7 columns

We can specify multiple conditions in the filter.

Each of the conditions is separated by a comma: here we are saying we want only the one observation for position 1, comma, where the country is the United States.

Each of these equals equals expressions is called an argument. This kind of double filter is useful for extracting a single observation you’re interested in.

You’ll be able to practice this in the exercises.

Let’s practice!

• siuba guide - filter
• siuba api - filter

Scroll down to investigate the music data with filter.

The arrange verb

In the last video you learned the filter verb, for extracting a subset of your observations based on a condition.

Now you’ll learn the arrange verb.

Arrange sorts the observations in a dataset, in ascending or descending order based on one of its variables.

This is useful, for example, when you want to know the most extreme values in a dataset.

The arrange verb

Sorting with arrange

(music_top200
  >> arrange(_.duration)
)

	country	position	track_name	artist	streams	duration	continent
10868	Slovakia	69	Klop Klop	Karlo	17222	65.631	Europe
4586	Greece	187	FENDI	iLLEOo	16786	76.099	Europe
9937	Poland	138	Mistrz ping-ponga	PRO8L3M	145143	83.360	Europe
...	...	...	...	...	...	...	...
535	Australia	136	Innerbloom	RÜFÜS DU SOL	260092	578.041	Oceania
1302	Brazil	103	Poesia Acústica #8: Amor e Samba	Pineapple StormTv	839192	614.615	Americas
11557	Turkey	158	Susamam	Şanışer	194804	851.871	Asia

12400 rows × 7 columns

Just like filter, you use the arrange verb after the pipe operator.

You would type music, then the pipe operator (two greater than symbols) and then arrange. Within those parentheses, you tell it what column you want to arrange by.

The observations are now sorted in ascending order, with the lowest duration songs appearing first.

Look at the second to rightmost column: it starts with 65.631, the smallest value in the dataset, then keeps increasing. Looking at the track name column, this track is called Klop Klop, and is the shortest track in the data.

Just like with filter, the music object itself is unchanged: arrange is just giving you a new, sorted dataset.

arrange descending

(music_top200
  >> arrange(-_.duration)
)

	country	position	track_name	artist	streams	duration	continent
11557	Turkey	158	Susamam	Şanışer	194804	851.871	Asia
1302	Brazil	103	Poesia Acústica #8: Amor e Samba	Pineapple StormTv	839192	614.615	Americas
535	Australia	136	Innerbloom	RÜFÜS DU SOL	260092	578.041	Oceania
...	...	...	...	...	...	...	...
9937	Poland	138	Mistrz ping-ponga	PRO8L3M	145143	83.360	Europe
4586	Greece	187	FENDI	iLLEOo	16786	76.099	Europe
10868	Slovakia	69	Klop Klop	Karlo	17222	65.631	Europe

12400 rows × 7 columns

arrange also lets you sort in descending order. To do that, you’d put a negative sign next to the variable you’re sorting by.

This lets us see that the track with the longest duration is Susamam, which is a hit in at least Turkey. It’s almost 15 minutes long!

However, we might be interested in looking at duration within a specific country.

arrange and filter

(music_top200
  >> filter(_.country == "United States")
  >> arrange(-_.duration)
)

	country	position	track_name	artist	streams	duration	continent
7841	United States	42	After Hours	The Weeknd	3672033	361.027	Americas
7915	United States	116	Life Is Good (feat. Drake, DaBaby & Lil Baby) - Remix	Future	2181930	315.346	Americas
7923	United States	124	SICKO MODE	Travis Scott	2085268	312.820	Americas
...	...	...	...	...	...	...	...
7832	United States	33	Strawberry Peels (feat. Young Thug & Gunna)	Lil Uzi Vert	4007781	115.350	Americas
7853	United States	54	CITY OF ANGELS	24kGoldn	3443366	112.493	Americas
7971	United States	172	Skechers	DripReport	1731265	106.000	Americas

200 rows × 7 columns

Suppose you wanted to find the longest duration song in the United States.

To do that, you can combine the two verbs you’ve already learned: filter, and arrange.

arrange and filter

(music_top200
  >> filter(_.country == "United States")

)

	country	position	track_name	artist	streams	duration	continent
7800	United States	1	The Box	Roddy Ricch	12987027	196.653	Americas
7801	United States	2	Myron	Lil Uzi Vert	9163134	224.955	Americas
7802	United States	3	Blueberry Faygo	Lil Mosey	8043475	162.547	Americas
...	...	...	...	...	...	...	...
7997	United States	198	Lights Up	Harry Styles	1606234	172.227	Americas
7998	United States	199	Without Me	Halsey	1606153	201.661	Americas
7999	United States	200	Enemies (feat. DaBaby)	Post Malone	1597824	196.760	Americas

200 rows × 7 columns

Longest duration song in the United States.

You start with the music dataset, then a pipe to give the dataset to filter. Then you specify that you want to filter for country equals equals United States.

Then you use another pipe step.

arrange and filter

(music_top200
  >> filter(_.country == "United States")
  >> arrange(-_.duration)
)

	country	position	track_name	artist	streams	duration	continent
7841	United States	42	After Hours	The Weeknd	3672033	361.027	Americas
7915	United States	116	Life Is Good (feat. Drake, DaBaby & Lil Baby) - Remix	Future	2181930	315.346	Americas
7923	United States	124	SICKO MODE	Travis Scott	2085268	312.820	Americas
...	...	...	...	...	...	...	...
7832	United States	33	Strawberry Peels (feat. Young Thug & Gunna)	Lil Uzi Vert	4007781	115.350	Americas
7853	United States	54	CITY OF ANGELS	24kGoldn	3443366	112.493	Americas
7971	United States	172	Skechers	DripReport	1731265	106.000	Americas

200 rows × 7 columns

Longest duration song in the United States.

The added pipe line takes the result of the filter, and gives it to arrange. You specify that you want to sort in descending order of duration.

arrange and filter

(music_top200
  >> filter(_.country == "United States")
  >> arrange(-_.duration)
)

	country	position	track_name	artist	streams	duration	continent
7841	United States	42	After Hours	The Weeknd	3672033	361.027	Americas
7915	United States	116	Life Is Good (feat. Drake, DaBaby & Lil Baby) - Remix	Future	2181930	315.346	Americas
7923	United States	124	SICKO MODE	Travis Scott	2085268	312.820	Americas
...	...	...	...	...	...	...	...
7832	United States	33	Strawberry Peels (feat. Young Thug & Gunna)	Lil Uzi Vert	4007781	115.350	Americas
7853	United States	54	CITY OF ANGELS	24kGoldn	3443366	112.493	Americas
7971	United States	172	Skechers	DripReport	1731265	106.000	Americas

200 rows × 7 columns

The result shows that the longest duration track in the United States is After Hours by The Weeknd.

We can explore many such questions with various combinations of dplyr verbs.

Over the course of these lessons, you’ll learn to pipe together multiple simple operations to create a rich and informative data analysis.

Let’s practice!

• siuba guide - arrange
• siuba api - arrange

The arrange verb

In the last video you learned the filter verb, for extracting a subset of your observations based on a condition.

Now you’ll learn the arrange verb.

Arrange sorts the observations in a dataset, in ascending or descending order based on one of its variables.

This is useful, for example, when you want to know the most extreme values in a dataset.

The arrange verb

Sorting with arrange

(music_top200
  >> arrange(_.duration)
)

	country	position	track_name	artist	streams	duration	continent
10868	Slovakia	69	Klop Klop	Karlo	17222	65.631	Europe
4586	Greece	187	FENDI	iLLEOo	16786	76.099	Europe
9937	Poland	138	Mistrz ping-ponga	PRO8L3M	145143	83.360	Europe
...	...	...	...	...	...	...	...
535	Australia	136	Innerbloom	RÜFÜS DU SOL	260092	578.041	Oceania
1302	Brazil	103	Poesia Acústica #8: Amor e Samba	Pineapple StormTv	839192	614.615	Americas
11557	Turkey	158	Susamam	Şanışer	194804	851.871	Asia

12400 rows × 7 columns

Just like filter, you use the arrange verb after the pipe operator.

You would type music, then the pipe operator (two greater than symbols) and then arrange. Within those parentheses, you tell it what column you want to arrange by.

The observations are now sorted in ascending order, with the lowest duration songs appearing first.

Look at the second to rightmost column: it starts with 65.631, the smallest value in the dataset, then keeps increasing. Looking at the track name column, this track is called Klop Klop, and is the shortest track in the data.

Just like with filter, the music object itself is unchanged: arrange is just giving you a new, sorted dataset.

arrange descending

(music_top200
  >> arrange(-_.duration)
)

	country	position	track_name	artist	streams	duration	continent
11557	Turkey	158	Susamam	Şanışer	194804	851.871	Asia
1302	Brazil	103	Poesia Acústica #8: Amor e Samba	Pineapple StormTv	839192	614.615	Americas
535	Australia	136	Innerbloom	RÜFÜS DU SOL	260092	578.041	Oceania
...	...	...	...	...	...	...	...
9937	Poland	138	Mistrz ping-ponga	PRO8L3M	145143	83.360	Europe
4586	Greece	187	FENDI	iLLEOo	16786	76.099	Europe
10868	Slovakia	69	Klop Klop	Karlo	17222	65.631	Europe

12400 rows × 7 columns

arrange also lets you sort in descending order. To do that, you’d put a negative sign next to the variable you’re sorting by.

This lets us see that the track with the longest duration is Susamam, which is a hit in at least Turkey. It’s almost 15 minutes long!

However, we might be interested in looking at duration within a specific country.

arrange and filter

(music_top200
  >> filter(_.country == "United States")
  >> arrange(-_.duration)
)

	country	position	track_name	artist	streams	duration	continent
7841	United States	42	After Hours	The Weeknd	3672033	361.027	Americas
7915	United States	116	Life Is Good (feat. Drake, DaBaby & Lil Baby) - Remix	Future	2181930	315.346	Americas
7923	United States	124	SICKO MODE	Travis Scott	2085268	312.820	Americas
...	...	...	...	...	...	...	...
7832	United States	33	Strawberry Peels (feat. Young Thug & Gunna)	Lil Uzi Vert	4007781	115.350	Americas
7853	United States	54	CITY OF ANGELS	24kGoldn	3443366	112.493	Americas
7971	United States	172	Skechers	DripReport	1731265	106.000	Americas

200 rows × 7 columns

Suppose you wanted to find the longest duration song in the United States.

To do that, you can combine the two verbs you’ve already learned: filter, and arrange.

arrange and filter

(music_top200
  >> filter(_.country == "United States")

)

	country	position	track_name	artist	streams	duration	continent
7800	United States	1	The Box	Roddy Ricch	12987027	196.653	Americas
7801	United States	2	Myron	Lil Uzi Vert	9163134	224.955	Americas
7802	United States	3	Blueberry Faygo	Lil Mosey	8043475	162.547	Americas
...	...	...	...	...	...	...	...
7997	United States	198	Lights Up	Harry Styles	1606234	172.227	Americas
7998	United States	199	Without Me	Halsey	1606153	201.661	Americas
7999	United States	200	Enemies (feat. DaBaby)	Post Malone	1597824	196.760	Americas

200 rows × 7 columns

Longest duration song in the United States.

You start with the music dataset, then a pipe to give the dataset to filter. Then you specify that you want to filter for country equals equals United States.

Then you use another pipe step.

arrange and filter

(music_top200
  >> filter(_.country == "United States")
  >> arrange(-_.duration)
)

	country	position	track_name	artist	streams	duration	continent
7841	United States	42	After Hours	The Weeknd	3672033	361.027	Americas
7915	United States	116	Life Is Good (feat. Drake, DaBaby & Lil Baby) - Remix	Future	2181930	315.346	Americas
7923	United States	124	SICKO MODE	Travis Scott	2085268	312.820	Americas
...	...	...	...	...	...	...	...
7832	United States	33	Strawberry Peels (feat. Young Thug & Gunna)	Lil Uzi Vert	4007781	115.350	Americas
7853	United States	54	CITY OF ANGELS	24kGoldn	3443366	112.493	Americas
7971	United States	172	Skechers	DripReport	1731265	106.000	Americas

200 rows × 7 columns

Longest duration song in the United States.

The added pipe line takes the result of the filter, and gives it to arrange. You specify that you want to sort in descending order of duration.

arrange and filter

(music_top200
  >> filter(_.country == "United States")
  >> arrange(-_.duration)
)

	country	position	track_name	artist	streams	duration	continent
7841	United States	42	After Hours	The Weeknd	3672033	361.027	Americas
7915	United States	116	Life Is Good (feat. Drake, DaBaby & Lil Baby) - Remix	Future	2181930	315.346	Americas
7923	United States	124	SICKO MODE	Travis Scott	2085268	312.820	Americas
...	...	...	...	...	...	...	...
7832	United States	33	Strawberry Peels (feat. Young Thug & Gunna)	Lil Uzi Vert	4007781	115.350	Americas
7853	United States	54	CITY OF ANGELS	24kGoldn	3443366	112.493	Americas
7971	United States	172	Skechers	DripReport	1731265	106.000	Americas

200 rows × 7 columns

The result shows that the longest duration track in the United States is After Hours by The Weeknd.

We can explore many such questions with various combinations of dplyr verbs.

Over the course of these lessons, you’ll learn to pipe together multiple simple operations to create a rich and informative data analysis.

Let’s practice!

• siuba guide - arrange
• siuba api - arrange

The mutate verb

You’ve already learned two verbs for manipulating data: filter for extracting a subset of the observations, and arrange for sorting them.

Suppose you want to change one of the variables in your dataset, based on the other ones. Or suppose you want to add a new variable.

For that, you would use the mutate verb.

The mutate verb

Using mutate to change a variable

(music_top200
  >> mutate(streams = _.streams / 1000)
)

First you’ll learn how to change an existing variable.

You use mutate like you would filter or arrange, after a pipe operator.

Inside this mutate statement, what’s on the right of the equals sign is what’s being calculated, and what’s on the left is what’s being replaced. Here, you’re calculating streams slash one thousand.

(Click the down arrow for a breakdown)

Using mutate to change a variable

(music_top200
  >> mutate(streams = _.streams / 1000)
)

The slash in Python performs division, so “streams divided by one thousand”.

Using mutate to change a variable

(music_top200
  >> mutate(streams = _.streams / 1000)
)

On the left, you’re saying you want to replace the existing streams column, by writing streams equals.

Using mutate to change a variable (result)

(music_top200
  >> mutate(streams = _.streams / 1000)
)

	country	position	track_name	artist	streams	duration	continent
0	Argentina	1	Tusa	KAROL G	1858.666	200.960	Americas
1	Argentina	2	Tattoo	Rauw Alejandro	1344.382	202.887	Americas
2	Argentina	3	Hola - Remix	Dalex	1330.011	249.520	Americas
...	...	...	...	...	...	...	...
12397	South Africa	198	Black And White	Niall Horan	11.771	193.090	Africa
12398	South Africa	199	When I See U	Fantasia	11.752	217.347	Africa
12399	South Africa	200	Psycho!	MASN	11.743	197.217	Africa

12400 rows × 7 columns

The result is the same table, but with streams replaced with a new value, one that’s much smaller than it was before.

This is how you manipulate existing variables in the table. That’s often necessary during data processing and cleaning.

Just like filter and arrange, you’re not altering the original music_top200 data, you’re just changing the value in this new data frame that’s being returned.

Using mutate to add a new variable

(music_top200
  >> mutate(ttl_stream_time = _.streams * _.duration)
)

	country	position	track_name	artist	streams	duration	continent	ttl_stream_time
0	Argentina	1	Tusa	KAROL G	1858666	200.960	Americas	3.735175e+08
1	Argentina	2	Tattoo	Rauw Alejandro	1344382	202.887	Americas	2.727576e+08
2	Argentina	3	Hola - Remix	Dalex	1330011	249.520	Americas	3.318643e+08
...	...	...	...	...	...	...	...	...
12397	South Africa	198	Black And White	Niall Horan	11771	193.090	Africa	2.272862e+06
12398	South Africa	199	When I See U	Fantasia	11752	217.347	Africa	2.554262e+06
12399	South Africa	200	Psycho!	MASN	11743	197.217	Africa	2.315919e+06

12400 rows × 8 columns

Alternatively, you may want to add a new variable.

For instance, suppose you want to know the total seconds people have listened to a track.

In the data you have a tracks duration in seconds, and its number of streams. This means total time listened is duration multiplied by streams.

You would use mutate in almost the same way. You pipe your music data to the mutate verb.

The asterisk in Python represents multiplication, so you write streams asterisk duration to multiply the two columns.

Using mutate to add a new variable (result)

(music_top200
  >> mutate(ttl_stream_time = _.streams * _.duration)
)

	country	position	track_name	artist	streams	duration	continent	ttl_stream_time
0	Argentina	1	Tusa	KAROL G	1858666	200.960	Americas	3.735175e+08
1	Argentina	2	Tattoo	Rauw Alejandro	1344382	202.887	Americas	2.727576e+08
2	Argentina	3	Hola - Remix	Dalex	1330011	249.520	Americas	3.318643e+08
...	...	...	...	...	...	...	...	...
12397	South Africa	198	Black And White	Niall Horan	11771	193.090	Africa	2.272862e+06
12398	South Africa	199	When I See U	Fantasia	11752	217.347	Africa	2.554262e+06
12399	South Africa	200	Psycho!	MASN	11743	197.217	Africa	2.315919e+06

12400 rows × 8 columns

Notice that to be informative, we have named the column ttl_stream_time, that’s what’s to the left of the equals sign in our code.

Column names need to be one word like this: you can’t use spaces.

Notice that in the results, there’s a brand new ttl_stream_time column, which is much larger than streams.

Answering a question

(music_top200
  >> mutate(ttl_stream_time = _.streams * _.duration)
  >> filter(_.country == "Costa Rica")
  >> arrange(-_.ttl_stream_time)
)

Let’s put together the three verbs you learned in this chapter to answer a question about our data.

Suppose we wanted to know for a country, like Costa Rica, what songs were listened to longest.

We would do this in three steps:

• create the column
• filter for country
• then sort

(Click the down arrow for a breakdown)

Answering a question

(music_top200
  >> mutate(ttl_stream_time = _.streams * _.duration)
 
 
)

	country	position	track_name	artist	streams	duration	continent	ttl_stream_time
0	Argentina	1	Tusa	KAROL G	1858666	200.960	Americas	3.735175e+08
1	Argentina	2	Tattoo	Rauw Alejandro	1344382	202.887	Americas	2.727576e+08
2	Argentina	3	Hola - Remix	Dalex	1330011	249.520	Americas	3.318643e+08
...	...	...	...	...	...	...	...	...
12397	South Africa	198	Black And White	Niall Horan	11771	193.090	Africa	2.272862e+06
12398	South Africa	199	When I See U	Fantasia	11752	217.347	Africa	2.554262e+06
12399	South Africa	200	Psycho!	MASN	11743	197.217	Africa	2.315919e+06

12400 rows × 8 columns

We use mutate to create the ttl_stream_time column.

Answering a question

(music_top200
  >> mutate(ttl_stream_time = _.streams * _.duration)
  >> filter(_.country == "Costa Rica")
 
)

	country	position	track_name	artist	streams	duration	continent	ttl_stream_time
2200	Costa Rica	1	Safaera	Bad Bunny	338078	295.177	Americas	9.979285e+07
2201	Costa Rica	2	Si Veo a Tu Mamá	Bad Bunny	244932	170.972	Americas	4.187651e+07
2202	Costa Rica	3	Ignorantes	Bad Bunny	233113	210.607	Americas	4.909523e+07
...	...	...	...	...	...	...	...	...
2397	Costa Rica	198	Ride It	Regard	21731	157.606	Americas	3.424936e+06
2398	Costa Rica	199	Sola	Manuel Turizo	21704	195.044	Americas	4.233235e+06
2399	Costa Rica	200	Nena Maldición (feat. Lenny Tavárez)	Paulo Londra	21684	228.875	Americas	4.962926e+06

200 rows × 8 columns

Then we’d use filter to restrict our analysis to only Costa Rica.

Answering a question

(music_top200
  >> mutate(ttl_stream_time = _.streams * _.duration)
  >> filter(_.country == "Costa Rica") 
  >> arrange(-_.ttl_stream_time) 
)

	country	position	track_name	artist	streams	duration	continent	ttl_stream_time
2200	Costa Rica	1	Safaera	Bad Bunny	338078	295.177	Americas	9.979285e+07
2202	Costa Rica	3	Ignorantes	Bad Bunny	233113	210.607	Americas	4.909523e+07
2222	Costa Rica	23	René	Residente	101872	457.592	Americas	4.661581e+07
...	...	...	...	...	...	...	...	...
2377	Costa Rica	178	Dónde Estás	KHEA	23177	153.560	Americas	3.559060e+06
2394	Costa Rica	195	Blueberry Faygo	Lil Mosey	21771	162.547	Americas	3.538811e+06
2397	Costa Rica	198	Ride It	Regard	21731	157.606	Americas	3.424936e+06

200 rows × 8 columns

Then we’d use arrange to sort the data in descending order of our new ttl_stream_time variable.

Answering a question (result)

(music_top200
  >> mutate(ttl_stream_time = _.streams * _.duration)
  >> filter(_.country == "Costa Rica") 
  >> arrange(-_.ttl_stream_time) 
)

	country	position	track_name	artist	streams	duration	continent	ttl_stream_time
2200	Costa Rica	1	Safaera	Bad Bunny	338078	295.177	Americas	9.979285e+07
2202	Costa Rica	3	Ignorantes	Bad Bunny	233113	210.607	Americas	4.909523e+07
2222	Costa Rica	23	René	Residente	101872	457.592	Americas	4.661581e+07
...	...	...	...	...	...	...	...	...
2377	Costa Rica	178	Dónde Estás	KHEA	23177	153.560	Americas	3.559060e+06
2394	Costa Rica	195	Blueberry Faygo	Lil Mosey	21771	162.547	Americas	3.538811e+06
2397	Costa Rica	198	Ride It	Regard	21731	157.606	Americas	3.424936e+06

200 rows × 8 columns

This gives us the answer we’re looking for.

The most streamed track in Costa Rica is still the first in our result.

However, it looks like the track normally in the 23rd position–René by Residente–is third, since it’s over 7 minutes long!

Let’s practice!

• siuba guide - mutate
• siuba api - mutate

Visualization with plotnine

In the last chapter, you used siuba to answer some questions about the music_top200 dataset.

You’ve been able to..

• filter for particular observations.
• arrange to find the highest or lowest values.
• mutate to add new columns.

However, most of the code so far has only printed out results as a table. Often a better way to understand and present this kind of data is as a graph.

In this chapter, you’ll learn the essential skill of data visualization using the plotnine package.

Visualizing with plotnine

In particular, this chapter will show you how to create scatterplots, like the one you see here.

Scatterplots compare two variables on an x- and y- axis.

Visualization and data wrangling are often intertwined, so you’ll see how the siuba and plotnine packages work closely together to create informative graphs.

Importing plotnine

from siuba import *
from plotnine import *

You’ll be creating plots using the plotnine package.

Just like the siuba package, you’ll have to load it with import.

Variables

billie = (
  track_features
  >> filter(_.artist == "Billie Eilish")
)

In this chapter, you’ll sometimes be visualizing subsets of the track_features data. For example, this code gets only tracks by the artist Billie Eilish.

When you’re working with just that subset, sometimes it’s useful to save the filtered data, as a new data frame.

To do this, you use the assignment operator.

Click down for a breakdown of assignment

Variables

(
  track_features
  >> filter(_.artist == "Billie Eilish")
)

First, write out the pipe with the filter verb.

This is the same as you did in the previous chapter.

Note that whether the name of the data, track_features is right after the ( or on its own line, the code works the same.

Variables

billie = (
  track_features
  >> filter(_.artist == "Billie Eilish")
)

Then, write the variable name, followed by the variable operator (an equal sign).

In this case, the variable name is billie.

Variables (result)

billie

	artist	album	track_name	energy	valence	danceability	speechiness	acousticness	popularity	duration
1273	Billie Eilish	dont smile ...	my boy	0.3940	0.3240	0.692	0.2070	0.472	44	170.852
2899	Billie Eilish	WHEN WE ALL...	listen befo...	0.0561	0.0820	0.319	0.0450	0.935	79	242.652
2950	Billie Eilish	lovely (wit...	lovely (wit...	0.2960	0.1200	0.351	0.0333	0.934	89	200.186
...	...	...	...	...	...	...	...	...	...	...
24857	Billie Eilish	WHEN WE ALL...	ilomilo	0.4230	0.5720	0.855	0.0585	0.724	79	156.371
24997	Billie Eilish	WHEN I WAS ...	WHEN I WAS ...	0.3320	0.0628	0.696	0.0425	0.853	71	270.520
25147	Billie Eilish	come out an...	come out an...	0.3210	0.1770	0.640	0.0931	0.693	74	210.376

27 rows × 10 columns

Now if you print the billie dataset, we can see that it’s another table.

But this one has only 27 rows: one for each track by Billie Eilish in the original data.

Visualizing with plotnine

(billie
 >> ggplot(aes("energy", "valence"))
  + geom_point()
  + labs(title = "Billie Eilish hit track features")
)

Suppose you want to examine the energy and valence of Billie Eilish’s songs.

You could do this with a scatterplot comparing two variables in our track_features dataset: energy on the x-axis and valence on the y-axis.

There are three parts to a plotnine graph.

(Click the down button to see a breakdown of the code)

Visualizing with plotnine

(billie
 >> ggplot(aes("energy", "valence"))
  + geom_point()
  + labs(title = "Billie Eilish hit track features")  
)

First is the data that we’re visualizing. In this case, that is the billie variable you just created.

Visualizing with plotnine

(billie
 >> ggplot(aes("energy", "valence"))
  + geom_point()
  + labs(title = "Billie Eilish hit track features")
)

Second is the mapping of variables in your dataset to aesthetics in your graph.

An aesthetic is a visual dimension of a graph that can be used to communicate information.

In a scatterplot, your two dimensions are the x axis and the y axis, so you write aes (for “aesthetic”), parentheses, x equals gdpPerCap, y = lifeExp, telling it which variables to place on which axes.

Visualizing with plotnine

(billie
 >> ggplot(aes("energy", "valence"))
  + geom_point()
  + labs(title = "Billie Eilish hit track features")
)

The third step is specifying the type of graph you’re creating. You do that by adding a layer to the graph: use a plus after the ggplot, and then geom underscore point.

The “geom” means you’re adding a type of geometric object to the graph, the “point” indicates it’s a scatter plot, where each observation corresponds to one point.

Together, these three parts of the code–the data, the aesthetic mapping, and the layer–construct the scatter plot you see here.

Let’s practice!

Scroll down to get started with practice!

Using plotnine geoms

Using plotnine geoms

(billie
 >> ggplot(aes("energy", "valence"))
  + geom_point()
)

In the previous lesson, you created this scatter plot comparing the energy of each song to its valence.

This plot communicates some interesting information: we can see that higher income countries tend to have higher life expectancy.

One problem with this plot, however, is that it’s impossible to tell which song is which. Adding labels to the point will let us identify songs in the plot.

Using geom_label

(billie
 >> ggplot(aes("energy", "valence", label = "track_name"))
  + geom_label()
)

The first option for plotting text is using geom_label(). This plots the text with a filled in box around it.

Unlike scanning a DataFrame of results, it’s easy to see pick out songs across a range of valence and energy.

For example, relative to other songs:

• “i love you” is low valence and low energy
• “bad guy” is high valence and high energy
• “bellyache” is extremely high energe and a moderate amount of valence

Using geom_text

(billie
 >> ggplot(aes("energy", "valence", label = "track_name"))
  + geom_text()
)

A second option for plotting text is with geom_text(). This plots text by itself, without a background box.

Notice the words in front don’t cover other words as much, but it can also be harder to read them.

Combining geoms

(billie
 >> ggplot(aes("energy", "valence", label = "track_name"))
  + geom_text(nudge_y = .1)
  + geom_point()
)

When you plot multiple geoms–like in this plot–ggplot goes line-by-line, from top to bottom.

For example, in the code here, it first puts down the words from geom_text(), then the points from geom_point().

Notice also that geom_text() has an option passed to it, nudge_y = ,1.

More on geom options

In general, there are many arguments you can pass to geoms, and different geoms can take different arguments.

The easiest way to understand what arguments geoms can take, and what they do, is to look at the plotnine documentation website.

Here are two useful parts of the docs:

• list of available geoms
• help document on geom_text

Let’s practice!

• plotnine docs
• geom_text() API page

In the exercises, you’ll practice creating other scatter plots to compare variables across countries, and in the rest of this chapter you’ll learn more ways to communicate information in a graph.

Using plotnine Aesthetics

In this lesson, you’ll use plotnine aesthetics to set more than just the position of points on a plot, but other features too–such as size and color.

Scatterplots

billie = filter(track_features, _.artist == "Billie Eilish")

(billie
  >> ggplot(aes("energy", "valence"))
   + geom_point()
)

You’ve learned how to create a scatter plot to compare two variables within your data using two visual aesthetics: energy x-axis, and valence on the y-axis.

Additional variables

billie

	artist	album	track_name	energy	valence	danceability	speechiness	acousticness	popularity	duration
1273	Billie Eilish	dont smile at me (Expanded Edition)	my boy	0.3940	0.3240	0.692	0.2070	0.472	44	170.852
2899	Billie Eilish	WHEN WE ALL FALL ASLEEP, WHERE DO WE GO?	listen before i go	0.0561	0.0820	0.319	0.0450	0.935	79	242.652
2950	Billie Eilish	lovely (with Khalid)	lovely (with Khalid)	0.2960	0.1200	0.351	0.0333	0.934	89	200.186
...	...	...	...	...	...	...	...	...	...	...
24857	Billie Eilish	WHEN WE ALL FALL ASLEEP, WHERE DO WE GO?	ilomilo	0.4230	0.5720	0.855	0.0585	0.724	79	156.371
24997	Billie Eilish	WHEN I WAS OLDER (Music Inspired By The Film ROMA)	WHEN I WAS OLDER - Music Inspired By The Film ROMA	0.3320	0.0628	0.696	0.0425	0.853	71	270.520
25147	Billie Eilish	come out and play	come out and play	0.3210	0.1770	0.640	0.0931	0.693	74	210.376

27 rows × 10 columns

But these aren’t the only variables in the track_features dataset: for example, you also have acousticness and popularity. You may want to examine relationships among all these variables in the same plot.

You already used the x-axis to represent energy and the y-axis to valence Now you’ll learn to add two more aesthetics–color and size–to communicate even more information in your scatter plot.

The color aesthetic

(billie
  >> ggplot(aes("energy", "valence", color = "acousticness"))
   + geom_point()
)

The scatterplot shows that songs with higher energy, tend to have higher valence. Another variable that might be related to energy is acousticness.

You can explore this relationship by setting the color of your points, like you see here. To use this aesthetic, you add color equals acousticness inside the aes, next to x equals energy and y equals valence.

Notice that ggplot2 automatically adds a legend to the plot, indicating which color represents which acousticness.

This communicates that lower energy tracks (toward the left of the plot) tend to be more acoustic. Note that brighter colors indicate more acoustic tracks.

The size aesthetic

(billie
  >> ggplot(aes("energy", "valence", color = "acousticness", size = "popularity"))
   + geom_point()
)

Another variable you may want to include in the graph is population, represented by the pop variable in the dataset.

One way to represent it is with the size of the points in the scatterplot, with higher popularity songs getting larger points.

Just like x, y, and color, you add size = “popularity” within the aes parentheses.

Note that to keep the length of each of the code lines reasonable, we put the size aesthetic on a second line, but this doesn’t make any difference, and you don’t have to do that in the exercises. You’ve now learned to use four aesthetics in a plot: x, y, color, and size: to communicate information about four variables in your dataset.

Aesthetics with multiple geoms

(billie
  >> ggplot(aes("energy", "valence", 
                color = "acousticness", size = "popularity",
                label = "track_name"))
   + geom_point()
   + geom_text(nudge_y = .1)
)

Notice that in this plot, the aesthetics set effect both the points and the text.

Let’s practice!

In the exercises, you’ll learn to mix and match aesthetics and variables to further explore the track features.

• filter artist, get characteristics
• try different combinations of characteristics–which ones seem most related?
• diagnose error (unquoted ggplot)
• diagnose error (verb without _)
• plot with aesthetics
• plot text instead
• labs?

Faceting

You’ve learned to use color in your scatterplots.

Now you’ll learn another way to explore your data. plotnine lets you divide your plot into subplots to get one smaller graph for each level of a variable.

This is called faceting, and it’s another powerful way to communicate relationships within your data.

Faceting

asia_top200 = (
  music_top200
  >> filter(_.continent == "Asia")
)
asia_top200

	country	position	track_name	artist	streams	duration	continent
4600	Hong Kong	1	WANNABE	ITZY	112648	191.242	Asia
4601	Hong Kong	2	Intentions (feat. Quavo)	Justin Bieber	104467	212.867	Asia
4602	Hong Kong	3	Señorita	Shawn Mendes	84196	190.960	Asia
...	...	...	...	...	...	...	...
12197	Viet Nam	198	Đưa Nhau Đi Trốn (Chill Version)	Đen	20750	241.959	Asia
12198	Viet Nam	199	Hôm Nay Tôi Buồn	Phùng Khánh Linh	20580	275.000	Asia
12199	Viet Nam	200	Kick It	NCT 127	20495	233.013	Asia

2600 rows × 7 columns

For this example, we’ll use Spotify top 200 track data for countries in Asia.

Faceting

(asia_top200
  >> ggplot(aes("position", "streams", color = "country"))
   + geom_point()
)

This plot shows the number of streams for each top 200 track in Asian countries.

There are so many countries that the plot hits two issues:

1. it has to use many different colors.
2. many points are on top of each other.

A facetted plot could help here, by giving each country its own small panel.

Faceting

(asia_top200
  >> ggplot(aes("position", "streams", color = "country"))
   + geom_point()
   + facet_wrap('~country')
)

You facet a plot by adding another option, with a +, to the end of your code, after geom_point.

You add facet underscore wrap, then “tilde country” within the parentheses.

In R, the tilde symbol typically means “by”, meaning that we’re splitting the plot by country, and you can usually find it on the upper left of your keyboard. This tells ggplot2 to divide the data into subplots based on the country variable.

Let’s practice!

Scroll down to get started with practice!

Summarizing data

In this chapter, you’ll return to the topic of data transformation with siuba to learn more ways to explore your data.

Data analysis

Analyses will usually involve a cycle between these steps of data transformation and visualization, as well as additional components of the data science workflow, like modeling (that this course won’t focus on).

Extracting data

(music_top200
  >> filter(_.country == "Japan", _.position == 1)
)

	country	position	track_name	artist	streams	duration	continent
6400	Japan	1	I LOVE...	Official HIGE DANdism	1591844	282.027	Asia

1 rows × 7 columns

You’ve learned to use the filter verb to pull out individual observations, such as the track in the first position for Japan.

Notice that the track’s duration is about 280 seconds. Is this is shorter or longer than the average track.

You’ll learn to answer this question by summarizing many observations into a single data point.

The summarize verb

(music_top200
  >> summarize(avg_duration = _.duration.mean())
)

	avg_duration
0	205.990073

1 rows × 1 columns

You might want to know if the 280 seconds we saw is longer than the average track across all countries.

You would do this with the summarize verb. Notice that summarize collapses the entire table down into one row.

In the output, we see the answer to our question: mean (or average) duration was around 205 seconds.

(Click down for a breakdown of the code)

The summarize verb

(music_top200
  >> summarize(avg_duration = _.duration.mean())
)

	avg_duration
0	205.990073

1 rows × 1 columns

Take your music_top200 data, pipe it into summarize.

Then, specify that you’re creating a summary column called avg_duration.

Notice that the highlighted code is similar to when created a new column with the mutate verb.

The summarize verb

(music_top200
  >> summarize(avg_duration = _.duration.mean())
)

(music_top200
  >> summarize(avg_duration = _.duration.mean())
)

	avg_duration
0	205.990073

1 rows × 1 columns

The “duration dot mean, followed by open and close parentheses” is worth examining.

This is calling the method mean on the variable duration. The mean method calculates the average of a set of values.

Summarizing one country

(music_top200
  >> filter(_.country == "Japan")
  >> summarize(avg_duration = _.duration.mean())
)

	avg_duration
0	250.53499

1 rows × 1 columns

If you think about it, it doesn’t really make sense to summarize across all countries. It may make more sense to compare our track to averages in the same country: Japan.

To answer this, you can combine the summarize verb with filter: filter your data for a particular country first, then summarize the result. This shows you that the average track duration in the Japan top 200 chart was about 250 seconds.

Summarizing into multiple columns

(music_top200
  >> filter(_.country == "Japan")
  >> summarize(
      avg_duration = _.duration.mean(),
      ttl_streams = _.streams.sum()
  )
)

	avg_duration	ttl_streams
0	250.53499	48942067

1 rows × 2 columns

You can create multiple summaries at once with the summarize verb.

For example, suppose that along with finding the average track duration in Japan, you want to find their chart’s total streams.

To do that, add a comma after the mean of the duration. Then, specify another column you’re creating.

You could give it a useful name like ttl_streams, and say that it’s equal to the sum–that’s another built-in function–of the pop variable.

Methods for summarizing

E.g. _.some_column.mean()

• .mean()
• .sum()
• .median()
• .min()
• .max()

The mean and sum are just two methods you could use to summarize a variable within a dataset.

Another example is median: the median represents the point in a set of numbers where half the numbers are above that point and half of the numbers are below.

Two others are min, for minimum, and max, for maximum.

In the exercises, you’ll use several of these functions to answer questions about the music_top200 dataset.

Let’s practice!

The group_by verb

In the last lesson, you used the summarize verb to calculate an average either,

• across all countries (the entire dataset)
• within a particular country (filtered data)

In this lesson you’ll learn the group_by verb, which opens up a new option for calculating an average:

• within each country

The group_by verb

You use group_by to define groupings of data (blue and green chunks in diagram).

Then, you would follow it with summarize to convert those groups to a single value (like a mean).

The summarize verb

(music_top200
  >> filter(_.country == "Japan")
  >> summarize(avg_duration = _.duration.mean()))

	avg_duration
0	250.53499

1 rows × 1 columns

For example, here you’re finding the average track duration in Japan.

What if we weren’t interested just in the average duration for Japan, but for each country in the dataset?

You could rerun this code and change the country each time, but that’s very tedious.

Summarizing by country

(music_top200
  >> group_by(_.country)
  >> summarize(avg_duration = _.duration.mean())
)

	country	avg_duration
0	Argentina	212.847855
1	Australia	204.795300
2	Austria	184.894870
...	...	...
59	United States	190.827500
60	Uruguay	210.796985
61	Viet Nam	217.222830

62 rows × 2 columns

Instead, you can use the group_by verb, which tells siuba to summarize within groups instead of summarizing the entire dataset. Notice that this replaces the filter country equals Japan with group_by country.

group_by(_.country) tells the summarize step that it should perform the summary within each country: within Argentina, then within Australia, and so on through Viet Nam, then combine the results.

Instead of getting one row overall, you now get one row for each country. There’s now a year variable along with the new avg_duration variable.

Summarizing by continent and position

(music_top200
  >> group_by(_.continent, _.position)
  >> summarize(
      min_streams = _.streams.min(),
      max_streams = _.streams.max()
  )
)

	continent	position	min_streams	max_streams
0	Africa	1	94422	94422
1	Africa	2	74689	74689
2	Africa	3	67552	67552
...	...	...	...	...
997	Oceania	198	44570	225951
998	Oceania	199	44364	225492
999	Oceania	200	44291	225179

1000 rows × 4 columns

You can also group by multiple variables.

Suppose you’re interested calculating min and max streams within each continent, and each position.

Now the output has one row for each combination of a continent and position.

(Click down to see what the steps leading to this code might look like)

Summarizing by continent and position

(music_top200

  >> summarize(
      min_streams = _.streams.min(),
      max_streams = _.streams.max()
  )
)

	min_streams	max_streams
0	1470	12987027

1 rows × 2 columns

You might start by summarizing min and max streams across all the data.

Summarizing by continent and position

(music_top200
  >> filter(_.continent == "Oceania", _.position == 1)
  >> summarize(
      min_streams = _.streams.min(),
      max_streams = _.streams.max()
  )
)

	min_streams	max_streams
0	321272	1757343

1 rows × 2 columns

Then, you might add a filter, so your summarize is over one specific grouping (position 1 tracks in Oceania).

Summarizing by continent and position

(music_top200
  >> group_by(_.continent, _.position)
  >> summarize(
      min_streams = _.streams.min(),
      max_streams = _.streams.max()
  )
)

	continent	position	min_streams	max_streams
0	Africa	1	94422	94422
1	Africa	2	74689	74689
2	Africa	3	67552	67552
...	...	...	...	...
997	Oceania	198	44570	225951
998	Oceania	199	44364	225492
999	Oceania	200	44291	225179

1000 rows × 4 columns

Finally, you might replace the filter with a group_by.

Note that you might save this result as a variable and use it in a new pipe or visualization. For example, the new pipe might filter to keep where position is 1, so you can inspect min and max streams for the top tracks across continents.

Let’s practice!

Visualizing summarized data

In this chapter you learned to use the group by and summarize verbs to summarize the music_top200 data by year, by continent, or by both.

Now you’ll learn how to turn those summaries into informative visualizations, by returning to the plotnine package from Chapter 2.

When visualizing raw data doesn’t work

(music_top200
  >> ggplot(aes("position", "streams", color = "country"))
   + geom_point()
)

Suppose we want to look at how many streams countries have. We could make a scatterplot showing each track in each country, like in the plot on the slide.

However, there are so many countries that it is hard to read, and half the plot is the legend that shows what color each country is.

Calculating min and max streams

by_position = (
  music_top200
  >> group_by(_.position)
  >> summarize(max_streams = _.streams.max(),
               min_streams = _.streams.min())
)
by_position

	position	max_streams	min_streams
0	1	12987027	13604
1	2	9163134	10801
2	3	8043475	9510
...	...	...	...
197	198	1606234	1472
198	199	1606153	1470
199	200	1597824	1470

200 rows × 3 columns

Rather than plotting the raw data, we can plot a summary.

For example, previously we summarized data by continent and position, to find the max and min streams.

Let’s do that again, but instead of viewing the summarized data as a table, let’s save it as an object called by_continent_position, so you can visualize the data using plotnine.

Plotting

(by_position
  >> ggplot(aes("position", "max_streams"))
   + geom_point()
   + labs(title = "Top 200 hits - max streams overall")
)

You would construct the graph with the three steps of plotnine:

• The data, which is by_year.
• The aesthetics, which puts year on the x-axis and total population on the y-axis.
• The type of graph, which in this case is a scatter plot, represented by geom_point.

Notice that the steps are the same as when you were graphing countries in a scatter plot, even though it’s a new dataset.

Plotting (result)

(by_position
  >> ggplot(aes("position", "max_streams"))
   + geom_point()
   + labs(title = "Top 200 hits - max streams overall")
)

The resulting graph of max streams by position, which is much easier to read than the previous plot with all the raw data.

The top track is (by definition) the most streamed, with about 100 million streams!

You might notice that the graph is a little misleading because it doesn’t include zero: It’s hard to get a a sense of how many streams the last position track had, since it is at the bottom of the graph.

Starting y-axis at 0

(by_position
  >> ggplot(aes("position", "max_streams"))
   + geom_point()
   + expand_limits(y = 0)
   + labs(title = "Top 200 hits - max streams overall"))

This is a good time to introduce another graphing option. By adding “expand underscore limits y = 0” to the end of the ggplot call, you can specify that you want the y-axis to start at zero.

Notice that you added it to the end just like you would with facet_wrap.

Now the graph makes it clearer that the top position track has over three times as many streams, as the lowest position one.

You could have created other graphs of summarized data, such as a graph of the average streams rather than max, by changing the y aesthetic.

Calculating min and max streams

by_continent_position = (
  music_top200
  >> group_by(_.continent, _.position)
  >> summarize(max_streams = _.streams.max(),
               min_streams = _.streams.min())
)
by_continent_position

	continent	position	max_streams	min_streams
0	Africa	1	94422	94422
1	Africa	2	74689	74689
2	Africa	3	67552	67552
...	...	...	...	...
997	Oceania	198	225951	44570
998	Oceania	199	225492	44364
999	Oceania	200	225179	44291

1000 rows × 4 columns

So far you’ve been graphing the by-position summarized data. But you have also learned to summarize after grouping by both position and continent, to see how much countries are streaming top tracks on different continents continents.

Visualize

(by_continent_position
  >> ggplot(aes("position", "max_streams", color = "continent"))
   + geom_point()
   + expand_limits(y = 0)
   + labs(title = "Top 200 hits - max streams overall"))

Since you now have data over position within each continent, you need a way to separate it in a visualization.

To do that you can use the color aesthetic you learned about in chapter two.

By setting color equals continent, you can show five separate trends on the same graph.

This lets us see that a country in the Americas has a hit tracks with a large number of streams, while there aren’t any countries in Africa doing a lot of streaming spotify hits (for more on companies competing to stream in Africa, see this article).

Let’s practice!

You’ll often combine siuba verbs and plotnine visualizations as part of an exploratory analysis, so it’s important to get into the habit of visualizing summarized or processed data.

Scroll down to investigate the music data with filter.