Illustrated Dan Denney in a circle

Spotify Data Project

January 27, 2015

  10 09 08 07 06 05 04 03 02 01

Feb 02

ee915e50041ce0dd1dfde71cc6e2c28e078aa8e8

Dangerous

b225984a1dec4cb4ea43345388ecae4f740cba0b

GDFR

f3cf87bfde0748d389702a69e0ba01f6091e2403

Cheerleader

c2e52266c4ed1541003babec2964a99ce0350aa3

Firestone

1c64a09fef47c954a6f5e779fbb21bf0245832dc

Love Me Like You Do

0e2747127c12222577e98ccec7ce44773c12c546

Outside

956a2c25a6a0cdfcf7e4d61af359b1b3e73575da

Sugar

646e9619750dfa3d1eadbbea959dc6f528a9109e

Thinking Out Loud

0080bec0581fe6edb83293c14d3a0d01942b1798

Take Me To Church

95f7f3bff78df3fbf5402ee304c9344aaec52890

Uptown Funk

  10 09 08 07 06 05 04 03 02 01

Jan 26

1c64a09fef47c954a6f5e779fbb21bf0245832dc

Love Me Like You Do

ee915e50041ce0dd1dfde71cc6e2c28e078aa8e8

Dangerous

f3cf87bfde0748d389702a69e0ba01f6091e2403

Cheerleader

b225984a1dec4cb4ea43345388ecae4f740cba0b

GDFR

956a2c25a6a0cdfcf7e4d61af359b1b3e73575da

Sugar

0e2747127c12222577e98ccec7ce44773c12c546

Outside

c2e52266c4ed1541003babec2964a99ce0350aa3

Firestone

0080bec0581fe6edb83293c14d3a0d01942b1798

Take Me To Church

646e9619750dfa3d1eadbbea959dc6f528a9109e

Thinking Out Loud

95f7f3bff78df3fbf5402ee304c9344aaec52890

Uptown Funk

  10 09 08 07 06 05 04 03 02 01

Jan 19

0e2747127c12222577e98ccec7ce44773c12c546

Blame

ee915e50041ce0dd1dfde71cc6e2c28e078aa8e8

Dangerous

ded7eb8e29384d0d75be7ffc2411bb8008f055b4

Love Me Harder

b225984a1dec4cb4ea43345388ecae4f740cba0b

GDFR

f3cf87bfde0748d389702a69e0ba01f6091e2403

Cheerleader

c2e52266c4ed1541003babec2964a99ce0350aa3

Firestone

0e2747127c12222577e98ccec7ce44773c12c546

Outside

646e9619750dfa3d1eadbbea959dc6f528a9109e

Thinking Out Loud

0080bec0581fe6edb83293c14d3a0d01942b1798

Take Me To Church

95f7f3bff78df3fbf5402ee304c9344aaec52890

Uptown Funk

  10 09 08 07 06 05 04 03 02 01

Jan 12

72d5bd1ecd38f261de6b7987283f35cd13d02fbd

Stay With Me

0e2747127c12222577e98ccec7ce44773c12c546

Blame

26b682beeb3bfefdfeed113bdbc0177bfb148b49

Lips Are Movin

ee915e50041ce0dd1dfde71cc6e2c28e078aa8e8

Dangerous

ded7eb8e29384d0d75be7ffc2411bb8008f055b4

Love Me Harder

c2e52266c4ed1541003babec2964a99ce0350aa3

Firestone

0e2747127c12222577e98ccec7ce44773c12c546

Outside

646e9619750dfa3d1eadbbea959dc6f528a9109e

Thinking Out Loud

0080bec0581fe6edb83293c14d3a0d01942b1798

Take Me To Church

95f7f3bff78df3fbf5402ee304c9344aaec52890

Uptown Funk

  10 09 08 07 06 05 04 03 02 01

Jan 05

c2e52266c4ed1541003babec2964a99ce0350aa3

Firestone

d7fe19b11c2814f98aa1449713cdc42668070a49

All About That Bass

ded7eb8e29384d0d75be7ffc2411bb8008f055b4

Love Me Harder

72d5bd1ecd38f261de6b7987283f35cd13d02fbd

Stay With Me

0e2747127c12222577e98ccec7ce44773c12c546

Blame

ee915e50041ce0dd1dfde71cc6e2c28e078aa8e8

Dangerous

0e2747127c12222577e98ccec7ce44773c12c546

Outside

646e9619750dfa3d1eadbbea959dc6f528a9109e

Thinking Out Loud

0080bec0581fe6edb83293c14d3a0d01942b1798

Take Me To Church

95f7f3bff78df3fbf5402ee304c9344aaec52890

Uptown Funk

Spotify Project: Today's Top Hits Top 100 Data

The Idea

Visualize the top 10 of Spotify’s Today’s Top Hits playlist throughout 2015.

In wanting to learn data visualization, it’s best for me to learn by projects. This process will help me understand data collection, sanitation, and visualization on a small scale. Plus, I love music and listen to this playlist often.

Manual Data Collection… yuk

In time, I’ll find out if there is an automated way of getting this information. For the moment, I’ll manually grab the information from the playlist every Monday.

I chose Mondays because Tuesday is the day for new music. It will allow 6 days for the new music to work through the list.

I quickly learned that the Today's Top Hits list is more of a curated list, so I switched to grabbing data from the Top 100 Playlist which appears to be by data. I was lucky that I had done some sampling before the start of the year or I would have missed out on a week of info.

Attributes

  • Date
  • Position
  • Title
  • Artist
  • ID
  • Image URL

Grabbing Data

I started by grabbing the information that Spotify makes easily available: HTTP Link, Spotify URI, and Embed Code. Here is an example of the information for the first song:

It seems that the ID is used for all of the attributes. There’s a lot of redundancy if I’m collecting 10 songs per week and it can all be generated if necessary automatically. All I need from that is the ID: 4bBrKDo8rchikwZOtmXbIH

Visualization Plans

I know that I’ll be using D3 because this won’t be a simple graph. Over time, I’m guessing that there will be some interesting info to be shown and that it will require customization.

While I have minimal data, the first decision is whether to display the dates horizontally or vertically. On a small screen, vertically makes the most sense. On a large screen, horizontal will allow for linear scrolling. Showing the path of a track through the weeks will always be tough on a small screen so that’ll be something to learn along the way.

Setup

Fortunately, this will be very simple. There’s no need for a build system. An HTML file, the data file, and D3 should suffice for now. If there ends up being a way to grab this data from an API I would need to set up an app, but I’ll deal with that later if necessary.

Everything will be in the repo:

Design Decisions

In playing with the sample data, there are already some decisions to make. While listing the #1 first feels right for the data, my eyes tend to follow the list to the right. If it stays horizontal, I think the #1 should be last.

screenshot of samples

  • Should dates run vertically or horizontally?
  • What data should be shown in the up state vs. a hover state?
  • What other data should be shown?
  • Where should additional data be shown?

January 18

Some changes are already in play. In playing with the arrangement and figuring out what data should be shown, it became clear to me that the individual song data is more interesting than the week-to-week.

This means that I need to structure the data in a way that allows for tracking an individual song's stats as well as each week's. I'm picturing that each week will be 10 vertical rows with each song showing it's meta data and a graph of it's performance through the list. I'll need a way of designating the entry point of the song because that will be interesting info to show as well.

I was using Numbers to track this, exporting to CSV, using a CSV to JSON automater, and then working with the JSON. Now that I'm using Middleman, I think I'm going to use YML for a bit because of the clean structuring. If I ever tried grabbing all of the info from Spotify's API, I'd likely need to use JSON but I'm not sure I can pull off the structuring I need.

For example, this week I'm going to try using their API to make the data collection faster. They don't link to the image URL so that part will still be manual. You can grab playlist info.

I'm slightly scared of the maintenance of the solution for tracking the progress of the track. It's simple but requires backwards updates. You're probably just not supposed to do things like this without a database. We shall see...

Final note for the day: I'm pretty sure that I want the embed available as an option. What's the fun of seeing a song if you can't play it, right? The embeds actually display all of the meta information that I am showing as well. They're so big, but I'm wondering what it'd look like with a bunch of them. So I'm going to replace the images for the moment and compare a couple things.

Final note for the day: I'm pretty sure that I want the embed available as an option. What's the fun of seeing a song if you can't play it, right? The embeds actually display all of the meta information that I am showing as well. They're so big, but I'm wondering what it'd look like with a bunch of them. So I'm going to replace the images for the moment and compare a couple things.

A page with images has 49 requests and weighs in at 1.1mb A page with embeds has 428 requests and weighs in at 4.4mb

It'd be irresponsible to do that for every set, so I'll have to work in the embeds in a different way.

January 19

Grabbing data from the Spotify API is pretty straight forward but it's a lot of data. I can trim it down but I still have to comb through a lot. The nice thing is that I think it would be possible to grab all of the meta that I want from it if I wanted to. It's set to grab the current state of a playlist, though, so this would have to be set to an automated job or I would have to do it weekly. It may be worthwhile to set that up just in case.

Already borked the data once. I put the concurrent weeks in the wrong order. Sooooo thankful for that numbers sheet. Added just the slightest bit of styles just so it didn't look like an explosion.

Made a mess of things tonight. I need to learn Bower. I was able to install Chartist from it, but I'm not sure how to reference it from Middleman so I ended up just duping it in the repo. Also, writing the Chartist JS in Coffee is odd. It requires weird structuring. The saddest part is that I'm likely not going to be able to use it. I love the simplicity of it, the weight, and the styles. However, I think I need the customization of d3. I want the scales to increase by 1 so you can see 10 places and I need to invert the scale so that 6 is higher than 8, for example.

January 20

I guess it needs to be design time. I have the information I need in a structure that seems to work. There are patterns emerging in the data that give me an idea of what I want to show. I can't use Chartist for it, so now I need to figure out how I want it to look so that I know what to learn in d3.

I've been stashing a bunch of interesting data viz work for inspiration and there was a piece that caught my eye today. There are so many good ideas in that set and a few patterns that make sense for this project.

January 21

Archiving screenshots of things that I'm killing. User tested the layouts and everyone (Cherrie and Jasmine) prefer the grid of album covers. So, I'm rolling with that. Also killing the chartist stuff since I can't use it.

screenshot of column layout

screenshot of column layout

January 22

Drew had exactly what I needed the whole time as I fought to make the classes from the song titles last night. middleman library

January 27

Negative numbers! It hit me today while driving that negative numbers would help with plotting and showing change from week to week. I still can't use Chartist because of the lack of control over the guides, but making 10 and 1 be -10 and -1 would allow for the graph to work properly.

screenshot of graph with negative plots

Showing the change from week to week would work too: 4 - 2 = 2, but it's really a drop of 2 spots. If -4 and -2 are used, -4 - -2 = -2. We'll see what I can do with that.

Feb 2

I played with a staggered view a bit and I'll keep it for now. So far it's the best representaiton of the positions but I'm not sold on it.

Some interesting data is starting to form. There are definitely things that I want to be able to show: highest jump in the charts, best starting point, and total number of individual tracks are a few. Those will definitely take some learning and I hope it can be done with the data as is.