This Week’s Program: Jan 9 - Jan 31

Last week, one of the things I resolved to do in 2017 was to spend some time learning and using D3. This was my focus this week.

I decided that for my first exploration into D3, I would use the metrics and stats of this Tinyletter as a basis for my data analysis. I have occasionally shared details and numbers about my readership. I subscribe to a handful of Tinyletters; I really have no idea where mine ranks in terms of subscribers and engagement. I have been a poor email marketer.

The first step was to find and extract some metrics for transformation into a data format that D3 can handle. I started by scraping the Tinyletter web application with a bit of jQuery, JSON.stringify, and the copy function (a handy utility made available in the Google Web Developer console). A JSON Array of Arrays can be transformed into a CSV using jq with:

jq -r '.[] @csv'

It turns out, I could do much better than scraping thanks to the wonderful tinystats command line utility, which provides several useful reports (see also the associated tinyapi Python library).

tinystats combined with csvkit got me just the right data to begin experimenting with visualization.

f041519e6c2795f5419d09771dd14da2

Here is my first D3 project. A column chart showing my subscriber count over time. Each letter is represented by a bar, with a time series going across the x axis. The color of the bar represents the open rate for the letter, with the brighter color representing a higher open rate.

For this project, I got to familiarize myself with a bunch of different subcomponents of D3 including scales and axes. I also decided to use some native ES6 features, so this might not show up at all in some browsers.

D3 is less of a data visualization framework and more of a toolkit of associated libraries that join data to HTML or SVG nodes. Having done a lot of work with SVG in Elm, I was feeling much more comfortable approaching this project. I struggled for a bit with one of D3’s foundational concepts: the data join. I found this article really helpful to get my mind wrapped around the concept.

I’m hosting this D3 project on Bl.ocks, which is a really handy way to host code from a GitHub Gist.

So what did I learn from my data visualization? Color is probably not the best way to visualize something like open rate. What this visualization tells me is perhaps that my earlier subscribers are the ones that tend to open my emails. I have, on average, 43 unique opens for each email. I’m not sure if that’s above or below the open rate for an average Tinyletter. Looking at my visualization, I also can’t tell what open rate correlates to. Subject line? Time of day the letter is sent? When the subscriber signed up? I’ll have to run some more reports and do some more visualizations to better understand those things. This is not a very useful visualization.

But for now, I’m satisfied. I immersed myself in D3 and feel like it’s something I have at least a decent grasp of. As I learn more about how to visualize data, I feel comfortable knowing that I have a toolset within reach that lets me implement those ideas.

Over the next few weeks, I’m going to be picking up Ansible again for managing my FreeBSD server, and I’ll begin learning the basic building blocks of Racket. I’ll also be writing up a post-mortem on hive-city and my work with Elm.

Have a good weekend
📊 Mark