I decided to build a workshop that introduces the D3.js library. I submitted a workshop proposal to the FOSS4GUK 2018 event committee and was proud to have had it accepted. At the time I had a few ideas, but I had not built what I was considering before, so the challenge was on to get the desired result then break it down into bite size chunks for workshop consumption. Another consideration was time, in Bonn I was given 4 hours, this time I had 2. I decided to adopt the cooking show approach and have pre-cooked partly built solutions ready to go. I wanted to deliver a workshop that would:

  • be fun
  • easy to follow
  • get people excited about D3.js
  • teach attendees a bunch of new stuff
  • give attendees a working solution to leave with

The following is a tutorial based on the workshop I delivered at the FOSS4GUK 2018 event in London. Thanks to those who attended, it was great to meet you and I hope you enjoyed it. Thanks to Mike Bostock, Scott Murray and Shimizu for all the great resources they have made available online. I would also like to thank the event organizers for supporting the workshop environment and 1Spatial for covering my accommodation, time and event entrance.

 

Lets get started!

This workshop is designed to use building block example HTML files to introduce concepts and highlight focal points that will provide a soft introduction to Leaflet and D3.js. Node.js is used to build a final solution and key for serving some of the examples.

We will take a look at the individual files in the browser and in a text editor of your choice, the instruction: open the file refers to opening the file in both browser and editor. There may be times when we will examine the HTML structure of the example, to do this use Ctrl+Shift+I to open the browsers developer tools.

Through each example, we will work our way to the final master plan, an interactive map and scatter-plot visualization.

Before we start, get the workshop material here and make sure you have Node.js correctly installed.

 

 

node

 

Node.js Installation

If you use Windows or OS X you can get the latest version from the Node.js website it should be a simple install and comes with the Node Package Manager, NPM which is really useful for installing the packages for your application. You will be using NPM a lot but don't worry it's fairly straightforward.

For Debian based distributions like Ubuntu, Node.js is available using the package manager. DigitalOcean is a great resource.

Open your terminal/command line and make a directory in which you will place your applications, I have made a directory called projects/nodejs. cd into the directory you have just made.

Next we are going to install the Express web framework - I found this framework really useful as it generates a collection of template files and folders that make sense and provide structure to our project.

sudo npm install express-generator -g

This command makes the Express functionality available globally so we can access it from any test application projects we build in the future. The global installation of Node.js Packages have to be run as root, so it must be run with admin priviledges.

To check if you have Node.js installed, type the following:

node --version

A successful installation will return the Node.js version.

 

 

Leaflet

 

Leaflet - 1_leaflet_simple.html

Let's begin with Leaflet. In the workshop material, navigate to the leaflet directory and open 1_leaflet_simple.html, you will see a slippy map of central London with zoom in/out buttons, let's take a look at the source code:

  • Standard HTML tags
  • A head tag that contains a link tag with a reference to the Leaflet style sheet and a script tag with a reference to the Leaflet JavaScript library
  • A div tag with id, width and height values
  • A script tag with a map variable, this is where the map is instantiated and assigned to the div with id="mapid", it is given a geographical point to center on and a zoom value
  • L.tileLayer is used to load and display tile layers on the map, in this example, and OpenStreetMap layer has been used for the basemap
Exercise L1 An alternative tile layer from Mapbox has been commented out, see if you can switch out the OpenStreetMap tile layer with one from Mapbox. The commented out lines that start with mapbox are different style options that can replace the tileLayer id value mapbox.pirates, if you have a moment have a look at these, some of the details at different extents are pretty cool.

 

Leaflet - 2_leaflet_quickstart.html

This example is from http://leafletjs.com/examples/quick-start/. Open the file, you will see 3 features and a popup, lets take a look at the source code:

  • The tileLayer style id option has been set to mapbox.light
  • Marker, circle and polygon functions each with an addTo and a bindPopup function
  • A popup variable
  • An onMapClick function
  • A click listener that calls the onMapClick function
Lets look at each of these individually

L.Marker is used to display clickable/draggable icons on the map, it instantiates a Marker object given a geographical point. The marker feature is then added to the map and given text to display in a popup - note that HTML can be used in the popup. The marker popup has an openPopup function that ensures the markers popup is displayed by default. No style options have been set so the default style is applied.

L.circle instantiates a circle marker object given a geographical point. It has a radius option which has been set to 500 and color, fillColor and fillOpacity options to style the feature. The circle feature is then added to the map and given the text "I am a circle." to display in a popup when clicked.

L.polygon a class for drawing polygon overlays on a map. Note that points you pass when creating a polygon shouldn't have an additional last point equal to the first one. The polygon feature is then added to the map and given the text "I am a polygon." to display in a popup when clicked. No style options have been set so the default style is applied.

Finally, the on function is used to listen for a click action, when the map is clicked the onMapClick function is called and generates a popup, gets the click lat lon value and adds it to the popup then displays the popup in the map. This is really powerful, you can generate all kinds of queries based on the map click coordinates, these can be georeferenced or submitted to a back end database to retrieve additional data you might wish to display, the output is not limited to a popup.

Notice that the feature bindPopup function takes priority over the map on click so if you click a feature the popup associated with the feature will return rather than the generic click popup.

Exercise L2 Create a map with two marker locations at places that mean something to you. At the workshop we used the Geovation Centre and the near by Bowler Pub. Assign them both popups with relevant text, bonus points for HTML and have one marker popup display by default. Hint: remember the zoom and center coordinates from the previous example.

 

This concludes the introduction to Leaflet section, we will use Leaflet in the next section when we add D3.js content. If you have time, take a look at the Mapbox Studio manual to learn how to create your own custom tile layer maps.

 

 

d3

 

Many of the D3 examples use JSON datasets to generate HTML content, to avoid cross-origin resource sharing (CORS) issues, Jez Nicholson has kindly put together a minimal node express server as a CORS work around. To use the node solution, ensure you have installed Node.js - see above, then navigate to the d3 directory of the coure material and run

node webserver

Navigate to http://0.0.0.0:3939/ in your browser to view the index page and click on the example you wish to view.

 

D3 - 1_d3_div_simple.html

In the d3 directory open 1_div_simple.html in a browser, what do you see? Nothing? Let's take a closer look, activate your developer tools - in Firefox and Chrome use Ctrl+Shift+I. Click on the Inspector or Elements tab and you will see a standard HTML format complete with html, head, body and script tags.

Lets take a look at the source code, apart from the block of code that has been commented out, the only thing worth mentioning here is the script reference to the D3.js V4 library. A lot of examples on the web reference the V3 library, there are some changes, so I have made sure that the examples in this tutorial all use the V4 library. There is plenty of material available on the differences.

Uncomment the code that is commented out - see below, and save the file.

var dataset = [ "a", "b", "c", "D3", "e" ];

d3.select("body").selectAll("div")
.data(dataset)
.enter()
.append("div")
.attr("id", function(d) {
return d.toLowerCase();
});

This is where the magic begins! Open the file in your browser - what do you see? Nothing? Lets take a closer look:

 

What you are looking at is the humble beginnings of our master plan!

 

Open your developer tools and take a look at the HTML, do you notice anything different? You will see 5 new div tags each with an id corresponding to a value in the dataset array. This is a pretty big deal!, D3 has added elements to our HTML this can be done on the fly and, when we add SVG to the mix, things really start to get exciting!

What the code is doing is selecting the body tag and appending a div tag for each position in the dataset array, which will create 5 new div tags. A function is used to get the value of each object in the array and return it as the div id value. A toLowerCase function is used to set the value of each object in the array to lower case so, if you re-visit the HTML, you will see one div tag has an id of "d3" and not D3 as it appears in the array. The toLowerCase function doesn't add much value in this example and simply shows that you can use JavaScript functions to assist with your D3 rendering tasks.

Exercise D1 Change the id names to fruit and make them all upper case.

 

D3 2_d3_css_class.html

This example has extended the Scott Murray Drawing divs tutorial. Open the 2_d3_css_class.html file, you should see the text: "hello". Take a look at the HTML, notice the single p tag with an id="here" and 5 div tags with class="bar".

Take a look at the source code - similar to the last example we are using the dataset to generate 5 divs, this time with an id value as the string "elem_" followed by the array value and class="bar". Notice that the values in the array are numeric rather than string as they were in the previous example, we will use these values shortly but first let's see how to combine style with D3. Un-comment the div.bar block - see below.

div.bar {
    display: inline-block;
    width: 20px;
    height: 75px;
    background-color: teal;
}

#here {
    background-color: red;
}

#elem_15 {
    background-color: red;
}

Save the file and refresh your browser, you should see the background color in which the text "hello" appears, has changed to red and a teal colored rectangle with a red vertical stripe has appeared. The rectangle is actually 5 rectangles with a width of 20px and a height value of 75px as defined in the style and assigned to the div tag with a bar class value. The div with the id="elem_15" has been assigned the background-color red.

CSS style can be used to style D3 content.

Exercise D2 Edit the style to create a space between the div rectangles. Edit the D3 code to assign the array values to the div height. Hint: look at margin-right and .style("height"

 

D3 3_d3_no_div.html

This example introduces SVG, this is where things get really fun! Open the 3_d3_no_div.html file, you should see 5 blue circles, take a look at the HTML, notice the addition of an svg tag, a group tag and 5 circle tags, these have been added in the same way we added div tags in the previous examples. The difference is that these are SVG elements. Scalable Vector Graphics (SVG) is an XML-based vector image format for two-dimensional graphics with support for interactivity and animation.

I am not going to go into much detail about SVG in this tutorial but it is at the core of the D3.js library so take some time to understand the basics if you decide to continue down the D3 path.

Lets take a look at the source code:

  • An SVG code block
  • A circle code block
  • A circle attribute assignment code block
The SVG block simply adds an svg tag with a width and height value to the HTML body tag and a group tag within the svg tag. The svg tag is now setup and ready to receive content.

 
The circle block appends svg circle elements to the svg tag just like we appended div tags to the body tag in the previous examples. The numeric dataset array is used to append the circles.

Finally, circles are assigned a center point defined by a cx and cy value and a radius value from the dataset array. Lastly, the value of "blue" has been assigned to the fill style attribute.

Very often you need to use multiple svg tags for operations like inter chart highlighting. My preference is to use divs to contain and position SVG elements, I find divs give me a slightly better handle on positioning.

Exercise D3 Change the SVG circle elements to rectangles, make a bar chart similar to the one we made using div tags in the previous exercise.

 

D3 4_d3_leaflet_popup.html
In this example we will use what we have learned about Leaflet so far and add some D3 content.

Open the file, you should see a Mapbox tilelayer with a single marker. Click on the marker, a popup containing a blue circle will open, if you mouse-over the circle it will change color from blue to red then back to red again when you mouse-out. This example demonstrates the ability to include SVG elements inside a Leaflet popup using D3.

Lets take a look at the source code, the majority of the code should look familiar. Looking before the on function, we can see that a marker group and a marker variable have been added. After the on function, the marker is added to the group and the markers group is added to the map.

You might recall from the 2_leaflet_quickstart.html example, we used the on function to display the map coordinates onclick. In this example we are using the on function to open the marker.

Notice the count variable just before the on function, this is used to ensure the SVG content is only created once, so only when count is equal to 1 will the content be created, the count ++ line is reached and increments the count value. After the first click on the marker the count variable is incremented to 2 and ensures that future use of the on function in any given session will recognize that the content has already been created.

The jQuery library has been introduced to assist with adding div and SVG content. We use the D3 code to append a circle element to the SVG in the div.

This time we are not using a dataset to generate the D3 content, we are simply creating a single SVG circle element. Notice the 2 additional on function references within the D3 code, these are used to handle mouseover and mouseout events, these provide a powerful way to interact with the SVG elements.

Exercise D4 Change the circle to a rectangle and change the color on click?

 

D3 5_d3_leaflet_mikeb.html
This example adds an SVG layer to the leaflet map and comes from the example provided by Mike Bostock - Mike is pretty much the godfather of D3.js. Some minor edits have been made to use the D3 v4 library. Open the file in your browser, you will see a US states SVG layer with an OSM tilelayer beneath. As you mouse-over a state its color changes from gray to brown, there is no click functionality.

Take a look at the source code:

  • 4 style references, these include the map extent, the map SVG layer and the SVG path style and hover functionality
  • An svg variable that uses a D3 select function to append svg and g elements to the map
  • A D3 json function, this is similar to the dataset we used in previous examples, except this time the data is being returned from a us-states.json file
  • A transform variable makes use of the project point function that uses Leaflet to implement a D3 geometric transformation
  • A path variable
  • A feature variable that constructs path elements based on the JSON dataset
  • A content on function with view reset argument calls the reset function that repositions the SVG to be in-line with the features. This essentially handles any changes to the map from actions like pan and zoom and resets the SVG layer accordingly
Exercise D5 Change the SVG layer features to orange and make each state appear blue on mouse-over.

 

D3 6_d3_leaflet_USA.html
This example is an extension on the last example, the difference being that the hover functionality has been shifted from the style to the D3 code block.

Open the file in a browser you will see a full screen map with a US states layer in place as it was in Mike Bostocks example, the colors are different and the map has click functionality in place.

Lets take a look at the source code, for the most part, it is the same as the previous example:

  • Style functionality has been shifted to D3 code block
  • The on function is used to handle mouseover mouseout and click events, this provides a bit more flexibility than the previous example
  • The reset function has been replaced by an update function
Exercise D6 Change the map data to the London dataset - borough_4326.geojson in the data directory of the course material.

 

Don't forget that we are working towards our master plan. At this stage, let's take a look at the blueprint of where we are aiming to get to.

blueprint

  • Split full screen solution
  • 70% map
  • 30% scatter-plot
  • Mouse-over functionality between plot and map
  • Mouse-over label display
 

D3 7_d3_scatter.html
Open the file, you will see a scatter-plot, lets take a look at the source code, you will notice some new content mostly related to the scatter-plot axes:

  • Style references for the plot axes
  • xScale and yScale variables , scaleLinear functions are used to map data from an input domain to an output range. These convert values from the data to a pixel location or length within an SVG or canvas
  • xAxis and yAxis variables, these provide reference lines and labels to a custom visualization
  • Append functions to append the x and y axes
Take a look at the dataset, how many objects are there? Now count the circles on the chart, you may notice one is missing - any ideas why? Take a look at the values of the last point and then look at the axes.

Exercise D7 Can you solve the mystery of the missing point?

Luckily D3 has great functions called min and max that figure out the minimum and maximum value of a list of values, this can then be used to plot the axes and ensure that they are automatically adjusted to accommodate any dataset:

  • Switch the .domain lines with the commented out .domain lines above that use the max function
  • Un-comment the labels section, this will display label content
  • Comment out the dataset variable and un-comment the random point generation dataset variable, each time you refresh your browser a new set of random points will be generated and the scatter-plot will adjust its axes to accommodate the random data.
 

D3 8_d3_scatter_borough.html
We will use the previous scatter-plot example to build a scatter-plot for the London borough dataset. Open the file in a browser, you will see two divs one containing the text "text goes here" and the one below that containing our scatter-plot of the London data with population on the x-axis and area on the y-axis.

Exercise D8 Display the name of the borough and its population density on point click, replace the "text goes here" text with the name of the borough the scatter-plot point represents.

 

D3 9_d3_highlight_order.html
Open the file, you will see a set of blue circles on top with a set of red circles below. If you mouse-over the blue circles nothing happens, if you mouse-over the red circles, the corresponding blue circle will change to yellow then back to blue on mouse-out. If you do this with your HTML format exposed in your browser developer tools, you can see the fill value changing in real time. We will use this functionality to connect our scatter-plot and map. The key is to have a common attribute to connect, for our final master plan we might use borough name or borough id. In this example, we have used the value because it is a common attribute in each of the red and blue datasets.

Let's take a look at the source code, most of the content will be familiar from previous examples. Two SVG tags have been added and appended to the two divs. The first dataset has a slightly different format but, d.val is used in both cases to return the value. For the blue circles, if you wanted to assign an id value of the dataset id rather than the value then you could use d.id rather than d.val.

The dataset id has been assigned to blue circles element id, take a look at the HTML. No ids have been assigned to the red circles however, the mouse-over and mouse-out functions are called using the id values assigned to the blue circles.

Exercise D9 The mouse-over relationship is currently in one direction - red to blue. Add mouse-over functionality to the blue set so that it changes the red set in some way. To do this you will need a way to generate unique ids using the datasets and the element ids. This is the final preparation before completing the master plan. Think mouse-over scatter-plot point to highlight borough and vice versa.

 

Exercise D10 If you have been working your way through the examples and exercises in this tutorial, you should have all the tools necessary to build the final visualization.

 

I hope you enjoy the tutorial, connect with me on twitter or Linkedin, I'd love to see where the workshop takes you.

 

Four items were raised in the workshop, these have all been addressed in this tutorial and include:

  1. The use of an ID name beginning with a number - although not best practice and built in haste to demonstrate D3.js append id attribute, it is actually a valid HTML5 ID format but is not valid in CSS, none the less, best to avoid the use of IDs beginning with a numeric character.
  2. A number of people attending the workshop had trouble with Cross-origin resource sharing (CORS) which according to [wikipedia](https://en.wikipedia.org/wiki/Cross-origin_resource_sharing), is a mechanism that allows restricted resources (e.g. fonts) on a web page to be requested from another domain outside the domain from which the first resource was served. Certain "cross-domain" requests, notably Ajax requests, are forbidden by default by the same-origin security policy. The issue related to the use of geojson data files in the local demo directory, this data would not display once the HTML demo files had rendered. Firefox users did not experience this issue. Thanks to Jez Nicholson for putting together a minimal node express server as a CORS work around for this tutorial.
  3. The leaflet_d3_popup example was opening and closing but not re-opening, the source code has been edited to resolve this behavior.
  4. Solution file paths were referencing the home directory rather than the data directory - the source files have since been adjusted.