Visualising the History of openSenseMap.org

Norwin Roosen

2018-10-20

This vignette serves as an example on data wrangling & visualization with opensensmapr, dplyr and ggplot2.

# required packages:
library(opensensmapr) # data download
library(dplyr)        # data wrangling
library(ggplot2)      # plotting
library(lubridate)    # date arithmetic
library(zoo)          # rollmean()

openSenseMap.org has grown quite a bit in the last years; it would be interesting to see how we got to the current 2407 sensor stations, split up by various attributes of the boxes.

While opensensmapr provides extensive methods of filtering boxes by attributes on the server, we do the filtering within R to save time and gain flexibility. So the first step is to retrieve all the boxes:

# if you want to see results for a specific subset of boxes,
# just specify a filter such as grouptag='ifgi' here
boxes = osem_boxes()

Plot count of boxes by time

By looking at the createdAt attribute of each box we know the exact time a box was registered. With this approach we have no information about boxes that were deleted in the meantime, but that’s okay for now.

…and exposure

Outdoor boxes are growing fast! We can also see the introduction of mobile sensor “stations” in 2017. While mobile boxes are still few, we can expect a quick rise in 2018 once the new senseBox MCU with GPS support is released.

Let’s have a quick summary:

exposure oldest newest count
outdoor 2015-02-18 16:53:41 2018-10-20 17:34:50 1919
indoor 2015-02-08 17:36:40 2018-10-18 10:34:43 361
mobile 2017-05-24 08:16:36 2018-10-18 19:45:02 107
unknown 2014-05-28 15:36:14 2016-06-25 15:11:11 20

…and grouptag

We can try to find out where the increases in growth came from, by analysing the box count by grouptag.

Caveats: Only a small subset of boxes has a grouptag, and we should assume that these groups are actually bigger. Also, we can see that grouptag naming is inconsistent (Luftdaten, luftdaten.info, …)

grouptag oldest newest count
Luftdaten 2017-03-14 17:01:16 2018-10-13 20:34:43 139
ifgi 2016-06-17 08:04:54 2018-09-11 22:17:28 38
TKS Bonn 2018-06-18 11:21:21 2018-10-19 06:20:50 35
PGKN 2018-03-22 16:44:00 2018-07-05 12:30:05 25
Luchtwachters Delft 2018-03-09 21:39:11 2018-10-15 19:38:22 23
luftdaten.info 2017-05-01 10:15:44 2018-08-22 20:39:16 16
Bad_Hersfeld 2017-07-18 13:32:03 2018-08-23 08:41:08 15
MakeLight 2015-02-18 16:53:41 2018-02-02 13:50:21 15
Luftdaten.info 2017-04-03 14:10:20 2018-09-09 09:16:03 12
Feinstaub 2017-04-08 06:38:25 2018-07-05 20:28:31 11
dwih-sp 2016-08-09 08:06:02 2016-11-23 10:16:04 11
Che Aria Tira? 2018-03-11 10:50:42 2018-03-11 23:11:20 10
Sofia 2017-04-11 04:40:11 2018-06-07 11:00:54 10
IKG 2017-03-21 19:02:11 2018-08-03 13:03:47 9
Raumanmeri 2017-03-13 11:35:39 2017-04-27 05:36:20 9

Plot rate of growth and inactivity per week

First we group the boxes by createdAt into bins of one week:

bins = 'week'
mvavg_bins = 6

growth = boxes %>%
  mutate(week = cut(as.Date(createdAt), breaks = bins)) %>%
  group_by(week) %>%
  summarize(count = length(week)) %>%
  mutate(event = 'registered')

We can do the same for updatedAt, which informs us about the last change to a box, including uploaded measurements. This method of determining inactive boxes is fairly inaccurate and should be considered an approximation, because we have no information about intermediate inactive phases. Also deleted boxes would probably have a big impact here.

inactive = boxes %>%
  # remove boxes that were updated in the last two days,
  # b/c any box becomes inactive at some point by definition of updatedAt
  filter(updatedAt < now() - days(2)) %>%
  mutate(week = cut(as.Date(updatedAt), breaks = bins)) %>%
  group_by(week) %>%
  summarize(count = length(week)) %>%
  mutate(event = 'inactive')

Now we can combine both datasets for plotting:

boxes_by_date = bind_rows(growth, inactive) %>% group_by(event)

ggplot(boxes_by_date, aes(x = as.Date(week), colour = event)) +
  xlab('Time') + ylab(paste('rate per ', bins)) +
  scale_x_date(date_breaks="years", date_labels="%Y") +
  scale_colour_manual(values = c(registered = 'lightgreen', inactive = 'grey')) +
  geom_point(aes(y = count), size = 0.5) +
  # moving average, make first and last value NA (to ensure identical length of vectors)
  geom_line(aes(y = rollmean(count, mvavg_bins, fill = list(NA, NULL, NA))))

We see a sudden rise in early 2017, which lines up with the fast growing grouptag Luftdaten. This was enabled by an integration of openSenseMap.org into the firmware of the air quality monitoring project luftdaten.info. The dips in mid 2017 and early 2018 could possibly be explained by production/delivery issues of the senseBox hardware, but I have no data on the exact time frames to verify.

Plot duration of boxes being active

While we are looking at createdAt and updatedAt, we can also extract the duration of activity of each box, and look at metrics by exposure and grouptag once more:

…by exposure

The time of activity averages at only 194 days, though there are boxes with 897 days of activity, spanning a large chunk of openSenseMap’s existence.

…by grouptag

grouptag duration_avg duration_min duration_max oldest_box
dwih-sp 735 days 581 days 802 days 802 days
Sofia 303 days 15 days 558 days 558 days
Feinstaub 294 days 4 days 561 days 561 days
IKG 292 days 70 days 514 days 578 days
Bad_Hersfeld 285 days 0 days 459 days 459 days
ifgi 274 days 0 days 769 days 855 days
Luftdaten 257 days 0 days 571 days 585 days
luftdaten.info 256 days 59 days 507 days 537 days
Che Aria Tira? 222 days 218 days 223 days 223 days
Luftdaten.info 194 days 25 days 565 days 565 days
Luchtwachters Delft 85 days 0 days 225 days 225 days
PGKN 73 days 0 days 212 days 212 days
Raumanmeri 45 days 7 days 318 days 586 days
TKS Bonn 43 days 0 days 124 days 124 days

The time of activity averages at only 225 days, though there are boxes with 802 days of activity, spanning a large chunk of openSenseMap’s existence.

…by year of registration

This is less useful, as older boxes are active for a longer time by definition. If you have an idea how to compensate for that, please send a Pull Request!

More Visualisations

Other visualisations come to mind, and are left as an exercise to the reader. If you implemented some, feel free to add them to this vignette via a Pull Request.