Data Web Services and a look at Pachube

I recently built an Arduino-based home electricity usage monitoring device, following the guides from the folks at OpenEnergyMon.

Being quite a data geek, I obviously wanted to be able to display various historical graphs and stats about the collected data. My device therefore has built-in Ethernet connectivity to be able to upload data on the Internet.

After doing a bit of code myself, I started to have a look to existing services that could help, which could be called Data Web Services. One really stands out, and it’s Pachube. Actually, it looks like it is currently the only service doing that. Simply put, it’s a web service which allows you to create data feeds, upload data through a REST-like API, and later retrieve this data. Everything is oriented around time series. The concept is quite neat and setting everything up was quite easy.

We want to analyse

Versatile storage and retrieving of stored data are obviously the basis of a Data Web Service. However,  I think it should provide more than just “store and retrieve” features.

Historical graphing first comes to mind. The user should be able to navigate in its data without limit and without excessive smoothing. The data from previous months

But everything becomes more interesting when you start doing more :

  • Computations : how much did I spend in electricity last week ? what was the average temperature last month ?
  • Comparison graphs

A platform or an end-user solution ?

Pachube is really a data platform. It does not know about what kind of data it handles, or what the user wants to do. This robust platform is obviously required, but I think the users want more. They want solutions, they want packaged applications, and this is where the real added value lies.

If you want to build a real home energy solution based on Pachube (and not just an history viewer), you have to build an application that uses the service as a data backend. To build this application, you have to do everything yourself (hosting, handling users)

An ideal Data Web Service should offer a platform for building applications based on its data. It should make it trivial to define user-centric dashboards.

The focus on the Internet Of Things and Open Data

Pachube’s main focus is on the Internet Of Things (IOT). It’s really designed from the ground up to handle data coming from embedded sensors and to allow some actions based on it. as put by their tagline “Manage real-time data from sensors, devices, and environments”.  Also, Pachube emphatizes the social and open data aspects.

I am a strong believer in the IOT. In a not-so-distant future, many of our devices will be Internet-enabled and having all of these devices interact and exchange data together is very exciting. It also joins the concept of Smart Grid, for energy usage monitoring and feedback.

However, not all data is open nor comes from devices. A Data Web Service should probably provide its users with robust and easy storage and analysis of private data. Wouldn’t you want to have a service that does detailed analysis of the evolution of your phone bills for example ?

A Data Web Service should also be able to retrieve large amounts of open data for public repositories and offer it to its users. Two major use cases come to mind :

  • Public open data repositories rarely come with sensible visualization services. Providing generic or specific (through the application building blocks) visualization of open data can help spread this idea and help citizens get a better insight into the data provided to them
  • Letting an user compare open data against its own private data can be of tremendous value. For example, Pachube became famous when a host of individuals started publishing radiation counters data following the Fukushima disaster. Providing comparisons between all of these and official figures is at least as interesting as the raw values from each counter.

What now ?

While I was thinking about this, several things happened. First, both Google and Microsoft shut down their services dedicated to home energy management (“Smart Grid”). Also, Pachube was acquired by LogMeIn, a provider of remote access and back-up services, both for personal and business users. While I am a bit puzzled by this acquisition, it definitely means that things are moving for Data Web Services.

There is space for new ideas, new concepts, new services. I am currently toying with the idea of creating a prototype for several of the ideas I have about this exciting service kind.

Software performance blog

Both in my work and in the various blog posts I read, I often find that people tend not to focus on the good questions when talking about software performance.

This can lead to wasting time, or even to actually worsen the performance of what you are trying to optimize.

Based on this, I started a new blog, called Software Performance. I intend to publish various tips and explanations about my vision of how to optimize your software, and how to do it the most efficiently possible.