To good to be true?

Recently, the Dutch Professor Diederik Stapel from Tilburg University was identified to be cheating with his micro data. There is an intermediate report about the case here (dutch).

The report says that two other professors did earlier qualify the research and its underlying data as "to good to be true". So there was doubt allover. The report mentions "strange, unlikely or impossible data patterns". This is intriguing. How does data look that is to good to be true? What did the other professors see? We would like to see it and visualize it, but the committee did not publish it....

They should...... they never heard of open data.....?
Only then we would be able to see how data that is too good to be true actually looks.......

The return of the CSV format

15 years ago we never thought that a flat format such as CSV would become the dominant format for open data and public data exchange, but these days it seems it is. That is remarkable since there have been so much attempts to create rich structured data formats. XML was getting popular as a hierarchical and easy to parse language. But these days it is heavily used for lower level data layers, but less for downloadable and easy accessible open public data. Excel used to be allover the world, but it turned out not to be a proper format for public data, since it is not open. Also, it is not well designed, mixing formatting constructs with data typing. RDF was promising, but was not the format of choice for hard-core data processing. SDMX tried in the field of statistics, but is still fighting for recognition. JSON is popular in the field of API's, not so much for the bare data download.

CSV is the winner...back to basics...easy to understand... plenty of tools to work on. Google's DSPL uses it, along with a rich model to describe the metadata of the data. The united nations database Undata uses it. Open data UK uses it and many more...

So the world of public data is getting more flat. The new data paradigm is to combine a simple flat data format with rich metadata about the flat data. Certain metadata, such as data types and hierarchical relationships among variables, can be included in the metadata but it is also possible that they can be derived from the flat data. This is how the treemapping tools of DrasticData work. They derive data types and hierarchical relationships from the bare flat data. Try the CSV examples included in the DrasticTreemapDesktop release and see how it works....

Dynamic treemapping on your desktop

Dynamic treemapping

DrasticData now offers a desktop application for dynamic treemapping, called DrasticTreemapDesktop. Downloaded it here:
http://www.drasticdata.nl/DDHome.php?m=drastictreemapdesktop

Very nice to explore the otherwise hidden patterns in your sales data, or other data.

Dutch Professions in 1899

This is an example of a visualization of a complex dataset, concerning the professions of Dutch people in 1899 in relation to the geographical location of people and their age. This visualization uses an interactive treemap, map and population pyramid to visualize the relationships between these variables.
See it here working in your browser.

DrasticTreemap component

DrasticTreemap is a dynamic treemap component that can be embedded into your web page to display your data. An example and free download can be found here.

Newsmap

This is an example of using treemap technology for the display of news in such a way that the most prominent items catch your eye, while it is still possible to have a feeling of the other news. The newsmap is refreshed automatically every few minutes, repositioning the news items and graphically showing the new items (wait a while to see it change).
You can see the in browser version operating on Google news here.
A full browser window version is here

Who populated the internet?

On the use of animated graphics for data visualization we made this example of internet users per country.

Animated stack graph

Fuel Consumption of cars

Some time ago we made this example of an interactive bar chart displaying the fuel consumption of cars in Australia. Although the data is quite old, it still shows how this kind of data can be represented so that the main characteristics quickly become clear.
See the example here.