World Map Stat Counter

If you visit https://planet.gnome.org/ and scroll to the bottom of the page, you will see a small picture of a world map with dots. The dots represent where the people in the GNOME community come from. A larger version is available here https://wiki.gnome.org/GnomeWorldWide .

This page describes a similar independent implementation - except that the dots depict where each of the website visit (aka stats, or "hits"), like this:

How to use in your own website

1. Requirements

  1. GD library (usually called "gd" or "libgd")
  2. netpbm tools (usually called "netpbm") - this gives you tools like pnmtojpeg, jpegtopnm, etc
  3. "Fly" - this is a command line interface to GD, from here: http://martin.gleeson.net/fly .

2. Preparing the data file

  1. Download the tarball.
  2. Extract the tarball to a directory.
  3. Get the maxmind.com's GeoIP Lite City CSV database from http://geolite.maxmind.com/download/geoip/database/GeoLiteCity_CSV/GeoLiteCity-latest.zip , then extract it somewhere
  4. Convert those CSV databases to binary databases like this:

    ./convert.sh /path/to/geoiplite/block-database.csv /path/to/geoiplite/location.csv
    
    When done, you will get ipinfo.dat and locinfo.dat in the current directory.

3. Preparing your website

  1. Copy ipgeocode, genimage.sh and map.png from the tarball to your webserver's cgi-bin directory.
  2. Copy the two 'dat' files you have prepared earlier to the cgi-bin directory too.
  3. You may want to examine genimage.sh whether the location of the data files and the image files are correct, if not, adjust them as needed.
  4. genimage.sh expects to fed with IP addresses on it standard input, one per line. It will produce a JPEG file on standard output. You will need to create a CGI script that provides this necessary information to genimage.sh as well as returning the output back to the webserver. This varies from webserver to another, here is one that works with SJPPLOG (the blog software I'm using for my blog):

    #!/bin/sh
    echo -ne "Content-Type: image/jpeg\r\n"
    echo -ne "Content-Disposition: inline; filename=\"stat.jpg\"\r\n"
    echo -ne "\r\n"
    awk -F"|"  '{print $1}' /path/to/your/sjpplog/online.ppl.uo 2> /dev/null | sort | uniq | ./genimage.sh
    
    Other webserver will be different.

How it was done

In principle it is very simple.

  1. We have a bunch of IP addresses.
  2. We need to convert those IP addresses to a location where they come from (IP geocoding), preferably in latitude / longitude coordinates.
  3. Mark the these coordinates in a map, using Mercator projection.

IP Geocoding

There are many ways to do IP Geocoding. There are plenty of online geocoding services (both free and paid). I choose not to use them because I will need to do a lot of geocoding so I probably couldn't meet the terms of service from free online services, and paying for this service just to provide a stat counter is silly. Another reason is time; it is quick to use WebServices when one only needs to geocode a single IP address, but I potentially have to geocode thousands of IP address in a single request.

Thanks to maxmind.com that provides the free (as in CC-BY-SA free) IP geocoding database, I can do the geocoding on my own (it's just a simple look-up table). I could use the data "as is" and perform the lookup with my favorite tools awk and grep, but the voluminous data makes me wanting of a more optimised solution.

Marking

Once I have the lat/long data, it is just a matter of plotting the markers on the map. This requires two things:

  1. The map (which I got the map from wikimedia (see notes below)).
  2. Knowing how to plot it correctly (explained here.
  3. Find the nice tool to do it. There are many tools available, I choose Fly because it is the most lightweight.

Miscellaneous notes

  1. map.png comes from http://upload.wikimedia.org/wikipedia/commons/thumb/7/74/Mercator-projection.jpg/310px-Mercator-projection.jpg which I converted to png format as Fly only recognises png.
  2. You can use another map as long as it is done using Mercator projection. Remember to change the image size inside genimage.sh.
  3. The tarball contains Linux 32-bit static binaries, that should work on all x86 Linuxes both 32 and 64-bit. If you need to compile, the sources are provided and are licensed under GNU GPL Version 3 or later.
  4. Fly unfortunately isn't included in most distributions so you have to compile it yourself. I would have included a static binary of fly were it not linked to other large system libs like libpng, libjpeg, libfreetype in addition to libgd.
    It is used in the final rendering of the "marks" in the map image; if you don't like to use it you can modify genimage.sh to use other command-line image manipulation tools like imagemagick, graphicsmagic, gmic, etc.
  5. The GeoIP Lite database is courtesy of maxmind.com. The conversion is purely for performance reasons; one can do exactly the same thing using awk and grep but the speed is much slower (about 2 orders of magnitude slower).

Originally posted here: