Question

Suppose I’ve been driving a set route with a 3g modem and GPS on my laptop, while my computer back at home records the ping delay. I’ve correlated ping with GPS lat/long, and now I’d like to visualise this data.

I’ve got about 80,000 points of data per day, and I’d like to display several month’s worth. I’m especially interested in displaying areas where ping consistently times out (ie ping == 1000).

Scatter plot

My first attempt was with a scatter plot, with one point per data entry. I made the size of the point 5x larger if it was a timeout, so it was obvious where these areas were. I also dropped the alpha to 0.1, for a crude way to see overlaid points.

# Colour
c = pings 
# Size
s = [2 if ping < 1000 else 10 for ping in pings]
# Scatter plot
plt.scatter(longs, lats, s=s, marker='o', c=c, cmap=cm.jet, edgecolors='none', alpha=0.1)

The obvious problem with this is that it displays one marker per data point, which is a very poor way to display large amounts of data. If I’ve drive past the same area twice, then the first pass data is just displayed on top of the second pass.

Interpolate over an even grid

I then had a try at using numpy and scipy to interpolate over an even grid.

# Convert python list to np arrays
x = np.array(longs, dtype=float)
y = np.array(lats, dtype=float)
z = np.array(pings, dtype=float)

# Make even grid (200 rows/cols)
xi = np.linspace(min(longs), max(longs), 200)
yi = np.linspace(min(lats), max(lats), 200)

# Interpolate data points to grid
zi = griddata((x, y), z, (xi[None,:], yi[:,None]), method='linear', fill_value=0)

# Plot contour map
plt.contour(xi,yi,zi,15,linewidths=0.5,colors='k')
plt.contourf(xi,yi,zi,15,cmap=plt.cm.jet)

From this example

This looks interesting (lots of colours and shapes), but it extrapolates too far around areas I haven’t explored. You can’t see the routes I’ve travelled, just red/blue blotches.

If I’ve driven in a large curve, it’ll interpolate for the area between (see below):

Interpolate over an uneven grid

I then had a try at using meshgrid (xi, yi = np.meshgrid(lats, longs)) instead of a fixed grid, but I’m told my array is too big.

Is there an easy way I can create a grid from my points?

---

My requirements:

• Handle large data sets (80,000 x 60 = ~5m points)
• Display duplicate data for each point either by averaging (I assume interpolation will do this), or by taking a minimum value for each point.
• Don’t extrapolate too far from data points

I’m happy with a scatter plot (top), but I need some way to average the data before I display it.

(Apologies for the dodgy mspaint drawings, I can’t upload actual data)

---

Solution:

# Get sum
hsum, long_range, lat_range = np.histogram2d(longs, lats, bins=(res_long,res_lat), range=((a,b),(c,d)), weights=pings)
# Get count
hcount, ignore1, ignore2 = np.histogram2d(longs, lats, bins=(res_long,res_lat), range=((a,b),(c,d)))
# Get average
h = hsum/hcount
x, y = np.where(h)
average = h[x, y]
# Make scatter plot
scatterplot = ax.scatter(long_range[x], lat_range[y], s=3, c=average, linewidths=0, cmap="jet", vmin=0, vmax=1000)

Answer 1

To simplify your question, you have two set of points, one for ping<1000, one for ping>=1000.
Since the count of points is very large, you can’t plot them directly by scatter(). I created some sample data by:

longs = (np.random.rand(60, 1) + np.linspace(-np.pi, np.pi, 80000)).reshape(-1)
lats = np.sin(longs) + np.random.rand(len(longs)) * 0.1

bad_index = (longs>0) & (longs<1)
bad_longs = longs[bad_index]
bad_lats = lats[bad_index]

(longs, lats) is points for ping<1000, (bad_longs, bad_lats) is points for ping>1000

You can use numpy.histogram2d() to count the points:

ranges = [[np.min(lats), np.max(lats)], [np.min(longs), np.max(longs)]]
h, lat_range, long_range = np.histogram2d(lats, longs, bins=(400,400), range=ranges)
bad_h, lat_range2, long_range2 = np.histogram2d(bad_lats, bad_longs, bins=(400,400), range=ranges)

h and bad_h are the points count in every little squere area.

Then you can choose many methods to visualize it. For example, you can plot it by scatter():

y, x = np.where(h)
count = h[y, x]
pl.scatter(long_range[x], lat_range[y], s=count/20, c=count, linewidths=0, cmap="Blues")

count = bad_h[y, x]
pl.scatter(long_range2[x], lat_range2[y], s=count/20, c=count, linewidths=0, cmap="Reds")

pl.show() 

Here is the full code:

import numpy as np
import pylab as pl

longs = (np.random.rand(60, 1) + np.linspace(-np.pi, np.pi, 80000)).reshape(-1)
lats = np.sin(longs) + np.random.rand(len(longs)) * 0.1

bad_index = (longs>0) & (longs<1)
bad_longs = longs[bad_index]
bad_lats = lats[bad_index]

ranges = [[np.min(lats), np.max(lats)], [np.min(longs), np.max(longs)]]
h, lat_range, long_range = np.histogram2d(lats, longs, bins=(300,300), range=ranges)
bad_h, lat_range2, long_range2 = np.histogram2d(bad_lats, bad_longs, bins=(300,300), range=ranges)

y, x = np.where(h)
count = h[y, x]
pl.scatter(long_range[x], lat_range[y], s=count/20, c=count, linewidths=0, cmap="Blues")

count = bad_h[y, x]
pl.scatter(long_range2[x], lat_range2[y], s=count/20, c=count, linewidths=0, cmap="Reds")

pl.show()

The output figure is: