GeoServer and efficient delivery of raster data (image pyramid layer) (update)

A perennial favorite on this blog is “GeoServer and efficient delivery of raster data (image pyramid layer)“. I am neither the last nor the first authority on this topic (check the GeoSolutions blog for authoritative work on GeoServer and raster, also look to the GeoServer documentation), but I’ve had some good experiences with serving rasters in GeoServer, especially using image pyramid layers

Read the original, as this will just augment, but here are some targets to hit with the retiling necessary for larger datasets. This is the command I currently use for the retiling:

gdal_retile.py -v -r bilinear -levels 4 -ps 6144 6144 -co "TILED=YES" -co "BLOCKXSIZE=256" -co "BLOCKYSIZE=256" -s_srs  EPSG:3734 -targetDir aerial_2011 --optfile list.txt

Don’t be afraid of big block sizes. Bump your memory up in your application container, stop worrying and learn to love the larger tif. I keep my total number of output tifs to no more than 2000, where I start to see performance issues in my implementation.
Also, give the image pyramid code a break. After retiling, do this:

mkdir 0
mv *.tif 0/.

Copying GWC Directory efficiently– Linux

There has been discussion recently amongst some of the web cache builders of switching from file-based tile caching to SQLite or other database driven systems.  I haven’t kept close tabs– perhaps this is already implemented for some.  For GeoWebCache, we’re working with files (lots of them), so copying efficiently if we need to move our cache is important.  The tar command helps here, e.g. (run in GWC cache directory)

 tar cf - * | ( cd /path-to-target-dir/; tar xfp -) 

 

WMS vs. WMTS and GetFeatureInfo Requests

How to get GetFeatureInfo requests to perform against GeoWebCache?  Just don’t.  I’m unclear as to whether it is problematic to do this because of a known bug or known implementation differences between the (unofficial) WMTS standard as implemented by OpenLayers and GeoServer, but in Googling for answers to how to perform a GetFeatureInfo request against GeoWebCache, I discovered a workaround (logged here for example).  The trick is to specify the url for the WMSGetFeatureInfo request as your WMS service, and layersUrls as your GeoWebCache, if that’s what you’re using for all your layers, like in my case.

Here is a control all fleshed out (thanks to my intern John for writing/modifying all but two of these lines… ):

var info = new OpenLayers.Control.WMSGetFeatureInfo({
	drillDown: false,
	queryVisible: true,
	url: GeoserverWMS,
	layerUrls: [GeowebcacheURL],
	eventListeners: {
		getfeatureinfo: function(event) {
			popup = new OpenLayers.Popup.FramedCloud(
				"popinfo",
				map.getLonLatFromPixel(event.xy),
				null,
				event.text,
				null,
				true
			);

			map.addPopup(popup, true);
		}
	}
});

map.addControl(info);

info.activate();

GeoWebCache Configuration

I’ve long delayed the configuration of GeoWebCache (GWC).  Automatic configuration within GeoServer’s deployment of it has been adequate until now.   Until now that I want to deploy tiles in something other than Google Mercator… .

I’ve developed a cache setup for Ohio State Plane North, Feet (epsg:3734).  At first it was a going to be a cache just for our service area, then I thought I’d include surrounding counties, and by the time I was done, I just decided that having a cache definition for the entire state plane ohio north coverage area made as much sense as anything else– it could make things easier to share and integrate as part of a regional standard for WMS-C, assuming I get buy-in from others… .  It could also be easily adapted for State Plane South.

The definition for a cache requires three things:  coordinate extent, resolutions (or equivalent), and tilesize (height and width).  For coordinate extent, I chose the extents of epsg:3734 as defined by spatialreference.org, and snapped them to the nearest 5000 foot interval:

1320000

160000

2525000

976000

Then, choosing resolution based on a series of measurable scales (1:600 or 1″=50′, 1:1200 or 1″=100′, etc) we get the resolutions:

918.6351706036746

229.65879265091866

58.20472440944883

45.931758530183735

29.102362204724415

22.047244094488192

14.220472440944883

11.023622047244096

8.818897637795276

8.267716535433072

5.511811023622048

4.409448818897638

2.204724409448819

1.1023622047244095

0.5511811023622047*

Finally, we’ll use a predictable tilesize:

<tileHeight>256</tileHeight>

<tileWidth>256</tileWidth>

Now we have a portable tilecache recommendation that should work for all of Northern Ohio:
GWC Example

GeoServer and efficient delivery of raster data (image pyramid layer)

One thing I’ve learned in the last few years is that there is no theoretical reason why (properly indexed and summarized) data cannot be displayed at all scales as quickly as at any scale. This is the principle at work behind the extraordinary efficiencies of delivering data and imagery through a slippy map interface like Google, Bing, and OpenLayers as well as efficient thick client interfaces like Google Earth. So, in principle, and largely in practice, serving spatial data for a whole county or the whole world shouldn’t be any more onerous than serving data for a particular site, so long as you have adequate storage for the pre-rendered and summarized data, and time to pre-render the data. As storage tends to be cheaper than processing and network speed, this is a no-brainer.

A number of great Open Source tools exist to help with serving large amounts of data efficiently, not the least of which is my favorite, GeoServer (paired with GeoWebCache). For serving imagery, in our case orthorectified 0.6-inch (0.1524 meter) aerial imagery, we have a few options. GeoServer does natively support GeoTiff, but for this large an area at this level of detail, we’d have to wade into the realm of BigTiff support through the GDAL extension, because we have 160GB imagery to serve. We could use wavelet compressed imagery, e.g. MrSid or ECW or Jpeg2000, but I don’t have a license to create a lossless version of these, and besides, storage is cheaper than processors– wavelet compressed imagery may be a good field solution, but for server side work, it doesn’t make a lot of sense unless it’s all you have available. Finally, there are two data source extensions to GeoServer meant for large imagery, the ImageMosaic Plugin, and the ImagePyramid Plugin. The ImageMosaic Plugin works well for serving large amounts of images, and has some great flexibility with respect to handling transparency and image overlap. The ImagePyramid extension is tuned for serving imagery at many scales. The latter is what we chose to deploy.

The ImagePyramid extension takes advantage of gdal_retile.py, a utility built as part of GDAL that takes an image or set of images and re-tiles them to a standardized size (e.g. 2048×2048) and creates overviews as separate images in a hierachy (here shown as outlines of the images):

But here’s the problem– for some reason, I can’t load all the images at once. If I do, only the low resolution pyramids (8-foot pixels and larger) load. If I break the area into smaller chunks, most of them fewer than 2000 images, they load fine.

1" = 50' scale snapshot

1:32,000 scale snapshot