Contours– Structuring PostGIS data for viewing with GeoServer

Naively structured data is my bane– the desire (and need) to get stuff done so often overtakes the time needed to do things the better way. So, we bootstrap.

A long time ago, we managed to load in a few tens of gigs of contour data into PostGIS, partitioned it into 2ft, 10ft, 20ft, 50ft, 100ft and 250ft tables using select queries with a modulus operator, e.g.

CREATE TABLE base.cuy_contours_10
	AS
	SELECT elevation, the_geom
		FROM base.cuy_contours_2
		WHERE base.cuy_contours_2.elevation % 10 = 0;

And then we built SLD‘s to display the data in GeoServer and formed the data into layer groups, and away we went… .

However, layer groups can often be replaced by properly structured PostGIS tables. For large (and homogenous) datasets like this, it’s the only way to go. In addition, properly structuring this allows us to take advantage of GeoServer’s ability to modify the legend based on zoom extent, which for representing contours at a range of scales is an almost “must have”.

To structure the table, we could be disciplined and build this out as a proper normalized relational dataset where our gid is used to determine if a contour is divisible by 10, 20, 50 etc., and while “I don’t wanna” isn’t a good enough reason not to do this, I think the computational overhead of a database view piecing these data back into a single table each time we need to access this would not be justified in the light of the static nature of the table. So database normalization be darned, disk space is cheap, full speed ahead. Let’s add some boolean fields for flagging whether a contour is divisible by our numbers and calculate that out:

UPDATE base.contours_2
	SET div_10 = CAST( contours_2.elevation % 10 AS BOOLEAN );

…

UPDATE base.contours_2
	SET div_250 = CAST( contours_2.elevation % 250 AS BOOLEAN );

yields the following (with highlights added):

Sort of the opposite of what I intended, the “falses” maybe should be “trues” and vice versa, but hey, it’ll work anyway. BTW, we did test doing this calculation a few different ways, and while I can’t remember all the details, doing this as a calculation instead of doing an update query with a while statement testing the modulus was much faster (3x faster).

Ok, so now we style an sld for it, and sit back and enjoy (pics later…).

GeoServer Optimization

As we move away from a simple stack of PostGIS/GeoServer/GeoWebCache/Openlayers to wrapping a MapFish print service into the stack, it’s time to think more seriously about optimizing and stabilizing GeoServer.

In preparation for this step, I’ve been setting up a series of VMWare ESX-hosted Debian Linux VMs to function as the cluster of geospatial services.

Fortunately for me, there’s plenty of great advice in Andre Aime’s 2009 Foss4G presentation GeoServer in Production. Here’s what I gleaned from the presentation (any mistakes are undoubtedly mine and not Aime’s), plus a little bit of expansion from me:

1) Control the requests coming into the system. In this case, Andre talks about application container requests, limiting, e.g. Tomcat concurrent requests to 20 instead of the default 200:

maxThreads="20" minSpareThreads="20"

2) Set up a high availability (HA) cluster. There are lots of ways to skin this beast, but a cheap and easy way is via the Ultimate Cheapskate Cluster. In Aime’s presentation, this is using vrrpd + balance, but with the current option of using “Pen“, stateful protocols like WFS-Transactional should be supported in addition to WMS.

3) Set up your java virtual machines intelligently. Most of this information get’s covered in GeoServer’s documentation page Running in a Production Environment. Additions from Andre’s presentation which might still be relevant are the following JVM flags:

-XX:NewRatio=2
-XX:+AggressiveOpt

If you use the second one, the JVM will use experimental optimizations, so test for stability before using this in a production environment.  The first one notifies the virtual machine that there will be many temporary objects.

(FYI, for the nubes like me out there– JVM flags for Tomcat are set in the Catalina.sh startup script.)

3a) This get’s a special subheading, ’cause I couldn’t figure out why my WMS rendering was slow and unstable when I switched from Windows to Linux: Install and Use JAI & JAI Image I/O.

4) Finally, make sure your data are structured properly. If it’s really big imagery (>2GB), use an Image Pyramid, but otherwise, take advantage of internal tiling and overviews.  Examples from gdal’s utilities include
gdal_translate -of GTiff -co "TILED=YES" utm.tif utm_tiled.tif
which creates internal tiling, and

gdaladdo -r average utm_tiled.tif 2 4 8 16 32 64 128 256

which adds overviews.  You might also look to optimize the size of internal tiling.

For vector data, use PostGIS (not shapefiles), and index on your geometry and any attributes that are used in your SLD as filters. Also, show simple symbology when “zoomed out”, and reserve the complex rules for closer zoom levels.

An alternative that Aime doesn’t mention is that for really complicated data, you can do additional optimization. You can create generalized geometry columns as alternate columns. This is the vector equivalent of overviews. The SLD can then be coded to use the alternate simplified geometry at coarser scales (see e.g. this post for info on how to specify the geometry column in an SLD).  I wish I could find the GeoServer post that originally advocated this technique… .

Hopefully this helps stabilize, optimize, and increase the availability of your GeoServer instance.  Hopefully it does so for mine as well… .

POV-ray for GIS Visualization

This is a “wouldn’t it be nice” post– i.e. a I-don’t-have-time-to-write-it-and-everything-else-on-my-list post. Wouldn’t it be nice if in addition to returning GeoRSS, WFS, OGC, KML, etc. GeoServer returned a POV-Ray file? It could be returned as part of a simple GET post like this:

http://localhost:8080/geoserver/wms?bbox=2100506,575982,2290532,739026&styles=&Format=application/povray&request=GetMap&version=1.1.1&layers=base:contours&width=800&height=644&srs=EPSG:102722&camera=2195519,657504,1000&look_at=2195029,657004,500

Vector symbolization could be a POV-Ray approximation of the WFS render styles using coordinates and POV primitives. Also, a PHP script could consume this return, and run a command line version of POV to render the file in a webserver implementation. This second part I’ll probably actually write with just a DEM and image overlay, but it would be cool to have the vector symbolization as well as part of another GeoServer protocol.  Of course, I recognize this is not a OGC protocol… .  Alternatively, a PHP script to consume WKT and convert it to a POV-Ray file might be easier than incorporating it in GeoServer.