Smathermather's Weblog

Remote Sensing, GIS, Ecology, and Oddball Techniques

Archive for the ‘GDAL’ Category

gdal_warp, cutlines, and cwhere– simple tip for use on Linux

Posted by smathermather on December 17, 2011

Mini GDAL tip of the day:

gdalwarp, especially in combination with gdal_merge, is a powerful tool for doing all sorts on nice aggregation (read: mosaic’ing) of spatial raster data.  Unfortunately, at least with a google search, there’s very little to be found on demonstrating the use of queries in conjunction with cutlines, probably because in general these queries are not difficult to figure out.

In a previous post, I demonstrated using this to stitch together USGS quads.  I found out when I tried to run this code on a Linux machine, the command like code was a little different.  Instead of:

gdal_merge -o usgs_merge.tif -createonly input1.tif input2.tif ...
gdalwarp -cutline quadrangle_index_24k.shp -cwhere "quadname_ = West_View" West_View.tif usgs_merge.tif

I found I needed some extra single quotes to get my string literal to not be interpreted as a field:


gdalwarp -cutline quadrangle_index_24k.shp -cwhere "quadname_ = 'West_View'" West_View.tif usgs_merge.tif

Ah, that’s better.  Now I have a nice big mosaic of DEMs for my region with perfect cuts where the overlaps used to be.  (The change is hard to see– look for single quotes around West_View in “quadname_ = ‘West_View’”)

Posted in GDAL, GIS | Tagged: , , , , | Leave a Comment »

GDAL Slopes– Local Hydrologic Slope vs. the Standard Approach

Posted by smathermather on December 15, 2011

Open Source software is not, of course just about warm and fuzzies, great support, rapid development cycles, shared costs, etc., it’s also about getting your hands dirty with someone else’s code and implementing stuff more quickly and more intelligently because of it, and hopefully learning something in the process.  You don’t have to poke under the hood to drive the car, but sometimes it’s nice to anyway.

The two normal approaches to calculating slope in GIS raster tools (used in applications from ESRI’s Spatial Analyst and GDAL’s gdaldem) are Horn’s formula and Zevenbergen & Thorne’s formula, both integrated calculations of slope that do a bit of generalization in the process of calculating slope.  This is particularly useful if you are using slope for more than a localized calculations, for example in doing estimates of local light conditions.  That said, if you want to match local hydrologic slope (something which will more closely approximate a field measurement of slope), you are better off using a different formula altogether.  An article a colleague sent me: “When GIS Slope is Not What You Think” in this month’s The Forestry Source tipped me off to this, and I thought I’d do some quick and dirty digging in the GDAL code to see if I could implement something a little different.

Here is the standard Horn approach as currently implemented in GDAL:

float GDALSlopeHornAlg (float* afWin, float fDstNoDataValue, void* pData)
{
    const double radiansToDegrees = 180.0 / M_PI;
    GDALSlopeAlgData* psData = (GDALSlopeAlgData*)pData;
    double dx, dy, key;

    using std::max;
    dx = (max((afWin[0], afWin[3], afWin[3], afWin[6])) -
          max((afWin[2], afWin[5], afWin[5], afWin[8])))/psData->ewres;

    dy = ((afWin[6], afWin[7], afWin[7], afWin[8]) -
          max((afWin[0], afWin[1], afWin[1], afWin[2])))/psData->nsres;

    key = (dx * dx + dy * dy);

    if (psData->slopeFormat == 1)
        return (float) (atan(sqrt(key) / (8*psData->scale)) * radiansToDegrees);
    else
        return (float) (100*(sqrt(key) / (8*psData->scale)));
}

Borrowing from gdaldem roughness we implement this:

float GDALSlopeHydroAlg (float* afWin, float fDstNoDataValue, void* pData)

{
    // Hydrologic Slope is the max
    //  local slope btw center cell and adjacent cells

    const double radiansToDegrees = 180.0 / M_PI;
    GDALSlopeAlgData* psData = (GDALSlopeAlgData*)pData;
                double key;

    float pafLocalMin = afWin[0];
     float pafLocalMax = afWin[0];

    for ( int k = 1; k < 9; k++)
    {
        if (afWin[k] < pafLocalMin)
        {
            pafLocalMin=afWin[k];
        }

        if (afWin[k] > pafLocalMax)
        {
            pafLocalMax=afWin[k];
        }
    }

    key = afWin[4] - pafLocalMin;

    if (psData->slopeFormat == 1)
        return (float) (atan(sqrt(key) / (2*psData->scale)) * radiansToDegrees);
    else
        return (float) (100*(sqrt(key) / (2*psData->scale)));
}

Output is rougher than the original, as expected:

Local Hydrologic Slope

Zevenbergen & Thorne’s Slope

And looking at the stats we see a larger standard deviation for local hydrologic slope:


Z:\>gdalinfo Hydrologic_slope.tif
Driver: GTiff/GeoTIFF
Files: Hydrologic_slope.tif
...

Metadata:
STATISTICS_MINIMUM=0
STATISTICS_MAXIMUM=46.168109893799
STATISTICS_MEAN=10.206419514979
STATISTICS_MEDIAN=2.8041767686283e-266
STATISTICS_MODE=1.4463930749435e-307
STATISTICS_STDDEV=5.3794018830967
LAYER_TYPE=athematic

Z:\>gdalinfo Zevenbergen_Thorne.tif
Driver: GTiff/GeoTIFF
Files: Zevenbergen_Thorne.tif
...
Metadata:
STATISTICS_MINIMUM=0
STATISTICS_MAXIMUM=52.424499511719
STATISTICS_MEAN=3.0666069784504
STATISTICS_MEDIAN=4.331070843283e-184
STATISTICS_MODE=-4
STATISTICS_STDDEV=3.567882216098
LAYER_TYPE=athematic

A couple of additional notes.  First, this is 2.5ft resolution data.  At this resolution, the differences in algorithms may be somewhat academic.  Also, of note– the range for the Zevenbergen-Thorn slope algorithm is greater than that for the local hydrologic slope.  This is an outcome I did not anticipate.

 

Update:

Hmmm, looks like I have more revisions and questions from the GDAL dev board. May have to revisit this… .  Forgot about the corners of my 3×3 window… .

Posted in Analysis, Ecology, GDAL, GIS | Tagged: , , , , , | Leave a Comment »

gdal_calc.py– Raster Calculator using Numby Functions

Posted by smathermather on December 1, 2011

gdal_calc is a python script that makes it easy to do band math and logical operations with gdal/numby. This ranges from basic arithemtic operations to logical functions. And while gdal_calc.py has been around since 2008, it is but is a recent and revelatory discovery for me. I had just today resigned myself to properly learning python so as to use the gdal bindings. But, my laziness may continue unabated. Not learning Python. Now that is lazy.

Anyway, want to do band math? It’s as simple as follows (straight from the python script itself):

# example 1 - add two files together
gdal_calc.py -A input1.tif -B input2.tif --outfile=result.tif --calc="A+B"

# example 2 - average of two layers
gdal_calc.py -A input.tif -B input2.tif --outfile=result.tif --calc="(A+B)/2"

# example 3 - set values of zero and below to null
gdal_calc.py -A input.tif --outfile=result.tif --calc="A*(A>0)" --NoDataValue=0

So, I had little modification to average my set of images from PovRay:

gdal_calc.py -A input.tif -B input2.tif -C input3.tif -D input4.tif -E input5.tif --outfile=result.tif --calc="(A+B+C+D+E)/5"

I might write some bash wrappers to create built in functions for basic things I’ll do repeatedly, like average a set of images based on a directory listing. Of course, a little voice inside says “You should just do that in Python.” We’ll see.

Posted in Analysis, GDAL, GIS, Image Processing | Tagged: , , , , , | 1 Comment »

GDAL, MrSid, and nearblack

Posted by smathermather on July 29, 2011

Translating MrSid lossy compressed files into uncompressed imagery has its drawbacks, including licensing and artifacts.
Old versions of fwtools, which includes the GDAL utilities (and more), were compiled with a license that allowed for the translation of MrSid into e.g. Erdas Imagine images or GeoTiff. The licensing changed on that library, so FWTools and MS4W don’t do this anymore.

If you have a compiled version of the GDAL utilities with a legal license for converting, you still run into the issue of artifacts, e.g.

Nearly, but not quite black pixels at the edge of MrSid lossy compressed image

GDAL’s nearblack is a great utility for correcting this problem. If we wanted to batch this up on a windows system, it might look something like this:


FOR %f IN (*.sid) DO nearblack -o %~nf.img %f

Posted in GDAL, GIS, Image Processing | Tagged: , , , , | 2 Comments »

Mapping places unknown– free global datasets and FOSS GIS are a great combo

Posted by smathermather on January 31, 2010

I wanted to put together a quick and dirty map of a biological reserve in Ecuador, sort of a laptop exploration of a place quite distant.  At first, I thought I’d use Shuttle Radar Topography Misssion data to get the elevation information.  Then I discovered the ASTER Global DEM which is 30m resolution for the whole world.  Wow.  Cool cool data. (I used the Japanese site to download, as I had trouble getting the US site to load).

So, first I downloaded and extracted the zipped geotiffs that come from the ASTER Global DEM.  I extracted the DEMs into the same directory, and mosaicked them into a single DEM:

gdalwarp AST*.tif ecuador_aster_dem.tif

Then I created a hillshade version of the DEM for mapping:

gdaldem hillshade ecuador_aster_dem.tif ecuador_aster_hillshade.tif

and then 100 and 50 meter contours for the site:

gdal_contour -a elev -3d -i 100 ecuador_aster_dem.tif ecuador_aster_contour_100.shp
gdal_contour -a elev -3d -i 50 ecuador_aster_dem.tif ecuador_aster_contour_50.shp

I loaded it all into Quantum GIS, symbolized it, and added some OpenStreetMap data.  To do this, I navigated to the location in OSM, chose the download tab, and chose OpenStreetMap XML Data as my output.  Using the OSM plug-in in QGIS, I imported the file, and walla– a nearly functional map.

Posted in GDAL, GIS | Tagged: , , , , | Leave a Comment »

Batch processing resampling of DEMs

Posted by smathermather on January 4, 2010

Yet again, working with that Ohio DEM dataset, I need some reduced resolution versions of the 2.5 foot DEMs.  At a grand total of ~600 DEMs for a given county, it was time to batch, this time in Windows Command prompt.  The neat thing is the variable substitution in the for-in-do command in the command prompt, where, e.g. for a file variable %f, the file name, minus extension can be invoked with %~nf.  As I write this,  I realize that I didn’t strictly need that variable for this dataset, as they were ESRI Grids, so they have no extension to trim off… .  Hehe.  I think I was being stupidly clever.  But, well, trim off the /D flag, and you can use this with any file extension… .  So, ya.  This is the more generic solution… .

I use bilinear resampling below for each tier from half the resolution.  Why bilinear when there are fancy resampling methods available?  I trust bilinear to do exactly what I want, and that is create an averaged tier that can be overlayed with the original data 100% (knock on wood… .).  I can quantify it’s behavior at each tier.  In other words, I’m fussy, and I understand bilinear, and haven’t invested the time to find out if I like anything else… .

for /D %f in (n*) do gdalwarp -tr 5 5 -r bilinear %f resampled\%~nf_05.tif
for /D %f in (n*) do gdalwarp -tr 10 10 -r bilinear resampled\%~nf_05.tif resampled\%~nf_10.tif
for /D %f in (n*) do gdalwarp -tr 20 20 -r bilinear resampled\%~nf_10.tif resampled\%~nf_20.tif
for /D %f in (n*) do gdalwarp -tr 40 40 -r bilinear resampled\%~nf_20.tif resampled\%~nf_40.tif
for /D %f in (n*) do gdalwarp -tr 80 80 -r bilinear resampled\%~nf_40.tif resampled\%~nf_80.tif
for /D %f in (n*) do gdalwarp -tr 160 160 -r bilinear resampled\%~nf_80.tif resampled\%~nf_160.tif

Posted in GDAL, GIS | Tagged: , , , , , | Leave a Comment »

DEM Processing with GDAL Utilities

Posted by smathermather on January 2, 2010

I hadn’t seen that PerryGeo’s work has been rolled into GDAL:

http://www.gdal.org/gdaldem.html

Nice work folks.  I especially like the addition of topological position index.  I’ll be playing with that… .

Posted in GDAL, GIS | 1 Comment »

Hillshade (cont.)

Posted by smathermather on December 23, 2009

In general, I like the results from the hillshade.  One problem that I forgot to take into account– hillshade algorithms typically leave a 1-pixel border along the edge where calculations cannot take place… .  For this Ohio dataset, that means that every 5000 feet or so (the x and y dimension of our DEM chunks), we have a 2-pixel wide line, resulting in a subtle and interesting grid.  Now this is a state plane grid, so I might be able to pretend it’s intentional… .  Maybe I need to mosaic the DEM and then compute hillshade.

Posted in GDAL, GIS | Tagged: , , , | 2 Comments »

Hillshade using PerryGeo’s ported GRASS utility

Posted by smathermather on December 22, 2009

Using the same LiDAR DEM from which we generated the contours, we can create hillshade tifs using  http://www.perrygeo.net/wordpress/?p=7.  It compiles easily on a mac, probably even easier on a Linux machine following his directions.  Then another simple loop:

#!/bin/bash
x=0
for f in $( ls *.txt); do
 x=`expr $x + 1`
 echo $x $f
 hillshade $f $f.tif
done

I’d like this all in a single tif:

gdal_merge.py *.tif

or in my case with such a large area:

gdal_merge.py -co "BIGTIFF=YES" *.tif

Then overviews:

gdaladdo -r average out.tif 2 4 8 16 32 64 128 256 512 1024

Posted in GDAL, GIS | Tagged: , , , | Leave a Comment »

GDAL Contours (cont.)

Posted by smathermather on December 21, 2009

Well, I made some mistakes in the last post, not the least of which is I used the wrong flag for creating an attribute field with elevation.  What follows is a little more sophisticated.  It takes us from a series of DEM tiles from which I want 2-foot and 5-foot contours (using gdal_contour), and then dumps those shapefiles into PostgreSQL using shp2pgsql.

First we prep the database.  Again, I used pre-built binaries from Kyng Chaos.

I created a database called “Vinton” using my favorite PostgreSQL management tool, PGAdminIII (actually, it’s the only one I’ve ever tried…).

So, let’s make Vinton spatially aware:

sudo su - postgres -c '/usr/local/pgsql/bin/createlang plpgsql Vinton'
sudo su - postgres -c '/usr/local/pgsql/bin/psql -d Vinton -f /usr/local/pgsql/share/contrib/postgis.sql'
sudo su - postgres -c '/usr/local/pgsql/bin/psql -d Vinton -f /usr/local/pgsql/share/contrib/spatial_ref_sys.sql'

Vinton county is in Ohio State Plane South.  Our dataset is a foot version of the State Plane South HARN.  Looking it up on SpatialReference.org, I get a description not unlike the PostGIS insert statement below, but I changed the EPSG number from 102323 to 1023236 (arbitrarily), and changed the units to feet (from the default for HARN, which is meters):

UNIT["Foot_US",0.30480060960121924]

so that the full insert is:

INSERT into spatial_ref_sys (srid, auth_name, auth_srid, proj4text, srtext) values ( 91023236, 'esri', 1023236, '+proj=lcc +lat_1=38.73333333333333 +lat_2=40.03333333333333 +lat_0=38 +lon_0=-82.5 +x_0=600000 +y_0=0 +ellps=GRS80 +units=m +no_defs ', 'PROJCS["NAD_1983_HARN_StatePlane_Ohio_South_FIPS_3402",GEOGCS["GCS_North_American_1983_HARN",DATUM["NAD83_High_Accuracy_Regional_Network",SPHEROID["GRS_1980",6378137,298.257222101]],PRIMEM["Greenwich",0],UNIT["Degree",0.017453292519943295]],PROJECTION["Lambert_Conformal_Conic_2SP"],PARAMETER["False_Easting",600000],PARAMETER["False_Northing",0],PARAMETER["Central_Meridian",-82.5],PARAMETER["Standard_Parallel_1",38.73333333333333],PARAMETER["Standard_Parallel_2",40.03333333333333],PARAMETER["Latitude_Of_Origin",38],UNIT["Foot_US",0.30480060960121924],AUTHORITY["EPSG","1023236"]]');

Now let’s create some tables to house our future contour data:

CREATE TABLE contours_2
(
 gid serial NOT NULL,
 id integer,
 elev numeric,
 the_geom geometry,
 CONSTRAINT contours_2_pkey PRIMARY KEY (gid),
 CONSTRAINT enforce_dims_the_geom CHECK (st_ndims(the_geom) = 2),
 CONSTRAINT enforce_geotype_the_geom CHECK (geometrytype(the_geom) = 'MULTILINESTRING'::text OR the_geom IS NULL),
 CONSTRAINT enforce_srid_the_geom CHECK (st_srid(the_geom) = 91023236)
)
WITH (OIDS=FALSE);
ALTER TABLE contours_2 OWNER TO postgres;

CREATE TABLE contours_5
(
 gid serial NOT NULL,
 id integer,
 elev numeric,
 the_geom geometry,
 CONSTRAINT contours_5_pkey PRIMARY KEY (gid),
 CONSTRAINT enforce_dims_the_geom CHECK (st_ndims(the_geom) = 2),
 CONSTRAINT enforce_geotype_the_geom CHECK (geometrytype(the_geom) = 'MULTILINESTRING'::text OR the_geom IS NULL),
 CONSTRAINT enforce_srid_the_geom CHECK (st_srid(the_geom) = 91023236)
)
WITH (OIDS=FALSE);
ALTER TABLE contours_5 OWNER TO postgres;

And finally, we can loop through all of our files, and dump the results first to a shapefile, then pipe that from shp2pgsql directly into the database:

#!/bin/bash
x=0
for f in $( ls *.txt); do
x=`expr $x + 1`
echo $x $f
echo Two Foot:
gdal_contour -i 2 -a elev $f $f.shp
/usr/local/pgsql/bin/shp2pgsql -s 91023236 -a $f.shp contours_2 | /usr/local/pgsql/bin/psql -U postgres -d Vinton
echo Five Foot:
gdal_contour -i 5 -a elev $f 5foot/$f.shp
/usr/local/pgsql/bin/shp2pgsql -s 91023236 -a 5foot/$f.shp contours_5 | /usr/local/pgsql/bin/psql -U postgres -d Vinton
done

Now, I’ll probably convert all those numeric contour fields to integer with a cast statement, but tonight, I let the little laptop do the work… .

Posted in Database, GDAL, GIS, PostGIS, SQL | Tagged: , , , , , , , , | Leave a Comment »

 
Follow

Get every new post delivered to your Inbox.

Join 198 other followers