At home, I work on a Mac, so when I want to do work here, my scripting language changes.  In this case, I prefer BASH as my control script.  I downloaded libLAS and did the standard Unix ./configure, make, sudo make install dance in the decompressed libLAS directory

./configure
make
sudo make install

Now I have some tools to manipulate LiDAR las files at home.  To convert all my las files to text:

#!/bin/bash
for f in $( ls *.las); do
 las2txt --parse xyzinrcpM --header pound $f $f.txt
done

There are more elegant ways to implement this, but it gets the job done… .  Now I have a few hundred text files with all the relevant LiDAR info in them for manipulation.  Since working with the whole enchilada in PostGIS failed, I think I’ll approach it by loading one section at a time, analyzing, and removing those data from the table.  More to come… .

Ok, so the INSERT statements were too numerous for inputing the LiDAR point dataset (about a billion points… .)  They kept crashing the postgres daemon.  So, I used copy from a CSV file:

COPY base.cuy_lidar_all FROM 'c:/path/cuy_lidar_ground_veg.insert' WITH CSV

Keep your fingers crossed… .

A unique constraint implemented as a trigger checking hashed geometry seems like a good idea, that is until I applied it to a multi-10GB dataset.  Not surprisingly, it starts off fast on inserts, and slows down a lot as time goes on.  So, I thought I’d approach duplicates another way, by deleting them once they exist.  So for my table:

base.cuy_contours_2

I have hashed my geometry, and added an index:

UPDATE base.cuy_contours_2 SET hash = MD5(ST_AsBinary(the_geom));
CREATE INDEX cuy_contours_2_hash_idx ON base.cuy_contours_2(hash);

I will follow by deleting the duplicate geometries by checking the hashed geometry:

DELETE
 FROM base.cuy_contours_2
 WHERE gid NOT IN
 (SELECT        MAX(dup.gid)
 FROM        base.cuy_contours_2 as dup
 GROUP BY    dup.hash
 );

But, after a very long time, I get an error:

ERROR:  could not read block 356366 of relation 1663/17185/12118368: Permission
denied

So, it’s a  down week, and most of my maps are cached, so we’ll take the postgres service down, and run it in single user mode:

postgres --single -D c:\postgre_data CM

And rerun the command on one line, per single user mode, with no semicolon:

DELETE FROM base.cuy_contours_2 WHERE gid NOT IN (SELECT MAX(dup.gid) FROM base.cuy_contours_2 as dup GROUP BY dup.hash );

and hope for the best… .

Oh and thanks to :

http://www.postgresonline.com/journal/index.php?/archives/22-Deleting-Duplicate-Records-in-a-Table.html

For help with this one.

Below is my metacode for iterating through a grid of images to render the vegetation surface model from povray.  The real code will be implemented in BASH and AWK, although this would be a perfect use of Python, if I knew how to use it… .

PostGIS Tables:

cuy_veg_points:        Load veg points into database and make 2D geometry
cuy_terrain_points:        Load terrain points into database and make 2D geometry
cuy_veg_height:        Use nearest neighbor to calculate tree height into veg point database
e.g. http://www.bostongis.com/?content_name=postgis_nearest_neighbor_generic

cuy_tile_centroids:        Generate tile centroids and load into table– contains coordinates and record area name

Process:

Read cuy_tile_centroids
For each centroid record,
     query view extent for cuy_veg_height
     output temporary tree_coords.inc
     write worldfile for image output from povray
     write povray file with camera position at centroid
     render povray file

Early on, I was working on the problem of rendering a virtual forest based on real LiDAR data in order to do things like a detailed, vegetation included, viewshed.  There are other things we can do with this data, including discovering the boundary of the canopy (and thus find canopy gaps) and rendering a vegetation surface model.

Let’s start with a single tree.  If we have a tree, height x, and we have a tree model, height 1, then we can place a tree in our scene height 1x.  If we want to use Povray to render a surface model, we can use a linear pattern applied to our tree to render what parts of the tree are closer and which are further.

#include "transforms.inc"

#declare Camera_Location = <0,80,0>;
#declare Camera_Lookat   = <0,0,0>;
#include "tree1.inc"

camera {orthographic
 location  Camera_Location
 look_at Camera_Lookat
 right 100
 up 100}

// Unioned box is used to normalize the scaling of the pigment
union {
 object { TREE
 scale 80
 }
 box {-1000,1000}
 pigment {
 gradient x color_map {
 [0 color rgb 1]
 [1 color rgb 0]
 }
 scale <vlength(Camera_Location-Camera_Lookat),1,1>
 Reorient_Trans(x, Camera_Lookat-Camera_Location)
 translate Camera_Location
 }
 finish {ambient 1}
}

My tree is the default tree from POV-Tree:  http://web.archive.org/web/20071101052625/propro.ru/go/Wshop/povtree/download.html.

Here’s my ini file:

All_Console=Off
All_File="false"
Antialias=Off
Bits_Per_Color=16
Bounding=On
Bounding_Threshold=10
Continue_Trace=Off
Create_Histogram=Off
Debug_Console=On
Debug_File="false"
Display=On
Draw_Vistas=On
End_Column=600
End_Row=600
Fatal_Console=On
Fatal_File="false"
Height=600
Jitter=Off
Light_Buffer=On
Output_Alpha=Off
Output_File_Type=n
Output_To_File=On
Quality=0
Remove_Bounds=On
Render_Console=On
Render_File="false"
Split_Unions=On
Start_Column=1
Start_Row=1
Statistic_Console=On
Statistic_File="false"
Verbose=On
Vista_Buffer=On
Warning_Console=On
Warning_File="false"
Width=600

You’ll notice I’m outputing as a 16-bit per channel PNG, so I can capture the full dynamic range of values without loosing too much of the precision in the distance values.  Now, just a little algebra, and we have distance from camera calculated, but I’ll leave that to a future post.

Here’s where I’m going– imagine a PostGIS database full of LiDAR vegetation points and ground points.  A nearest neighbor search and a little arithmetic tells us the local height of the vegetation point.  Now we iterate through a grid extracting sections of this dataset, using a simple query and a little AWK scripting to output the include file, the povray file, and render the image of tree height.  The tree height interpolator is then a simple 3D tree shaped stamp that we scale, rotate, and stamp all across our landscape, thus creating a PovRay generated digital surface model of vegetation.

Edit:

Oh, and I must credit the source for my height mapping:

http://news.povray.org/povray.advanced-users/thread/%3Cweb.49c79d64a563150ecb94416e0@news.povray.org%3E/

For a variety of reasons, I want to load my whole LiDAR vegetation dataset into PostGIS.  I have it in raw, unclassified LAS files, and broken into classified, space delimited text files as well.  The unclassified LAS is nice, if I ever want to hone the data, but for now, we’ll assume the classified data are correctly classified by someone smarter than I am (a likely scenario).

First I wanted to concatenate the text files (here in Windows Command):

copy *.xyz cuy_lidar_veg.txt

Now, I really wanted them in CSV form to make my insert statements easy, so I cheated and used lastools to do the job, assuming it would be more efficient than a stream based script, like awk or sed:

txt2las -i cuy_lidar_veg.txt -o cuy_lidar_veg.las -parse xyz
las2txt -i cuy_lidar_veg.las -o cuy_lidar_veg.csv -sep comma-parse xyz

converting from this:

2114483.860 612559.340 794.720
2114470.510 612558.420 794.420
2114449.970 612565.010 793.900
2114451.540 612569.640 793.800
2114481.010 612571.740 794.390
2114473.230 612572.950 794.130 ...

to this:

2114483.86,612559.34,794.72
2114470.51,612558.42,794.42
2114449.97,612565.01,793.9
2114451.54,612569.64,793.8
2114481.01,612571.74,794.39
2114473.23,612572.95,794.13 ...

I need a table to accept the data so in psql:

CREATE TABLE base.cuy_veg_lidar(
 x numeric,
 y numeric,
 z numeric
);

Now to use awk to format a series of insert statements and pass them through psql right into the database:

less cuy_lidar_veg.csv | awk '{ print "INSERT INTO base.cuy_veg_lidar (x,y,z) VALUES (" $0 ");" };' | psql -U postgres -d CM

This creates a series of command which look like this:

INSERT INTO base.cuy_veg_lidar (x,y,z) VALUES (2114120.67,612570.62,791.11);

Oh, and this last step I ran in CYGWIN command prompt so I could easily access my favorite unix utilities… .  I’m still a nube in the Windows world.  I need my crutches… .

Why so much work when I could just use Postgres’ COPY command?  Permissions, permissions, permissions… .  I like my database to have very little file access ability… .

Next we add a geometry column for 3 dimensional points:

SELECT AddGeometryColumn('base','cuy_veg_lidar','the_geom',9102722,'POINT', 3);

And populate that with our XYZ information:

update base.cuy_veg_lidar set the_geom = ST_SetSRID(ST_MakePoint(x,y,z),102722);

An actual example of using the hashed geometry to uniquely constrain geometry (nothing really new here, just an example of using what we learned in the last post with a new problem):

I’ve got a contours dataset from the county where I work that is in 20 different shapefiles ranging in size from 500MB to 2GB.  I want to put them into a PostGIS database in a single table, so that I can use PostGIS to slice and dice them into reasonable chunks for use by our engineers in AutoDesk products, by ArcGIS users in several divisions, and to improve spatial indexing when dump them into one table in our spatial database as well (ala Regina Obe/Boston GIS’s Map Dicing).  This is a really detailed LiDAR-based contour dataset which used breaklines to make a really detailed and accurate product– which sadly is unusable in its current form because it takes too long to load any given section.

But, today, no slicing and dicing just yet.  I loaded all the data and discovered duplicate geometries.  It seems that some of these blocks that the data come in have overlaps.  I don’t have a screen shot of it, but when I viewed the data in uDig, you can see the duplicate geometries because the transparency of the contour lines in the duplicated areas is less transparent, i.e. the lines are darker against a white background.  I could search for and remove duplicates, but that seems heavy handed.  I’m going to recreate the table with a unique constraint applied to the hashed geometry (and hope for no hash collisions).

First we create the table:

CREATE TABLE base.cuy_contours_2
(
 gid serial NOT NULL,
 elevation smallint,
 shape_leng real,
 the_geom geometry,
 hash character(32),
 CONSTRAINT cuy_contours_2_pkey PRIMARY KEY (gid),
 CONSTRAINT geom_uniq UNIQUE (hash),
 CONSTRAINT enforce_dims_the_geom CHECK (ndims(the_geom) = 4),
 CONSTRAINT enforce_geotype_the_geom CHECK (geometrytype(the_geom) = 'LINESTRING'::text OR the_geom IS NULL),
 CONSTRAINT enforce_srid_the_geom CHECK (srid(the_geom) = 9102722)
)
WITH (OIDS=FALSE);
ALTER TABLE base.cuy_contours_2 OWNER TO postgres;

CREATE INDEX cuy_contours_2_the_geom_idx
 ON base.cuy_contours_2
 USING gist
 (the_geom);

Now we create our trigger that creates a hashed version of the geometry, “the_geom”, in a column called “hash”.

DROP TRIGGER hasher on base.cuy_contours_2;
CREATE OR REPLACE FUNCTION hash_geom() RETURNS TRIGGER AS $hasher$
BEGIN
 IF(TG_OP='INSERT') THEN

 UPDATE base.cuy_contours_2 SET hash = MD5(ST_AsBinary(the_geom));

 END IF;
 RETURN NEW;
END;
$hasher$ LANGUAGE plpgsql;

CREATE TRIGGER hasher AFTER INSERT ON base.cuy_contours_2 FOR EACH STATEMENT EXECUTE PROCEDURE hash_geom();

And finally we add a unique constraint:

ALTER TABLE base.cuy_contours_2 ADD CONSTRAINT geom_uniq UNIQUE (hash);

Now, sadly, I have to use windows where I work, so the following is Windows Command language to load all my shapefiles into the PostGIS database.  That said, I think the command, in this rare case, is more elegant than similar code in, say, BASH scripting:

for %f in (*.shp) do shp2pgsql -s 102722 -a -S -N skip %f base.cuy_contours_2 | psql -h bobs_server -d CM -U postgres

using the shp2pgsql -a flag for append, -S to enforce simple geometries, -N skip to ensure that we keep all records with non-null geometries to compensate for potential errors in the input dataset).

In an earlier post, I griped about the limits to Postgres B-tree indexing, in that for large PostGIS geometries, I couldn’t create a unique constraint for PostGIS geometry.  Abe Gillespie (blame me not him for any mistakes  in what follows), suggested that I hash the geometry and create an index on that.  In a follow-up e-mail, he suggested a series of tests in a trigger statement, first test on the bbox, then on the hashed geometry, then finally on the geometry itself upon insert, to determine if the geometry is unique or not.

Well, I’ve completed the first suggestion, but not yet the second.  Here’s my trigger statement:

DROP TRIGGER dissolver on public.reservation_boundaries_public_private_cm;
CREATE OR REPLACE FUNCTION dissolve_bounds() RETURNS TRIGGER AS $dissolver$
BEGIN
   IF(TG_OP='INSERT') THEN

     UPDATE reservation_boundaries_public_private_cm SET hash = MD5(ST_AsBinary(the_geom));

   END IF;
 RETURN NEW;
END;
$dissolver$ LANGUAGE plpgsql;

CREATE TRIGGER dissolver BEFORE AFTER INSERT ON public.reservation_boundaries_public_private_cm FOR EACH STATEMENT EXECUTE PROCEDURE dissolve_bounds();

So, each time I insert a new geometry, the field “hash” is updated with an MD5 hashed version of the geometry.  Now we want to test against this so we add a constraint:

ALTER TABLE public.reservation_boundaries_public_private_cm ADD CONSTRAINT geom_uniq UNIQUE (hash);

And it works a charm. The only problem is that if I were to get a hash collision, I would reject a record and not mean to. I’m not sure the statistical chances of this, but implementing the above flow chart would prevent such a possibility.  If my reading is right, the chances of a hash collision is nearly 1 in 2^128, so maybe I won’t worry this year… .

Yup, I’m going for hits with the above title.  So, I’ve been playing with triggers in PostGIS (PostgreSQL) trying to streamline a process that I go through quarterly.  We have a geographic boundary layer for our public entity:

The green area is the area with public access.  The purple without.  I want to maintain a version of the boundary with this very simple distinction, and another version of it in which the boundary is dissolved.  Triggers seem ideal here.  When I update the first, I get a version of the second (see below)

The command to do this is as follows:

CREATE TABLE public.reservation_boundaries_public_private_cm_dissolved
 AS
 SELECT res, ST_Union(the_geom) AS the_geom
  FROM public.reservation_boundaries_public_private_cm
   GROUP BY res;

So far, so good (sorry about the awful table names).  Now it needs to run each time there is an insert on the original table.  Here is what I came up with:

DROP TRIGGER dissolver ON public.reservation_boundaries_public_private_cm;

CREATE OR REPLACE FUNCTION dissolve_bounds() RETURNS TRIGGER AS $dissolver$

BEGIN
 IF(TG_OP='INSERT') THEN

 DROP TABLE IF EXISTS public.reservation_boundaries_public_private_cm_dissolved;

 CREATE TABLE public.reservation_boundaries_public_private_cm_dissolved
 AS
 SELECT res, ST_Union(the_geom)
   AS the_geom
   FROM public.reservation_boundaries_public_private_cm
   GROUP BY res;

 END IF;
 RETURN NEW;
END;

$dissolver$ LANGUAGE plpgsql;

CREATE TRIGGER dissolver
  AFTER INSERT ON public.reservation_boundaries_public_private_cm
  FOR EACH STATEMENT EXECUTE PROCEDURE dissolve_bounds();

My only complaint here is speed– it takes about 45 seconds (instead of close to 0.3 seconds) now to insert all my records.  I hoped that I could speed it up with the addition of

FOR EACH STATEMENT EXECUTE PROCEDURE

instead of

FOR EACH RECORD EXECUTE PROCEDURE

but the difference in speed is undetectable.  Any suggestions on how to speed this puppy up?

Not much to say here. I had a bit of inspiration, remembering a Christoph Hormann tutorial http://www.imagico.de/pov/icons.html on making transparent icons in povray and realized that what he was describing was the final piece in the povray for viewsheds milleau: no_image option for an object forces it to interact with the scene in every way, but to be seen by the camera. I’ll post the revised code coming up, but it’s pretty simple. Add the following option inside the code to the tree object:
no_image .

http://www.povray.org/documentation/view/3.6.1/328/