Comments on: PostGIS Gripe—Limits to Postgre’s B-tree indexing

By: smathermather

smathermather — Fri, 05 Dec 2008 04:29:49 +0000

I had to do some reading in Wikipedia to make sure I understood this response, but I think it will work, just so long as there aren’t hash collisions (a term which I’d never heard before but now understand… shouldn’t confess to such Nube-ness).

So, my interpretation of Abe’s comment is that an alternative approach is to create that unique index on a proxy for the geometry field, without having to index something as long as the geometry field. So the idea is to turn the geometry field into a hash representation to compress it, hope there aren’t any false duplicates when I do inserts later (hash collisions), and index and constrain that. Brilliant approach, and a new tool in my toolbox. So, maybe something like this:

So first binary straight from the database…

ST_AsBinary(the_geom)

then encode to text:

encode()

and then convert this to an MD5 hash

MD5()

for a complete command like this:

MD5(encode(ST_AsBinary(the_geom)))

resulting in something like this:
900150983cd24fb0 d6963f7d28e17f72

Which is much shorter than the original values, and might be index-able.

To be tested ASAP (next week most likely…).

By: Abe

Abe — Fri, 05 Dec 2008 00:18:16 +0000

Why not hash the geometry value and place that into another column, say “the_geom_hash”. Then put a unique constraint on the_geom_hash. Now create an insert trigger that does the hashing and sets the value of the_geom_hash column.

I realize it’s possible to have hash collisions, but they should generally be unique. If your data can tolerate this very slim possibility of error then it should work fine.

By: Regina

Regina — Mon, 01 Dec 2008 20:00:40 +0000

Hmm I suppose that would work. Though may not work for large geometries. Not sure about the floating point case. You could also try this trick I mentioned – which would also use a spatial index so might perform better. Though admittedly I haven’t tried it. Also check out other people’s suggestions.

http://postgis.refractions.net/pipermail/postgis-users/2008-November/021891.html

This is not something I run into often so haven’t really stress tested any of these tricks as far as geometries go.

hope that helps,
Regina

By: smathermather

smathermather — Sun, 30 Nov 2008 23:59:58 +0000

So, ignorant question, why wouldn’t this work (other than my table would be too large to index):
alter table base.contours add constraint unique_geom unique ST_AsEWKB(the_geom);

By: smathermather

smathermather — Sun, 30 Nov 2008 14:31:53 +0000

This is very helpful, thank you. I can’t offhand think of any case where bbox instead of a full comparison would get me in trouble, but it’d always be out there lurking as a possibility, and there’s nothing worse than a subtle lurking problem.

Now a nagging question about floating point values: if the comparison is done on well known binary, do we have to worry about comparison of floating point values? What I mean is, can we be certain that each time a float value is represented, it will be represented identically to the last time, and thus we can detect when two values are equal? How does PostGIS handle this?

By: Regina

Regina — Sat, 29 Nov 2008 05:10:07 +0000

Actually using DISTINCT the_geom would use the same bbox = logic that Vincent mentioned, so that won’t work either.

If you want to go the DISTINCT route, you’ll need to do a
INSERT INTO newtable(the_geom)
SELECT DISTINCT ST_AsEWKB(the_geom)
FROM oldtable

that will force it into binary mode for the distinct thus preventing a bbox = thingy, but on insert into the table it will be cast back to a geometry.

By: Vincent

Vincent — Thu, 27 Nov 2008 18:21:06 +0000

Hi,
Be careful that when you declare a unique constraint on the_geom, you are actually not eliminating exact geometry duplicates, but geometries with the same bounding box.
As a matter of fact, it uses the = operator, which in postgis is defined on geometries as a bbox comparison.
bye,
Vincent