Smathermather's Weblog

Remote Sensing, GIS, Ecology, and Oddball Techniques

Landscape Position using GDAL — PT 3

Posted by smathermather on November 25, 2014

More landscape position pictures — just showing riparianess. See also

http://smathermather.wordpress.com/2014/11/22/landscape-position-using-gdal/

and

http://smathermather.wordpress.com/2014/11/24/landscape-position-using-gdal-pt-2/

valleyz3 valleyz2 valleyz1 valleyz

Map tiles by Stamen Design, under CC BY 3.0. Data by OpenStreetMap, under ODbL

Posted in Other, POV-Ray, Landscape Position, Ecology, GDAL, Analysis | Tagged: , , , , , , , , , , | Leave a Comment »

Landscape Position using GDAL — PT 2

Posted by smathermather on November 24, 2014

Just pretty pics today of estimated riparianess. If you prefer a bit of code, see previous post http://smathermather.wordpress.com/2014/11/22/landscape-position-using-gdal/

valleys_improved valleys_improved_1 valleys_improved_2 valleys_improved_3 valleys_improved_3.1 valleys_improved_4 valleys_improved_5

Posted in Analysis, Ecology, GDAL, Landscape Position, Other, POV-Ray | Tagged: , , , , , , , , , , | 4 Comments »

Landscape Position using GDAL

Posted by smathermather on November 22, 2014

Hat tip again to Seth Fitzsimmons. I’ve been looking for a good, easy to use smoothing algorithm for rasters. Preferably something so easy, I don’t even need to write a little python, and so efficient I can run it on 30GB+ datasets and have it complete before I get distracted again by the next shiny project (a few hours).

Seth’s solution? Downsample to a low resolution using GDAL, then sample back up to a higher resolution in order to smooth the raster. My innovation to his approach? Use Lanczos resampling to keep location static, and get a great smooth model:

Unsmoothed DEM

Unsmoothed DEM

Smoothed DEM

Smoothed DEM

Code to do this in gdal follows. “-tr” sets our resamping resolution, “-r lanczos” sets our resampling algorithm, and the “-co” flags are not strictly necessary, but I’ve got a 30GB dataset, so it helps to chop up the inside of the TIFF in little squares to optimize subsequent processing.

gdalwarp -tr 50 50 -srcnodata "0 -32767" -r lanczos  -co "BLOCKXSIZE=512" -co "BLOCKYSIZE=512" oh_leap_dem.tif oh_leap_dem_50.tif
gdalwarp -tr 10 50 -srcnodata "0 -32767" -r lanczos  -co "BLOCKXSIZE=512" -co "BLOCKYSIZE=512" oh_leap_dem_50.tif oh_leap_dem_10-50.tif

At first this excited me for cartographic reasons. We can use this to simplify contours, and then use simplified contours at different zoom levels for maps:

But, we can also use this for analyses. For example, if we difference these smoothed images with our original digital elevation model, we get a measurement of local elevation difference, the first step in establishing where valleys, ridges, and other land forms are.

# Resample to lower resolution
gdalwarp -tr 328.0523587211646 328.0523587211646 -srcnodata "0 -32767" -r lanczos  -co "BLOCKXSIZE=512" -co "BLOCKYSIZE=512" oh_leap_dem.tif oh_leap_dem_328.tif
# Upsample again to get nicely smoothed data
gdalwarp -tr 3.048293887897243 3.048293887897243 -srcnodata "0 -32767" -r lanczos  -co "BLOCKXSIZE=512" -co "BLOCKYSIZE=512" oh_leap_dem_328.tif oh_leap_dem_3-328.tif
# Merge two datasets together into single image as separate bands to ensure they are the same dimensions
# (gdal_calc, as a wrapper for numpy requires this)
gdal_merge -separate -o oh_leap_dem_3-328_m.tif oh_leap_dem.tif oh_leap_dem_3-328.tif
# And now we'll use gdal_calc to difference our elevation model with the smoothed one to get relative elevation 
gdal_calc -A oh_leap_dem_3-328_m.tif -B oh_leap_dem_3-328_m.tif --A_band=1 --B_band=2 --outfile=oh_leap_dem_lp_328.tif --calc="A-B"

So, if we want a good proxy for riparian zones, we can use a technique like this, instead of buffering our streams and rivers a fixed distance (in this case, I’ve used 4 different distances:

Map of landscape position estimated valleys in Cuyahoga County, Ohio

Map of landscape position estimated valleys in Cuyahoga County, Ohio

Pretty snazzy riparian finder. It seems to also find upland headwater wetlands (most of them historic and drained for Cuyahoga County). I am now running on 4 million acres of Ohio at a 10ft (~3 meter) resolution. It’s that efficient.

Addendum: It also finds escarpment edges, like the Portage Escarpment in the above, so it is a mix of a few landforms. Darn handy nonetheless.

Posted in Analysis, Ecology, GDAL, Landscape Position, Other, POV-Ray | Tagged: , , , , , , , , , , | 2 Comments »

Quick (and likely apocryphal) post on versioning and databases

Posted by smathermather on November 15, 2014

This is a quick blog post about technologies that I don’t know well… so please comment if you know better. GeoGig and dat are great tools for addressing versioning in data, so what’s the difference?

Screen shot of GeoGig website

GeoGig is built on Java and meant for any “simple features” geometry (points, lines, polygons).

It’s strength is that it is built from the ground up to handle geometries well, going beyond CRUD functions to specifically address geospatial problems in versioning. Think of it as git for geospatial data.

There’s a hosted version in pre-release from BoundlessGeo called Versio and meant to be the GitHub for geospatial data. You can run your local version from http://geogig.org/

From the website:

“Users are able to import raw geospatial data (currently from Shapefiles, PostGIS or SpatiaLite) in to (sic) a repository where every change to the data is tracked. These changes can be viewed in a history, reverted to older versions, branched in to sandboxed areas, merged back in, and pushed to remote repositories.”

—————————————————————————
Ok, so how about dat?

Screen shot of dat website

dat is built on javascript, meant to do streaming data and some other cool features and does CRUD versioning. Think of it as git for data built by web people. Therefore, it “defines an API for reading, writing and syncing datasets”, as opposed to a repository into which one would import data.

“Dat is an open source project that provides a streaming interface between every file format and data storage backend.”

A cursory look indicates it will work for geospatial data, but effectively as blobs, with no special handling for changes within features like GeoGig. But, it does what GeoGig does not, and that is to make datasets automatically syncable.

Like all projects, each has its strengths. Choose your project wisely.

Posted in dat, Database, GeoGig, git, Versio | Tagged: , , , , | 9 Comments »

ICCM — International Conference of Crisis Mappers

Posted by smathermather on November 9, 2014

Screen shot of Crisis Mappers landing page

Screen shot of Crisis Mappers landing page

The mark of a great conference is one that not only is well run and orchestrated, interesting from a content perspective, and full of bright minds, but also one that experiments with elements of interactions to maximize the value delivered to attendees. By these measures, the International Conference of Crisis Mappers (ICCM) succeeds. Normally, I struggle with the call to attend sessions vs. the conference hallway conversations which can be of such great value. ICCM provided enough context and structure for enjoying both, plus a number of other participatory structures in which to interact with and learn from other ICCM attendees.

For attending ICCM, I had plans to only hallway chat about http://OpenDroneMap.org, in order to get a measure of the culture and needs, and also of the interest for such a project from digital humanitarians in an effort to get enough interest to begin to garnish users and contributors. Thanks to some last minute slots opening up, and on the kind recommendation of Jen Ziemke, I was able to show OpenDroneMap at the Tech & Analysis Fair at Google at the start of the conference and also do a well attended deep dive session on Saturday. In each, I got great questions and recommendations, did a brainstorming session on use in the deep dive session, and had a thoroughly great time.

More later, but I will end with this — if you like technology, have empathy and a desire to apply empathetic design, development, and sweat equity to humanitarian needs, then get signed up at http://crisismappers.net, become a participant in HOT, join the Standby Task Force, and / or all the other work done by digital humanitarians, and make your way to Boston next year for the next ICCM.

Bravo, all.

Posted in 3D, International Conference of Crisis Mappers, OpenDroneMap | Tagged: , , | Leave a Comment »

Airspace — A deep rabbit hole

Posted by smathermather on October 25, 2014

In previous maps we looked at Class B, C, and D airspace. Let’s add in Class E0 and E5… (not yet in 3D):

(Map tiles by Stamen Design, under CC BY 3.0. Data by OpenStreetMap, under ODbL)

Map showing Class B, C, D, E0, and E5 airspace

Map showing Class B, C, D, E0, and E5 airspace

Previous posts:

http://smathermather.wordpress.com/2014/10/25/airspace-is-complicated-and-so-i-abuse-postgis-once-again/

and

http://smathermather.wordpress.com/2014/10/25/airspace-is-complicated-and-so-i-abuse-postgis-once-again-reprise/

Posted in 3D, Cartography, Database, Drone, PostGIS, PostgreSQL, SQL, UAS | Tagged: , , , , , | Leave a Comment »

Airspace is complicated — and so I abuse PostGIS once again — Reprise…

Posted by smathermather on October 25, 2014

In the previous post: http://smathermather.wordpress.com/2014/10/25/airspace-is-complicated-and-so-i-abuse-postgis-once-again/ we explore the 3D shape and complexity of controlled airspace.

Now here’s the rest of the code. We’ll add our affine transformation ala Seth Fitsimmons:

    SELECT 
        ST_Affine(
            ST_Rotate(geom, -pi() / 2),
            -- isometric
            cos(pi() / 6), -cos(pi() / 6), 0,
            sin(pi() / 6), sin(pi() / 6), 1,
            0, 0, 0,
            0, 0, 0
        )
    AS geom

And integrate that into our original function:

-- Inputs are a geometry and an elevation to move the geometry to
CREATE OR REPLACE FUNCTION threed_iso(footprint geometry, elevation numeric)
  RETURNS geometry AS
$BODY$

-- Force 3D, then translate to the input elevation
WITH floor AS
(
    SELECT ST_Translate( ST_Force3DZ(footprint), 0, 0, elevation ) AS geom
),
-- Now make isometric (and begin Seth Code)
iso AS
(
    SELECT 
        ST_Affine(
            ST_Rotate(geom, -pi() / 2),
            -- isometric
            cos(pi() / 6), -cos(pi() / 6), 0,
            sin(pi() / 6), sin(pi() / 6), 1,
            0, 0, 0,
            0, 0, 0
        )

    AS geom
    FROM floor
)
-- We'll force it back to 3D so QGIS is happy
SELECT ST_Force2D(geom) FROM iso
;
$BODY$
  LANGUAGE sql VOLATILE
  COST 100;

And voila!

DROP TABLE IF EXISTS class_c_isoc;

CREATE TABLE class_c_isoc AS
	SELECT gid, airspace, name, lowalt, highalt, threed_iso(geom, lowalt::numeric * 5) AS geom
	FROM class_c_subset;

Let’s take a look at Washington, DC and surrounds, another nice complicated example:

3D Figure of DC controlled airspace

3D Figure of DC controlled airspace

And again with map tiles by Stamen Design, under CC BY 3.0. Data by OpenStreetMap, under ODbL:

3D Figure of DC controlled airspace

3D Figure of DC controlled airspace

 

Posted in 3D, Cartography, Database, Drone, PostGIS, PostgreSQL, SQL, UAS | Tagged: , , , , , | 1 Comment »

Airspace is complicated — and so I abuse PostGIS once again

Posted by smathermather on October 25, 2014

Let’s ignore for a moment the drone hobbiest / enthusiast. What is the shape of airspace for airplanes and commercial and government unmanned aircraft flying under Certificates of Authorization, and how can we visualize it?

Thanks to Anita in the last post, we have the Class B,C,D,E Airspace Shape Files which helps us define the overall shape of controlled airspace.

Map of Detroit, Cleveland, and Pittsburgh Class B Airspace

Map of Detroit, Cleveland, and Pittsburgh Class B Airspace

But, these are 3D things. I want to visualize them thus. Let us put some constraints on the problem. Let’s do it all in PostGIS, that way we can see it in QGIS or on the web. Let’s not use full PostGIS 3D (i.e. the SFCGAL library), not because it isn’t awesome (it truly is) but because it can be hard to install at the moment (although see https://github.com/vpicavet/docker-pggis for an easy way with docker). Finally, true 3D with unconstrained viewing angles and 100% flexibility is… is… well it usually sucks. So, we’ll stick to isometric viewing (thanks to Seth Fitzsimmons of Stamen http://stamen.com/ for his PostGIS isometric code which will be released upon his word). (Update — all the code is there…):

-- Inputs are a geometry and an elevation to move the geometry to
CREATE OR REPLACE FUNCTION threed_iso(footprint geometry, elevation numeric)
  RETURNS geometry AS
$BODY$

-- Force 3D, then translate to the input elevation
WITH floor AS
(
    SELECT ST_Translate( ST_Force3DZ(footprint), 0, 0, elevation ) AS geom
),
-- Now make isometric (and begin Seth Code)
iso AS
(
    SELECT
        ST_Affine(
            ST_Rotate(geom, -pi() / 2),
            -- isometric
            cos(pi() / 6), -cos(pi() / 6), 0,
            sin(pi() / 6), sin(pi() / 6), 1,
            0, 0, 0,
            0, 0, 0
        )

    AS geom
    FROM floor
)
-- We'll force it back to 3D so QGIS is happy
SELECT ST_Force2D(geom) FROM iso
;
$BODY$
  LANGUAGE sql VOLATILE
  COST 100;

Ok, now let’s rock some geometries with this bad function:

DROP TABLE IF EXISTS class_c_isoc;

CREATE TABLE class_c_isoc AS
	SELECT gid, airspace, name, lowalt, highalt, threed_iso(geom, lowalt::numeric * 5) AS geom
	FROM class_c_subset;

And what do our controlled airspaces look like?

Isometric view of Cleveland controlled airspace

Isometric view of Cleveland controlled airspace

Kind of conical, in this case with some “wings” that serve as approaches. It makes sense, I guess. At the center, where the airport is, controlled airspace goes from the ground up. As we get further from the airport, a set of concentric rings starting at higher elevations provide a controlled space that allows for landing and taking off.

Image of Cleveland airspace with Stamen Toner map as backdrop

Map tiles by Stamen Design, under CC BY 3.0. Data by OpenStreetMap, under ODbL.

There are more complicated ones, naturally. We need look no further than Detroit for additional complexity:

Visualization of Detroit controlled airspace

Visualization of Detroit controlled airspace

airspace_detroit_toner

No wonder every time I fly through Detroit, no matter where I’m going, I end up flying over Lake Erie.

If we want really complicated, we need look no further than Cincinnati:
airspace_cinci

What is going on there? Is that due to shape of the hills, or city boundaries?

Finally, what does airspace look like over Ohio, West Virginia, and Pennsylvania (etc.), overall?
airspace_is_complicated_regional

And while the following map isn’t quite right, here is a figure including many of the small airports sans controlled airspace:

View of all controlled and uncontrolled airspace across Ohio and neighbors.

View of all controlled and uncontrolled airspace across Ohio and neighbors.

May an aeronautical pox be upon you!
The above line was not intended in bad taste, but just an homage to the “red dot fever” of early neo-geography (neo-geography which I’m informed definitionally doesn’t exist). Only a few minutes ago, it dawned on me that the deleted phrase could be misinterpreted these days… .

On a related note, if you want an interesting analysis of ebola, read Zeynep Tufekci’s analysis.

(for the record, all heights are exagerated by 5x, for clarity of reading).

Also, in case it wasn’t obvious: Map tiles by Stamen Design, under CC BY 3.0. Data by OpenStreetMap, under ODbL.

Posted in 3D, Cartography, Database, Drone, PostGIS, PostgreSQL, SQL, UAS | Tagged: , , , , , | 4 Comments »

Someone is wrong on the internet…

Posted by smathermather on October 23, 2014

Ok @Mapbox, I’ve given you enough grace. I pulled the grumpy-old-man thing with Kenneth Field from ESRI a couple of weeks ago, and ended up apologizing. If i only give ESRI a few hours of grace, 3 months of grace for you is more than fair.

Now, to raise my foot up. We’ll see if it ends up in my mouth again.

Ok, so Mapbox and drones. Mapbox and drones. Those folks are excited about drones. They’ve got a toolchain their working on, and integration into their existing services, and lots of posts about drones. Downright giddiness I say:

https://duckduckgo.com/?q=site%3Amapbox.com+drones&t=canonical

And then this:

Really? 3 months ago, you launch “Don’t Fly Here”, create a repo for feedback and improvements to the data, but the biggest problem that you haven’t even tried to fix is the map really under represents the restrictions on where you should fly if you are trying to stay out of controlled air space, and the major update you do is adding temporary flight restrictions. Glitz and glory over getting the fundamentals right.

Here’s the current Mapbox no fly zones:

not_quite_right

But, we are missing a lot of smaller airports that also have controlled airspace. Let’s be considerate to our hobbiest drone friends and put them in with 3-mile buffers and refine our 5-mile buffers on medium and large airports to include the edges of the airport, just to be on the safe side:

a_little_righter

Well, that’s a bit less inviting. But hey, still plenty of places to fly, right? Uh, no. We forgot, according to the Aeronautical Information Manual, in certain busy airspace, there is a Mode C requirement for all craft flying in the airspace. This means you need to have a transponder to fly in this space between ground and 10,000ft above mean sea level. This is a transponder that weighs several times what your drone can lift — in otherwords a “non-starter”, effectively making these zones no-fly zones as well. What does Hopkins look like with it’s 30 nautical mile Mode C requirement?:

even_righter

Now, to be fair, this is murky legal territory at best and I am not a lawyer. How much applies to hobbiests given the 1981 Advisory Circular I can’t say I know. But, if we are to propose a map to clarify where we can and cannot fly as hobbiests, then we should be including as much information (in a simple and easy to use way) as we can. On these grounds, “Don’t Fly Here” fails.

It’s cool though. It’s an open source data project in an open source community. Community contributions and knowledge will fix all mistakes in time, so my pull request to get to the second map will be reviewed and rejected on solid grounds or integrated, right?

https://github.com/mapbox/drone-feedback/pull/40

Three months later, I’m still waiting… . Fix your map please. Engage your repo users. Do this right. Please.

Posted in Drone, UAS | Tagged: , , | 11 Comments »

Government 2.0 is not the platform

Posted by smathermather on October 8, 2014

(In which Steve vehemently disagrees with a major thought leader on the nature of Civic Technology… .)

(TLDR: Open Data is the platform. Governance participates in, contributes to, and benefits from Open Data, but government is not the platform, Open Data is the platform.)

Tower of people standing on each others shoulders

I’m sorry, Tim O’Reilly. You are a brilliant man, have some truly great ideas, and some successes I hear as a publisher and VC, but on Gov 2.0, you are wrong, or at least 90 degrees away from right.

I like Open Data. I hate Government as a Platform. Why? Two reasons. First (selfish reason), being a platform as your default position does not always have the best views.

To be fair, though, it’s deeper than just the views. Public servants are, after all, like all good persons, servants, so we can’t whine too much about platform / infrastructure services. This is part of what we signed up for.

Counterpoint: historically, government is not just about infrastructure (roads, bridges, sewers, data), but also has a role to play in services. Great Open Data is served not just in raw form, but with great APIs, great interfaces, great front ends. I think this is particularly true (or should be) in the Parks and Recreation sector. Domain expertise matters, and we should be leveraging the domain expertise of our hard working public servants where we can.

But, I can’t win this on selfish arguments alone. Perhaps that is our role in the public sector. We do the pedantic, boring work, we provide the data, and clever, brilliant people who are necessarily outside in the private sector do the value added work.

Let us start with the Classic Example — Google and Portland’s Trimet put together the GTFS standard that allows for transit agencies across the US to share their transit information and thus make it available through services like Google Maps. Thus it is that the platform is governments serving GTFS feeds, private industry (Google) does their magic in creating a common mapping interface for people to easily navigate complicated transit systems, and an angel gets it’s wings.

I would not suggest that this arrangement is bad. Is it a commodification of the commons by means of Open Data? Yes? Do we all benefit from it? Ya. Mostly (although use the transit app for Seoul, Korea for truly great transit software). It’s an acceptable trade (ignoring the commodification of all of our private lives in the trade, but that’s a separate issue from Trimet and GTFS). (The trade works in part, too, because Google derives value from being it’s own privatized Commons, but that’s an analysis for another time, and perhaps a PhD or two)

But if this is the only model we put forth as our Open Data model, we are missing some really important elements. Succinctly, it is not Government that is or should be the platform, any more than OpenStreetMap’s public domain inputs are the most valuable. Open Data is the platform. Governance participates in, contributes to, and benefits from Open Data, but within the context of data, government is not the platform, Open Data is the platform.

It is easy to loose sight of this within the context of Open Data as a new Commons. We have come to some terms with Open Source Software as a Commons. We understand, sometimes with large swaths of apocryphal sociological analysis, elements of the value of Commons within the context of Open Source Software. It is now time to understand Open Data with similar breadth and depth.

When I think of Open Data, I do not want Transit! I want transitivity. Transitivity between legal structures (private and public) and between sectors (health, education, technology, human services, etc.). I want Open Data to be a new Commons that we all contribute to from multiple sectors and structures.

Tonight, as we discussed the future of Open Data in Cleveland, I was refreshed by the interest in placing Open Data as that hub, as that Commons, within the context of the verticals and plays that we seek to engage in a “loosely coupled coalition of the willing”. Bravo to Lev Gonick and others at the meeting who articulated those interlacing pieces of the coalition as follows:

#OpenEd
#OpenGov
#OpenHealth
#OpenEcology
#OpenSustainability

And so, it bears repeating: Open Data is the platform. Governance participates in, contributes to, and benefits from Open Data, but government is not the platform, Open Data is the platform. A new Commons. These are exciting times.

 

 

Posted in Other | Tagged: , , | Leave a Comment »

 
Follow

Get every new post delivered to your Inbox.

Join 814 other followers