BGT 2 GBKN

The BGT.

BGT in PDOK.

BGT - CityGML format.

 

Step 1: Polygons 2 lines.

Converting polygons to lines is a no-brainer for a seasoned FME user, but you do need some tricks up your sleeve to successfully accomplish that for all of the polygon objects.

Result DWG.

This demo is based on a solution already used by the municipality of Gorinchem.
The solution created allows the user to transform the polygon map into a DWG defined by a template file.

The solution also provides the user the ability to reorder the generated lines, set new visualization rules and decide with layers should be included in the output.

If you don't believe me, just ask Hans......

Small tip for reading the BGT in FME, use the CityGML reader with the imgeo xsd provided by Geonovum or simply download this workspace and follow the instructions.

Georeferencing evaluation with FME.

FME and data evaluation.

FME is a great tool to validate and evaluate data (next to the many things you can do with FME)
There are plenty of resources available on the subject demonstrating FME's data validation and Q&A capabilities.

Data evaluation can involve different aspects and have many forms.
For this post I choose to evaluate how well a publicly available data set can be georeferenced (if you can add value to it and put it on a map, why shouldn't you...)
For any serious conclusions, you'll have to work it out yourself, since my main intention is to demonstrate FME capabilities (and not bad mouth anybody particularly...)

Data source.

The Dutch government publishes many data sets openly and the numbers are increasing all the time.
I choose to use the data set of the national education registry since it is highly dynamic and it contains addresses, which makes it possible to potentially georeference the features.

The data used is available in csv format, which can easily be accessed online via the CSV reader (just point it to the url). For limiting sorting and filtering the incoming data, see my previous post: Where clause on text.
This results in a continuously updated data source, which is great to have but poses a challenge when displaying the results.

 Georeferencing the data.

For georeferencing the source data I am using the BAG Geocoding service, available via the National SDI.

An easy way in FME to access the service is by a HTTPFetcher transformer.

By constructing the URL in the transformer's text editor and making use of attributes values, a very flexible solution is created.

BAG Geocoding service results.

The BAG Geocoding service returns the location(s) in an XML snippet that translates into geometry and attributes. In case of ambiguity or lack of sufficient input, the service returns an aggregate geometry.

Somewhere in the aggregate geometry the corresponding location and attributes can be found (well most of the times...)

Using the total count of both georeferenced and failed features, simple statistics (percentage of correct georeferencing) can be gathered and used for display.

HTTPFetcher

Interpreting the results.

Some of the 'failed' to georeference features do actually exist (BAG Web) and can be correctly geocoded by slightly changing the address used, see for example georeferenced (note the street tag) and not georeferenced (note URL used =  input address)

Displaying the results.

I am using Google Fusion Tables to display  the results since it is an easy way to share geographical information (article is in dutch)

Also non-spatial data can be shared this way and the failed features are saved into a non-spatial Google Fusion Table. Needless to say FME supports both spatial and non spatial reading and writing of this format.
Some limitations of this format are the number of features supported and that it is still considered an experimental format, something that unfortunately makes it less reliable.
As mentioned before the input source data is updated frequently and in contrast the displayed results are static and present a moment in time.


Map of results, created 1-10-2014.

Findings and (possible) future developments.

A way to keep the displayed results up to date would be to use FME Cloud, something I still have not got around to try. I imagine that using FME Cloud to run this workspace would not require almost any resources or adaptations, since most of the data is on-line.
Some of the finding are:

• Saving the source csv data is necessary due to memory issues (something that is easily done in the HTTPFetcher)
• Another curious issue found is that using the postcode in the request string actually results in less features georeferenced.

Don't forget that by the time you read this post the output might look very different.

All PDOK atom feeds in FME.

Recently I was approached about accessing atom feeds services from the Dutch national SDI (PDOK).

After writing several posts on the subject I realized that having access to all of the atom feeds services in FME, would be a nice challenge and a useful addition.
So I set about figuring out what would be the best way to set it up. I realized that I wanted to have the option to select an individual service or several of them.

How to get all the URL's

Normally the GEORSS reader is used to access a XML feed.
To be able to access a service you would have to select the URL from the web page and paste it into the reader.
This approach works fine, if you access a couple of services, but I would like to have them all selected.
Especially when new services are added or deleted.
Once I have all of them I can choose the ones I would like to use in my workspace.

Well since FME is great at accessing data on the web, Why not use it to retrieve the URL's of all the services?

The PDOK site is built with a certain logic, all of the services are alphabetically ordered.
Using that logic in FME enables you to select the services by accessing the HTML pages displaying the services on the PDOK website.

PDOK web page HTML

The XMLFragmenter can slice through the web page HTML like any other XML document.
Parsing the XML (another great feature of FME) enables the final selection of the URL's.

FME workspace

Result

So after some XML parsing I got a list of all of the current PDOK atom feed services.
You might think, well I can just sit and copy paste the URL's into a text file.....well sure you can! nobody is stopping you :)
But by using FME and it's XML capabilities, you always end up with an updated list.
When services are deleted or new ones introduced, the resulting list will reflect that.
Beside that, once you have the services list available, you can continue your data transformation, something which is not really possible with a text file :)

Atom feeds services list.

Atom feeds in FME

As with anything FME, there are more ways to skin the proverbial cat ;)
One way of using the services list in a workspace, can be by using a startup python script.
Yet another way would be to wrap the workspace into a custom reader, so that it can become part of your regular FME readers.

And finally you can just opt to continue developing the workspace.
There are quite a lot of FME transformers dedicated to web services, two of them are specific for GEORSS feeds.
In this scenario the GeoRSSFeatureReplacer is very useful to extract all the information from the server response into attributes.

Services selection

The services selection should be done at run time to provide extra flexibility.
This is where a TestFilter and user parameter combo comes in handy.
To retrieve the parameter value(s) into my workspace, I use the so adequately named :) > ParameterFetcher
Then it's a simple matter of setting up the correct test to retrieve the selected services.

So to wrap it up:

1. Now I have a new PDOK atom feeds reader in FME, and I know for sure that it will always provide me with an updated list of services. 
2. As you know FME is a no code approach. 
3. With just 8 transformers the workspace is super simple.

Life made easy the FME way, interested? give me a call.

Combining BAG and AHN2 Point cloud data.

Data sources

In previous posts , I demonstrated how the BAG data and AHN2 rasters can be accessed by FME.

What I would like to demonstrate now, is how to fetch the AHN2 point cloud data and combine it with the BAG buildings.
This in effect will show how easy it is to add the elevation values to the BAG buildings.
The additional elevation information makes it possible for example to classify the buildings roof type (flat vs.slanting) and transform the 2D BAG buildings into 3D objects.

BAG

The BAG data was accessed for a small area of interest (AOI), this area contains 2D footprints of buildings.

AHN2 Point Cloud

Much like the AHN2 raster data, the on-line point cloud data can be easily accessed via FME. One of the differences in this workspace is that the FeatureReader transformer is used instead of the RasterReader.

 

Combining the data

For spatially relating features there are a few options, you can go for the 'old fashion' method by clipping the point cloud data for each building or by using the SpatialRelator, but my preferred way is to use the SpatialFilter.
Why? well mainly due to performance issues and the fact that no extra transformers are necessary as is the case with the SpatialRelator.
So after relating the features, the elevation information is in fact added to the buildings. (whether it represents the correct height is another matter, which is not addressed here).

There are lies, damned lies and statistics 

- Mark Twain.

So how to go about adding more than just the elevation information?

Well after spatially relating the point cloud to each building, a number of statistics can be computed with the help of the StatisticsCalculator.

This added information can be used for initial classification purposes, for example buildings with a low range value can be classified to have flat roofs.

The Workspace.

After creating the AOI (Creator) the BAG data is fetched off the Internet in the BAG custom transformer.
For more info on how to do that see this previous post.
The point cloud are, in much the same way, fetched from the web, unfortunately it is not possible to grab only the point cloud features of the AOI. 

Point cloud AHN2 custom transformer.

Once all data is read, combining it is done with the SpatialFilter and the StatisticsCalculator finishes the job by adding additional elevation statistics (don't forget the Group By setting)
Note that I have opted for the summary port of the StatisticsCalculator, since I am no longer interested in the points themselves. (tip for good practice, drop anything you don't need ASP!!)

To be able to share some results I have created a 3D pdf  that contains a building footprint (select it to view elevation statistics), point cloud data and additionally derived features (TIN, contours) (tip: download it, and open with Adobe, the web brouwser cannot display it correctly)
Notice the spikes in the point cloud data and derived features, some of them can be attributed to the roof material, others to windows and lastly to vegetation (trees), how do I know? well here is a hint (switch to satellite and head north)