PDOK Geocoder Service

The new geocoding service (Locatieserver)

The Dutch National SDI (Spatial Data Infrastructure) PDOK has recently introduced a new geocoding service (Locatieserver) that will in due time replace the current one.
The new service is very well documented, based on open source stack, available to all and the PDOK community is available for information about the service and eventual feedback. Naturally most of the information available is in Dutch.
For these same reasons I am not going to go into details about the services.

FME and the Locatieserver

Essentially  the geocoding service is an API and communicating with a API's is well demonstrated in the Safe Software Blog.

There are however a few issues that needs some mentioning, especially if you can't read Dutch ;)

• The geocoding service is based on the BAG, the national building and addresses registration system.
• The geocoding service enables you to choose from two different types of services. The free service, which is comparable to the current geocoding service and the suggest & lookup service that is composed of two endpoints that essentially work together.
• The geocoding services offer a wide range of options that are not available in the current geocoding service.

Putting it all together

Back in 2013 when I started writing about FME and share examples on how to use it, there was no real easy way how to share your workspaces, beside the making them available for download of mailing them.

Luckily nowadays the FME Hub is incorporated into the Workbench application, which makes it very easy to grab a transformer created by somebody else.

For demonstration purposes I have actually decided on sharing a template and not a transformer.
My intention is to demonstrate how you can make a simple start with this service and FME especially when taking in consideration all the options this services provides.

PDOKLocatieserver template

The template

A FME template is a great way of sharing workspaces because you can incorporate the source data in it. In this specific case the source is an online CSV file that contains addresses of educational facilities.
As simple validation and clearing some duplicate data is done and then the service type selection is done via a Tester transformer and user parameter.
The geocoding services return JSON by default, but XML is also available for the XML savvies.

Both types of services return multiple results and each result as a score. The highest the score value the highest the probability that it's the best match.

Once the result with the highest score is sorted per request it a simple matter to transform it into a point feature with the GeometryReplacer transfomer.

I have selected to use the geometrie_rd attribute to which contains the coordinate values in the Dutch national coordinate system, if you are more internationally oriented the service also returns a set of coordinates in WGS84.

The results are written into a SQLlite database, a format that is supported by most of the open source and propriety GIS applications.

What to try it out yourself? first of all download FME 2017 (still in Beta while writing this post) and check out FME Hub for the template.

Have Fun!

    

BGT via FMECloud

FME Cloud

What is FME Cloud? Simply put : FME Cloud FME Server in the Cloud. FME Cloud is the first location-aware iPaaS ( Integration Platform as a Service) and that means you have all the FME Server functionality at your disposal, without the hardware . 

See the following link for more information on FME Cloud.

One of the many advantages of FME Cloud is the ability to activate processes via an external trigger (e.i. event driven ) . Examples of such an external trigger can be an e-mail or a sensor value change

BGT

The creation of the BGT ( or The Registration Large Scale Topography as Goggle translate calls it ) is in full swing , many of the stake holders are feverishly working on the assembly of this nation scale polygon map. The first stage is expected to be completed on January 2016. Stake holders deliver their part of the BGT to the national facility (LV) and it is from there made ​​available in CityGML format via PDOK (Dutch national SDI). For an overview of the already available BGT in the LV see: BGTWeb 

The CityGML format cannot be directly used in GIS applications or loaded into a database,  a format conversion (something FME is great at ) must first take place.

Event driven BGT2FGDB conversion.

FME is the tool for spatial data conversion and FME Cloud is the cloud tool for spatial data conversion :)

The trigger

The format conversion is carried out with FME Cloud and is started via a trigger, in this case the trigger is an e-mail . There are some conditions that the trigger has to meet in order to start the process, for example a selected map tile link should be provided.

The conversion

During the format conversion a number of format -specific adjustments are done. Examples of these changes include: removing specific format invalid characters , conversion to uppercase , null values ​​mapping, date conversions, new geometry is created (think of house numbering and street names labels)

The result.

The result is a uniform and complete BGT data set in FileGeodatabase ( FGDB ) format. The FGDB is a widely used and efficient GIS format . This format can be used in both closed- and open-source GIS applications. After performing the format conversion , the result is made ​​available via an e-mail with a download  link.

Advantages and properties of the conversion are : 

• Easy to use. 
• No application is required. 
• Online and always available. 
• Reliably and quickly. 
• Outcome ready for database read-in action and GIS applications. 
• Unique : There is no comparable data distribution method ( in the Netherlands)

Results displayed in QGIS

This product is designed for the Dutch market if you are interested and want to set up something similar please contact us via info@etlsolution.nl

BGT 2 GBKN

The BGT.

BGT in PDOK.

BGT - CityGML format.

 

Step 1: Polygons 2 lines.

Converting polygons to lines is a no-brainer for a seasoned FME user, but you do need some tricks up your sleeve to successfully accomplish that for all of the polygon objects.

Result DWG.

This demo is based on a solution already used by the municipality of Gorinchem.
The solution created allows the user to transform the polygon map into a DWG defined by a template file.

The solution also provides the user the ability to reorder the generated lines, set new visualization rules and decide with layers should be included in the output.

If you don't believe me, just ask Hans......

Small tip for reading the BGT in FME, use the CityGML reader with the imgeo xsd provided by Geonovum or simply download this workspace and follow the instructions.

Georeferencing evaluation with FME.

FME and data evaluation.

FME is a great tool to validate and evaluate data (next to the many things you can do with FME)
There are plenty of resources available on the subject demonstrating FME's data validation and Q&A capabilities.

Data evaluation can involve different aspects and have many forms.
For this post I choose to evaluate how well a publicly available data set can be georeferenced (if you can add value to it and put it on a map, why shouldn't you...)
For any serious conclusions, you'll have to work it out yourself, since my main intention is to demonstrate FME capabilities (and not bad mouth anybody particularly...)

Data source.

The Dutch government publishes many data sets openly and the numbers are increasing all the time.
I choose to use the data set of the national education registry since it is highly dynamic and it contains addresses, which makes it possible to potentially georeference the features.

The data used is available in csv format, which can easily be accessed online via the CSV reader (just point it to the url). For limiting sorting and filtering the incoming data, see my previous post: Where clause on text.
This results in a continuously updated data source, which is great to have but poses a challenge when displaying the results.

 Georeferencing the data.

For georeferencing the source data I am using the BAG Geocoding service, available via the National SDI.

An easy way in FME to access the service is by a HTTPFetcher transformer.

By constructing the URL in the transformer's text editor and making use of attributes values, a very flexible solution is created.

BAG Geocoding service results.

The BAG Geocoding service returns the location(s) in an XML snippet that translates into geometry and attributes. In case of ambiguity or lack of sufficient input, the service returns an aggregate geometry.

Somewhere in the aggregate geometry the corresponding location and attributes can be found (well most of the times...)

Using the total count of both georeferenced and failed features, simple statistics (percentage of correct georeferencing) can be gathered and used for display.

HTTPFetcher

Interpreting the results.

Some of the 'failed' to georeference features do actually exist (BAG Web) and can be correctly geocoded by slightly changing the address used, see for example georeferenced (note the street tag) and not georeferenced (note URL used =  input address)

Displaying the results.

I am using Google Fusion Tables to display  the results since it is an easy way to share geographical information (article is in dutch)

Also non-spatial data can be shared this way and the failed features are saved into a non-spatial Google Fusion Table. Needless to say FME supports both spatial and non spatial reading and writing of this format.
Some limitations of this format are the number of features supported and that it is still considered an experimental format, something that unfortunately makes it less reliable.
As mentioned before the input source data is updated frequently and in contrast the displayed results are static and present a moment in time.


Map of results, created 1-10-2014.

Findings and (possible) future developments.

A way to keep the displayed results up to date would be to use FME Cloud, something I still have not got around to try. I imagine that using FME Cloud to run this workspace would not require almost any resources or adaptations, since most of the data is on-line.
Some of the finding are:

• Saving the source csv data is necessary due to memory issues (something that is easily done in the HTTPFetcher)
• Another curious issue found is that using the postcode in the request string actually results in less features georeferenced.

Don't forget that by the time you read this post the output might look very different.

All PDOK atom feeds in FME.

Recently I was approached about accessing atom feeds services from the Dutch national SDI (PDOK).

After writing several posts on the subject I realized that having access to all of the atom feeds services in FME, would be a nice challenge and a useful addition.
So I set about figuring out what would be the best way to set it up. I realized that I wanted to have the option to select an individual service or several of them.

How to get all the URL's

Normally the GEORSS reader is used to access a XML feed.
To be able to access a service you would have to select the URL from the web page and paste it into the reader.
This approach works fine, if you access a couple of services, but I would like to have them all selected.
Especially when new services are added or deleted.
Once I have all of them I can choose the ones I would like to use in my workspace.

Well since FME is great at accessing data on the web, Why not use it to retrieve the URL's of all the services?

The PDOK site is built with a certain logic, all of the services are alphabetically ordered.
Using that logic in FME enables you to select the services by accessing the HTML pages displaying the services on the PDOK website.

PDOK web page HTML

The XMLFragmenter can slice through the web page HTML like any other XML document.
Parsing the XML (another great feature of FME) enables the final selection of the URL's.

FME workspace

Result

So after some XML parsing I got a list of all of the current PDOK atom feed services.
You might think, well I can just sit and copy paste the URL's into a text file.....well sure you can! nobody is stopping you :)
But by using FME and it's XML capabilities, you always end up with an updated list.
When services are deleted or new ones introduced, the resulting list will reflect that.
Beside that, once you have the services list available, you can continue your data transformation, something which is not really possible with a text file :)

Atom feeds services list.

Atom feeds in FME

As with anything FME, there are more ways to skin the proverbial cat ;)
One way of using the services list in a workspace, can be by using a startup python script.
Yet another way would be to wrap the workspace into a custom reader, so that it can become part of your regular FME readers.

And finally you can just opt to continue developing the workspace.
There are quite a lot of FME transformers dedicated to web services, two of them are specific for GEORSS feeds.
In this scenario the GeoRSSFeatureReplacer is very useful to extract all the information from the server response into attributes.

Services selection

The services selection should be done at run time to provide extra flexibility.
This is where a TestFilter and user parameter combo comes in handy.
To retrieve the parameter value(s) into my workspace, I use the so adequately named :) > ParameterFetcher
Then it's a simple matter of setting up the correct test to retrieve the selected services.

So to wrap it up:

1. Now I have a new PDOK atom feeds reader in FME, and I know for sure that it will always provide me with an updated list of services. 
2. As you know FME is a no code approach. 
3. With just 8 transformers the workspace is super simple.

Life made easy the FME way, interested? give me a call.

Combining BAG and AHN2 Point cloud data.

Data sources

In previous posts , I demonstrated how the BAG data and AHN2 rasters can be accessed by FME.

What I would like to demonstrate now, is how to fetch the AHN2 point cloud data and combine it with the BAG buildings.
This in effect will show how easy it is to add the elevation values to the BAG buildings.
The additional elevation information makes it possible for example to classify the buildings roof type (flat vs.slanting) and transform the 2D BAG buildings into 3D objects.

BAG

The BAG data was accessed for a small area of interest (AOI), this area contains 2D footprints of buildings.

AHN2 Point Cloud

Much like the AHN2 raster data, the on-line point cloud data can be easily accessed via FME. One of the differences in this workspace is that the FeatureReader transformer is used instead of the RasterReader.

 

Combining the data

For spatially relating features there are a few options, you can go for the 'old fashion' method by clipping the point cloud data for each building or by using the SpatialRelator, but my preferred way is to use the SpatialFilter.
Why? well mainly due to performance issues and the fact that no extra transformers are necessary as is the case with the SpatialRelator.
So after relating the features, the elevation information is in fact added to the buildings. (whether it represents the correct height is another matter, which is not addressed here).

There are lies, damned lies and statistics 

- Mark Twain.

So how to go about adding more than just the elevation information?

Well after spatially relating the point cloud to each building, a number of statistics can be computed with the help of the StatisticsCalculator.

This added information can be used for initial classification purposes, for example buildings with a low range value can be classified to have flat roofs.

The Workspace.

After creating the AOI (Creator) the BAG data is fetched off the Internet in the BAG custom transformer.
For more info on how to do that see this previous post.
The point cloud are, in much the same way, fetched from the web, unfortunately it is not possible to grab only the point cloud features of the AOI. 

Point cloud AHN2 custom transformer.

Once all data is read, combining it is done with the SpatialFilter and the StatisticsCalculator finishes the job by adding additional elevation statistics (don't forget the Group By setting)
Note that I have opted for the summary port of the StatisticsCalculator, since I am no longer interested in the points themselves. (tip for good practice, drop anything you don't need ASP!!)

To be able to share some results I have created a 3D pdf  that contains a building footprint (select it to view elevation statistics), point cloud data and additionally derived features (TIN, contours) (tip: download it, and open with Adobe, the web brouwser cannot display it correctly)
Notice the spikes in the point cloud data and derived features, some of them can be attributed to the roof material, others to windows and lastly to vegetation (trees), how do I know? well here is a hint (switch to satellite and head north)

Fetching all of the AHN2 raster data with FME.

AHN2

Recently the AHN2 dataset was released to the public via the Dutch SDI (PDOK)

This was a perfect opportunity to set FME to work on this newly available resource.

This time the rasters and point cloud data was made available via an Atom feed XML.

This means that you can download the 5 meter GeoTIFF rasters and LAZ point cloud data on a national scale. 

Well I had to try that...just for a starters, to see how long it would take me to build the workspace and download the raster data.

Workspace

Building the workspace took about 5 minutes, the tricky part was to make sure you are grabbing the correct part of the XML, after that it's a walk in the park, letting FME get the data off the Internet. 

The workspace itself (4 transformers) starts with a Creator to jump start the HTTPFeatcher, that grabs the XML document.(AtomFeed)

After finding the correct XML tag and extracting the information (URL) with an XMLFragmenter the string is passed to retrieve the rasters (RasterReader).

Data

The raster data took a while longer to download but that is mainly due to the fact that the rasters need to be fetched off the Internet.

Another aspect that I wanted to test was the ability to save the data, so I made use of the (very much welcomed back!!) capability to save the data directly from the Data Inspector.

The FFS is the native FME format, which supports vectors and rasters. The ability to compress the data was applied to see how compact such a big dataset can be made.

After about 15 minutes the Data Inspector happily reported finishing the job of saving the data, I was pleasantly surprised to find out that only about 6 gigabyte of space was necessary.

Results and development

 After downloading the data, I wanted to make the workspace easy use to use and flexible for any spatial selection (My guess is that not many people are interested in the all dataset, but mainly the data in a specific area of interest)

AHN2 raster data.

So what can be better than combining additional services containing rasters bounding boxes and boundaries.
That way you can select your area of interest be it a municipality border or a selected area of interest.

The Dutch SDI provides the possibility to download data in much the same way, but once you have the data in your FME workbench, you can do much more that just download.

This is actually where it starts to get interesting, the opportunities that the height data provides are numerous.

More about downloading the point cloud data and combining the different SDI services on the next post.

By the way did I mention the the rasters are read from a zip file? Well no because that is so 2013 ;)