All PDOK atom feeds in FME.

Recently I was approached about accessing atom feeds services from the Dutch national SDI (PDOK).

After writing several posts on the subject I realized that having access to all of the atom feeds services in FME, would be a nice challenge and a useful addition.
So I set about figuring out what would be the best way to set it up. I realized that I wanted to have the option to select an individual service or several of them.

How to get all the URL's

Normally the GEORSS reader is used to access a XML feed.
To be able to access a service you would have to select the URL from the web page and paste it into the reader.
This approach works fine, if you access a couple of services, but I would like to have them all selected.
Especially when new services are added or deleted.
Once I have all of them I can choose the ones I would like to use in my workspace.

Well since FME is great at accessing data on the web, Why not use it to retrieve the URL's of all the services?

The PDOK site is built with a certain logic, all of the services are alphabetically ordered.
Using that logic in FME enables you to select the services by accessing the HTML pages displaying the services on the PDOK website.

PDOK web page HTML

The XMLFragmenter can slice through the web page HTML like any other XML document.
Parsing the XML (another great feature of FME) enables the final selection of the URL's.

FME workspace

Result

So after some XML parsing I got a list of all of the current PDOK atom feed services.
You might think, well I can just sit and copy paste the URL's into a text file.....well sure you can! nobody is stopping you :)
But by using FME and it's XML capabilities, you always end up with an updated list.
When services are deleted or new ones introduced, the resulting list will reflect that.
Beside that, once you have the services list available, you can continue your data transformation, something which is not really possible with a text file :)

Atom feeds services list.

Atom feeds in FME

As with anything FME, there are more ways to skin the proverbial cat ;)
One way of using the services list in a workspace, can be by using a startup python script.
Yet another way would be to wrap the workspace into a custom reader, so that it can become part of your regular FME readers.

And finally you can just opt to continue developing the workspace.
There are quite a lot of FME transformers dedicated to web services, two of them are specific for GEORSS feeds.
In this scenario the GeoRSSFeatureReplacer is very useful to extract all the information from the server response into attributes.

Services selection

The services selection should be done at run time to provide extra flexibility.
This is where a TestFilter and user parameter combo comes in handy.
To retrieve the parameter value(s) into my workspace, I use the so adequately named :) > ParameterFetcher
Then it's a simple matter of setting up the correct test to retrieve the selected services.

So to wrap it up:

1. Now I have a new PDOK atom feeds reader in FME, and I know for sure that it will always provide me with an updated list of services. 
2. As you know FME is a no code approach. 
3. With just 8 transformers the workspace is super simple.

Life made easy the FME way, interested? give me a call.

FME and Open Geospatial Consortium (OGC) Sensor Observation Service (SOS)

Sensor Observation Service (SOS)

The Open Geospatial Consortium (OGC) implemented the SOS standard
that basically returns sensor and observation data via XML. (yey!)
3 main functions are supported:

1. GetCapabilities - what server provides
2. DescribeSensor - information about a sensor
3. GetObservation - observation data

Naturally with FME all of these functions can be accessed and the data retrieved to any supported GIS and database format.

FME and web services.

There is already a great webinar displaying how, without a single line of code, a variety of web services are accessed and made with FME. In the webinar the observation data is displayed in the Data Inspector (with a background map) and the table view is used to sort the measurement values.
As with all FME webinars, the workspaces are made available.

Demo workspace

IOOS Demo Workspace.

After viewing the webinar, downloading the workspaces and exploring them, I wanted to expand the SOS demo by retrieving the sensor information.
The demo workspace is well annotated and the use of bookmarks is demonstrated.
By the way, did you notice that once you use bookmarks, you actually have your technical design?

Demo workspace +

 In the developed version of the workspace the upper part is the same as in the demo.
I did , however, change the location of the creator and ParameterFetcher, to be able to distinguish between the service functions.

The added parts are far from complicated, and that is actually the beauty of it, with no code and a minimal number of transformers all data can be easily retrieved.


The sensor information retrieved also contains the sensor location (among other things) to be able to share these results I have created this Google map (with FME, naturally...) that shows the location of the stations.
One of my favorite locations is Sand Island, Midway Islands. Does anybody know what species are all those bird chicks..? 

Google Street View

 

Combining BAG and AHN2 Point cloud data.

Data sources

In previous posts , I demonstrated how the BAG data and AHN2 rasters can be accessed by FME.

What I would like to demonstrate now, is how to fetch the AHN2 point cloud data and combine it with the BAG buildings.
This in effect will show how easy it is to add the elevation values to the BAG buildings.
The additional elevation information makes it possible for example to classify the buildings roof type (flat vs.slanting) and transform the 2D BAG buildings into 3D objects.

BAG

The BAG data was accessed for a small area of interest (AOI), this area contains 2D footprints of buildings.

AHN2 Point Cloud

Much like the AHN2 raster data, the on-line point cloud data can be easily accessed via FME. One of the differences in this workspace is that the FeatureReader transformer is used instead of the RasterReader.

 

Combining the data

For spatially relating features there are a few options, you can go for the 'old fashion' method by clipping the point cloud data for each building or by using the SpatialRelator, but my preferred way is to use the SpatialFilter.
Why? well mainly due to performance issues and the fact that no extra transformers are necessary as is the case with the SpatialRelator.
So after relating the features, the elevation information is in fact added to the buildings. (whether it represents the correct height is another matter, which is not addressed here).

There are lies, damned lies and statistics 

- Mark Twain.

So how to go about adding more than just the elevation information?

Well after spatially relating the point cloud to each building, a number of statistics can be computed with the help of the StatisticsCalculator.

This added information can be used for initial classification purposes, for example buildings with a low range value can be classified to have flat roofs.

The Workspace.

After creating the AOI (Creator) the BAG data is fetched off the Internet in the BAG custom transformer.
For more info on how to do that see this previous post.
The point cloud are, in much the same way, fetched from the web, unfortunately it is not possible to grab only the point cloud features of the AOI. 

Point cloud AHN2 custom transformer.

Once all data is read, combining it is done with the SpatialFilter and the StatisticsCalculator finishes the job by adding additional elevation statistics (don't forget the Group By setting)
Note that I have opted for the summary port of the StatisticsCalculator, since I am no longer interested in the points themselves. (tip for good practice, drop anything you don't need ASP!!)

To be able to share some results I have created a 3D pdf  that contains a building footprint (select it to view elevation statistics), point cloud data and additionally derived features (TIN, contours) (tip: download it, and open with Adobe, the web brouwser cannot display it correctly)
Notice the spikes in the point cloud data and derived features, some of them can be attributed to the roof material, others to windows and lastly to vegetation (trees), how do I know? well here is a hint (switch to satellite and head north)

Fetching all of the AHN2 raster data with FME.

AHN2

Recently the AHN2 dataset was released to the public via the Dutch SDI (PDOK)

This was a perfect opportunity to set FME to work on this newly available resource.

This time the rasters and point cloud data was made available via an Atom feed XML.

This means that you can download the 5 meter GeoTIFF rasters and LAZ point cloud data on a national scale. 

Well I had to try that...just for a starters, to see how long it would take me to build the workspace and download the raster data.

Workspace

Building the workspace took about 5 minutes, the tricky part was to make sure you are grabbing the correct part of the XML, after that it's a walk in the park, letting FME get the data off the Internet. 

The workspace itself (4 transformers) starts with a Creator to jump start the HTTPFeatcher, that grabs the XML document.(AtomFeed)

After finding the correct XML tag and extracting the information (URL) with an XMLFragmenter the string is passed to retrieve the rasters (RasterReader).

Data

The raster data took a while longer to download but that is mainly due to the fact that the rasters need to be fetched off the Internet.

Another aspect that I wanted to test was the ability to save the data, so I made use of the (very much welcomed back!!) capability to save the data directly from the Data Inspector.

The FFS is the native FME format, which supports vectors and rasters. The ability to compress the data was applied to see how compact such a big dataset can be made.

After about 15 minutes the Data Inspector happily reported finishing the job of saving the data, I was pleasantly surprised to find out that only about 6 gigabyte of space was necessary.

Results and development

 After downloading the data, I wanted to make the workspace easy use to use and flexible for any spatial selection (My guess is that not many people are interested in the all dataset, but mainly the data in a specific area of interest)

AHN2 raster data.

So what can be better than combining additional services containing rasters bounding boxes and boundaries.
That way you can select your area of interest be it a municipality border or a selected area of interest.

The Dutch SDI provides the possibility to download data in much the same way, but once you have the data in your FME workbench, you can do much more that just download.

This is actually where it starts to get interesting, the opportunities that the height data provides are numerous.

More about downloading the point cloud data and combining the different SDI services on the next post.

By the way did I mention the the rasters are read from a zip file? Well no because that is so 2013 ;) 

Fetching all BAG WFS features

Fetch boy fetch!

 

 

 

 

 

Origin 

The idea for this post originates from a tweet by a FME user on a blog post in which information is given about accessing the Dutch SDI (PDOK) OGC webservices. For original post (in dutch) see brentjesgeoict.

Curiosity

In the post a number of handy tips are given concerning fetching all features residing in the national registration for addresses and buildings service (BAG)
I must admit that until now I have used OGC services strictly as input data source without actually realizing that there might be a limitation. After getting curious about it I started finding out more about the possibilities of smart usage of OGC services and the ability to make the use easy via FME.

 OGC

I am not going to give you a detailed overview about OGC, for that use the Internet, but I will mention the fact that the OGC standards are unique in the GeoSpatial sector for being widely accepted as created standards (in contrary to the de facto standards e.i. shapefile)

Making it easy

The recommended way to overcome the service restrictions (for this case there is a restriction on the service that results in a maximum of 15000 features returned) is to make use of the count and startIndex parameters of the GetFeature request.(WFS 2.0.0)
To make sure that all of the features are returned, an initial query ("resulttype=hits") returns the total number of features served.
With that a simple calculation can be done that ensures the correct number of requests are always sent.
The use of parameters in the workbench makes it easy to adapt the GetFeature request syntax to any available feature type.
Allowing the area of interest to be selected (municipality, province or national) at runtime and the calculated requests make it all care free and really easy to use.

FME Workbench

Results

To make the results accessible I have created a Google map.
With the help of the filter option simple questions such as: "all the buildings built last year?"* can be directly answered and visualized.






* Results are based on a temporary service, and may not be updated or complete.
Interested in the workspace, give me a call. 

FME 2013 in retrospect.

FME 2014 release.

Now with FME 2014 available, it is for me a time to reflect back on the changes made in the 2013 release.

Thinking back on the adaptations made for FME 2013, especially the ones that I have used the most, I can truly say which are my favorites.

1. Conditional processing FME 2013 SP1.

Original FME Evangelist post

Funny enough my first question on the FME Community was about such conditional processing.
At the time it was only possible to achieve it with multiple transformers, testing and setting/mapping values (or an elaborate sql statement)
With the introduction of conditional processing, the amount of transformers for such processing was reduced to 1 e.i. the AttributeCretor.
The AttributeCretor has been replacing more and more transformers on the canvas, mainly testing and mapping transformers obviously.

2.  Data Inspector FME 2013 SP1.

Right from the start of 2013 I have been using the Data Inspector instead of the Universal Viewer.
Since viewing the data in question, is such an essential part of the process of transformation, it's obvious that a capable tool for the job is needed.

The Data Inspector for me is such a tool, especially since it did not only replaced the Universal Viewer, but outdated it too. The addition of the table view to the Data Inspector, did turn out to be a very useful addition that added a certain maturity to the all concept of data visualization within FME.

The last Universal Viewer remnant, that I was missing, is now also been made available in the new FME 2014 release. Saving from the Inspector is now possible in two ways, saving all the features, or just the ones viewed.

3.  New Excel reader/writer FME 2013 SP2.

Blog post

Several posts have been published in the past year about the new Excel reader and writer. It is truly an major improvement to the old reader/writer, supporting formulas, colors and a wider range of data types (XYZ coordinates to geometry), just to mention a few.

Surprisingly the new reader does not support a where clause, as the old reader did, I am still puzzled about this, especially since Excel is treated as a database. Possibly the underlying technology change is the cause of this reader parameter missing in the new reader.

The handling of multiple schema, especially originating from multiple feature types, is still an unresolved issue. Even then the advantages of the new Excel reader/writer capabilities greatly outweigh these few introduced glitches.



So there you have it, these are my favorite and most used FME 2013 improvements I wanted to share.
Now with that out of the way I can send more time on FME 2014 in future posts exploring the sea of opportunities.

Have a feeling the it will all be about nothing.....

FME 2014 - look and feel

FME 2014 is released and available. I have only been using it for the past week, this post will be about my first impressions of the new release.

My first impression of FME 2014 is the amazing speed in which the new look makes  FME 2013 look so chunky and outdated.
This might not be (and probably isn't) the most important addition in the new release, but its a striking one, right from the start.
Its is very easy to get used to the smoothness introduced with this new FME release.

Again hat's off for the people at Safe!!

A few more apparent improvements\changes are:

• Bookmarks and annotations have had a major touch up which makes them more user friendly and easier to use, supporting links and adjustable text properties.

•  Naturally a new theme is also expected with every release of FME, as announced the lizard came back to vanquished the zipster into the sands of the red planet.

• Another immediately apparent change to me, is the option to save your workspace while the translation is running. This is for me a major user interface improvement, that can save a lot of awkward moments when running unsaved workspaces crash.

• The selection of features on the workbench canvas is another major improvement, don't forget to activate it! (Tools > FME Options >Workbench)

Well thats it for now,  as I continue to use FME 2014 and get better acquainted with all the new additions, I  expect additional posts about FME 2014 additions.