New annotation option in FME 2017

Annotations in FME

Annotations are an easy way to include description in a workspace.

There are 3 types of annotations in FME:

1. Header annotations that are only generated when a workspace is generated.
2. Summary annotations that are dynamic and reflect the changes in a workspace component (transformer, feature type, etc)
3. Custom annotations are generally known as user annotations.

The image below shows all 3 annotation types.

The header annotations are above the reader and writer feature types and connection, the summary annotations is blue and describes the reader feature type, while the user annotation is by default yellow and empty of content.

The Magic of FME 2017

The #FMEWT is about to finish for 2017 and I had the luck to present in it, if you are interested in my presentation here is a link

During the preparation of the presentation I came across an unknown (for me) new functionality in the Workbench application.

This new functionality is the ability to transform ;) a summery annotation into a user annotation.

Just right click on the summary annotation and select 'Convert to Attached Annotation'

If you are a Copy& Paste kinda of a person, as I am, then this is mana from heaven.....

Now I can create a summary annotation and edit it, something that can save a lot of time.

So next to the Parameter Editor this is another excellent addition that can boost your FME development productivity.

Have Fun!

PDOK Geocoder Service

The new geocoding service (Locatieserver)

The Dutch National SDI (Spatial Data Infrastructure) PDOK has recently introduced a new geocoding service (Locatieserver) that will in due time replace the current one.
The new service is very well documented, based on open source stack, available to all and the PDOK community is available for information about the service and eventual feedback. Naturally most of the information available is in Dutch.
For these same reasons I am not going to go into details about the services.

FME and the Locatieserver

Essentially  the geocoding service is an API and communicating with a API's is well demonstrated in the Safe Software Blog.

There are however a few issues that needs some mentioning, especially if you can't read Dutch ;)

• The geocoding service is based on the BAG, the national building and addresses registration system.
• The geocoding service enables you to choose from two different types of services. The free service, which is comparable to the current geocoding service and the suggest & lookup service that is composed of two endpoints that essentially work together.
• The geocoding services offer a wide range of options that are not available in the current geocoding service.

Putting it all together

Back in 2013 when I started writing about FME and share examples on how to use it, there was no real easy way how to share your workspaces, beside the making them available for download of mailing them.

Luckily nowadays the FME Hub is incorporated into the Workbench application, which makes it very easy to grab a transformer created by somebody else.

For demonstration purposes I have actually decided on sharing a template and not a transformer.
My intention is to demonstrate how you can make a simple start with this service and FME especially when taking in consideration all the options this services provides.

PDOKLocatieserver template

The template

A FME template is a great way of sharing workspaces because you can incorporate the source data in it. In this specific case the source is an online CSV file that contains addresses of educational facilities.
As simple validation and clearing some duplicate data is done and then the service type selection is done via a Tester transformer and user parameter.
The geocoding services return JSON by default, but XML is also available for the XML savvies.

Both types of services return multiple results and each result as a score. The highest the score value the highest the probability that it's the best match.

Once the result with the highest score is sorted per request it a simple matter to transform it into a point feature with the GeometryReplacer transfomer.

I have selected to use the geometrie_rd attribute to which contains the coordinate values in the Dutch national coordinate system, if you are more internationally oriented the service also returns a set of coordinates in WGS84.

The results are written into a SQLlite database, a format that is supported by most of the open source and propriety GIS applications.

What to try it out yourself? first of all download FME 2017 (still in Beta while writing this post) and check out FME Hub for the template.

Have Fun!

    

2016 in retrospect

The end of the year is nearing and it is time to look back at the passing year and list my top 3 of FME 2016 functionalities.
At the same time its also a time to look forward to the coming year and the upcoming FME functionalities in FME 2017.

This year I actually wanted to combine both aspects into a double top 3, first my top 3 of FME 2016 functionalities and a second top 3 which involves a new FME 2017 functionality.


I hope I am not giving away a functionality that Safe intends to blog about, so if that is the case....my apologies in advance.
But personally I think that there will be so many awesome additions in 2017 that the little bit I am using doesn't even make a small dent in the pile of 2017 goodies that Safe can use to blog about.

My FME 2016 top 3

1. FeatureWriter

2016 started with the announcement (for me it felt more like a meteor fall...) that in the future FME will be used without any Readers and Writers
You probably all know what I am referring to, the birth of the FeatureWriter,
the transformer that would ".... shake most FME users to their very core!"

I have to admit that initially I didn't see what all the excitement was about, but as the year worn on and I started using it more and more, I can tell you right now that I wouldn't know how I could have done without it! (well probably using a lot more workspacesto get the same job done)

So it's no surprise that it is by far the number 1 on my top 3 for FME 2016 functionalities. Personally I think Safe succeeded in delivering what they promised and that the time of no Readers and Writers workspaces is nearing.

2. AttributeManager

The AttributeManager  was another FME 2016 functionality which slowly changes our approach into data transformation and in a good way.

No more AttributeCopier, AttributeCreator, AttributeRenamer, etc, etc necessary, a single super transformer to replace them all.

Personally I really like the AttributeManager for it capabilities, but there is one small annoying issue with it and that is the fact that it requires my attention way to often when the data schema changes.

So I have resorted to update the AttributeManager content when the workspace is finished, instead of continuously updating it. Despite that the AttributeManager makes so much possible with one single transformer and that is why it's my number 2 on the top 3 of FME 2016 functionalities.

3.WFS Paging settings

My previous post was on this awesome functionality (secretly?) added to the WFS reader .
If you use WFS OGC services a lot then I bet you are as exited as I am about this hidden gem.

In the past I have demonstrated how to use ResponcePaging in FME, but that required some inventive workaround to get all the features and overcoming the service's limitations.

Nowadays it's a matter of setting the reader settings accordingly, lean back and enjoy the logging happily passing by, while the service is queried.

For that sole reason, making my (and hopefully anybody that uses OGC WFS services) life easier I am giving the bronze medal to this FME 2016 functionality (my number 3)



This concludes my personal top 3 of FME 2016, I had lots of fun playing around with data this past year thanks to Safe Software and their great product.
I am expecting to have as much, if not more, fun with the upcoming FME 2017 functionalities.

The other top 3

Since the FME beta is always available and it's a great place to find out about new functionalities, I was searching for an idea for a post when I saw this tweet from @MadMansson which made me remember an old post where a list of certified FME professionals was created by parsing the HTML page of the Safe Software site.

So where does this all come together?  In FME 2017! where we can easily parse HTML with the brand new HTMLExtractor transformer (note to myself get cracking on CSS selectors).

So as a small homage to the previous post I put together a small workspace in which the countries with FME certified professionals are ordered by a ratio of the number of certified professionals per country divided by the country's area.

You might wonder what are the area units, well I just grabbed the first hit on Google for world countries shape and it is in the LL-WGS84 coordinate system.

Another thing to mention is that when I initially made the workspace Luxembourg was number one, but as you see recently somebody joined the club (welcome!) and now it is Singapore leading the list.

According to the same shapefile there are 43 other countries with a smaller area than Singapore, so if you are from Macau and you plan to get certified, I can promise you an eternal first on this list :)

The HTMLExtractor transformer makes it easy to grab information from the web pages and I personally think we will see more web related functionalities coming in FME 2017.

Looking forward to it!

Have a great New Year!

WTwFs?

HU?

I wanted to share with you something I accidentally came across while playing around with FME.
After finding this I was astonished that no special (spatial?) reference to this addition was made other than the standard documentation.

Doing a bit of snooping around I've found this addition was already added to the 2016.1.0.0 (build 16492) FME Desktop version.



Well what's it all about? well lets say that I consider it to be as much as a game changer as the introduction of THE game changer of  FME 2016, the FeatureWriter.

It might not be as all encompassing as the FeatureWriter itself, but it does make it a lot easier for many people that work with OGC services on a daily basis.

OGC

Several OGC (Open Geospatial Consortium) services formats have been supported for some time now in FME Desktop.

The Web Map Service (WMS) is great as a background map in your Data Inspector, but other than that there is not much use for it.

Now the WFS (Web Feature Service) that's more like it! Data that we can play around with in FME to transform, combine and write to our favorite formats.

The WFS comes in different versions and each version allows for a different interaction with the service.


The services are usually restricted to allow a certain maximum number of features to return per request. That is done so that the system serving the data should not be overrun by requesting all the features in one go.

The 2.0.0 version allows for ResponsePaging which is the mechanism that allows users to request all the data but in a orderly manner.

ResponsePaging 

In the past I have shown how it is possible to use this mechanism in FME, but now its made even easier!

Now is response paging incorporated into the WFS reader, so that once you select the 2.0.0 services version, new settings specific for response paging are available.


The Start Index and Count parameters enable you to control the response paging, sit back and relax as FME manages the data requests for you.

First of all excellent addition! but what blows my mind is that NO MENTION of this was reported or communicated!!
I really think you guys at FME headquarters need to take some more credit.


I know for sure that you just made my life and lots of others easier so that we can do this...

while FME is doing the heavy lifting.

XML is all around us

The Past

In 2014 I wrote about how to get all of the Dutch SDI (PDOK) services into FME by creating a custom reader.

With the new XML abilities in FME 2016, I was wondering if a better job can be done.

I set about creating a second custom reader that not only returns constantly updated results (services url's) but provides them in such a way that can be directly used futher down the transformation.

The Present

The new XML abilities in FME 2016 makes life so much easier. It's still early in the year but I am already sure that the new ability to scan the input XML is one of my top 3 of FME 2016.

Couple that with the setting to limit the maximum schema features to scan, and you get a very powerful tool that makes the days of wrestling with large XML files history.

The PDOK URL Reader

As with the previous custom reader, all of the services provided by PDOK are returned by the reader.

During the development I actually got a verification that the reader works properly since 2 new services were automatically added to the reader's results.

As mentioned before, I wanted to improve the previous reader, this time an attribute (ServiceFeatureType) containing the WFS feature types is provided.

This attribute, along with the url attribute, can be directly used in the FeatureReader transformer to access the WFS service.

Why only the WFS services? the short answer is that FME is all about data transformation and there is no point in making the feature types of the other services types (WMS, WCS and WMTS) available, because you cannot do much with them anyway.

The Future

The future is already here! (well if you were a kid in the 80's)

Anyway what I mean to say is that nowadays is it much easier to share.

The FME Hub is a new feature that makes sharing and publishing much easier. I wonder if it is the future of the FME Store, but that can only be verified by somebody from Safe.

The FME hub is where you can also find my PDOKURLReader among other amazing stuff.

This reader is self-updating and lets you for example access all of the PDOK WFS features of a specific area.

The following workspace shows how you could go about achieving that.

Example FME workspace

The results look like this.

BAG and AAN WFS features

Have fun and let me know what you think of it.

"FME" is a registered trademark of Safe Software Inc.

2015 in retrospect

2015 in retrospect

The new year is upon us and it is time for me to look back on the passing year.
About this time, a year ago, I published a post on my top 3 of FME 2014, this time I have something totally different in mind.

FME Cloud

For me the past year was without a doubt a FME Cloud year.
This all started with me trying out FME Cloud, after my curiosity about FME Cloud got the better of me. I wondered what I could do with FME Cloud.

One of the many great things of FME Cloud (and Safe Software) is that you get some 'playing around' credits to test it without any need to invest your own capital.

Did you fill in the FME Server questionnaire last year? and this year? well I did and guess what?
I got even more credit to play around with FME Cloud......that was just awesome!


I soon found out that having my own FME Cloud instance made me a power player.
Having all of the transformation powers of FME Server at my fingertips and not even have to pay for it! I couldn't believe my luck.

I realized that I need a good product to be able to make a dent in the over monopolized dutch GIS sector.
That's when I came up with the idea of a self serve data transformation, which also was my first contribution the the FMEWT.

Not that self serve data transformation is something new for FME Server, but what was new about it was (and still is) is the data transformed (BGT).

Up until today there is no such service available for the BGT and even the big guns of the GIS sector are still struggling with how to service the data to its customers.
The transformation trigger for this product is an email with a link in it, quite simple and easy.

Self Serve

Self serve BGT

After the FME Cloud BGT transformation the flood gates of self serve transformations were flung open.

This resulted in a second product in which the degree of complexity got a tiny bit more complex, this time it was not a link to the data, but the data itself that got the transformation started.

This was a very important learning step for me, since working with the resources section of FME Server is crucial when switching to FME Cloud.

The third product (SPLITS Translator) was again more complex, to develop. This time the layer within the CAD trigger data needed to be specified and correctly accessed.

Fortunately the options to use parameters and transfer them between workspaces is easily done (once you figure it out) and a must have for advance FME Cloud usage.

Lastly my final 2015 FME Cloud based product (Geocoder) was the most complex of them all.
This time not only did I need to develop buit in checks to make sure that the schema of the trigger data is as required but also multiple output formats are possible.

2016

With the release of FME 2016 (desktop) and the soon to be released FME Server I am sure lots of more FME'ing will be done this year.

Happy FME'ing!



"FME Cloud" is a registered trademark of Safe Software Inc.

SPLITS Translator

SPLITS Translator

The SPLITS Translator is a FME Cloud based translation service for creating SPLITS compliant GML.

As with previous products the trigger is an email (content + attachment) which starts the translation.

The results are returned via an email with a download link.

SPLITS

The division of property lots is a normal procedure when real estate sales and housing development activities are involved. 

The Dutch cadastral web application SPLITS enables property lots splitting by sending all the data to the cadaster. 

The web application supports the drawing of new (temporary) boundary lines and also the use of GML (Geography Markup Language) as input. 

SPLITS application.

The use of GML has the following advantages: 

• The drawing can take place within a professional CAD environment making it more flexible to make adjustments.

• Multiple plots can be easily divided (think big size projects)

• Fast processing within the application. 

FME Cloud and SPLITS

In my previous posts I have shown how you can start an event-driven ETL transformation via email + content and email + attachment. 

This time the FME Cloud transformation is actually a combination of the two types.

The email is the trigger and now the attachment and e-mail content are used. 

FME is the master of formats.

Master kick.

Normally you would use a separate (dedicated) reader per format, but there is also a generic reader that offers the possibility to read multiple formats with one reader. 

In this transformation I use such a generic reader that supports the most common CAD formats (DWG, DXF and DGN). 

Using the generic reader makes it is very easy to adjust the workspace in order to support other (desired) formats. 

This product has been added to the growing ETL Solution FME Cloud based products, see www.etlsolution.nl.

Also interested in event triggered FME Cloud based ETL transformations? contact me via itay@etlsolution.nl