New annotation option in FME 2017

Annotations in FME

Annotations are an easy way to include description in a workspace.

There are 3 types of annotations in FME:

1. Header annotations that are only generated when a workspace is generated.
2. Summary annotations that are dynamic and reflect the changes in a workspace component (transformer, feature type, etc)
3. Custom annotations are generally known as user annotations.

The image below shows all 3 annotation types.

The header annotations are above the reader and writer feature types and connection, the summary annotations is blue and describes the reader feature type, while the user annotation is by default yellow and empty of content.

The Magic of FME 2017

The #FMEWT is about to finish for 2017 and I had the luck to present in it, if you are interested in my presentation here is a link

During the preparation of the presentation I came across an unknown (for me) new functionality in the Workbench application.

This new functionality is the ability to transform ;) a summery annotation into a user annotation.

Just right click on the summary annotation and select 'Convert to Attached Annotation'

If you are a Copy& Paste kinda of a person, as I am, then this is mana from heaven.....

Now I can create a summary annotation and edit it, something that can save a lot of time.

So next to the Parameter Editor this is another excellent addition that can boost your FME development productivity.

Have Fun!

WTwFs?

HU?

I wanted to share with you something I accidentally came across while playing around with FME.
After finding this I was astonished that no special (spatial?) reference to this addition was made other than the standard documentation.

Doing a bit of snooping around I've found this addition was already added to the 2016.1.0.0 (build 16492) FME Desktop version.



Well what's it all about? well lets say that I consider it to be as much as a game changer as the introduction of THE game changer of  FME 2016, the FeatureWriter.

It might not be as all encompassing as the FeatureWriter itself, but it does make it a lot easier for many people that work with OGC services on a daily basis.

OGC

Several OGC (Open Geospatial Consortium) services formats have been supported for some time now in FME Desktop.

The Web Map Service (WMS) is great as a background map in your Data Inspector, but other than that there is not much use for it.

Now the WFS (Web Feature Service) that's more like it! Data that we can play around with in FME to transform, combine and write to our favorite formats.

The WFS comes in different versions and each version allows for a different interaction with the service.


The services are usually restricted to allow a certain maximum number of features to return per request. That is done so that the system serving the data should not be overrun by requesting all the features in one go.

The 2.0.0 version allows for ResponsePaging which is the mechanism that allows users to request all the data but in a orderly manner.

ResponsePaging 

In the past I have shown how it is possible to use this mechanism in FME, but now its made even easier!

Now is response paging incorporated into the WFS reader, so that once you select the 2.0.0 services version, new settings specific for response paging are available.


The Start Index and Count parameters enable you to control the response paging, sit back and relax as FME manages the data requests for you.

First of all excellent addition! but what blows my mind is that NO MENTION of this was reported or communicated!!
I really think you guys at FME headquarters need to take some more credit.


I know for sure that you just made my life and lots of others easier so that we can do this...

while FME is doing the heavy lifting.

XML is all around us

The Past

In 2014 I wrote about how to get all of the Dutch SDI (PDOK) services into FME by creating a custom reader.

With the new XML abilities in FME 2016, I was wondering if a better job can be done.

I set about creating a second custom reader that not only returns constantly updated results (services url's) but provides them in such a way that can be directly used futher down the transformation.

The Present

The new XML abilities in FME 2016 makes life so much easier. It's still early in the year but I am already sure that the new ability to scan the input XML is one of my top 3 of FME 2016.

Couple that with the setting to limit the maximum schema features to scan, and you get a very powerful tool that makes the days of wrestling with large XML files history.

The PDOK URL Reader

As with the previous custom reader, all of the services provided by PDOK are returned by the reader.

During the development I actually got a verification that the reader works properly since 2 new services were automatically added to the reader's results.

As mentioned before, I wanted to improve the previous reader, this time an attribute (ServiceFeatureType) containing the WFS feature types is provided.

This attribute, along with the url attribute, can be directly used in the FeatureReader transformer to access the WFS service.

Why only the WFS services? the short answer is that FME is all about data transformation and there is no point in making the feature types of the other services types (WMS, WCS and WMTS) available, because you cannot do much with them anyway.

The Future

The future is already here! (well if you were a kid in the 80's)

Anyway what I mean to say is that nowadays is it much easier to share.

The FME Hub is a new feature that makes sharing and publishing much easier. I wonder if it is the future of the FME Store, but that can only be verified by somebody from Safe.

The FME hub is where you can also find my PDOKURLReader among other amazing stuff.

This reader is self-updating and lets you for example access all of the PDOK WFS features of a specific area.

The following workspace shows how you could go about achieving that.

Example FME workspace

The results look like this.

BAG and AAN WFS features

Have fun and let me know what you think of it.

"FME" is a registered trademark of Safe Software Inc.

Georeferencing evaluation with FME.

FME and data evaluation.

FME is a great tool to validate and evaluate data (next to the many things you can do with FME)
There are plenty of resources available on the subject demonstrating FME's data validation and Q&A capabilities.

Data evaluation can involve different aspects and have many forms.
For this post I choose to evaluate how well a publicly available data set can be georeferenced (if you can add value to it and put it on a map, why shouldn't you...)
For any serious conclusions, you'll have to work it out yourself, since my main intention is to demonstrate FME capabilities (and not bad mouth anybody particularly...)

Data source.

The Dutch government publishes many data sets openly and the numbers are increasing all the time.
I choose to use the data set of the national education registry since it is highly dynamic and it contains addresses, which makes it possible to potentially georeference the features.

The data used is available in csv format, which can easily be accessed online via the CSV reader (just point it to the url). For limiting sorting and filtering the incoming data, see my previous post: Where clause on text.
This results in a continuously updated data source, which is great to have but poses a challenge when displaying the results.

 Georeferencing the data.

For georeferencing the source data I am using the BAG Geocoding service, available via the National SDI.

An easy way in FME to access the service is by a HTTPFetcher transformer.

By constructing the URL in the transformer's text editor and making use of attributes values, a very flexible solution is created.

BAG Geocoding service results.

The BAG Geocoding service returns the location(s) in an XML snippet that translates into geometry and attributes. In case of ambiguity or lack of sufficient input, the service returns an aggregate geometry.

Somewhere in the aggregate geometry the corresponding location and attributes can be found (well most of the times...)

Using the total count of both georeferenced and failed features, simple statistics (percentage of correct georeferencing) can be gathered and used for display.

HTTPFetcher

Interpreting the results.

Some of the 'failed' to georeference features do actually exist (BAG Web) and can be correctly geocoded by slightly changing the address used, see for example georeferenced (note the street tag) and not georeferenced (note URL used =  input address)

Displaying the results.

I am using Google Fusion Tables to display  the results since it is an easy way to share geographical information (article is in dutch)

Also non-spatial data can be shared this way and the failed features are saved into a non-spatial Google Fusion Table. Needless to say FME supports both spatial and non spatial reading and writing of this format.
Some limitations of this format are the number of features supported and that it is still considered an experimental format, something that unfortunately makes it less reliable.
As mentioned before the input source data is updated frequently and in contrast the displayed results are static and present a moment in time.


Map of results, created 1-10-2014.

Findings and (possible) future developments.

A way to keep the displayed results up to date would be to use FME Cloud, something I still have not got around to try. I imagine that using FME Cloud to run this workspace would not require almost any resources or adaptations, since most of the data is on-line.
Some of the finding are:

• Saving the source csv data is necessary due to memory issues (something that is easily done in the HTTPFetcher)
• Another curious issue found is that using the postcode in the request string actually results in less features georeferenced.

Don't forget that by the time you read this post the output might look very different.

FME and 3D printing.

3D printing.

3D printing has been around since the 1980's, but it is only in the last few years that a real boom in its application has taken place especially for domestic use.
The most commonly used format (e.i de facto standard) is the STL, which comes in two types (ASCII and Binary).

3D print.

STL format and FME.

If you search for this format in the Readers and Writers documentation, you will find that FME does not support it.
Well not directly since no reader or writer are available, but indirectly, because it is plain text and FME can create and write STL files.

3DS to STL transformation.

There are lots of 3D files available on the web, and since it was around lunch time... I have selected this fork for testing.

3DS in the Data Inspector.

The 3DS fork is represented as a IFMEMesh geometry, which is to say that it is composed of parts that do not necessarily have a topological or spatial relationship.

These are also the parts that need to be represented in the STL format, adhering to simple format definitions.

So its not surprise that with FME the mesh geometry can be transformed and written to the STL format.

Workspace.

The workspace is quite simple and there are but a few steps necessary.

1. Firstly decomposing the geometry to its components.
2. the coordinates are extracted into attributes, here is also where I drop the geometry since it is not needed anymore (get rid of anything unnecessary, another FME rule).
3. Some formatting is taking place to represent the coordinates as floating point numbers (StringFormatter), although some applications export into STL without the floating point representation.
4. Finally creating the output by aggregating the coordinate values and concatenation.

A text file writer is all that is necessary to write the STL format.

 Result.

Result STL file.

Since FME does not support the STL format it cannot be viewed in the Data Inspector, but no worries there are plenty of visualization tools available.
I am using the freely available Meshmixer to display the result.

I dont have a 3D printer, so in case you have tried yourself to create a STL file with FME and printed the result, it would be great to know.

All PDOK atom feeds in FME.

Recently I was approached about accessing atom feeds services from the Dutch national SDI (PDOK).

After writing several posts on the subject I realized that having access to all of the atom feeds services in FME, would be a nice challenge and a useful addition.
So I set about figuring out what would be the best way to set it up. I realized that I wanted to have the option to select an individual service or several of them.

How to get all the URL's

Normally the GEORSS reader is used to access a XML feed.
To be able to access a service you would have to select the URL from the web page and paste it into the reader.
This approach works fine, if you access a couple of services, but I would like to have them all selected.
Especially when new services are added or deleted.
Once I have all of them I can choose the ones I would like to use in my workspace.

Well since FME is great at accessing data on the web, Why not use it to retrieve the URL's of all the services?

The PDOK site is built with a certain logic, all of the services are alphabetically ordered.
Using that logic in FME enables you to select the services by accessing the HTML pages displaying the services on the PDOK website.

PDOK web page HTML

The XMLFragmenter can slice through the web page HTML like any other XML document.
Parsing the XML (another great feature of FME) enables the final selection of the URL's.

FME workspace

Result

So after some XML parsing I got a list of all of the current PDOK atom feed services.
You might think, well I can just sit and copy paste the URL's into a text file.....well sure you can! nobody is stopping you :)
But by using FME and it's XML capabilities, you always end up with an updated list.
When services are deleted or new ones introduced, the resulting list will reflect that.
Beside that, once you have the services list available, you can continue your data transformation, something which is not really possible with a text file :)

Atom feeds services list.

Atom feeds in FME

As with anything FME, there are more ways to skin the proverbial cat ;)
One way of using the services list in a workspace, can be by using a startup python script.
Yet another way would be to wrap the workspace into a custom reader, so that it can become part of your regular FME readers.

And finally you can just opt to continue developing the workspace.
There are quite a lot of FME transformers dedicated to web services, two of them are specific for GEORSS feeds.
In this scenario the GeoRSSFeatureReplacer is very useful to extract all the information from the server response into attributes.

Services selection

The services selection should be done at run time to provide extra flexibility.
This is where a TestFilter and user parameter combo comes in handy.
To retrieve the parameter value(s) into my workspace, I use the so adequately named :) > ParameterFetcher
Then it's a simple matter of setting up the correct test to retrieve the selected services.

So to wrap it up:

1. Now I have a new PDOK atom feeds reader in FME, and I know for sure that it will always provide me with an updated list of services. 
2. As you know FME is a no code approach. 
3. With just 8 transformers the workspace is super simple.

Life made easy the FME way, interested? give me a call.

Fetching all BAG WFS features

Fetch boy fetch!

 

 

 

 

 

Origin 

The idea for this post originates from a tweet by a FME user on a blog post in which information is given about accessing the Dutch SDI (PDOK) OGC webservices. For original post (in dutch) see brentjesgeoict.

Curiosity

In the post a number of handy tips are given concerning fetching all features residing in the national registration for addresses and buildings service (BAG)
I must admit that until now I have used OGC services strictly as input data source without actually realizing that there might be a limitation. After getting curious about it I started finding out more about the possibilities of smart usage of OGC services and the ability to make the use easy via FME.

 OGC

I am not going to give you a detailed overview about OGC, for that use the Internet, but I will mention the fact that the OGC standards are unique in the GeoSpatial sector for being widely accepted as created standards (in contrary to the de facto standards e.i. shapefile)

Making it easy

The recommended way to overcome the service restrictions (for this case there is a restriction on the service that results in a maximum of 15000 features returned) is to make use of the count and startIndex parameters of the GetFeature request.(WFS 2.0.0)
To make sure that all of the features are returned, an initial query ("resulttype=hits") returns the total number of features served.
With that a simple calculation can be done that ensures the correct number of requests are always sent.
The use of parameters in the workbench makes it easy to adapt the GetFeature request syntax to any available feature type.
Allowing the area of interest to be selected (municipality, province or national) at runtime and the calculated requests make it all care free and really easy to use.

FME Workbench

Results

To make the results accessible I have created a Google map.
With the help of the filter option simple questions such as: "all the buildings built last year?"* can be directly answered and visualized.






* Results are based on a temporary service, and may not be updated or complete.
Interested in the workspace, give me a call. 

FME 2013 in retrospect.

FME 2014 release.

Now with FME 2014 available, it is for me a time to reflect back on the changes made in the 2013 release.

Thinking back on the adaptations made for FME 2013, especially the ones that I have used the most, I can truly say which are my favorites.

1. Conditional processing FME 2013 SP1.

Original FME Evangelist post

Funny enough my first question on the FME Community was about such conditional processing.
At the time it was only possible to achieve it with multiple transformers, testing and setting/mapping values (or an elaborate sql statement)
With the introduction of conditional processing, the amount of transformers for such processing was reduced to 1 e.i. the AttributeCretor.
The AttributeCretor has been replacing more and more transformers on the canvas, mainly testing and mapping transformers obviously.

2.  Data Inspector FME 2013 SP1.

Right from the start of 2013 I have been using the Data Inspector instead of the Universal Viewer.
Since viewing the data in question, is such an essential part of the process of transformation, it's obvious that a capable tool for the job is needed.

The Data Inspector for me is such a tool, especially since it did not only replaced the Universal Viewer, but outdated it too. The addition of the table view to the Data Inspector, did turn out to be a very useful addition that added a certain maturity to the all concept of data visualization within FME.

The last Universal Viewer remnant, that I was missing, is now also been made available in the new FME 2014 release. Saving from the Inspector is now possible in two ways, saving all the features, or just the ones viewed.

3.  New Excel reader/writer FME 2013 SP2.

Blog post

Several posts have been published in the past year about the new Excel reader and writer. It is truly an major improvement to the old reader/writer, supporting formulas, colors and a wider range of data types (XYZ coordinates to geometry), just to mention a few.

Surprisingly the new reader does not support a where clause, as the old reader did, I am still puzzled about this, especially since Excel is treated as a database. Possibly the underlying technology change is the cause of this reader parameter missing in the new reader.

The handling of multiple schema, especially originating from multiple feature types, is still an unresolved issue. Even then the advantages of the new Excel reader/writer capabilities greatly outweigh these few introduced glitches.



So there you have it, these are my favorite and most used FME 2013 improvements I wanted to share.
Now with that out of the way I can send more time on FME 2014 in future posts exploring the sea of opportunities.

Have a feeling the it will all be about nothing.....

FME 2014 - look and feel

FME 2014 is released and available. I have only been using it for the past week, this post will be about my first impressions of the new release.

My first impression of FME 2014 is the amazing speed in which the new look makes  FME 2013 look so chunky and outdated.
This might not be (and probably isn't) the most important addition in the new release, but its a striking one, right from the start.
Its is very easy to get used to the smoothness introduced with this new FME release.

Again hat's off for the people at Safe!!

A few more apparent improvements\changes are:

• Bookmarks and annotations have had a major touch up which makes them more user friendly and easier to use, supporting links and adjustable text properties.

•  Naturally a new theme is also expected with every release of FME, as announced the lizard came back to vanquished the zipster into the sands of the red planet.

• Another immediately apparent change to me, is the option to save your workspace while the translation is running. This is for me a major user interface improvement, that can save a lot of awkward moments when running unsaved workspaces crash.

• The selection of features on the workbench canvas is another major improvement, don't forget to activate it! (Tools > FME Options >Workbench)

Well thats it for now,  as I continue to use FME 2014 and get better acquainted with all the new additions, I  expect additional posts about FME 2014 additions.

XML the xfMap path

Most of the people I meet in the Geo-information sector are aware of FME as a translation tool  from one format to another.

That is originally what FME was built for (20 years ago) and is still one of the jobs where FME transcend all the rest of the available tools.

But FME is much more than a format translation tool, is as good as any GIS software available , and in my opinion it blows away most of them.


I constantly get amazed looks from costumers, when explaining them that most of their requirements can be done with FME, and in most cases in a single workspace.

The ability to dazzle with FME is not restricted to the spatial domain, FME's support of non spatial formats (for example the Excel writer, the shapefile of non spatial) makes it possible for non spatial people to benefit from FME when a serious user is around.

XML is another format where FME can be of great benefit, since almost any imaginable operation on XML can be done in FME, the combination of spatial and non spatial makes FME a truly versatile tool.

Since nowadays you can find XML almost everywhere, from OGC services styling to cadaster data sources all is served via XML.
The only main difference in the data is the schema (data model) and that can range from simple to extremely complex.

In FME the advised way of approaching XML data is by using feature paths, this makes XML handling very easy and does not require any in depth knowledge of the data schema.
However in some cases using feature path doesn't (and sometimes can't) handle the XML as good as xfMap.
xfMaps are the basis of XML handling in FME and if you know your xfMap then handling XML becomes super easy. The main drawback is that you do need in depth knowledge of the data schema.

The big advantage of using  xfMap over feature paths is the ability to read very large XML files, I suspect that when using feature paths the file is read into the computers memory, this can be an issue when dealing with huge volumes of data, even if your machine is supercharged and all of your FME settings are in place.

When using xfMap the file that crashed your machine trying to handle it with feature paths, will now easily load the same data.

I have tested this with the new BRK (new cadastral registration) data, available from the Dutch Kadaster.
Trying to parse the XML (260MB) with feature paths, just crashed my machine (and I do have a reasonable machine with the FME settings set properly).
With a very simple (and powerful) xfMap the same file loaded within 10 seconds and was ready to be transformed.

So to wrap it up, use xfMap if:

1. The data volume is large and the feature paths configuration doesn't comply.
2. The data schema is static
3. You have too much time on your hands.

This is the last post for 2013, if you have already had some glimpses of what is to be expected for us with FME 2014, then I am sure you are as excited as I am!

A prosperous 2014 to us all.

Itay

Bye Bye Universal Viewer

What is the first thing they teach you in a FME course? To have a look at the data !

All the way back to my first careful steps with FME, the Universal Viewer was there to help visualize and query the data I was working with.

A great tool and without it, it would have been unthinkable to work with any type of data.

 FME Universal Viewer

Since the first appearance of the next generation viewing application of FME (Data Inspector), I have been hesitant to make the transition from the known and familiar Universal Viewer to the new Data Inspector.

Sure there have been moments when I thought OK now I am finally going over to the Inspector, I mean its fancy and you can view in 3D (That was back in the days when the 3D was hot). But somehow I never got around to finally take the final step.

???

Since FME 2013 was released I have been only using the Data Inspector (I remember somewhere in the beginning of 2013 that the FME Evangelist himself made the transition and naturally recommended it to us all).
Throughout 2013 a number of features have been added to the Data Inspector (background maps, table view) that made the Universal Viewer really seem old fashioned.
The only Universal Viewer functionality I still miss in the Data Inspector is the ability to save the viewed features.

So bye bye good old Universal Viewer I suspect I wont see much of you anymore (wasn't FME 2013 also the last release with the Universal Viewer?)

And for those of you that still cling to the Universal Viewer, take the final step it's worth it!

It's something unpredictable, but in the end is right, I hope you had the time of your life.
- Green Day

Rotten Apple

As an ETL specialist you are mostly involved in moving data, doesn't matter what you do with the data you are always moving it in some way.

There is always a starting point (E) and an end point (L) to the transformation (T), this time I would like to share some of my experience concerning the ETL process.

This will no be so much about how to do this or that, but more about the delicate and sometime awkward situations that can arise in the process.

 If you are doing custom transformation work or it is just part of your daily work, the hardest part of the ETL process can sometimes be the process around the technicality of it.

Since you are responsible for the 'black box' for most people (and managers especially) you are bound to get the blame if it goes wrong.

 Lesson 1: make it clear from the start what your responsibilities are!

Make sure that the parameters and expected results of your job are well defined before you even open FME!

This can take some time and you will be seen as a pain in the .... but it can save you a lot of pain and it will make your goals clear and well defined.

In most cases involving large and complicated database schema and design, you are dependent on others to provide you an insight about the database schema or even better: queries to preform on the database.

Most of the times you get a reaction like: " I thought you are the ETL specialist...." or " Don't you know how to do it yourself?", sounds familiar?

Lesson 2: clearly define the input and expected results (especially if large and complicated database schema is involved)

OK you got over the first hurdles and created your workspace and it works or at least you think it does, but wait a minute after some time it turns out that it is not providing the expected results (oops embarrassing moment) but hey we are all human and we make mistakes.

So you rack your brain trying to find where the mistake is done. Eventually you find no fault in your work and the workspace does what it is suppose to do.

In fact you have delivered what was possible based on the information you got, the problem arises when you are not supplied the correct information.

Lesson 3: document the defined input, expected results, responsibilities and ETL process.

Despite it all it can always go bad, there is always a rotten apple.....somewhere in between it all.

FME Template

The ability to save workspaces as templates was added in FME 2011.

If you keep preforming the same actions, for example use a reader's bounding box settings, then this is something you should consider.

A template is basically a saved workspace, in which you can have as many transformers and readers in it as you like or none at all.

None at all? well yes actually the one I use the most doesn't have any transformers (yet) and only one reader and an inspector.

If you are a heavy database user then consider the following:

• You constantly need to access data from different services with different users/passwords and you want a to spatially select the data and tables from where the data comes from.

You can create a workspace from scratch each time, but if you hate repetitive and not particularity efficient tasks (like me) then a template can save you lots of time (which you can spend on the fun part = transformation)

To make the template as flexible as possible I make use of many parameters (private and published).
These parameters assist me to define the database service, user name, password, feature types (tables), where clause and location (bounding box) when running the workspace (prompt and run)

Prompt window

The only action I actually have to do is to select my translation source parameters and run, viola!