WTwFs?

HU?

I wanted to share with you something I accidentally came across while playing around with FME.
After finding this I was astonished that no special (spatial?) reference to this addition was made other than the standard documentation.

Doing a bit of snooping around I've found this addition was already added to the 2016.1.0.0 (build 16492) FME Desktop version.



Well what's it all about? well lets say that I consider it to be as much as a game changer as the introduction of THE game changer of  FME 2016, the FeatureWriter.

It might not be as all encompassing as the FeatureWriter itself, but it does make it a lot easier for many people that work with OGC services on a daily basis.

OGC

Several OGC (Open Geospatial Consortium) services formats have been supported for some time now in FME Desktop.

The Web Map Service (WMS) is great as a background map in your Data Inspector, but other than that there is not much use for it.

Now the WFS (Web Feature Service) that's more like it! Data that we can play around with in FME to transform, combine and write to our favorite formats.

The WFS comes in different versions and each version allows for a different interaction with the service.


The services are usually restricted to allow a certain maximum number of features to return per request. That is done so that the system serving the data should not be overrun by requesting all the features in one go.

The 2.0.0 version allows for ResponsePaging which is the mechanism that allows users to request all the data but in a orderly manner.

ResponsePaging 

In the past I have shown how it is possible to use this mechanism in FME, but now its made even easier!

Now is response paging incorporated into the WFS reader, so that once you select the 2.0.0 services version, new settings specific for response paging are available.


The Start Index and Count parameters enable you to control the response paging, sit back and relax as FME manages the data requests for you.

First of all excellent addition! but what blows my mind is that NO MENTION of this was reported or communicated!!
I really think you guys at FME headquarters need to take some more credit.


I know for sure that you just made my life and lots of others easier so that we can do this...

while FME is doing the heavy lifting.

XML is all around us

The Past

In 2014 I wrote about how to get all of the Dutch SDI (PDOK) services into FME by creating a custom reader.

With the new XML abilities in FME 2016, I was wondering if a better job can be done.

I set about creating a second custom reader that not only returns constantly updated results (services url's) but provides them in such a way that can be directly used futher down the transformation.

The Present

The new XML abilities in FME 2016 makes life so much easier. It's still early in the year but I am already sure that the new ability to scan the input XML is one of my top 3 of FME 2016.

Couple that with the setting to limit the maximum schema features to scan, and you get a very powerful tool that makes the days of wrestling with large XML files history.

The PDOK URL Reader

As with the previous custom reader, all of the services provided by PDOK are returned by the reader.

During the development I actually got a verification that the reader works properly since 2 new services were automatically added to the reader's results.

As mentioned before, I wanted to improve the previous reader, this time an attribute (ServiceFeatureType) containing the WFS feature types is provided.

This attribute, along with the url attribute, can be directly used in the FeatureReader transformer to access the WFS service.

Why only the WFS services? the short answer is that FME is all about data transformation and there is no point in making the feature types of the other services types (WMS, WCS and WMTS) available, because you cannot do much with them anyway.

The Future

The future is already here! (well if you were a kid in the 80's)

Anyway what I mean to say is that nowadays is it much easier to share.

The FME Hub is a new feature that makes sharing and publishing much easier. I wonder if it is the future of the FME Store, but that can only be verified by somebody from Safe.

The FME hub is where you can also find my PDOKURLReader among other amazing stuff.

This reader is self-updating and lets you for example access all of the PDOK WFS features of a specific area.

The following workspace shows how you could go about achieving that.

Example FME workspace

The results look like this.

BAG and AAN WFS features

Have fun and let me know what you think of it.

"FME" is a registered trademark of Safe Software Inc.

Transformation made easy: the FME Cloud way.

FME Cloud provides the possibilities to have data transformed and served without any detailed knowledge of the data schema or even an application.

In my previous post, based on my FMEWT 2015 presentation, I have demonstrated how easy it is to send FME an email and let FME do the heavy lifting of getting the data, transforming it and making it available.
In that scenario the user only had to select the data and send the url.
In the next scenario the user already has some data which he needs translated. So here the email is sent with an attachment.

FME Server's email capabilities are described by a series of tutorials on the FME Knowledge Center (latest name for the old FMEPedia)
In the tutorials all the information necessary for making the most out of the incoming email is addressed.

The AERIUS project  was also presented on the FMEWT, since FME is everywhere....

The AERIUS calculator is one of the components of the national PAS project which results in a GML file.
Reading GML should not be an issue for a seasoned FME user and with the help of the Knowledge Center a beginner shouldn't have much of a problem figuring out how to point to the schema document and let FME do the work.



But what if you don't have FME or you are only interested in usable results?


Well the answer to that is: FME Cloud !

To demonstrate this I have used FME's powers to build a simple and easy to use AERIUS2FGDB translation.
The translation is event triggered which means you need to start it.





How? well that's easy:

1. send an email to: fme@etlsolution.nl
2. Email topic: AERIUS2FGDB_b28ba3b0-0da6-11e5-a040-028deac61efd
3. Email attachment : AERIUS GML

Result?
The AERIUS GML gets translated to filegeodatabase (FGDB) and is made available via an email with a download link.

Are just interested in testing the methodology behind the product? I have already a AERIUS GML available for you to send as an attachment.

Interested in applying this for your organisation and your specific needs? dont esitate to contact me via itay@etlsolution.nl.

Free testing this product will be available from 8/6/2015 until 10/6/2015 between 9:00 AM and 16:00 PM (CET)

BGT via FMECloud

FME Cloud

What is FME Cloud? Simply put : FME Cloud FME Server in the Cloud. FME Cloud is the first location-aware iPaaS ( Integration Platform as a Service) and that means you have all the FME Server functionality at your disposal, without the hardware . 

See the following link for more information on FME Cloud.

One of the many advantages of FME Cloud is the ability to activate processes via an external trigger (e.i. event driven ) . Examples of such an external trigger can be an e-mail or a sensor value change

BGT

The creation of the BGT ( or The Registration Large Scale Topography as Goggle translate calls it ) is in full swing , many of the stake holders are feverishly working on the assembly of this nation scale polygon map. The first stage is expected to be completed on January 2016. Stake holders deliver their part of the BGT to the national facility (LV) and it is from there made ​​available in CityGML format via PDOK (Dutch national SDI). For an overview of the already available BGT in the LV see: BGTWeb 

The CityGML format cannot be directly used in GIS applications or loaded into a database,  a format conversion (something FME is great at ) must first take place.

Event driven BGT2FGDB conversion.

FME is the tool for spatial data conversion and FME Cloud is the cloud tool for spatial data conversion :)

The trigger

The format conversion is carried out with FME Cloud and is started via a trigger, in this case the trigger is an e-mail . There are some conditions that the trigger has to meet in order to start the process, for example a selected map tile link should be provided.

The conversion

During the format conversion a number of format -specific adjustments are done. Examples of these changes include: removing specific format invalid characters , conversion to uppercase , null values ​​mapping, date conversions, new geometry is created (think of house numbering and street names labels)

The result.

The result is a uniform and complete BGT data set in FileGeodatabase ( FGDB ) format. The FGDB is a widely used and efficient GIS format . This format can be used in both closed- and open-source GIS applications. After performing the format conversion , the result is made ​​available via an e-mail with a download  link.

Advantages and properties of the conversion are : 

• Easy to use. 
• No application is required. 
• Online and always available. 
• Reliably and quickly. 
• Outcome ready for database read-in action and GIS applications. 
• Unique : There is no comparable data distribution method ( in the Netherlands)

Results displayed in QGIS

This product is designed for the Dutch market if you are interested and want to set up something similar please contact us via info@etlsolution.nl

FME and Open Geospatial Consortium (OGC) Sensor Observation Service (SOS)

Sensor Observation Service (SOS)

The Open Geospatial Consortium (OGC) implemented the SOS standard
that basically returns sensor and observation data via XML. (yey!)
3 main functions are supported:

1. GetCapabilities - what server provides
2. DescribeSensor - information about a sensor
3. GetObservation - observation data

Naturally with FME all of these functions can be accessed and the data retrieved to any supported GIS and database format.

FME and web services.

There is already a great webinar displaying how, without a single line of code, a variety of web services are accessed and made with FME. In the webinar the observation data is displayed in the Data Inspector (with a background map) and the table view is used to sort the measurement values.
As with all FME webinars, the workspaces are made available.

Demo workspace

IOOS Demo Workspace.

After viewing the webinar, downloading the workspaces and exploring them, I wanted to expand the SOS demo by retrieving the sensor information.
The demo workspace is well annotated and the use of bookmarks is demonstrated.
By the way, did you notice that once you use bookmarks, you actually have your technical design?

Demo workspace +

 In the developed version of the workspace the upper part is the same as in the demo.
I did , however, change the location of the creator and ParameterFetcher, to be able to distinguish between the service functions.

The added parts are far from complicated, and that is actually the beauty of it, with no code and a minimal number of transformers all data can be easily retrieved.


The sensor information retrieved also contains the sensor location (among other things) to be able to share these results I have created this Google map (with FME, naturally...) that shows the location of the stations.
One of my favorite locations is Sand Island, Midway Islands. Does anybody know what species are all those bird chicks..? 

Google Street View

 

FME 2013 in retrospect.

FME 2014 release.

Now with FME 2014 available, it is for me a time to reflect back on the changes made in the 2013 release.

Thinking back on the adaptations made for FME 2013, especially the ones that I have used the most, I can truly say which are my favorites.

1. Conditional processing FME 2013 SP1.

Original FME Evangelist post

Funny enough my first question on the FME Community was about such conditional processing.
At the time it was only possible to achieve it with multiple transformers, testing and setting/mapping values (or an elaborate sql statement)
With the introduction of conditional processing, the amount of transformers for such processing was reduced to 1 e.i. the AttributeCretor.
The AttributeCretor has been replacing more and more transformers on the canvas, mainly testing and mapping transformers obviously.

2.  Data Inspector FME 2013 SP1.

Right from the start of 2013 I have been using the Data Inspector instead of the Universal Viewer.
Since viewing the data in question, is such an essential part of the process of transformation, it's obvious that a capable tool for the job is needed.

The Data Inspector for me is such a tool, especially since it did not only replaced the Universal Viewer, but outdated it too. The addition of the table view to the Data Inspector, did turn out to be a very useful addition that added a certain maturity to the all concept of data visualization within FME.

The last Universal Viewer remnant, that I was missing, is now also been made available in the new FME 2014 release. Saving from the Inspector is now possible in two ways, saving all the features, or just the ones viewed.

3.  New Excel reader/writer FME 2013 SP2.

Blog post

Several posts have been published in the past year about the new Excel reader and writer. It is truly an major improvement to the old reader/writer, supporting formulas, colors and a wider range of data types (XYZ coordinates to geometry), just to mention a few.

Surprisingly the new reader does not support a where clause, as the old reader did, I am still puzzled about this, especially since Excel is treated as a database. Possibly the underlying technology change is the cause of this reader parameter missing in the new reader.

The handling of multiple schema, especially originating from multiple feature types, is still an unresolved issue. Even then the advantages of the new Excel reader/writer capabilities greatly outweigh these few introduced glitches.



So there you have it, these are my favorite and most used FME 2013 improvements I wanted to share.
Now with that out of the way I can send more time on FME 2014 in future posts exploring the sea of opportunities.

Have a feeling the it will all be about nothing.....

XML the xfMap path

Most of the people I meet in the Geo-information sector are aware of FME as a translation tool  from one format to another.

That is originally what FME was built for (20 years ago) and is still one of the jobs where FME transcend all the rest of the available tools.

But FME is much more than a format translation tool, is as good as any GIS software available , and in my opinion it blows away most of them.


I constantly get amazed looks from costumers, when explaining them that most of their requirements can be done with FME, and in most cases in a single workspace.

The ability to dazzle with FME is not restricted to the spatial domain, FME's support of non spatial formats (for example the Excel writer, the shapefile of non spatial) makes it possible for non spatial people to benefit from FME when a serious user is around.

XML is another format where FME can be of great benefit, since almost any imaginable operation on XML can be done in FME, the combination of spatial and non spatial makes FME a truly versatile tool.

Since nowadays you can find XML almost everywhere, from OGC services styling to cadaster data sources all is served via XML.
The only main difference in the data is the schema (data model) and that can range from simple to extremely complex.

In FME the advised way of approaching XML data is by using feature paths, this makes XML handling very easy and does not require any in depth knowledge of the data schema.
However in some cases using feature path doesn't (and sometimes can't) handle the XML as good as xfMap.
xfMaps are the basis of XML handling in FME and if you know your xfMap then handling XML becomes super easy. The main drawback is that you do need in depth knowledge of the data schema.

The big advantage of using  xfMap over feature paths is the ability to read very large XML files, I suspect that when using feature paths the file is read into the computers memory, this can be an issue when dealing with huge volumes of data, even if your machine is supercharged and all of your FME settings are in place.

When using xfMap the file that crashed your machine trying to handle it with feature paths, will now easily load the same data.

I have tested this with the new BRK (new cadastral registration) data, available from the Dutch Kadaster.
Trying to parse the XML (260MB) with feature paths, just crashed my machine (and I do have a reasonable machine with the FME settings set properly).
With a very simple (and powerful) xfMap the same file loaded within 10 seconds and was ready to be transformed.

So to wrap it up, use xfMap if:

1. The data volume is large and the feature paths configuration doesn't comply.
2. The data schema is static
3. You have too much time on your hands.

This is the last post for 2013, if you have already had some glimpses of what is to be expected for us with FME 2014, then I am sure you are as excited as I am!

A prosperous 2014 to us all.

Itay

Rotten Apple

As an ETL specialist you are mostly involved in moving data, doesn't matter what you do with the data you are always moving it in some way.

There is always a starting point (E) and an end point (L) to the transformation (T), this time I would like to share some of my experience concerning the ETL process.

This will no be so much about how to do this or that, but more about the delicate and sometime awkward situations that can arise in the process.

 If you are doing custom transformation work or it is just part of your daily work, the hardest part of the ETL process can sometimes be the process around the technicality of it.

Since you are responsible for the 'black box' for most people (and managers especially) you are bound to get the blame if it goes wrong.

 Lesson 1: make it clear from the start what your responsibilities are!

Make sure that the parameters and expected results of your job are well defined before you even open FME!

This can take some time and you will be seen as a pain in the .... but it can save you a lot of pain and it will make your goals clear and well defined.

In most cases involving large and complicated database schema and design, you are dependent on others to provide you an insight about the database schema or even better: queries to preform on the database.

Most of the times you get a reaction like: " I thought you are the ETL specialist...." or " Don't you know how to do it yourself?", sounds familiar?

Lesson 2: clearly define the input and expected results (especially if large and complicated database schema is involved)

OK you got over the first hurdles and created your workspace and it works or at least you think it does, but wait a minute after some time it turns out that it is not providing the expected results (oops embarrassing moment) but hey we are all human and we make mistakes.

So you rack your brain trying to find where the mistake is done. Eventually you find no fault in your work and the workspace does what it is suppose to do.

In fact you have delivered what was possible based on the information you got, the problem arises when you are not supplied the correct information.

Lesson 3: document the defined input, expected results, responsibilities and ETL process.

Despite it all it can always go bad, there is always a rotten apple.....somewhere in between it all.

XML

I used to be a bit cautious when dealing with XML in the past, but since I have started using FME with it, it just broke down all the barriers.

I actually find it to be a very exiting data type to work with, it is simple to understand and with FME (what else) it is simple to update, create and completely transform.

There is a all bunch of FME transformers for dealing with XML in the transformer gallery.

I have to admit that I only use a couple of them on a regular basis (XMLTemplater, XMLFormatter and the XMLValidatior) which ,as the names suggest, help me to create, validate and make then human readable.

I guess that Don's love and enthusiasm for XML has affected me too!

First rumblings

Where to start.....


I guess a bit of free typing wont hurt at this stage, I mean nobody is actually going to read this? right?
My conclusion is quite easy for me to understand and argue, but for any other some clarifications are needed.

My initial idea was to create a personal log that one day might take shape into a full grown blog, this soon took another form, an blog to promote/add exposure to my self.

Shortly, I am (or trying to be) a freelance spatial ETL professional, I live in the Netherlands one of the most advanced geospatial nation in Europe. 
Because of that and the amount of competition in the geospatial market, exposure is critical for a start up business.

BTW did I mention FME?

More on that later....