Tuesday, August 1, 2017

Java debugging in CloverETL

CloverETL allows you to create a simple data transformation with just Drag&Drop UI, but sometimes you need something more complex and then writing code in CTL (Clover Transformation Language) comes in handy.

You can say that CTL is simplified Java, with some limitations - no inheritance, no exceptions, focused on a standard issues found in data manipulation processes.

And when you are writing code, you would like to have some debugging functionality. Before version 4.3.0 it was a bit old school. printLog function all over the place and digging through log output in Console view.

Since 4.3.0 CTL debugging was introduced and you can read about it in this great blog post http://blog.cloveretl.com/code-debugging-cloveretl-designer .

CloverETL also offer functionality of writing your transformation in Java if needed. Common use cases are implementing non-existent functionality in CTL or re usability of existing Java code. Java transformation could be used pretty much anywhere you can use CTL (Reformat, Denormalizer, Normalizer components etc.).

Debugging functionality is implemented not only for CTL, but for Java code too. Whole debugging experience should be really familiar if you ever debugged Java code in the Eclipse.

I won't repeat content of a mentioned blog, which describes all available functionality, this should only be a short brain dump to remind me that CTL debugging is in CloverETL designer out of box, you need just put 'breakpoint' into your ctl code and run your graph in Debug mode. What I keep forgetting is if you need to debug Java code, you need to enable Java debugging in Window > Preferences. (It uses communication on port 4444 by default, so if you are running CloverETL on a tightly locked box you would need to enable that port on your firewall!)

Allow me to steal picture worth of thousands words:

Tuesday, July 25, 2017

CloverETL Releases Version 4.7.0.M1

Version 4.7 Milestone release is all about this new feature called a Data Service. Unfortunately, you don't have access to the new data service unless you have already purchased a CloverETL Server or Cluster (or are planning to). Let's talk about what exactly the data service is...

The developers have given end users the ability to expose any graph as a REST API endpoint so users can create smarter applications. The REST service will allow you to process data, perform business logic, or create custom transformations at runtime and expose the result set to your application.

When I think about this service, it's truly amazing. Let's take Salesforce as an example. If you want to get fancy, you can create a force.com application for Salesforce that's built using Apex (a proprietary Java-like programming language). Let's say you would like to integrate your customer support tickets with the Salesforce accounts, but don't want to store that information in Salesforce (because, well it costs money). So, you can create a force.com application within Salesforce, expose a CloverETL graph as a REST API that will go to your customer support portal, get the latest tickets from the account in Salesforce where the request was made, and display all of the tickets for the customer on-demand! That is just one example of where the data service would be relevant.

There are a number of other use cases that I would be happy to share of where the Data Service could be relevant. Think big: IoT, Edge Computing, iPaaS.

One other improvements from this release:
-Graphs and jobflows are now automatically validated immediately after a change is made in your designer. If you have ever executed a graph and received an error, you know that the error symbol always remained on the component until you actually saved your graph again to reset the state. That has been improved.

Thursday, March 30, 2017

CloverETL 4.5.0 is released!

I haven't written in a while and wanted to update everyone on what is new in the latest CloverETL production release (4.5.0). I will prioritize the new features from most functional and powerful, to less exciting (in my opinion of course).

1. Restart-ability of jobs/graphs from the CloverETL Server. Have you ever run a job and it failed? I hope so because you cannot truly develop and/or test a solution properly without failing. Now, you can restart jobflows and graphs from within your Server Console. If you passed parameters into the graph/jobflow, you can re-use the same values! This is a great accomplishment and something that the users desperately needed. Just think, you are responsible for operations of the CloverETL solution, and your job fails due to a network outage from connecting to the database. You can easily restart the job and continue on with your day.
Restart a graph from CloverETL Server

2. Salesforce Wave Connector - If you are immersed in the Salesforce ecosystem, you will undoubtedly heard of Wave Analytics. Now, CloverETL has added support for writing directly to your Wave Analytics dashboard directly from CloverETL.

Salesforce Wave Analytics Component

3. Event Listeners UI change - The CloverETL developers have now consolidated all listeners into one main Event Listeners screen where you can see all of your listeners in one view.

4. Added batch size and concurrency mode to the Salesforce components. Now, you can limit how much data and if you want to send everything in parallel or serial mode. You may have not noticed this to be an issue for you, but if you are seeing errors when you are writing to Salesforce, try these new features out to control data being written to Salesforce. 

As always, make sure you plan for the upgrade! If you have a development environment, upgrade in your development environment before you try it out in production. 

Let me know if you have any questions or would like to see the full list of changes.