Wednesday, September 28, 2016

CloverETL and Football - Are they related?


I was recently asked to compare CloverETL to football, and I thought this would be a fun exercise to share with a wider audience. This concept of relating normal everyday things to technology is not uncommon, and in fact has been scientifically proven that the concepts are easy to learn if you can relate them to the individual who are trying to comprehend all of the terms. Why is this the case? Most research shows that if you can speak the same language it’s easier to understand and comprehend the topic. That makes sense doesn’t it, but why are more people not doing this? The answer: it’s very difficult to find a commonality between your audience, so you have to make more generalizations than you would like. However, I don’t have that problem here because I am only talking to people who may not understand everything about CloverETL, but understand American football (the world’s most popular sport).

Web Application Container (Tomcat, JBOSS, etc) - A web application container will host web applications. When relating this term to football, I think of the stadium for the same reason. The stadium is hosting football games. Fairly straightforward and simple.

Web Application - A web application is a software application that runs in a web browser. When relating this term to football, I think of it as an actual act of playing football because this can happen in many different formats: NFL, Canadian Football, Soccer, Rugby, and many others could be considered in this.

Java - A popular programming language that is used for creating distributed applications. I think this relates to the entity of the NFL because the NFL is the governing body of American football and is a particular brand of football. Java is one of the most widely used programming languages today, and the NFL is a brand of American football that is the most popular professional sport in the world.

CloverETL Designer - The CloverETL Designer is an engine-based ETL desktop application where a user can utilize the drag-and-drop interface to build data workflows. The CloverETL Designer can be thought as the Head Coach because the head coach must build all the data workflows from the ground-up. Yes, they will utilize others to help with the process, but the entirety of the CloverETL Designer acts as a head coach would for a football team.

CloverETL Server - The CloverETL server is a web server application which can schedule and orchestrate various CloverETL jobs. The server is in charge of managing the entire load of the system and defining all aspects of your data integration workflows. The analogy for the CloverETL Server in football terms, in my opinion, would be the General Manager. The general manager is in charge of all aspects of football operations: player management, coach management, contracts, salary cap, and many other areas.

CloverETL Cluster - The CloverETL cluster is a set of CloverETL server instances running concurrently to improve performance, add high availability, and allow for greater scalability. The CloverETL Cluster can be related to an NFL Franchise owner because they are always looking for ways to improve performance and efficiency throughout the entire organization. They are also responsible for managing all personnel as well as all financial implications that come with running a business.

CloverETL Engine - The CloverETL engine is an embedded application which is responsible for the execution of all graphs/jobflows. This closely resembles the Quarterback of the team. The quarterback must be the interpreter of all plays on/off the field for their teammates much like the engine would do as it receives instructions from both the Designer, Server, or Cluster.

Sandbox - The Sandbox is where the CloverETL project lives. This will store all jobflows, graphs, database connections, metadata, and data files. Essentially, the sandbox will contain all information that will be used for a particular project. I think this mostly resembles the playbook that is used by the coaching staff and players to execute on the field. The playbook contains all plays, options for plays, personnel, and scouting reports for a successful game.

Jobflow - The CloverETL jobflow is the orchestration layer for conditional job execution. I believe this relates to a game plan that is designed by the coaches for a successful game. This requires proper architecting and planning using the playbook to come up with the best possible game plan to be successful on the field.

Graph - The CloverETL graph is defined as a workflow that is designed for a specific business rule. This is typically where CloverETL interacts with data -- reading from data sources, transforming data, and writing to another data source to satisfy a business requirement. When you think of a CloverETL graph in terms of football, I would consider a graph as a play that is called by coaches and executed by the players. Each play must be carefully architected and executed as designed in order to be successful, but is only a small piece of the entire solution (or game plan).

Palette - The CloverETL Palette is a pre-defined list of components which are available for drag-and-drop use to design your CloverETL graphs/jobflows. I like to think about the palette as your roster where you can utilize available players for use in plays.

Component - A CloverETL component is an out of the box functionality that is pre-programmed to complete a specific task. Due to the nature of what a component actually does, it’s only natural to relate a component to an individual player. Each player has a position, skill, and task that they are given when they are signed onto a team.

Edge - An edge in CloverETL connects components and is where data will flow along. This is more of a conceptual topic than a physical topic which is why I am relating an edge to a Coordinator who will tell each of the players know how they must interact with each other. They must connect 2 individual players (or components), but also have to be aware of the entire play (or graph). If you are looking for more of a physical comparison, I would have to say an edge would be considered like the football. The football directly relates to data that will be passed along the edge. This is the most precious item in the environment (both football for the game, and data for integration) and should be handled with care.

Metadata - The CloverETL definition of metadata is the structure of data that connects two components. This is probably the most difficult concept to relate to football, but I think the closest thing in football to this is the huddle. The huddle is where a play is called in from the sidelines to the quarterback and the quarterback must let the other players know the play call as well as be able to describe the play to individual players if they do not understand the call. This concept physically defines how the players will interact during the next play, and the huddle is where the understanding occurs.

Phases - The CloverETL phasing defines in order the execution order of components in graphs/jobflows. I think this closely resembles a down in football because downs must also go in order. A team has 4 plays to reach the 10-yard mark before they turn the ball over.

Successful Execution - A successful execution for CloverETL means that your graph and/or jobflow was successful in running. You did not have any errors or unforeseen consequences as a result of your design. In my opinion, this would be equivalent to scoring a touchdown on a drive. Some may think that this would mean winning the game, but you must remember that this is only a short term victory and you must continue to improve, expand and define other ways to be successful.

Failed Execution - A failed execution for CloverETL means that a graph and/or jobflow failed to execute as designed. This could be a problem with the design of the graph or an unforeseen consequence in your solution. This directly relates to a turnover in football as it was a result that you did not want or plan for. This is where you have an opportunity to improve, learn, and grow from this experience both on the football field and when designing your jobflows/graphs.

To sum this up, are all of my analogies perfect, probably not. Would you like to argue against some of my logic? I hope so because that means that I sparked your interest and did my job.

Wednesday, September 14, 2016

Evernote to Apple Notes


Migrate from Evernote to Apple Notes and keep notes tags, location and other metadata


Do you want to migrate your notes from Evernote to Apple Notes and keep all notes metadata like tags, location, web page source, author and other valuable information?
Until now, this was a problem, as Apple Notes do not support tags, location, notes author, web page source and other metadata. That information is lost during import of Evernote exported notes into Apple Notes desktop.
In an effort to constantly improve the lives of our existing and potential users, we prepared an example CloverETL graph that will migrate your Evernote notes into Apple Notes desktop app in four steps:

  1. Reading your exported Evernote notes (.enex file)
  2. Looking for notes metadata (note tags, location, author, note source, etc.)
  3. Copying those metadata into note text in user and machine readable form
  4. Generating a new .enex file, which can be easily imported into Apple Notes desktop

Here is a detailed user manual how convert your notes:
  1. Download and install free 45-days trial of CloverETL 4.3.0
  2. Download and import example CloverETL migration graph
  3. Export your notes from Evernote desktop
  4. Run example CloverETL migration graph
  5. Import your notes into Apple notes desktop


1) DOWNLOAD FREE 45-DAYS TRIAL OF CLOVERETL DESIGNER 4.3.0

You can download 45-days trial of CloverETL 4.3.0 for OSX / macOS, Windows or Linux here.

2) INSTALL CLOVERETL 4.3.0 DESIGNER

Installation of CloverETL Designer is pretty simple. In OSX open downloaded .dmg file and follow installation instructions. Once you accept user agreement, please perform typical OSX installation by copying CloverETL Designer App into Application folder.




When you open CloverETL Designer icon, you may receive security notification, please click on “Open”.


CloverETL Designer will ask you to chose disk and folder of workspace, where you will store:
  • CloverETL example graph
  • Notes you exported from Evernote (as .enex file)
  • Processed .enex file ready to be imported to Apple Notes
In this manual we use example folder /Users/YourUserName/Documents/CloverETLworkspace.

Once you register your copy of CloverETL Designer you can start to work with CloverETL Designer.


3) DOWNLOAD AND IMPORT EXAMPLE CLOVERETL MIGRATION GRAPH
Download example CloverETL project files: Evernote Example.

In CloverETL Designer chose “File –> Import –> CloverETL –> Import external CloverETL projects” and press button “Next”.


Select archive (downloaded) example file and press button “NextExample CloverETL graph is now imported into your CloverETL Designer. 



In you CloverETL workspace, you will see new folder:
/Users/YourUserName/Documents/CloverETLworkspace/EvernoteExample

There are several sub-folders. Three of them are important for you:
  • /data-in/ you will save all your Evernote exported .enex files there
  • /data-out/ CloverETL will create updated .enex files there
  • /graph/  CloverETL transformation graph is located there.
After import, go to left-upper part of CloverETL Designer screen called “Navigator”. Go to folder /graph/ and click on graph “EvernoteTransformer”.  You should see this screen:




4) CHANGE PROPERTIES OF CLOVERETL DESIGNER
In CloverETL Designer go to “CloverETL Designer -> Preferences -> CloverETL -> ETL Runtime” and:
  • Change parameter “Max heap size” into 3000 (MB) . This allows you to process large files up to 250 MB.
  • Set-up CloverETL properties file as "EvernoteExample/CloverETL.properties".
  • Click "Apply”.
  • CloverETL Designer runtime will be restarted (aprx. 10-15 sec.).



5) EXPORT YOUR NOTES FROM EVERNOTE DESKTOP
  1. Select group of notes or whole notebook (recommended) and select “Export Notes from”  (important: size of  one note including attachments should not exceed 25MB).
  2. Save your notes as Evernote export “.enex” export file into your CloverETL Designer workspace folder into /data-in/ directory in your workspace.
  3. You can save as many files as you want.
  4. Go back to CloverETL Designer.


6) Run example CloverETL migration graph
Click  on green “Play” button to execute ETL graph.




You will be asked to enter file name. Please enter name of one of your exported files in  /data-in/ directory in your workspace. Press “OK” to run graph.



Once graph run has been successfully executed, console window (bottom center window) turn light green and you will receive console message:
INFO  [JobFinalizer_5] Finished Status: FINISHED_OK”.




7) IMPORT YOUR NOTES INTO APPLE NOTES
  1. Open Apple Notes, chose “File -> Import to Notes ->”.
  2. Go to your CloverETL Designer workspace into /data-out/ directory.
  3. Select updated .enex file you want.

8) KNOWN LIMITATIONS

  • The graph was built in CloverETL Designer 4.2.0.
  • Size of one note including attachments should not exceed 25MB.
  •  Only one .enex file can be processed in one graph execution.
  • Encrypted Evernote notes are transformed blank.
  • Evernote encrypted notes are migrated blank.
  • The XSD schema of Evernote .enex file is predefined and ready to use without any changes.
  • Changes in notes design are done by Apple Notes app, not by CloverETL.


9) TROUBLESHOOTING:
Console message
Solution
File is unreachable:
You entered wrong file name of Evernote file to be migrated
SAX parsing exception
Realocation of CloverBuffer failed. 

Requested capacity is 99900090 and maximum capacity is 80000000.
Edit both values in file “/EvernoteExample/CloverETL.properties/

Restart CloverETL runtime or re-launch CloverETL designer
Java heap space Heap:
Go to “CloverETL Designer -> Preferences -> CloverETL -> ETL Runtime

Change parameter “Max heap size” into larger volume.  Please be noted, that CloverETL runtime max allocation depends on your macbook real RAM size

Restart CloverETL runtime or re-launch CloverETL designer


FINAL REMARKS  
Did you enjoy the experience with Clover? Let us know in the comments.

Do you want to use Clover for something different? Visit the academy (add hyperlink: http://www.cloveretl.com/learn/walkthrough/) to get a taste of how to create graphs yourself and start implementing your own solutions.
Remember, imagination is your only limit.