Wednesday, September 28, 2016

CloverETL and Football - Are they related?


I was recently asked to compare CloverETL to football, and I thought this would be a fun exercise to share with a wider audience. This concept of relating normal everyday things to technology is not uncommon, and in fact has been scientifically proven that the concepts are easy to learn if you can relate them to the individual who are trying to comprehend all of the terms. Why is this the case? Most research shows that if you can speak the same language it’s easier to understand and comprehend the topic. That makes sense doesn’t it, but why are more people not doing this? The answer: it’s very difficult to find a commonality between your audience, so you have to make more generalizations than you would like. However, I don’t have that problem here because I am only talking to people who may not understand everything about CloverETL, but understand American football (the world’s most popular sport).

Web Application Container (Tomcat, JBOSS, etc) - A web application container will host web applications. When relating this term to football, I think of the stadium for the same reason. The stadium is hosting football games. Fairly straightforward and simple.

Web Application - A web application is a software application that runs in a web browser. When relating this term to football, I think of it as an actual act of playing football because this can happen in many different formats: NFL, Canadian Football, Soccer, Rugby, and many others could be considered in this.

Java - A popular programming language that is used for creating distributed applications. I think this relates to the entity of the NFL because the NFL is the governing body of American football and is a particular brand of football. Java is one of the most widely used programming languages today, and the NFL is a brand of American football that is the most popular professional sport in the world.

CloverETL Designer - The CloverETL Designer is an engine-based ETL desktop application where a user can utilize the drag-and-drop interface to build data workflows. The CloverETL Designer can be thought as the Head Coach because the head coach must build all the data workflows from the ground-up. Yes, they will utilize others to help with the process, but the entirety of the CloverETL Designer acts as a head coach would for a football team.

CloverETL Server - The CloverETL server is a web server application which can schedule and orchestrate various CloverETL jobs. The server is in charge of managing the entire load of the system and defining all aspects of your data integration workflows. The analogy for the CloverETL Server in football terms, in my opinion, would be the General Manager. The general manager is in charge of all aspects of football operations: player management, coach management, contracts, salary cap, and many other areas.

CloverETL Cluster - The CloverETL cluster is a set of CloverETL server instances running concurrently to improve performance, add high availability, and allow for greater scalability. The CloverETL Cluster can be related to an NFL Franchise owner because they are always looking for ways to improve performance and efficiency throughout the entire organization. They are also responsible for managing all personnel as well as all financial implications that come with running a business.

CloverETL Engine - The CloverETL engine is an embedded application which is responsible for the execution of all graphs/jobflows. This closely resembles the Quarterback of the team. The quarterback must be the interpreter of all plays on/off the field for their teammates much like the engine would do as it receives instructions from both the Designer, Server, or Cluster.

Sandbox - The Sandbox is where the CloverETL project lives. This will store all jobflows, graphs, database connections, metadata, and data files. Essentially, the sandbox will contain all information that will be used for a particular project. I think this mostly resembles the playbook that is used by the coaching staff and players to execute on the field. The playbook contains all plays, options for plays, personnel, and scouting reports for a successful game.

Jobflow - The CloverETL jobflow is the orchestration layer for conditional job execution. I believe this relates to a game plan that is designed by the coaches for a successful game. This requires proper architecting and planning using the playbook to come up with the best possible game plan to be successful on the field.

Graph - The CloverETL graph is defined as a workflow that is designed for a specific business rule. This is typically where CloverETL interacts with data -- reading from data sources, transforming data, and writing to another data source to satisfy a business requirement. When you think of a CloverETL graph in terms of football, I would consider a graph as a play that is called by coaches and executed by the players. Each play must be carefully architected and executed as designed in order to be successful, but is only a small piece of the entire solution (or game plan).

Palette - The CloverETL Palette is a pre-defined list of components which are available for drag-and-drop use to design your CloverETL graphs/jobflows. I like to think about the palette as your roster where you can utilize available players for use in plays.

Component - A CloverETL component is an out of the box functionality that is pre-programmed to complete a specific task. Due to the nature of what a component actually does, it’s only natural to relate a component to an individual player. Each player has a position, skill, and task that they are given when they are signed onto a team.

Edge - An edge in CloverETL connects components and is where data will flow along. This is more of a conceptual topic than a physical topic which is why I am relating an edge to a Coordinator who will tell each of the players know how they must interact with each other. They must connect 2 individual players (or components), but also have to be aware of the entire play (or graph). If you are looking for more of a physical comparison, I would have to say an edge would be considered like the football. The football directly relates to data that will be passed along the edge. This is the most precious item in the environment (both football for the game, and data for integration) and should be handled with care.

Metadata - The CloverETL definition of metadata is the structure of data that connects two components. This is probably the most difficult concept to relate to football, but I think the closest thing in football to this is the huddle. The huddle is where a play is called in from the sidelines to the quarterback and the quarterback must let the other players know the play call as well as be able to describe the play to individual players if they do not understand the call. This concept physically defines how the players will interact during the next play, and the huddle is where the understanding occurs.

Phases - The CloverETL phasing defines in order the execution order of components in graphs/jobflows. I think this closely resembles a down in football because downs must also go in order. A team has 4 plays to reach the 10-yard mark before they turn the ball over.

Successful Execution - A successful execution for CloverETL means that your graph and/or jobflow was successful in running. You did not have any errors or unforeseen consequences as a result of your design. In my opinion, this would be equivalent to scoring a touchdown on a drive. Some may think that this would mean winning the game, but you must remember that this is only a short term victory and you must continue to improve, expand and define other ways to be successful.

Failed Execution - A failed execution for CloverETL means that a graph and/or jobflow failed to execute as designed. This could be a problem with the design of the graph or an unforeseen consequence in your solution. This directly relates to a turnover in football as it was a result that you did not want or plan for. This is where you have an opportunity to improve, learn, and grow from this experience both on the football field and when designing your jobflows/graphs.

To sum this up, are all of my analogies perfect, probably not. Would you like to argue against some of my logic? I hope so because that means that I sparked your interest and did my job.

No comments:

Post a Comment