New from Inquidia Labs: Log Helper for Pentaho Data Integration

Wednesday, December 7, 2016 - 07:00

Earlier this year, Inquidia announced Inquidia Reflex℠, a set of best practices for Pentaho Data Integration (PDI) to ensure consistent and quality delivery of PDI projects. Among these best practices are several related to logging enabling engineering and operations to understand what is happening when a transformation is executed, including:

  • Parent level jobs should always log the parameters they are using to the log. This enables operations to see the values that were used when running the job, settings that would otherwise be invisible.

  • Job logs should be stored so that the operations can use them to review job results and perform necessary troubleshooting. When possible the log or a pointer to the log should be included in failure notification systems.

Pentaho Data Integration is a powerful tool, and both of these best practices can be accomplished out the box. The writing parameter to a log best practice can be accomplished through the use of a write to log step. However, anytime parameters are added, this step must be manually updated. Including the log on failure notifications can be accomplished through result files, but this requires multiple steps, forces the inclusion of the entire log, and does not enable you to filter for only the error messages.

This is why Inquidia Labs developed the newly released Log Helper plugin for Pentaho Data Integration. This plugin is a set of 5 steps to make these and other logging needs easier and faster to implement while reducing associated maintenance. These steps include:

  • Write Variables to Log Job Entry - This job entry gets all of the variables or parameters that are defined in your environment and writes the name and value to the log. It includes filters to choose between parameters and all variables. The job entry also contains a regex filter on the variable names.

  • Write Variables to Log Step - Same as the Write Variables to Log Job Entry, but it works in transformations.

  • Send Log to Variable Job Entry - This job entry will get the job’s log and add it to a variable. The variable can then be used in notifications allowing you to include the actual log on email, Slack, or other notification channels. This job entry includes configurable limits on the number of log lines and log levels.

  • Write Result Rows to Log Job Entry - Result rows are a concept in Pentaho used to pass data between jobs and transformations. This job entry allows you to see these rows by writing them to the log, something that would normally require a sub-transformation.

  • Write Result Files to Log Job Entry - PDI maintains a list in memory of files that were touched as part of a job. This job entry writes these result file names to the log.

Inquidia Labs is proud to announce the immediate availability of the Log Helper Plugin on GitHub here and via the Pentaho Marketplace.

 

Contact us today to find out how Inquidia can show you how to collect, integrate and enrich your data. We do data. You can, too.

Would you like to know more?

Sign up for our fascinating (albeit infrequent) emails. Get the latest news, tips, tricks and other cool info from Inquidia.