In this lesson you will
- Read multiple files using a wildcard (filename*.csv)
- Enable “debug mode” on edges to inspect data passing through (Data Inspector)
- Use Reformat to manipulate data (to remove dashes ‘-‘ from account numbers)
- Learn to use CTL – CloverDX Transformation Language (if-then and basic string functions)
- Write rejected records to a file
Reading files using a wildcard (file*.csv) 0:57
Readers (in our case the UniversalDataReader for CSV, TXT and other text or binary structured data) support reading multiple files or even wildcards using ‘*’ for ‘any’ at once. Files are read in sequence.
Debugging - Data Inspector 1:29
You can use Data Inspector to view data on components and edges. Typically, you’d use it to check a reader or data flow through edges. Remember that you need to turn on “Debug” for edges to collect data while the transformation is running before you can view the data.
Right-click the edge you want to debug.
Select “Enable Debug“.
Run (Don’t forget the transformation needs to re-run in order to collect debug data!).
Finally, switch to Data tab and click on edges or components you want to inspect.
Programming in Reformat (CTL) 3:53
Many components (Reformat being one of the most common) support adding pieces of code to add business logic.
In our example, we’ll be checking Account_number for dashes (‘-‘). If we find them, we will remove them by concatenating parts of the string to the left and to the right of the dash and finally we’ll convert the result from string to integer.
Basics of CTL
- Please refer to CTL – Language Reference