Until now, we have looked at the normal way a component is used. However, most components have very powerful features that have arisen out of the need to process real world data and such data comes in all shapes, sizes and flavors.

Fancy Reading

Let's take a quick look at the way we can process multiple files that need to be read in, for example. In the File URL property of the Reader, we open up the dialog and can then specify either a group of files individually or even a folder where we might expect to find any number of files (of the same type probably) and we want to process the lot of them.

Here is the screenshot of the way we set up the reading of 2 files, cust0001.csv and cust0002.csv. Note for anything to be selected, it has to be in the File URLs list at the bottom of the dialog.

To read ALL files in the folder, the easiest way is to just set the property directly in the main component property list. See below and note how we have set it to ${DATAIN_DIR}/batch/*.*

Fancy Writing

Something you may very well want to do is to read in a large file and then generate separate output file names based on a field in the main dataset. Using our familiar customers file, let's say we want to create an output file for each country we encounter in the Country field of the customer data. This is called partitioning and in order to explain how this is done I suggest you watch the video as explaining it as text would require you to read a lot of dull lines of text..