Preserving an audit trail of events

This solution describes how to design a flow that preserves an audit trail of flow and stage events that occur.

Store event records to preserve an audit trail of the events that occur. You can store event records from any event-generating stage. For this solution, say you want to keep a log of the files written to HDFS by the following flow:

Flow with a Hadoop FS target

To do this, you simply:

Configure the Hadoop FS target to generate events.
On the General tab, select the Produce Events property

Now the event output stream becomes available, and the target generates an event each time it closes a file. For this target, each event record includes fields for the file name, file path, and size of the closed file.
You can write the event records to any target, but let's assume you want to write them to HDFS as well:

You could be done right there, but you want to include the time of the event in the record, so you know exactly when the Hadoop FS target closed a file.
All event records include the event creation time in the sdc.event.creation_timestamp record header attribute, so you can add an Expression Evaluator to the flow and use the following expression to include the creation time in the record:
```
${record:attribute('sdc.event.creation_timestamp')}
```
The resulting flow looks like this:

Note that event creation time is expressed as an epoch or Unix timestamp, such as 1477698601031. And record header attributes provide data as strings.

Tip: You can use time functions to convert timestamps to different data types. For more information, see Functions.