Running a streams flow and monitoring its metrics
After you run your streams flow, the Metrics page will provide information about the general health of your streams flow, real-time metrics, runtime views, and embedded logging and error notification.
Let’s take a deeper look at what’s in the Metrics page. The following screen capture is an example.
-
Streams Flow pane shows a dynamic, bird’s-eye view of the data as it flows among operators in your streams flow. You can see the throughput of the flow. In the previous example, the mouse pointer is hovering over the flow between the Sample Data and the Filter operators. While the pointer is hovering over a flow, you can see the current throughput and the total events of that flow. You can drill down even further to view the events that flow in the Flow of Events table.
-
Flow of Events table shows the events that flow between operators in both table and JSON formats. This table opens when you click the data flow between operators in the Streams Flow pane. This table is helpful if you need to understand the data better or to tweak an operator. In the previous screen capture, two tables are shown because both the flow between the Sample Data and the Filter operators and the flow between the Filter and the Debug operators were clicked.
-
Ingest Rate graph shows the number of events that are submitted to the streams flow per second. For example, a Sample Data operator might submit one million events per second to the streams flow. If several Sample Data operators submitted data, then the ingest rate is the sum of the events of those operators.
-
Throughput graph shows the throughput of a selected operator’s data flow (input, output, or both), the number of dropped events from the network, and the number of events that are not valid for any reason. In the previous example, the Filter operator is selected and the input and output flows are shown in the graph. You can zero in on different operators in the streams flow to see their input and output rates, and to see if any errors are occurring.
From the Metrics page you can start, stop, duplicate, and save a streams flow. If your streams flow needs to be tweaked to give you the exact information that you need, you can go directly to the edit canvas. The Notifications icon can indicate compilation or runtime errors. In addition, you can view the job graph.
Prerequisites
Procedure
To monitor or work with a streams flow, perform the following steps:
-
In the Metrics page, you can do any of the following actions:
a. Make a duplicate of the streams flow, and then save it with a different name in the same project folder.
b. Save the streams flow to your code archive. The file is in STF format. Then, you can create a new streams flow in a different project by importing the saved file.
c. Editing and saving a streams flow creates an updated version with the same name. If the streams flow is running and you edit and save the streams flow, a new draft of the streams flow is saved. You can redeploy with the updated streams flow.
d. Show the throughput, number of events, and ingestion rate by doing the following actions.
-
To see all operators and the flow of data between them in the streams flow, see the Streams Flow pane. Hover your mouse pointer over a data flow to see its throughput rate and total number of events per second. When the Status is in “running” state, each data flow has a distinct color.
-
To see the number of events that are submitted to the streams flow per second for each streams flow source, see the Ingest Rate graph. Each streams flow source has a distinct color.
-
To see the throughput of a specific operator, click the operator in the Streams Flow pane. The throughput of the operator is displayed in the Throughput graph. You can see the number of events that flow into or out from the specific operator, the number of events that are dropped from the network, and the number of events that are not valid for any reason.
-
To see the events in a data flow, click anywhere in the flow. The Flow of Events table opens to show events in table and in JSON formats. Use this table to see what events are going into and out of an operator. This information is useful to debug your streams flow by comparing the incoming and outgoing events from a specific operator.
e. The messages there indicate as clearly as possible what and where the problem is. The icon is enabled only when the streams flow has errors. See Troubleshooting streams flow for specific types of errors.
In the Notifications taskbar, click any of the following icons:
-
Streams instance icon
to check the instance. In the Manage page, you can view the job graph. -
Save user log icon
to save the user log file.
The user log contains logging messages that you put into the Code operator and the Python Model operator. -
Save logs icon
to save system log files. The log files are saved to your local disk in a compressed format. These logs are needed if you contact Support.
g. Delete the streams flow. When a streams flow is running, clicking on the Delete icon in the Metrics page will stop the running job before deleting the streams flow.
-