Data loaders
Data loaders collect operational data from data sources at regular intervals. EGO data loaders load the data into tables in a relational database, and IBM® Spectrum Conductor data loaders load the data into elasticsearch.
Data gathering methods
The data loaders use different methods of gathering data, depending on the types of data sources from which the data loaders gather data. They handle daylight savings automatically by using Greenwich mean time (GMT) when gathering data.- Collect
- A data collecting loader has full control over what data is gathered from the data sources.
- Retrieve
- A data retrieving loader does not have full control over what data is gathered and needs to send a request to the data sources. The data sources send the requested set of logs or events back to the data loader.
- Sample
- A data sampling loader does not have full control over what data is gathered and needs to send a request to the data sources. The data sources send the requested system status information back to the data loader.
Default behavior
| Data loader name | Data type | Data gathering interval | Data loads to this parameter | Loader type | Data gathering method |
|---|---|---|---|---|---|
| Consumer resource (egoconsumerresloader) | resource allocation | 5 minutes | CONSUMER_DEMAND CONSUMER_RESOURCE_ALLOCATION CONSUMER_RESOURCELIST |
polling | sample |
| Dynamic metric (egodynamicresloader) | host-related dynamic metric | 5 minutes | RESOURCE_METRICS RESOURCES_RESOURCE_METRICS |
polling | sample |
| EGO allocation events (egoeventsloader) | resource allocation | 5 minutes | ALLOCATION_EVENT | polling | sample |
| EGO service instance metric (egoservicesimetricsloader) | service instance related usage metrics | 1 minute | EGO_SI_RESOURCE_METRIC | polling | sample |
| Static attribute (egostaticresloader) | host-related static attribute | 1 hour | ATTRIBUTES_RESOURCE_METRICS RESOURCE_ATTRIBUTES |
polling | sample |
| Data loader name | Data type | Data gathering interval | Data gathering method |
|---|---|---|---|
| Spark resource usage (sparkresusageloader) | Spark activity resource usage metrics | 30 seconds | sample |
| Spark activity information (sparkactivityinfoloader) | Activity information | 3.5 minutes | sample |
| Spark resource usage aggregation (sparkresusageaggloader) | Spark application resource metrics | 10 minutes | sample |
Interactions
The scheduling of the data loaders is controlled by the loader controller service. Sampling data loaders request cluster operation data from the data sources while other data loaders obtain it directly. The data loaders store this information as raw data in tables within the relational database.