Cloudability Cloudability Cost and Usage Data Availability in Reporting

Overview

To report on Cloudability cost and usage data, the billing files from the vendor must first be processed through our data pipeline. The process begins with raw cost and usage data ingestion. Once that data is ingested, it goes through a series of data normalization, transformation, aggregation, and decoration steps before being made available in reporting.

Cost Pipeline Details

Cost and usage data ingestion in Cloudability begins after successful vendor credentialing where you grant Cloudability access to your billing data. Once Cloudability confirms that it has appropriate permissions from the credentialing process, it begins retrieving data from the vendor. Cloudability checks for new data multiple times a day for each credentialed account. As soon as new data is discovered on the vendor side, the ingestion process begins to pull the latest available data.

After the raw billing data ingestion, Cloudability ’s data pipeline proceeds with various data processing steps. This next stage of the pipeline is responsible for enriching the raw data with many of the valuable data points and measures you use every day, such as tag mappings, business dimensions, account groups, and business metrics among others. Also, if you have clusters provisioned for use with Cloudability Container Insights, this is when container cost allocation is performed. Finally, the resulting processed and decorated cost data is loaded into our data platform where it can be used by Cloudability reporting, dashboards, and Apptio BI.

Frequently Asked Questions (FAQ)

What impacts my data processing time?

Data processing time is directly impacted by data volume. In addition the overall vendor raw file size and record count per credentialed account, each of the following are important considerations regarding the processing time:

• Number of provisioned clusters and associated pod count (if provisioned w/ Cloudability ).

  • Container cost allocation generates additional granularity which leads to many more records.

• Customer-defined dimensions such as Business Mappings and Tag Mappings.

  • Mapping quantity and statement complexity.

• Time of the month – as month goes on, MTD data volume increases.

  • It is common for end of month processing to take longer due to data volume being at its peak.

When can I expect my new data to be available for reporting?

With respect to the inconsistent frequency in which the billing files are updated by the vendor and the variability of the overall data size per customer, we cannot guarantee an exact time that the new data will become available in Cloudability. Generally, this is within an average of 24 hours after a new billing file update is published by the vendor.

What is a reprocess?

A typical customer action is a reprocess , which is a manually triggered action that performs the post-ingestion processing again on historical data . This means that we do not ingest any new vendor data during this operation, but instead process it through the latter part of our pipeline to decorate and transform the data again with updated information. A common use case for a reprocess would be to update Business Mappings on historical data in Cloudability. A reprocess is not necessary during the current month as it is processed automatically as explained previously. Reprocesses are performed in month units, and can take longer than the business as usual processing depending on the number of months requested.

What is a refetch?

A refetch is where we go through the entire pipeline process again, including retrieval the raw vendor data – this should only be performed in rare circumstances such as backfilling missing vendor data due to an error. Or, onboarding a new payer account in which historical data exists in the vendor’s storage and needs to be pulled into Cloudability. If required, work with your TAM or open an Apptio Support case.