QHMM overview

Query History Monitoring and Management (QHMM) is a service that stores and manages the diagnostic data such as, queries history and query event-related information of the Presto(Java) and Presto(C++) engine in a storage bucket. You can retrieve the stored history files for analysis, debugging and monitoring purpose.

QHMM primarily aims to address the issue of data persistence for serviceability data. When the engine restarts or goes offline, there are chances of losing valuable diagnostic data they generate. QHMM resolves the issue by storing such data in object storage solutions like Cloud Object Storage (COS), Amazon S3, Google Cloud Storage (GCS), or Minio buckets. The data is organized in a structured folder hierarchy, making it easily accessible for users to retrieve and analyze.

QHMM allows to retrieve the following diagnostic data:
  • Logs generated for the following query events in Presto:
    • Query created event - event logged when a query is initiated.
    • Split completed event - split correspond to an individual task in a query execution. An event is logged when a split or a task is completed.
    • Query completed event - event logged when a query execution is completed.
    • Query Optimiser event - event logged when the query optimizer is enabled.
  • Histories of the queries executed by Presto in the form of a json file.
  • Query history table created where user can execute a query to view the histories of the queries executed by Presto.
  • Heap dump by using API.

Data organization

QHMM maintains a structured folder hierarchy within the chosen object storage solution to store data efficiently. The structure is as follows:

<base_path>/ <wxdInstanceID>/ <engine or service type>/ <engine or service ID>/ <recordtype>/ dt=<dd-MM-yyyy> <records>

  • <base_path> : Default base_path is qhmm and you can change the path from the QHMM configuration page.
  • <wxdInstanceID>: A unique identifier for the instance.
  • <engine or service type>: The type of engine sending the data.
  • <engine or service ID>: A unique identifier for the engine.
  • date<dd-MM-yyyy>: The date on which the data is recorded.
  • <recordtype>: The type of diagnostic data (e.g., query history, query events, dumps).
  • <records>: The actual data records stored.

To store query history details, one more folder is added: <base_path>/ <wxdInstanceID>/ <engine or service type>/ <engine or service ID>/ <recordtype>/ dt=<dd-MM-yyyy> <user> <records>

  • <user>: The username of the person who executed the query.