Historical Data for Analytical Tooling Extended-(HDAT-Ext)

  • Domain Portfolio: Observations
  • Domain: Historical
  • U Sage Classification: Restricted
  • Geography: Global
  • Attribution Required: No
  • Attribution Requirements: n/a

Overview

The "Historical Data for Analytical Tooling - Extended" API outputs training data for use by Data Science teams to perform Initial Data Exploration and train Machine Learning models.

HTTP Headers and Data Lifetime - Caching and Expiration

For details on appropriate header values as well as caching and expiration definitions, please see The Weather Company® Data | API Common Usage Guide.

Details

Once models are trained using data provided by this API, Predictive Analytics is often performed using the forecast counterpart of this API, Forecast Data for Analytical Tooling. The data provided by this API consists of the Average, Max and Min values, for each of 7 daily “periods” (expressed in local time relative to the postalKey or geolocation supplied in the call), for each of the specified weather aspects (“products”), for each day in the date range specified in the call. See Appendix A for a visualization of these aggregations and the daily periods over which they occur.

New data arrives hourly, and min/max/avg values for a given “daypart” (daily “period”) become available as soon as there are sufficient hours of data to compute them. The lag time of data availability is approximately 2 hours. The data used in this API is composed of 11 products from the gCOD (gridded Currents on Demand) dataset, and 2 products from the AgE (Agriculture and Energy) dataset. Datasets, and even weather products within datasets, have varying start dates, as can be seen in the Data Dictionary for Product IDs section of this document. Given that the dayparts need certain hours of data to compute their min/max/avg values, and that a given queried point will imply an offset from Zulu time, it is important to note that queries near the arrival of a product in a dataset may have incomplete dayparts which would not be present in the response.

A paging mechanism is employed to allow clients to fetch years of historical data while keeping individual HTTP transactions within a time period that Weather’s authentication mechanisms will allow. On the first request the next query parameter is omitted or empty. If the request results in too much data to be satisfied in a single HTTP transaction a X-Next-Token header is returned in the HTTP response. This token may be echoed to subsequent calls to the same URL to fetch subsequent data. When no data remains to be fetched the X-Next-Token is not set.

This API only supplies a CSV response. The CSV response is formatted such that the first row of data defines the columns for all subsequent values. It is up to the client to determine how to handle entries for which there was no data (e.g. if not all weather products were selected). See Appendix B: CSV Sample and Appendix C: CSV Column Titles.

The products endpoint can be used to discover valid values for the productId parameter to the data endpoint. The (case-insensitive) word “all” can be used to include all available products in the response.

The characteristics of the underlying data can be found in a subsection of the HoD Utility API description.

For periods where there is no data, the data will be represented by an empty field. For averaged time periods, the average is derived from the hourly samples that are collected and does not include periods where no data is collected. For more, see: Quick Reference and Best Practices .

Why is it called “Extended” / How does it differ from the original HDAT API?

This API is the extended version of the original, because it offers additional layers of data. The details can be discovered by calling the Product Discovery endpoint, which reveals the expressions that can be used as values for the productId request parameter to the Data Request endpoint.

URL Construction

Atomic API URL Examples:
https://api.weather.com/v3/wx/observations/historical/analytical/ext?geocode=39.86,-104.67&productId=Evapotranspiration&startDate=20200310&endDate=20200315&language=en-US&units=s&format=csv&apiKey=yourApiKey
Data Request by Postal Key: Required Parameters:postalKey, productId, language, units, format, apiKey ,startDate, endDate, | Optional Parameter: next
https://api.weather.com/v3/wx/observations/historical/analytical/ext?postalKey=81657:US&productId=Evapotranspiration&startDate=20200310&endDate=20200315&language=en-US&units=s&format=csv&apiKey=yourApiKey
Product Discovery: Required Parameters: apiKey
https://api.weather.com/v3/wx/observations/historical/analytical/ext/products?apiKey=yourApiKey
Data Request by Geocode: Required Parameters:geocode, productId, language, units, format, apiKey, startDate, endDate | Optional Parameter: next

Valid Parameter Definitions

Parameter Name Valid Parameter Value Description
geocode ex: 42.0,-89.5 The GPS coordinates (or the latitude and longitude) of a physical address
postalKey 81657:US Colon-delimited Postal code and Country Code combination
productId Any productId returned by the Product Discovery endpoint Weather aspect for which aggregate values are to be retrieved. Can be individual (single) product, comma delimited list of multiple products, or the (case-insensitive) word “all” to get them all. Derived from /products endpoint
startDate 20180101 ISO8601 formatted date
endDate 20180101 ISO8601 formatted date
next 20180101 or empty Page token
language en-US Only en-US supported at this time
units s (SI: Système international d'unités) The unit system in which numerical values should be expressed
format csv Output format = csv is valid

Data Dictionary for Product IDs

Weather Variable Description Unit First Available
MSLP Hourly mean sea level pressure. Archive begins from July 17, 2017 Pa 2017-07-17 15:00 Z
RelativeHumidity The relative humidity of the air, which is defined as the ratio of the amount of water vapor in the air to the amount of vapor required to bring the air to saturation at a constant temperature. Relative humidity is always expressed as a percentage. % 2015-06-29 16:00 Z
Temperature The temperature of the air, measured by a thermometer 1.5 meters (4.5 feet) above the ground that is shaded from the other elements. K 2015-06-29 16:00 Z
UVIndex UV index 0 to 2 is Low, 3 to 5 is Moderate, 6 to 7 is High, 8 to 10 is Very High, 11 to 16 is Extreme Enumerated Integer 2015-06-29 16:00 Z
Dewpoint The temperature which air must be cooled at constant pressure to reach saturation. The Dew Point is also an indirect measure of the humidity of the air. The Dew Point will never exceed the Temperature. When the Dewpoint and Temperature are equal, clouds or fog will typically form. The closer the values of Temperature and Dew Point, the higher the relative humidity. K 2015-06-29 16:00 Z
FeelsLike

feels like temperature.

An apparent temperature. It represents what the air temperature "feels like" on exposed human skin due to the combined effect of the wind chill or heat index.

K 2015-06-29 16:00 Z
WindSpeed The wind information reported in the hourly current conditions corresponds to a 10-minute average called the sustained wind speed. Sudden or brief variations in the wind speed are known as "wind gusts" and are reported in a separate data field (gust). m/s 2015-06-29 16:00 Z
Visibility The horizontal visibility at the observation point. Visibilities can be reported as fractional values particularly when visibility is less than 2 miles. Visibilities greater than 10 statute miles(16.1 kilometers) which are considered “unlimited” are reported as “999”; in your feed. You can also find visibility values that equal zero. This occurrence is not wrong. Dense fogs and heavy snows can produce values near zero. Fog, smoke, heavy rain and other weather phenomena can reduce visibility to near zero miles or kilometers. m 2015-06-29 16:00 Z
Gust The maximum expected wind gust speed. m/s 2015-06-29 16:00 Z
PrecipAmount Hourly rate of observed liquid precipitation in millimeters per hour (mm/h) mm/h 2015-06-29 16:00 Z
SnowAmount Hourly rate of observed snowfall in meters per hour (m/h) m/h 2015-06-29 16:00 Z
Evapotranspiration Reference crop evapotranspiration. This is a rate that gives the amount of water lost by a reference crop (a hypothetical grass surface) in units of mm of water per hour. mm/hr 2020-03-04 17:00 Z
GlobalHorizontalIrradiance The total amount of shortwave (solar) radiation received by a horizontal surface on the Earth. W/m2 2020-03-04 17:00 Z

HTTP Headers/Data Elements & Definitions (See below for full list of headers)

Header/Field Name Description Type Range Sample Nulls Allowed
X-Next-Token

An opaque string that the caller of the HDAT api may include as the query parameter next to fetch data that could not be returned in an earlier data request.

By “opaque” we mean that the string shall have no necessary meaning to the caller other than that it was returned by the HDAT service and may be echoed back to that service to continue data iteration.

The empty string submitted as a request shall mean that the first page of data is desired.

The empty string returned as a response shall mean that no further data is available.

[string] Any valid HTTP header string. 20180101 Y

Appendix A: Daily Aggregation Periods

Appendix A: Daily aggregation periods

Appendix B: CSV Sample

All numerical data are decimal numbers.
Appendix B: CSV sample

Appendix C: CSV Column Titles

Appendix C: CSV column titles