Keeping people safer in extreme weather

The Weather Company stays up and running with IBM Cloud
by Kristin Johnson
8-minute read

Many companies build their web platforms based on the assumption that predictable factors will drive the traffic they serve. For example, a retailer expects extended seasonal peaks around the winter holidays, with sharp spikes for major events such as Black Friday. Scaling to handle increased demand can be challenging, but long-term planning makes it a solvable problem.

Weather websites like weather.com® and wunderground.com®, however, face an exponentially harder scalability challenge — and their performance during periods of intense demand can literally spell the difference between life and death.

Even on a day of normal weather, the level of demand on the two sites — owned and operated by The Weather Company, an IBM Business — can verge on the extraordinary. Consider this: on an average day, The Weather Company processes over 400 TB of data (enough to fill 75 million 400-page novels in printed form), delivers more than 50 billion requests for weather information and produces 25 billion forecasts. And when severe weather strikes, that demand can shift dramatically, with page views surging from an average of 30 million to 150 million per day.

Chris Hill, Vice President and Chief Information and Technology Officer for IBM Watson Advertising and Weather, explains: “Our sites are unique because they add the greatest value when they’re operating at scale. The most critical time to provide accurate, timely information is during hurricanes, tornadoes and severe winter storms, because that’s when weather insight really helps people keep themselves and their families safe.”

During severe weather, scales to serve

5X

the average number of page views

On an average day, processes

400

terabytes of data

During major weather events, The Weather Company relies on its infrastructure to deliver exceptionally high performance when its servers are under the most intense load. Extreme weather systems are often fast-moving and difficult to predict, so to handle massive, short-notice peak workloads, the company depends on the ability to scale up rapidly at any time.

Since weather is a hyperlocal phenomenon, the scalability of the web platform needs to be localized, too. As The Weather Company expands its operations into markets such as Japan, India and Eastern Europe, it must be able to harness local multizone regions to bring its endpoints closer to the user to minimize latency.

IBM Cloud is the perfect engine to power the world’s largest weather websites and deliver the fastest, most accurate weather insight to millions of users around the globe.
Chris Hill
VP, Chief Information and Technology Officer, IBM Watson Advertising and Weather
Umbrella

Localizing its regions adds further complexity to the scalability challenge. Each local instance needs to be able to scale independently: a tornado in the US Midwest creates demand peaks in that geographical area, with little effect on operations elsewhere.

In 2012, The Weather Company started moving its infrastructure into the cloud, building most of its systems around the services offered by its cloud provider. The cloud platform helped address the company’s scalability challenges, but also locked it into using proprietary technologies.

By 2018, better options that use open-source software for building cloud solutions had become available — and one company’s solution stood out in that regard. “We chose to migrate to IBM Cloud because it’s based on open-source technology at the core,” says Hill, “so we get the advantage of our software staying current, with fewer person hours required to achieve that. Focusing on open source as the basis for many of our platform and product components, as well as for our frameworks, allows us to take advantage of massive communities that are growing and evolving those technologies. Ultimately, this is a big step in keeping our technology modern and up to date.”

Designing an open-source cloud platform

Prior to moving to IBM Cloud®, many of The Weather Company’s web servers were running in portable Docker containers on the company’s existing cloud platform, making them relatively straightforward to migrate to a different cloud provider. The team viewed the project as both an important first step in its journey to IBM Cloud and a strong test of the capabilities of the new platform.

The Weather Company made the critical decision to structure its new environment on the IBM Cloud Kubernetes Service, which orchestrates container management across an extensive cluster of IBM Cloud Virtual Servers. Kubernetes, the open-source container management platform that has recently become a de facto standard, provides a powerful, declarative way to define how containers should be started, shut down and monitored, and what hardware resources they can utilize. IBM provides Kubernetes as a managed service, significantly reducing the burden on The Weather Company’s DevOps resources.

“In a traditional microservices world, we’d need a lot more DevOps pipeline to handle all this management work,” says Hill. “Since we migrated to IBM Cloud using IBM’s managed Kubernetes service, we’ve seen a 20% reduction in the DevOps resources needed to manage the infrastructure.”

Server room
Umbrellas on a crowded street

To store static assets — such as images and videos — that weather.com and wunderground.com serve to users, The Weather Company uses IBM Cloud Object Storage. The storage solution provides highly compatible application programming interfaces (APIs) that make it easy to interoperate with other cloud storage platforms — reducing the need for The Weather Company to make significant code changes to its front-end web applications.

“IBM’s Cloud Object Storage product is a technology we’re really excited about,” says Hill. “It follows industry-standard interfaces, so integration was easy, and the global replication and topology of the product will be a major part of our resiliency story for the petabytes of data we will store there.”

One significant lesson learned during the implementation was that although many companies think of the cloud as a commodity, in practice, each cloud provider has a unique approach that influences the architecture of the client systems it hosts. For IBM Cloud, one of the strongest emphases is enterprise-class security.

The IBM Kubernetes managed service provides ISO 27001 security out of the box. With many cloud providers’ solutions, users have to configure security themselves, requiring tremendous effort to attain the level of security and compliance that’s included in IBM’s solution.

“IBM is synonymous with security — that’s why its clients have so much trust in the IBM brand,” says Hill. “We have spoken with government and financial services clients about future projects, and the security that the IBM Cloud offers is a major selling-point.”

From a support perspective, IBM Cloud provides sophisticated dashboards and monitoring tools to help The Weather Company team manage its extensive web server estate. The team is using IBM Cloud logging and monitoring together with familiar open-source tools that are easy to integrate with the IBM Cloud stack.

“As a 24x7 shop, having the right level of support services and tools is vital,” says Wendy Frazier, Head of Consumer Web Development and Content Delivery at The Weather Company. “The IBM Cloud team helped us make the transition seamless. We didn’t need to add any resources to our own team, and the infrastructure has been rock-solid.”

With the IBM Cloud infrastructure in place, The Weather Company has completed the migration of its web properties and is redesigning other aspects of its infrastructure. The fact that The Weather Company’s systems can coexist easily across multiple clouds and on-premises infrastructure demonstrates IBM Cloud’s ability to handle the intensive demands of enterprise workloads.

developer working on computer in server room
Lightning

Under the covers, the company manages an extensive set of core weather systems and APIs, which ingest and process petabytes of data from weather radar and other sources. Additionally, the company has expanded its use of IBM Watson® services, such as the IBM Watson live captioning capability for adding closed captions to its vast video library and IBM Watson Assistant technology to build AI-powered chatbots to help online users. The company is also harnessing IBM Watson Studio to augment its machine learning and deep learning capabilities and embed AI into its applications.

With its global reach, IBM can offer The Weather Company access to an extensive range of 28 multizone regions in 19 countries. This global reach helps the company extend its local footprint in emerging markets and continue its strategy of minimizing latency by keeping web endpoints as close to the user as possible.

IBM Cloud also gives The Weather Company access to offerings such as IBM Cloud Bare Metal Servers, making it more straightforward to migrate services that can’t be containerized easily, and support high-performance workloads that require specific hardware profiles or custom configuration. Currently the team is applying this offering to facilitate high-volume science workloads for its forecast model — the IBM Global High-Resolution Atmospheric Forecasting (GRAF) System — for the purposes of taking advantage of GPU-type computing. Fueled by the IBM Power9™ processor, IBM GRAF brings the same technology to weather forecasting that is behind some of the world’s most powerful supercomputers.

A long-term commitment

The weather.com and wunderground.com sites have been fully in production on IBM Cloud since 2018. In the meantime, they have seamlessly handled high-volume events generated by dozens of hurricanes, floods, and tropical and winter storms, experiencing 100% uptime during the 2018 and 2019 hurricane seasons. This has given The Weather Company the confidence to extend its use of IBM Cloud to all of its cloud-based products and services.

“The migration is about 30% complete, and so far, we have been impressed with the scalability and reliability of IBM Cloud,” says Hill. “The world’s most accurate weather provider is now serving its forecasts using IBM Cloud, giving us the elasticity and global redundancy to handle the most dramatic spikes in traffic, regardless of where they occur.”

IBM Cloud’s open-source capabilities are key. “Our products are all built on open-source technologies, allowing us to focus on building products that our customers love rather than on building support components from scratch,” Hill continues. “Using open-source components ensures that they can leverage the scale of open-source communities to keep them up to date.”

A developer at the computer coding
Foggy city night by the bay

At the core of it all is the IBM Cloud Kubernetes Service, which pulls in the upstream Kubernetes versions at a rate much faster than competitors’ solutions are capable of, keeping The Weather Company’s compute platform current and up to date. “Everything we do is based on that,” says Hill, “and it gives us great confidence that we have the foundation to build all of our products reliably.”

A major gain, beyond pure time-savings and scalability, is the cultural change. The IBM Cloud Kubernetes Service creates less separation between development and operations teams. Developers can build a new system and take it all the way to the container — a paradigm shift that can significantly boost development velocity.

With IBM managing the Kubernetes environment and the underlying IBM Cloud Virtual Servers, The Weather Company team has more time to focus on rolling out new features. Moreover, when the company needs to move fast on new growth initiatives, developers and testers have immediate access to the additional compute resources they need, without the wait time for requesting traditional hardware and standing up virtual machines (VMs).

Instead of having to do a major cut-over to the new IBM Cloud platform, workloads running in The Weather Company’s existing VMs and its newer containers are able to coexist in a true hybrid cloud environment. The company can maintain system availability without unexpected downtime and without disruptions when it needs to roll out new features and updates.

Over the course of the project, The Weather Company team has forged close relationships with the IBM Cloud engineering team, providing ongoing feedback and insight that helps shape the future of the cloud platform. The company also continuously evaluates projects from the open-source community to understand how new tools can increase velocity and performance.

Today, the company is in the midst of adopting another pivotal IBM Cloud technology, the Red Hat® OpenShift® container platform. “With the 2019 acquisition of Red Hat by IBM, we’ve made the decision to use OpenShift as a key technology for our business,” says Hill. “We have migrated the weather.com consumer website into it now, which enables us to leverage our investment in implementing IBM Cloud Kubernetes Service, the managed component of OpenShift on IBM Cloud.”

And starting in late 2020, the company will begin migrating its Storage Utility Network (SUN) platform to IBM Cloud. The SUN platform serves weather API and other data to global customers at a sustained volume of over one million requests per second.

But at the heart of everything The Weather Company does are the people it serves. “The most important thing is that we can continue to scale our platforms to help people stay safe when extreme weather hits,” Hill concludes. “IBM Cloud is the perfect engine to power the world’s largest weather websites and deliver the fastest, most accurate weather insight to millions of users around the globe.”

Inside Etihad airplane economy

Serverless queries, unlimited storage

Clouds

The Weather Company is also benefiting from IBM Cloud technologies within individual teams and projects. The company is in the process of rolling out a new version of its History on Demand – Conditions offering that relies heavily upon the IBM Cloud SQL Query and IBM Cloud Object Storage solutions.

Accessible through a web API, the History on Demand – Conditions application gives users access to global, high-resolution datasets of past weather conditions. The Weather Company collects weather data across the globe at the rate of 34 million records per hour, and the History on Demand application serves that historical weather data to users via an API, averaging 600,000 requests per day. Clients can apply that data to machine learning and data analytics for research in areas such as climate science, transportation, insurance, and energy and utilities. Ultimately, the findings help clients make better business and safety decisions around issues involving weather.

Within this environment, the IBM Cloud SQL Query solution provides serverless, interactive querying for analyzing data in IBM Cloud Object Storage. The Weather Company expects to achieve significant cost savings and performance improvements with the addition of these cloud-based managed services. The IBM Cloud SQL Query pay-as-you-go model means that users only pay for data scanned from IBM Cloud Object Storage. By leveraging leading-edge data skipping technology, together with a fully integrated catalog and geospatial library, SQL Query avoids scanning irrelevant data. This has enabled The Weather Company to achieve considerable reductions in cost, while extending coverage and significantly accelerating performance.

The Weather Company logo

About About The Weather Company, an IBM Business

The Weather Company, an IBM Business, helps people make informed decisions and take action in the face of weather. The powerful combination of IBM’s advanced AI and cloud capabilities with The Weather Company’s high volume of weather data helps people, businesses and communities around the world prepare for and mitigate the cost of weather. The world’s most accurateExternal Link forecaster globally, the company offers more than 25 billion forecasts daily with personalized and actionable weather data and insights. The Weather Company is committed to trust and transparency, and its digital properties from The Weather Channel (weather.comExternal Link) and Weather Underground (wunderground.comExternal Link) are trusted by hundreds of millions of people to provide accurate, timely forecasts that help them make critical decisions every day – and has even been named one of the most trustedExternal Link brands.

Solution components
The Weather Company logo

About About The Weather Company, an IBM Business

The Weather Company, an IBM Business, helps people make informed decisions and take action in the face of weather. The powerful combination of IBM’s advanced AI and cloud capabilities with The Weather Company’s high volume of weather data helps people, businesses and communities around the world prepare for and mitigate the cost of weather. The world’s most accurateExternal Link forecaster globally, the company offers more than 25 billion forecasts daily with personalized and actionable weather data and insights. The Weather Company is committed to trust and transparency, and its digital properties from The Weather Channel (weather.comExternal Link) and Weather Underground (wunderground.comExternal Link) are trusted by hundreds of millions of people to provide accurate, timely forecasts that help them make critical decisions every day – and has even been named one of the most trustedExternal Link brands.

Solution components