The Weather Company’s websites serve millions of users per day. When extreme weather hits and usage peaks, the sites must be at their fastest and most reliable to provide the information people need to stay safe.
To optimize for elasticity in handling extreme spikes in demand, The Weather Company worked with IBM to migrate its web platform quickly and seamlessly from its existing cloud provider to the IBM Cloud.
Unlockssignificant cost savings on cloud hosting and support
Acceleratesdeployment of new services with Kubernetes
Expandsglobal reach with access to a larger number of data centers in local markets
Business challenge story
Instant insight when it matters most
Many companies build their web platforms on the assumption that the traffic they need to serve will be driven by predictable factors. For example, a retailer expects extended seasonal peaks around the winter holidays, with sharp spikes for major events such as Black Friday. Scaling to handle increased demand can be challenging, but long-term planning makes it a solvable problem.
Weather websites like weather.com and wunderground.com, however, face an exponentially harder scalability challenge—and their ability to perform at a high level during periods of intense demand can literally spell the difference between life and death.
Chris Hill, VP and Chief Information and Technology Officer for IBM Watson Media and Weather, explains: “Our sites are unique because when they’re operating at scale is when they add the greatest value. The most critical time to provide accurate, timely information is during hurricanes, tornadoes and severe winter storms, because that’s when weather insight really helps people keep themselves and their families safe.”
During major weather events, The Weather Company relies on its infrastructure to deliver very high performance when its servers are under the most intense load. Extreme weather systems are often fast-moving and difficult to predict, so to handle massive, short-notice peak workloads, the company depends on the ability to scale up rapidly at any time.
Wendy Frazier, Head of Consumer Web Development and Content Delivery at The Weather Company, explains: “During Hurricane Sandy, our sites served close to 9 petabytes of video data in one day. It’s really important that the system can scale because our volumes can go from one extreme to the other, overnight.”
Since weather is a hyper-local phenomenon, the scalability of the web platform needs to be localized, too. As The Weather Company expands its operations into markets such as Japan, India and Eastern Europe, it needs to be able to harness local data centers to bring its endpoints closer to the user to minimize latency. Localizing its data centers adds further complexity to the scalability challenge, because each local instance must be able to scale independently: a tornado in the US Midwest creates demand peaks in that region, with little effect on operations elsewhere.
Five years ago, The Weather Company started moving its infrastructure into the cloud, architecting most of its systems around the services offered by its cloud provider. The cloud platform helped to address the company’s scalability challenges, but also locked it into using proprietary technologies.
Jagmeet Chawla, Global Head of Architecture Office and Cloud Engineering at The Weather Company, says: “Today there are much better options for building cloud solutions on open-source software. To put ourselves in a stronger position to meet future growth and scalability challenges, we decided to re-architect all our cloud services using open-source tools in the IBM Cloud. Our web platform was the perfect place to start.”
Designing an open-source cloud platform
Many of The Weather Company’s web servers were running in portable Docker containers on the company’s existing cloud platform, making them relatively straightforward to migrate to a different cloud provider. The team viewed the project as both an important first step on its journey to the IBM Cloud, and a strong test of the capabilities of the new platform.
The Weather Company made the critical decision to architect its new environment on IBM Cloud Kubernetes Service, which orchestrates container management across an extensive cluster of IBM Cloud Virtual Servers. Kubernetes, the open-source container management platform that has recently become a de facto standard, provides a powerful, declarative way to define how containers should be started, shut down and monitored, and what hardware resources they can utilize. IBM provides Kubernetes as a managed service, significantly reducing the burden on The Weather Company’s DevOps resources.
“In a traditional microservices world, we’d need a lot more DevOps pipeline to handle all this management work,” says Chris Hill. “Once we start setting up new services with IBM Cloud Kubernetes Service, we expect to see a significant time-saving for our DevOps team.”
For example, developers can now experiment easily in the languages of their choice, pushing changes to dev and test systems that are scaled out on separate clusters. Those pushes are automated with open toolchains and IBM Cloud Continuous Delivery, which automates and accelerates build and deployment processes.
Jagmeet Chawla agrees: “The way IBM has engineered Cloud Kubernetes Service, it frees our team to focus on the business requirements, rather than worrying about the operational details of how to run the cluster.”
To store static assets—such as images and videos—that weather.com and wunderground.com serve to users, The Weather Company uses IBM Cloud Object Storage. The new storage solution provides highly compatible APIs that make it easy to interoperate with other cloud storage platforms—reducing the need for The Weather Company to make significant code changes to its front-end web applications.
“We really like the cross-region replication that IBM Cloud Object Storage System provides,” says Jagmeet Chawla. “It’s easy to configure, and it adds an extra level of resilience. Even if a whole data center went offline, we could still serve content to our users.”
One significant lesson learned during the project is that although many companies think of the cloud as a commodity, in practice, each cloud provider has a unique approach that influences the architecture of the client systems they host. For IBM Cloud, one of the strongest emphases is enterprise-class security.
“With many cloud providers, if you want security, you have to explicitly configure it yourself—their approach is to leave things open until you specify otherwise,” says Jagmeet Chawla. “With IBM, it’s the opposite: you start with full security and relax it only as much as you need to. It’s easier to build systems that are secure by default.”
“IBM is synonymous with security—that’s why its clients have so much trust in the IBM brand,” adds Chris Hill. “We are already speaking with government and financial services clients about future projects, and the security that the IBM Cloud offers is a major selling-point.”
From a support perspective, IBM Cloud provides sophisticated dashboards and monitoring tools to help The Weather Company team manage its extensive web server estate. The team is using IBM Cloud Logging and Monitoring, together with familiar third party tools such as Sysdig and Prometheus, which are easy to integrate with the IBM Cloud stack.
“As a 24/7 shop, having the right level of support services and tools is vital,” says Wendy Frazier. “The IBM Cloud team has helped us make the transition seamless. We haven’t needed to add any resources to our own team, and the infrastructure has been rock-solid.”
With the IBM Cloud infrastructure in place, The Weather Company is focusing on completing the migration of its web properties before moving on to re-architect other aspects of its infrastructure. The fact that The Weather Company’s systems can coexist easily across multiple clouds and on-premises infrastructure demonstrates IBM Cloud’s ability to handle the intensive demands of enterprise workloads.
Under the covers, the company manages an extensive set of core weather systems and APIs, which ingest and process petabytes of data from weather radar and other sources. Additionally, the company is planning to expand its use of IBM Watson® services, such as IBM Watson Captioning for adding closed captions to its vast video library, and Watson™ Assistant to build AI-powered chatbots to help online users. The company is also considering harnessing IBM Watson Studio to augment its machine learning and deep learning capabilities and embed AI into its applications.
“The web platform is a huge environment, but it’s really just the tip of the iceberg,” says Jagmeet Chawla. “IBM Cloud will help us rebuild our core systems around open-source technologies, such as IBM Compose for PostgreSQL and MySQL, and IBM Analytics for Apache Spark, and then extend these with the latest AI and interactive capabilities.”
With its global reach, IBM can also offer The Weather Company an extensive range of 28 data center options in 19 countries. In the future, this can help the company extend its local footprint in emerging markets and continue its strategy of minimizing latency by keeping web endpoints as close to the user as possible.
IBM Cloud also gives The Weather Company access to infrastructure-as-a-service offerings such as IBM Cloud bare metal servers. In the future, this could help the team migrate services that can’t be containerized easily, and support high-performance workloads that require specific hardware profiles or custom configuration.
“As far as possible, we want to focus on harnessing the open source platform that IBM Cloud provides,” says Chris Hill. “Nevertheless, it’s a major advantage that IBM Cloud bare metal servers are available if we need them. It opens up options that we just didn’t have with our previous cloud provider.”
A bright prospect for the future
The weather.com and wunderground.com sites are now fully in production on the IBM Cloud, and The Weather Company team is impressed with the performance and scalability of the platform so far.
“Our migration has made a fantastic start,” says Jagmeet Chawla. “The world’s top weather property now runs on IBM Cloud, giving us the hyper-scale elasticity we need to handle even the most dramatic spikes in traffic.
“Everything is now built around open-source technology, and we are freeing ourselves from technical debt,” he adds. “The IBM Cloud Kubernetes Services is absolutely the right strategic direction for us, and IBM has done a great job of engineering a service that runs well and is easy to use.”
The biggest gain, beyond pure time-savings, is the cultural change. The IBM Cloud Kubernetes Service creates less separation between development and operations teams. Developers can build a new system and take it all the way to the container—a paradigm shift that can significantly boost development velocity.
With IBM managing the Kubernetes environment and the underlying IBM Cloud Virtual Servers, The Weather Company team has more time to focus on rolling out new features. Moreover, when the company needs to move fast on new growth initiatives, developers and testers have immediate access to the additional compute resources they need, without the wait time for requesting traditional hardware and standing up VMs.
Instead of having to do a major cut-over to the new IBM Cloud platform, workloads running in The Weather Company’s existing VMs and its newer containers are able to coexist in a true hybrid cloud environment. The company is able to maintain system availability without unexpected downtime and without disruptions when it needs to roll out new features and updates.
Over the course of the project, The Weather Company team has forged close relationships with the IBM Cloud engineering team, providing feedback and insight that is helping to shape the future of the cloud platform. The company is also evaluating projects from the open source community to understand how tools like Istio and SPIFFE can increase velocity and performance.
“The most important thing is that we can continue to scale our platforms to help people stay safe when extreme weather hits,” Chris Hill concludes. “IBM Cloud is the perfect engine to power the world’s largest weather websites and deliver the fastest, most accurate weather insight to millions of users around the globe.”
The Weather Company, an IBM Business
The Weather Company, an IBM Business, helps people make informed decisions and take action in the face of weather. The company offers the most accurate forecasts globally with personalized and actionable weather data and insights to millions of consumers, as well as thousands of marketers and businesses via Weather’s API, its business solutions division, and its own digital products from The Weather Channel (weather.com) and Weather Underground (wunderground.com).
The company delivers around 25 billion forecasts daily. Its products include the world’s most downloaded weather app, a network of 250,000 personal weather stations, a top-20 U.S. website, one of the world’s largest IoT data platforms, and industry-leading business solutions.
Weather Means Business™. The world’s biggest brands in aviation, energy, insurance, media, and government rely on The Weather Company for data, technology platforms and services to help improve decision-making and respond to weather’s impact on business.
For more, visit business.weather.com.