Characterizing the Evolution of Tasks Within Occupations

Share this post:

AI has started to transform occupations, with the potential to make both labor and machines more efficient and productive. To better understand this transformation, our team at the MIT-IBM Watson AI Lab presented a new study at the AAAI Conference on AI, Ethics, and Society on “Learning Occupational Task-Shares Dynamics for the Future of Work” that shows how to predict changes in the economy’s demand for different tasks. As AI continues to evolve and tasks transform, the next decade is expected to see significant changes in task requirements across occupations.

Our team is studying how task demands of different occupations have changed over the last decade by leveraging a large dataset of online job postings. To understand how the occupations are evolving, we dive deeper into how tasks within them are shifting. The job postings provide information about the prevalence of tasks within each occupation. This is used to generate time-series data of tasks counts for each occupation. This provides a measure of employer demand over time for workers who can perform specific tasks. We additionally incorporate wages and employment shares data from the Bureau of Labor Statistics (BLS), which publishes annual statistics of the average wages and number of employees for 964 occupations. We normalize the task demand time-series data by the share of workers employed in each occupation to derive the unique task-shares dynamics data for each task-occupation pair. The changes in the occupations during that period are further characterized by the evolution of the task-shares within each occupation. This allows us to document trends in occupations and tasks as well as occupational wage terciles (low, medium, high).

Figure A. Task share dynamics of different Information Technology task clusters across HML wage occupations [1].

Figure A. Task share dynamics of different Information Technology task clusters across HML wage occupations [1].

To study how AI and related technologies are impacting the labor market at the initial phase of adoption, this study focuses on the Information Technology (IT) task cluster family to look at specific task clusters. Figure A illustrates the evolution of task-shares of selected clusters within the IT task family across high, mid and low (HML) wage occupations. Although task clusters for “SQL Databases and Programming,” “Java,” and “JavaScript & jQuery” have the highest shares in high and mid wage occupations, their demand is steadily declining. In contrast, even though the “Artificial Intelligence” and “Big Data” task clusters have low task-shares in the high wage occupations, their demand increased at a very high rate during 2010-2017. Task clusters like “Scripting Languages” (includes Python) and “Cloud Solutions” gained task-shares in high wage occupations, whereas, most IT task clusters lost task-shares in low wage occupations. This evolution of IT task demands confirms the industry trends towards developing AI-based products and services in the cloud requiring workers to perform AI, Big Data, Scripting Languages, and Cloud Solutions based tasks, while focusing less on traditional software products and services that require workers to perform SQL, Java, and Data Management oriented tasks.

Figure B. One-step ahead predictions of task-shares of selected task clusters families across HML wage occupations [1].

Figure B. One-step ahead predictions of task-shares of selected task clusters families across HML wage occupations [1].

We use the temporal aspects of the task-shares data to learn the dynamics of tasks and occupations, and then quantitatively predict the task-shares for the near future with confidence bounds. We train an autoregressive integrated moving average (ARIMA) model on this data to learn the dynamics of the task-shares of different task cluster families across High-Mid-Low wage occupations over its first 72 months (2010-2015). We then use this ARIMA model to make one-month-ahead predictions of the task-shares. The paper reports less than 5% mean absolute percentage error (MAPE) of predictions. Figure B shows the task-share forecasts (black lines) with 95% confidence intervals (grey areas) to compare against the true task-shares (dotted lines) for a few selected task cluster families across High (red line), Mid (green line), and Low (blue line) wage occupations. The accuracy of the task-share predictions is a clear indicator of the benefit of developing robust and more accurate forecasting models to characterize the evolution of occupations and their tasks. Such predictive capabilities on the labor market might help the workers reskill themselves, corporations retrain their employees, or new graduates to learn the skills to be able to execute the tasks of the future.

This empirical research sheds new light on the transformation of work by characterizing occupations in terms of task-shares dynamics. There are still many open questions remaining in the study. Today, we know the changes AI and related technologies will bring to the labor market are relatively small, but real. To prepare for continued adoption and advancements in technologies, the development of accurate, comprehensive and robust predictive models are crucial to provide guidance to educational institutions, workers, employers, and new graduates on skills and tasks of the future.


[1] Subhro Das, Sebastian Steffen, Wyatt Clarke, Prabhat Reddy, Erik Brynjolfsson, Martin Fleming. “Learning Occupational Task-Shares Dynamics for the Future of Work”. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (AIES), 2020.

VP, Chief Economist Finance and Operations, IBM

Subhro Das

Research Staff Member MIT-IBM Watson AI Lab, IBM Research

More AI stories

New research helps make AI fairer in decision-making

To tackle bias in AI, our IBM Research team in collaboration with the University of Michigan has developed practical procedures and tools to help machine learning and AI achieve Individual Fairness. The key idea of Individual Fairness is to treat similar individuals well, similarly, to achieve fairness for everyone.

Continue reading

MIT and IBM announce ThreeDWorld Transport Challenge for physically realistic Embodied AI

MIT Brain and Cognitive Sciences, in collaboration with the MIT-IBM Watson AI Lab, has developed a new Embodied AI benchmark, the ThreeDWorld Transport Challenge, which aims to measure an Embodied AI agent’s ability to change the states of multiple objects to accomplish a complex task, performed within a photo- and physically realistic virtual environment.

Continue reading

Mimicking the brain: Deep learning meets vector-symbolic AI

To better simulate how the human brain makes decisions, we’ve combined the strengths of symbolic AI and neural networks. Specifically, we combined the learning representations that neural networks create with the symbol-like entities represented by high-dimensional and distributed vectors. The idea is to guide a neural network to represent unrelated objects with dissimilar high-dimensional vectors.

Continue reading