What is zero-shot prompting?

29 January 2025

Authors

Meredith Syed

AI Developer Advocate Lead

Vrunda Gadesha

Technical Content Writer, IBM

What is zero-shot prompting?

Zero-shot prompting is a prompt engineering method that relies on the pretraining of a large language model (LLM) to infer an appropriate response. In contrast to other prompt engineering methods such as few-shot prompting, models aren’t provided examples of output when prompting with the zero-shot technique.1

How zero-shot prompting works

One of the promises of foundation models is that they can be adapted to many use cases without requiring additional training data. In zero-shot prompting, the model is prompted to generate a response without receiving an example of the desired output for the use case. Zero-shot prompting is an application of zero-shot learning, a machine learning pattern that asks models to make predictions with zero training data.

In the example below, IBM’s granite-3-8b-instruct model from IBM's Granite Series is prompted by using zero-shot prompting on a classification use case.2 In this case, the model is asked to classify an IT issue as belonging to either the class “High” or the class “Low,” indicating the urgency of the issue. In this example, the model infers the classification label “High” for the prompt (as depicted by the bold output), an appropriate response in this context.

Note: The prompt was executed in IBM watsonx.ai Prompt Lab using the “Freeform” mode.

Set the Class name for the issue described to either High, Medium or Low. Predict only the Class name for the last issue. Give a short description for why that Class name was chosen.

Class name: High
Description: An issue that has a high business cost, impacts many users or both.

Class name: Medium
Description: An issue that falls somewhere in between high and low.

Class name: Low
Description: An issue that has an impact on a few users, does not have a high business cost or both.

Issue: Users are reporting that they are unable to upload files.

Class: High

Description: This issue is considered high because it impacts many users and has a high business cost. The inability to upload files can prevent users from completing their tasks, which can lead to delays and decreased productivity. Additionally, this issue may impact multiple departments or teams, further increasing the business cost.

A zero-shot prompting strategy is often the default prompt engineering strategy when tasking a model with a new problem.1 This example demonstrates how the model is expected to generate an appropriate inference without any explicit examples of responses. The zero-shot prompting scenario mimics human learning in applying prior knowledge to solve new problems.3

Components of a prompt

While the model isn’t provided with example prompts, it is provided with details about how to complete the task:1

Instruction: First, the instructions provided to the model are to “Set the Class name for the issue described...”

Context: Next, the context for the model includes a description of class names.

Input data: The model receives input data to run the classification task with the prompt of “Issue: Users are reporting that they are unable to upload files.”

Output indicator: Optionally, the model can receive an output indicator, in this case the text "Class:" which cues the model to respond with the class name of the issue. Output indicators indicate to the model what type of output to produce for a specific type of response.

The customized format of this prompt is for the classification problem at hand. Other use cases might require a different format for the prompt and they might not contain the same instruction, context, input data and output indicator components.1 Different models can require different formats for a prompt. Be sure to follow any instructions given for how to format a prompt for a specific model. In this example, as a result of the pretraining of the model and this well-crafted prompt using the described components, the model responds with appropriate output for this task.

Zero-shot prompting vs few-shot prompting

In contrast to zero-shot prompting, few-shot prompting provides the model with examples of expected input and output for the task.1 The preceding image diagrams the difference between zero-shot prompting and few-shot prompting, with one-shot prompting also shown as a special case. 

By using the same IBM granite-3-8b-instruct model, examples of “Issues” and their corresponding “Classes” for this task are provided.2 The three examples of input and output that are provided illustrate the few-shot prompting strategy. In response to this prompt, the model makes a reasonable prediction of the class “Medium” (as highlighted in bold).

Note: The prompt was executed with IBM watsonx.ai Prompt Lab by using the “Freeform” mode.

Set the Class name for the issue described to either High, Medium or Low. I've provided a few examples of issues and their corresponding Class names. Predict only the Class name for the last issue. Give a short description for why that Class name was chosen.

Class name: High
Description: An issue that has a high business cost, impacts many users or both.

Class name: Medium
Description: An issue that falls somewhere in between high and low.

Class name: Low
Description: An issue that has an impact on a few users, does not have a high business cost or both.

Issue: New users report that they cannot create accounts.
Class: High

Issue: A user reports that labels are rendering overlapping in the app's analytics function.
Class: Low

Issue: Users in the United States are reporting that the app is crashing when trying to make a purchase.
Class: High

Issue: Users report that images are not loading in the app.

Class: Medium

Explanation: The issue of images not loading in the app impacts users' experience and might cause frustration, but it does not have a high business cost or impact many users. Therefore, it is classified as a medium-level issue.

Few-shot prompting, which is a strategy derived from the few-shot learning paradigm, is typically used to improve the performance of the model over zero-shot prompting on a task.1 In this example, the AI model produces useful inferences in both the zero-shot and few-shot scenarios. In deciding whether to use zero-shot or few-shot prompting, one should consider the constraints of the problem and the demonstrated performance of both strategies. Reynolds and McDonell (2021) have found that with improvements in prompt structure, zero-shot prompting can outperform few-shot prompting in some scenarios.4 Schulhoff et al. (2024) find different results comparing the performance of several prompting strategies.5

Advantages and limitations of zero-shot prompting

Zero-shot prompting is a popular approach because of its advantages.6 Researchers continue to develop techniques to address the limitations of this prompting technique.8

Advantages

  1. Simplicity: Prompts are simple to construct and easy to understand. This approach allows for users to experiment with different prompts without deep prompt engineering knowledge. 
  2. Ease of use: Zero-shot prompting doesn't require any additional data, making it valuable in cases where relevant data is difficult to source or scarce.
  3. Flexibility: Prompts are easy to adapt as needed. Improving a prompt or updating a prompt due to changing circumstances requires low effort.

Limitations

  1. Performance variability: While zero-shot prompting can be effective, its performance can vary significantly depending on the complexity and specificity of the task. Models might struggle with tasks that require deep domain knowledge, nuanced understanding or highly specific outputs, leading to suboptimal results compared to task-specific fine-tuned models.
  2. Dependence on pretrained model quality: The success of zero-shot prompting heavily depends on the quality and comprehensiveness of the pretrained language model. If the model lacks sufficient exposure to certain topics, languages or contexts during pretraining, its zero-shot performance on related tasks will likely be poor.

Advances in training methods for LLMs have improved model output for zero-shot prompting across various use cases.7

Improvements in zero-shot prompting performance

Zero-shot prompting relies on the foundation model’s pretrained knowledge and its flexibility to adapt to the requested prompt and to provide an appropriate response.1

Improving responses in the zero-shot scenario is a focus of researchers.Zero-shot prompt response accuracy is often used to benchmark model performance while testing new model training methods.7 Two improvements that have resulted in better zero-shot prompting performance are instruction tuning and reinforcement learning with human feedback (RLHF).89

In instruction tuning, a model is fine-tuned by using supervised learning on a dataset that includes instructions for various tasks and outcomes for these tasks. The dataset includes tasks such as text summarization, conversion and reading comprehension. This strategy of fine-tuning with an instructional dataset has resulted in better zero-shot prompting performance on new tasks in these categories.8

Another example of using fine-tuning to improve zero-shot prompting outcomes is RLHF fine-tuning, in which reinforcement learning learns a policy that guides the model to better outputs. In this 3-step process, the model is first fine-tuned by using an instructional dataset in which humans have provided the target responses. Then, the model projects outputs to several prompts ranked by humans. Lastly, the ranked outputs are used to train a reinforcement learning model that learns a policy to select best outputs based on these human-provided rankings.12

The final step uses reinforcement learning’s ability to use the consequences (rewards or punishments) of actions (decision or path taken) to learn a strategy (or policy) for making good decisions. In this case, the problem space is all of the potential strategies that might be used in the model’s selection of a good output as its response.9

Applications of zero-shot prompting

In comparison to traditional supervised machine learning for natural language processing (NLP), zero-shot prompting does not require labeled training data. Artificial intelligence practitioners and data scientists can use the generative AI technology of large language models in the zero-shot prompting scenario for various use cases including:10

Text classification

As shown in the prior example classifying the priority of IT issues with IBM’s granite-3-8b-instruct model, the model can achieve classification without prior examples belonging to the different classes. This capability is ideal for situations where limited or no labeled training data exists. This zero-shot classification tutorial shows an implementation of this use case.

Information extraction

Given a body of text and a question, an LLM can extract the requested information in accordance with a prompt.

Question answering

Using the pretrained knowledge of a model, a user can prompt for a response to a question.

Text summarization

Given text and an instruction for text summarization, large language models can run this task in the zero-shot prompt scenario without requiring example summaries of other texts.

Generation

LLMs generate data in the form of text, code, images and more for specified use cases.

Conversation

Typically using models tuned for chat (such as the well-known chat-GPT series), LLMs can interact with a user in chat mode, accomplishing many of the preceding use cases.

Other prompt engineering methods

For complex use cases such as multistep reasoning tasks, both zero-shot prompting and few-shot prompting might fail to produce an appropriate response from the model. Advanced prompting techniques including chain-of-thoughts and tree-of-thoughts can be more successful in these cases.

Chain-of-thoughts: Chain-of-thought (CoT) prompting is a strategy that poses a task to the model by specifying the larger task as a series of discrete steps to solve. This exposition of intermediary steps improves the ability of the model to generate a correct response. CoT also allows better transparency into the problem-solving process because of the elucidation of the intermediate steps. This prompt engineering technique shows success in areas including improving customer service chatbot performance, helping organize the thoughts of researchers and writers and generating step-by-step descriptions for math and science educational problems.11

Tree of thoughts: Tree-of-thought (ToT) prompting generates a branching text tree of potential next steps and corresponding possible solutions to the problem. This tree structure allows the model to explore multiple paths and backtrack, if necessary, when a path doesn’t result in an acceptable solution. It is designed to approximate human reasoning strategies when comparing potential paths to a solution. Common strategies for exploring solutions are breadth-first-search (BFS) and depth-first-search (DFS) along with the heuristic search and reinforcement learning approaches. Researchers have used this application to solve puzzles such as sudoku and the Game of 24.1213

 

Footnotes

1. E. Saravia. "Prompt Engineering Guide." https://github.com/dair-ai/Prompt-Engineering-Guide (accessed Oct 2024).

2. "Granite 3.0 Language Models," IBM Research, Yorktown Heights, NY, Oct 2024. https://github.com/ibm-granite/granite-3.0-language-models/blob/main/paper.pdf

3. B. Romera-Paredes, P. Torr, "An embarrassingly simple approach to zero-shot learning," in ICML, 2015, pp. 2152–2161, https://proceedings.mlr.press/v37/romera-paredes15.pdf.

4. L. Reynolds, K. McDonell, "Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm", Feb 2021, https://doi.org/10.48550/arXiv.2102.07350.

5. S. Schulhoff, M. Ilie, N. Balepur, K. Kahadze, A. Liu, C. Si, Y. Li, A. Gupta, H. Han, S. Schulhoff, P. S. Dulepet, S. Vidyadhara, D. Ki, S. Agrawal, C. Pham, G. Kroiz, F. Li, H. Tao, A. Srivastava et al. "The Prompt Report: A Systematic Survey of Prompting Techniques," Dec 2024,  https://doi.org/10.48550/arXiv.2406.06608.

6. Y. Li, Yinheng, "A Practical Survey on Zero-shot Prompt Design for In-context Learning," in RANLP, July 2023, pp. 641‑647, https://acl-bg.org/proceedings/2023/RANLP%202023/pdf/2023.ranlp-1.69.pdf.

7. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi`ere, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave and G. Lample, “LLaMA: Open and efficient foundation language models,” Feb 2023, https://doi.org/10.48550/arXiv.2302.13971.

8. J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai and Q. V. Le, "Finetuned Language Models are Zero-Shot Learners," in ICLR, 2022, https://doi.org/10.48550/arXiv.2109.01652.

9. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe, “Training language models to follow instructions with human feedback,” in NeurIPS, 2022,   https://proceedings.neurips.cc/paper_files/paper/2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf.

10. P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig, “Pre-train, prompt and predict: A systematic survey of prompting methods in Natural Language Processing,” ACM Computing Surveys, vol. 55, no. 9, pp. 1–35, Jan. 2023, https://dl.acm.org/doi/pdf/10.1145/3560815.

11. J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” Jan 2023, https://doi.org/10.48550/arXiv.2201.11903.

12. J. Long, "Large Language Model Guided Tree-of-Thought," May 2023, https://doi.org/10.48550/arXiv.2305.08291.

13. S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y. Cao, and K. Narasimhan, "Tree of Thoughts: Deliberate Problem Solving with Large Language Models," Dec 2023, https://doi.org/10.48550/arXiv.2305.10601.

Related solutions IBM® watsonx.ai™

Train, validate, tune and deploy generative AI, foundation models and machine learning capabilities with ease and build AI applications in a fraction of the time with a fraction of the data.

IBM Consulting™ services

Redefine how you work with AI for business. IBM Consulting™ is working with global clients and partners to co-create what’s next in AI. Our diverse, global team of more than 20,000 AI experts can help you quickly and confidently design and scale cutting edge AI solutions and automation across your business. ​

IBM's AI solutions

IBM’s artificial intelligence solutions help you build the future of your business. These include: IBM® watsonx™, our AI and data platform and portfolio of AI-powered assistants; IBM® Granite™, our family of open-sourced, high-performing and cost-efficient models trained on trusted enterprise data; IBM Consulting, our AI services to redesign workflows; and our hybrid cloud offerings that enable AI-ready infrastructure to better scale AI.

Resources

What is prompt-tuning?
Blog

Get started

What is prompt engineering?
Related topic

Get started

What is fine-tuning?
Related topic

Get started

IBM watsonx.ai: Open source, pre-trained foundation models make AI and automation easier than ever before
Blog

Get started

Think Newsletter

 

The latest AI and tech insights from Think

Sign up today
Take the next step

Train, validate, tune and deploy generative AI, foundation models and machine learning capabilities with IBM watsonx.ai, a next-generation enterprise studio for AI builders. Build AI applications in a fraction of the time with a fraction of the data.

Explore watsonx.ai Book a live demo