What is Google Gemma?

Because they’re leaner than Gemini models, Gemma models can be deployed on laptops and mobile devices, but they’re also optimized for NVIDIA graphics processing units (GPUs) and Google Cloud tensor processing units (TPUs). Yet unlike Gemini, Gemma is not multilingual nor multimodal.

These text-to-text artificial intelligence (AI) models derive their name from the same Latin word, which means “precious stone.” Gemma is a group of open models, with Google providing free access to the model weights, and the models are freely available for individual and commercial use and redistribution.¹

Gemma’s first-generation models were introduced in February 2024,¹ while the second-generation models were announced in June 2024.²

Gemma’s collection of AI models includes Gemma and Gemma 2 at its core, plus a few more specialized models that have been optimized for specific tasks and have a different architecture underpinning them. Models in the Gemma line have base or pretrained variants and instruction-tuned variants.

Gemma is the first generation of the Gemma models. Gemma 2B is the smallest at 2 billion parameters, while Gemma 7B has 7 billion parameters. These models were trained on code and math datasets and mostly English-language content from web documents.³

Gemma 2 is the second generation of the Gemma family. According to Google, Gemma 2 has better performance and is more efficient at AI inferencing (when a model generates a response to a user’s query) compared to its predecessor.²

The model is available in 2, 9 and 27 billion parameter sizes. Their training datasets encompass English-language web documents, code and science articles.⁴

This text-to-code model is fine-tuned for coding tasks. It supports multiple programming languages, including C++, C#, Go, Java, JavaScript, Kotlin, Python and Rust.⁵

CodeGemma has a 7B pretrained variant for code completion and code generation, a 7B instruction-tuned variant for natural language code chat and instruction following and a 2B pretrained variant for swift code completion.⁵

DataGemma is composed of fine-tuned Gemma and Gemma 2 models that supplement their responses with data from Google’s Data Commons, a repository of public statistical data. DataGemma RIG models apply retrieval-interleaved generation to create natural language queries for getting data from Data Commons. Meanwhile, DataGemma RAG models employ retrieval-augmented generation for fetching data from Data Commons that can augment the models’ prompts.⁶

This vision-language model accepts both images and text as input and produces text as output. As such, it’s ideal for answering questions about images, detecting objects within images, generating image captions and reading text embedded in images. Its underlying architecture consists of a vision transformer image encoder and a transformer text decoder initialized from Gemma 2B.⁷

PaliGemma has a general-purpose set of pretrained models and a research-oriented set of models fine-tuned on certain research datasets. Google notes that most PaliGemma models require fine-tuning, and outputs must be tested before deployment to users.⁸

RecurrentGemma uses a recurrent neural network architecture developed by Google researchers. This makes it quicker at inferencing—particularly when generating long sequences—and requires less memory than Gemma. It comes in 2B and 9B pretrained and instruction-tuned models.⁹

CodeGemma and PaliGemma have their own specific use cases. But in general, people can use Gemma for natural language processing (NLP) and natural language understanding tasks, including:

• Building conversational AI assistants and chatbots
• Editing and proofreading
• Question answering and research
• Text generation, such as emails, advertising copy and other content
• Text summarization, especially for lengthy documents and huge volumes of reports or research papers

Gemma is based on a transformer model, a neural network architecture that originated from Google in 2017.¹⁰

Here’s a brief overview of how transformer models work:

• Encoders transform input sequences into numerical representations called embeddings that capture the semantics and position of tokens in the input sequence.

• A self-attention mechanism allows transformers to “focus their attention” on the most important tokens in the input sequence, regardless of their position.

• Decoders use this self-attention mechanism and the encoders’ embeddings to generate the most statistically probable output sequence.

However, Gemma uses a variation of the transformer architecture known as the decoder-only transformer.¹¹ In this model, input sequences are fed directly into the decoder, which still uses embeddings and attention mechanisms to generate the output sequence.

Gemma’s first-generation models improve upon transformers through a few architectural elements:

• Each layer of the neural network applies rotary positional embeddings instead of absolute positional embeddings. Embeddings are also shared across inputs and outputs to compress the model.³

• Gemma 7B employs multihead attention, with multiple “attention heads” having their own keys and values to capture different types of relationships between tokens. In contrast, Gemma 2B employs multiquery attention, where all attention heads share a single set of keys and values, thereby enhancing speed and lessening the memory load.¹¹

Gemma 2 uses deeper neural networks than Gemma. Here are some other notable architectural differences:⁴

• For every other layer of its neural network, Gemma 2 alternates between a local sliding window attention and global attention. Local sliding window attention is a dynamic mechanism for focusing on certain fixed-size “windows” of input sequences, allowing models to concentrate on only a few words at a time. Global attention, meanwhile, attends to every token in the sequence.

• Gemma 2 also employs grouped-query attention, a divide-and-conquer approach that splits queries into smaller groups and computes attention within each group separately.

• Also, the Gemma 2 2B and 9B models apply knowledge distillation, which entails “distilling” a larger model’s knowledge into a smaller one by training the smaller model to emulate the larger model’s reasoning process and match its predictions.

In terms of instruction tuning, which primes the model to better follow instructions, both Gemma and Gemma 2 apply supervised fine-tuning and reinforcement learning from human feedback (RLHF).⁴ Supervised fine-tuning uses labeled examples of instruction-oriented tasks to teach the model how to structure its responses. Meanwhile, RLHF uses a reward model to translate quality ratings from human evaluators into numerical reward signals, helping models learn which responses will garner positive feedback.

Evaluations of Gemma 7B’s performance in LLM benchmarks spanning code generation, commonsense reasoning, language understanding, mathematical reasoning and question answering indicate that it is comparable to SLMs of a similar scale such as Llama 3 8B and Mistral 7B. Gemma 2 9B and 27B performed even better, surpassing both Llama 3 8B and Mistral 7B in most benchmarks.¹²

However, Llama 3.2 3B and Ministral 3B, the latest SLMs from Meta and Mistral, respectively, have surpassed Gemma 2 2B in various benchmarks.¹³ Microsoft’s Phi-3-mini, a 3.8-billion-parameter language model, also gained higher performance than Gemma 7B.¹⁴

Gemma models can be accessed through these platforms:

• Google AI Studio

• Hugging Face (also integrated into Hugging Face Transformers)

• Kaggle

• Vertex AI Model Garden

Also, developers can implement the models in open source machine learning frameworks such as JAX, LangChain, PyTorch and TensorFlow, and through application programming interfaces (APIs) like Keras 3.0. In addition, because Gemma includes optimization across NVIDIA GPUs, developers can use NVIDIA tools, including the NeMo framework to fine-tune models and TensorRT-LLM to optimize them for efficient inferencing on NVIDIA GPUs. 

For enterprise AI development, Gemma models can be deployed on Google Cloud Vertex AI and Google Kubernetes Engine (GKE). For those with limited computational power, Google Colab provides free cloud-based access to computing resources like GPUs and TPUs.

Like other AI models, Google Gemma continues to grapple with the risks of AI, including:

• Bias: Smaller models can learn from the bias present in their larger counterparts, and this domino effect can reflect in their results.

• Hallucinations: Verifying and monitoring the outputs of SLMs like Gemma is essential to make sure what they produce is accurate and factually correct.

• Privacy violations: Google notes that the training datasets for Gemma and Gemma 2 have been filtered to remove certain personal information and other sensitive data.⁴ However, individual users and enterprises must still be cautious with the data they use to fine-tune Gemma to avoid leaking any personal or proprietary data.

When it comes to safety and security, Google evaluated Gemma on several metrics, including offensive cybersecurity, CBRN (chemical, biological, radiological and nuclear) knowledge, self-proliferation (the ability to autonomously replicate) and persuasion. Gemma’s knowledge in CBRN domains is low. Similarly, the model has low capabilities in offensive cybersecurity, self-proliferation and persuasion.⁴

Google also released a Responsible Generative AI Toolkit to help AI researchers and developers build responsible and safe AI applications.¹