Google Gemini MedGemma

MedGemma & Vertex AI Model Garden for Healthcare

Here’s a Friend Link to read this member-only story.

I attended a session by Google, and here’re what I’ve learnt. Based on the demonstration by a Google Customer Engineer, here’s a simplified explanation of how to use Google Cloud’s generative AI and Large Language Models (LLMs).

How to Access Generative AI on Google Cloud

There are three main ways to work with LLMs on Google Cloud:

• Agent Builder: This is a simple, visual tool that lets you create and customize AI assistants without needing to write code. You can drag and drop different components to build an AI agent, especially for tasks like Retrieval Augmented Generation (RAG).
• Model Builder: This is for those who want more control. You can take an existing model, like one of Google’s foundational models, and fine-tune it. This means you can train it further on your specific data to make it better at a particular task. For example, AISG created a “Sealion” model based on Google’s Gemma model that is specialized for local languages.
• Model Garden: This is like a marketplace for various AI models. It gives you access to a wide range of models from different providers, including third-party ones from places like Hugging Face, as well as specialized models like MedGemma.

The Bigger Picture: Google Cloud Services

The image below shows that Google Cloud is a comprehensive platform with many services that work together:

Source: https://images.app.goo.gl/pieT2QxqvA1wJVbTA

• Virtual Machines (AI Hypercomputer): These are the powerful computers that run the AI models. They use special hardware like GPUs and TPUs, which are essential for the massive calculations required for training and running AI.
• Vertex AI: This is a platform-as-a-service (PaaS) that brings together many AI tools, including the Agent Builder, Model Builder, and Model Garden, to help you develop and deploy your AI applications.
• BigQuery: A service designed for analyzing huge amounts of data, even databases the size of petabytes.
• Looker: A tool for visualizing data from BigQuery, creating dashboards and reports to help you understand your data better.
• Gemini: Google’s most advanced LLM, which is integrated into many products to allow users to perform tasks using simple, natural language commands instead of code.
• Gemini for Google Cloud Console: Helps you manage your cloud services with natural language.
• Gemini for Google Workspace: Assists with tasks in applications like G.Docs and G.Sheets, such as automatically creating a pivot table.
• Gemini for Code Assist: Helps developers by generating code snippets.

Retrieval Augmented Generation (RAG) Explained

RAG is a clever technique used to make LLMs more knowledgeable and up-to-date. Here’s how it works:

1. Connect to Data: First, you connect to your own data sources, which could be websites, PDF files, or enterprise databases.
2. Parse and Chunk: The system reads and understands your documents, breaking them down into smaller pieces or “chunks.”
3. Vector Embedding: These chunks are then converted into numerical representations called “vector arrays” or “embeddings.” This is a way of storing the meaning of the text.
4. Query and Retrieve: When you ask a question (your “query”), the system also converts your query into a vector. It then finds the most relevant chunks of data from your stored embeddings.
5. Generate Output: Finally, the LLM uses both your original query and the (newly retrieved) relevant information to generate a precise and well-informed response.

The Vertex AI Search Agent Builder can automate this entire RAG process for you.

Overcoming LLM Limitations

One common issue with LLMs is that their knowledge is based on the data they were trained on, which has a “knowledge cut-off” date. For example, the Gemini 2.5 model might not know about events that happened after February 2025. To get around this, you can:

• Ground on private documents and data in Vertex AI Search: Connect the LLM to your private documents and data using Vertex AI Search, so it can access the most current and specific information.
• Ground with Google Search: For general, real-time information, the model can be connected to Google Search to find the latest data.

Customizing and Fine-Tuning LLMs

You can customize LLMs in two ways:

(1) Training: This is the process of teaching a model from scratch or from a very basic state, using a large dataset. For example, showing a model many pictures of cats and dogs and labeling them. This is a very resource-intensive process, requiring powerful GPUs like the A100 or H100.

(2) Fine-tuning: This is a more targeted approach where you take an existing model and further train it on a smaller, specific dataset. This makes it better at a particular task without starting over. Techniques include:

• Prompt Design: Crafting the best questions to get the desired output.
• Supervised Tuning: Providing dataset of correct question-answer pairs.
• Reinforcement Learning with Human Feedback: Humans provide feedback to help the model learn what is a good or bad response.
• Model Evaluation: Testing the model to see how well it performs.

Note: While some models like Gemini 2.0 Flash support fine-tuning, the newer Gemini 2.5 does not yet.

Data Ownership, Security, and Indemnity

Data Ownership: When you use Google Cloud’s AI services, you own and control your data. Google does not use your data for training its own models. Your data is stored in a special “adapter layer” within Vertex AI Model Builder, which lets you customize a model while keeping your sensitive data private and secure.

Security: Google provides strong security controls, including Virtual Private Cloud Service Controls (VPC-SC), Customer-Managed Encryption Keys (CMEK), and Access Transparency to protect your data.

Indemnity: This is a legal protection Google offers. If a customer is sued for copyright infringement related to the training data used to create Google’s models or the outputs generated by those models, Google will take on the legal responsibility, as long as the services were used responsibly.

• Training data indemnity covers Google’s use of training data to create Google models utilized by all our Generative AI services.
• Generated output indemnity covers the generated output created by our customers.

Data Residency Requirements

Data residency means keeping your data within a specific geographic location. This is often a legal or regulatory requirement.

• A model might run in the U.S. but if the data is stored in a server in Singapore, the data is considered to “reside” in Singapore.
• For stricter requirements, you can use models like Gemini 1.5 and 2.5 that are hosted and processed entirely within Singapore.

MedGemma: A Specialized Medical Model

MedGemma is a special version of the Gemma model trained specifically for understanding medical text and images. MedGemma can be accessed via OpenAI SDK or VertexAI SDK. It’s available in Singapore to help organizations meet data residency requirements. It has different variants:

• 4B (billion) parameter model that can handle both text and images (i.e. multimodal).
• 27B text only instruction tuned
• 27B multimodal

Its uses include interpreting medical images, understanding clinical notes, and can be customized further through fine-tuning. MedGemma can be accessed using standard software development kits (SDKs) like those for OpenAI or Vertex AI.

Calculating Costs

To understand the cost of running these services, you can use the Google Cloud Pricing Calculator.

Compute estimates from Google Cloud Pricing Calculator

For example, to estimate the cost of running a specific type of virtual machine with GPUs (an “accelerator-optimized” G2 series machine)

• Add to estimate: Compute Engine
• Machine type: Accelerator-optimized
• Series: G2
• Machine type: g2-standard-48

You would find that running it for a full month, 24/7, costs approximately $2,922.22. This gives you a clear idea of the expenses involved. There is an option to run the virtual machine during daylight hours only, thus saving some running costs.

In summary, Google Cloud provides multiple ways to access generative AI and LLMs, from easy-to-use tools like Agent Builder for no-code solutions to more advanced options like Model Builder and Model Garden for fine-tuning and using third-party models. These services are built on a powerful infrastructure with Gemini integrated across various products to enable natural language use. Techniques like Retrieval Augmented Generation (RAG) and grounding with private data or Google Search help overcome the limitations of a model’s knowledge cut-off. Google ensures data privacy and security, offering indemnity for copyright challenges. Additionally, it provides solutions like locally hosted models to meet data residency requirements, as exemplified by the specialized MedGemma model.

Thank you for taking time to read my stories! Let’s connect on LinkedIn!