AI Agents · Multimodal · Google
Google AI Studio · Gemini Developer API

Gemini API Architecture:
Interactions, Live, Batch, and Media

The Gemini Developer API is a direct path to Google's model family through a single API key from AI Studio. This covers the full interaction surface: generateContent, streaming, tool use loops, the File API, Imagen image generation, rate tier boundaries, and where cost accumulates.

The Developer API Path

Google offers two ways to call Gemini models in production. The Developer API runs through a single API key you create in Google AI Studio. Calls go to generativelanguage.googleapis.com. The Vertex AI path uses Google Cloud service accounts and ADC and sits behind a different endpoint. The two are not interchangeable. This article covers the Developer API path only.

┌─────────────────────────────────────────────────────────────────┐ │ GEMINI DEVELOPER API TOPOLOGY │ └─────────────────────────────────────────────────────────────────┘ Google AI Studio generativelanguage.googleapis.com ┌──────────────┐ ┌──────────────────────────────┐ │ Create key │ ──GEMINI_API──► │ /v1beta/models/:model │ │ (per GCP │ KEY env var │ :generateContent │ │ project) │ │ :streamGenerateContent │ └──────────────┘ │ :batchEmbedContents │ │ │ @google/genai SDK │ /v1beta/files │ ┌──────────────────────┐ │ (File API - upload/get) │ │ ai.models │ │ │ │ .generateContent() │◄───────►│ /v1beta/models │ │ .generateImages() │ │ (list available models) │ │ .embedContent() │ └──────────────────────────────┘ │ ai.files.upload() │ └──────────────────────┘ Auth: x-goog-api-key header (SDK handles this automatically) NOT: Google Maps key, service account JSON, ADC, OAuth
One Credential, Multiple Failure Modes

A Google Maps API key returns 403 API_KEY_SERVICE_BLOCKED on the Gemini endpoint. A Vertex AI service account JSON is a completely different credential type and won't work here either. The key must come from AI Studio, created against a Google Cloud project with the Generative Language API enabled.

SDK Setup

The current unified SDK is @google/genai (JS/TS) and google-genai (Python). The older @google/generative-ai package is deprecated. Do not mix them in the same project.

npm install @google/genai

// lib/ai.ts  (server-side only)
import { GoogleGenAI } from "@google/genai";

export const ai = new GoogleGenAI({
  apiKey: process.env.GEMINI_API_KEY,
});

One singleton per process. The key reads from the environment. Never pass it to the browser or bundle it in a client component.

Model Family

Gemini model IDs are versioned strings. A bare name like gemini-2.5-flash may resolve to a different underlying version over time. Pin to a dated alias for production workloads where behavior stability matters. List live models with a GET to /v1beta/models.

gemini-2.5-flash

  • Fast text and multimodal reasoning
  • Best default for most API use cases
  • Supports tool use, streaming, JSON output
  • Lower cost per token than Pro tier
  • Context window: 1 million tokens

gemini-2.5-pro

  • Higher-quality reasoning, code, analysis
  • Same 1M token context window
  • Slower; higher cost per token
  • Reach for it when Flash gives inconsistent output on a complex task

gemini-2.5-flash-image

  • Multimodal model that generates images via generateContent
  • Response: inlineData.data (base64 PNG)
  • Used for quick image synthesis alongside text reasoning
  • Billed per image generated, not per token

imagen-4.0-generate-001

  • Dedicated high-quality text-to-image model
  • Called via ai.models.generateImages()
  • Fast variant: -fast-001, ultra: -ultra-001
  • Returns imageBytes (base64)
  • Higher quality than flash-image at higher cost
Model ID Drift

A bare model alias like gemini-2.5-flash resolves to a specific snapshot on Google's side. That mapping changes when they release a new version under the same alias. If your production app depends on stable output, pin to a specific dated version ID (e.g. gemini-2.5-flash-001) and check the models list periodically.

Interaction Patterns

generateContent: Unary Call

The simplest path. One request, one response. Use this for short outputs where the client can wait.

const res = await ai.models.generateContent({
  model: "gemini-2.5-flash",
  contents: "Explain M-Pesa STK Push in two sentences.",
});
const text = res.text; // convenience accessor on first candidate

res.text is undefined when the response has no text part (safety block, tool call, empty content). Always check res.candidates?.[0]?.finishReason in production.

streamGenerateContent: Streaming

For long outputs or better perceived latency, stream the response chunk by chunk. Each chunk has a partial text accessor.

const stream = await ai.models.generateContentStream({
  model: "gemini-2.5-flash",
  contents: "Write a 500-word product description.",
});

for await (const chunk of stream) {
  process.stdout.write(chunk.text ?? "");
}

In a Next.js API route, pipe chunks into a ReadableStream and return a streaming Response. The client reads them with the Fetch API's body reader.

Structured JSON Output

Set responseMimeType: "application/json" and responseJsonSchema to force the model to return valid JSON matching your schema. No parsing failures; the SDK retries internally if the model drifts.

import { GoogleGenAI, Type } from "@google/genai";

const res = await ai.models.generateContent({
  model: "gemini-2.5-flash",
  contents: "List three Kenyan mobile payment providers.",
  config: {
    responseMimeType: "application/json",
    responseJsonSchema: {
      type: Type.ARRAY,
      items: {
        type: Type.OBJECT,
        properties: {
          name:        { type: Type.STRING },
          description: { type: Type.STRING },
        },
        propertyOrdering: ["name", "description"],
      },
    },
  },
});

const providers = JSON.parse(res.text ?? "[]");

Multimodal: Image + Text Input

Pass inlineData parts alongside text to analyze an image, PDF page, or audio clip. For files under a few MB, encode as base64 and inline them. For larger or reusable assets, use the File API instead.

import { readFileSync } from "fs";

const imageBase64 = readFileSync("screenshot.png").toString("base64");

const res = await ai.models.generateContent({
  model: "gemini-2.5-flash",
  contents: {
    parts: [
      { text: "Describe the UI issues you see in this screenshot." },
      { inlineData: { mimeType: "image/png", data: imageBase64 } },
    ],
  },
});
01

Build contents

Construct a contents array of parts. Each part is text, inlineData, or a fileData URI from the File API.

02

Choose model

Flash for speed and cost. Pro for quality on hard tasks. Flash-image or Imagen for image generation.

03

Call the API

Unary for short output. Stream for long content or better UX. Both share the same model and contents shape.

04

Handle response

Check finishReason. STOP is normal. SAFETY means no content. TOOL_CALLS means a tool loop is in progress.

Tool Use and Function Calling

Tool use in Gemini follows the same ReAct pattern as other models: declare a set of function schemas, the model decides which to call, your code executes the function, and you return the result for another model turn. The loop continues until the model stops calling tools and produces a final text response.

TOOL USE LOOP Client Gemini API ────── ────────── │ │──generateContent(contents, tools)──────────►│ │ │ Model decides: call get_weather(city="Nairobi") │◄─────────────────────────────────────────── │ │ finishReason: TOOL_CALLS │ candidates[0].content.parts[0].functionCall │ { name: "get_weather", args: { city: "Nairobi" } } │ │ [execute get_weather("Nairobi") locally → "22°C, Sunny"] │ │──generateContent(contents + functionResponse)►│ │ │ Model reads result, produces text │◄────────────────────────────────────────── ──│ │ finishReason: STOP │ text: "It's 22°C and sunny in Nairobi today." │

Declaring Tools

Pass a tools array in the config. Each tool has a functionDeclarations list with JSON Schema-compatible parameter definitions.

const tools = [{
  functionDeclarations: [{
    name: "get_exchange_rate",
    description: "Return the current KES to USD exchange rate.",
    parameters: {
      type: "OBJECT",
      properties: {
        base:   { type: "STRING", description: "Base currency code" },
        target: { type: "STRING", description: "Target currency code" },
      },
      required: ["base", "target"],
    },
  }],
}];

const res = await ai.models.generateContent({
  model: "gemini-2.5-flash",
  contents: "What is the current KES to USD rate?",
  config: { tools },
});
Safety Blocks in Tool Loops

If the model triggers a safety filter mid-loop, finishReason becomes SAFETY and the tool loop halts with no function call. Handle this explicitly or the calling code will spin waiting for a tool call that never arrives.

Returning Function Results

After calling the function locally, append the result to the contents array as a functionResponse part and call generateContent again. The model reads the result and produces its next action.

const call = res.candidates[0].content.parts[0].functionCall;
const result = await executeLocally(call.name, call.args);

const next = await ai.models.generateContent({
  model: "gemini-2.5-flash",
  contents: [
    ...originalContents,
    { role: "model", parts: [{ functionCall: call }] },
    { role: "user",  parts: [{ functionResponse: {
        name: call.name, response: result
    }}]},
  ],
  config: { tools },
});

Media and the File API

Inlining large files as base64 in every request is wasteful. The File API lets you upload a file once and reference it by URI across multiple calls. Uploaded files persist for 48 hours. After that they're deleted automatically.

Upload and Reference

// Upload once
const file = await ai.files.upload({
  file: new Blob([pdfBytes], { type: "application/pdf" }),
  config: { displayName: "Q2 Financial Report" },
});

// Reference in multiple calls for 48h
const res = await ai.models.generateContent({
  model: "gemini-2.5-flash",
  contents: {
    parts: [
      { text: "Summarize the revenue trends in this report." },
      { fileData: { mimeType: "application/pdf", fileUri: file.uri } },
    ],
  },
});
Method Best for Size limit Lifetime
inlineData Small images, quick analysis ~20 MB per request Per-request only
File API + fileData PDFs, large images, video, audio; reuse across calls 2 GB per file 48 hours then auto-deleted
File API Gotcha: 48-Hour Window

Files expire after 48 hours. If you cache a file URI for later use and call it after expiry, the API returns a 404. For long-lived assets, upload again before each batch of calls, or store the raw bytes in your own storage (R2, S3) and re-upload on demand.

Supported Media Types

  • Images: PNG, JPEG, WebP, HEIC, HEIF, BMP, GIF
  • Video: MP4, MPEG, MOV, AVI, FLV, MKV, WebM, WMV, 3GPP (audio track extracted for audio-only analysis)
  • Audio: WAV, MP3, AIFF, AAC, OGG, FLAC
  • Documents: PDF (each page is processed as an image)
  • Text: Plain text files

Image Generation

Two paths exist for generating images via the Gemini Developer API. They have different quality profiles, latencies, and cost structures.

Imagen 4 (dedicated model)

Highest-quality text-to-image. Three variants:

  • imagen-4.0-generate-001 (standard)
  • imagen-4.0-fast-001 (faster, lower cost)
  • imagen-4.0-ultra-001 (best quality, highest cost)

Called via ai.models.generateImages(). Response: generatedImages[0].image.imageBytes (base64).

gemini-2.5-flash-image

Multimodal model that emits image parts in a standard generateContent response. Useful for combined text + image output in one call.

Response: candidates[0].content.parts[].inlineData.data (base64 PNG).

Lower quality than Imagen 4 but works in the same request as reasoning.

Imagen 4 Call

const res = await ai.models.generateImages({
  model: "imagen-4.0-generate-001",
  prompt: "A dark-themed dashboard UI with cyan accents, isometric 3D view",
  config: { numberOfImages: 1, outputMimeType: "image/png" },
});

const bytes = res.generatedImages?.[0]?.image?.imageBytes; // base64
// Decode and store in R2 / S3 to avoid regenerating on every request

Cost Reality Check: Images

  • Image generation is billed per image generated, not per token.
  • One Imagen 4 standard image costs significantly more than a typical text generation call.
  • Generate thumbnails or marketing images once and cache them in object storage (Cloudflare R2, S3). Regenerating on every page load or API call will run up bills quickly.
  • The free tier allows a limited number of images per day. Exceeding that returns 429 errors. Enable billing before production launch.
  • Exact pricing is at ai.google.dev/pricing and changes. Check before budgeting.

Rate Limits and Error Handling

The Gemini API returns standard HTTP codes with a canonical status name. Three codes are transient and worth retrying. The rest indicate a bad request or bad credentials: retrying just wastes quota.

Code Status Meaning Retry?
400 INVALID_ARGUMENT Malformed request, bad field, unsupported MIME type No: fix the call
403 PERMISSION_DENIED Wrong key type (e.g. Maps key), API not enabled No: fix the key
429 RESOURCE_EXHAUSTED Rate limit or quota exceeded Yes: exponential backoff
500 INTERNAL Unexpected error on Google's side Yes: exponential backoff
503 UNAVAILABLE Service temporarily overloaded Yes: exponential backoff
504 DEADLINE_EXCEEDED Request timed out Maybe: raise client timeout or shrink prompt

Backoff Pattern

The SDK throws an ApiError with a .status field (the HTTP code). Branch on status, not string messages. Cap retries at 5. Apply exponential backoff with jitter.

import { ApiError } from "@google/genai";

const RETRYABLE = new Set([429, 500, 503]);
const sleep = (ms) => new Promise(r => setTimeout(r, ms));

async function withBackoff(fn, maxRetries = 5) {
  for (let attempt = 0; ; attempt++) {
    try {
      return await fn();
    } catch (err) {
      const status = err instanceof ApiError ? err.status : undefined;
      if (attempt >= maxRetries || !RETRYABLE.has(status)) throw err;
      // 1s, 2s, 4s, 8s... capped at 30s, plus up to 1s jitter
      await sleep(Math.min(2 ** attempt * 1000, 30_000) + Math.random() * 1000);
    }
  }
}
Safety Blocks Are Not Errors

When the model blocks a response for safety reasons, the API returns HTTP 200 with finishReason: "SAFETY" and no text content. res.text is undefined. This is not a transient error: retrying the same prompt returns the same block. Handle it as a separate case from network errors.

Free Tier vs Paid Tier

The free tier has tight per-minute (RPM) and per-day (RPD) quotas. The specific numbers vary by model and change over time. Check your project's live limits in the AI Studio console. Once billing is enabled, limits are much higher but you pay per token (text) and per image (images).

On the free tier, Google may use request and response data for model improvement. For any production workload handling PII or confidential user data, enable billing and verify the current data-use terms apply the appropriate opt-out.

Key Security

The GEMINI_API_KEY is server-side only. Call the API from Next.js Server Actions, API routes, or standalone scripts. Never reference the key in client components or let it land in the browser bundle. Restrict the key in the Google Cloud console to specific APIs and IP ranges where possible.

Cost Reality Check

Where Cost Accumulates

  • Text input tokens: counted against the prompt including any inline files or conversation history. Long context (1M tokens) means expensive calls if you stuff the full context every turn.
  • Text output tokens: usually priced higher than input. Streaming doesn't reduce token count; it just changes when you receive the bytes.
  • Cached tokens: Gemini supports prompt caching for repeated prefixes. Cached tokens are billed at a lower rate. Worth setting up for system prompts or large document prefixes you reuse across many calls.
  • Image generation: billed per image, not per token. Generate once and cache in object storage.
  • File API uploads: files themselves are not billed for storage during the 48-hour window. You pay for the tokens when the file is processed in a generateContent call.
Behavior Cost Impact Mitigation
Sending full chat history on every turn Grows linearly with turns Summarize old turns or use caching prefix
Inlining the same large PDF on every call Repeated token cost Upload once via File API, reuse URI for 48h
Regenerating images on each request Billed per image every time Generate once, cache bytes in R2/S3
No prompt caching on fixed system prompt Full system prompt billed every call Enable prompt caching on static prefix
Using Pro model for simple tasks 2-4x token cost vs Flash Default to Flash; escalate to Pro only when quality needs it

Key Takeaways

Principles

  1. The Developer API and Vertex AI are different paths. A key from AI Studio works only on generativelanguage.googleapis.com. A Maps key returns 403. A Vertex credential won't work here at all.
  2. Use @google/genai, not @google/generative-ai. The old package is deprecated. Mixing both in the same project causes confusing import collisions.
  3. Default to Flash, escalate to Pro. Gemini 2.5 Flash handles the vast majority of tasks at lower cost. Reach for Pro only when Flash gives inconsistent quality on a specific workload.
  4. Streaming doesn't change token count. Use it for UX (perceived latency), not for cost savings. The total billed tokens are the same either way.
  5. Inline small files; use File API for large ones. The 48-hour expiry means you need a strategy for files older than two days: re-upload or store raw bytes in your own storage.
  6. Generate images once and cache them. Image generation is billed per image. Generating the same image on every API call or page load is an easy cost leak to avoid with a CDN or object storage cache.
  7. Only retry 429, 500, 503. Retrying a 400 or 403 with no changes is pointless. Retrying a 429 without backoff makes the quota hole deeper.
  8. Keep the API key server-side. Never reference GEMINI_API_KEY in client components or anywhere that ends up in the browser bundle.