Groq API Tutorial: Build Ultra-Fast LLM Apps in Python (2026 Guide)

Groq is not another LLM provider in the same category as OpenAI or Anthropic. It is a hardware company that built a chip called the LPU (Language Processing Unit) specifically designed for transformer inference. The result is LLM speeds that are genuinely in a different class: 300 to 800 tokens per second depending on the model, versus 40 to 80 tokens per second from OpenAI. If your application has any real-time component - voice interfaces, live coding assistants, interactive agents - Groq changes what is possible. I use it in production for LingoScribe, my Urdu speech-to-text pipeline, and have run it under serious load. This guide covers everything you need to get started.

Why Groq Is Fast: The LPU

Standard GPU inference is fast at parallel matrix multiplication but has high memory bandwidth overhead for sequential token generation. LLMs generate one token at a time, which means the bottleneck during inference is memory access, not compute. The LPU is designed around this: it eliminates the memory bandwidth bottleneck that throttles GPU inference, which is why you get 10x the speed with equivalent model quality. The models running on Groq are the same open-weight models you can run elsewhere - the hardware is what changes.

Available Models on Groq

• -llama-3.3-70b-versatile - Best all-around. Strong reasoning, follows instructions well, fast. The default choice for most tasks.
• -llama-3.1-8b-instant - Extremely fast, very low cost. Use for classification, extraction, and simple tasks where speed matters more than nuance.
• -mixtral-8x7b-32768 - 32k context window. Good for long document processing.
• -gemma2-9b-it - Google Gemma 2 on Groq hardware. Strong on structured output tasks.
• -whisper-large-v3 - OpenAI Whisper for audio transcription. This is what LingoScribe uses for Urdu STT.
• -whisper-large-v3-turbo - Faster Whisper, slightly lower accuracy. Good for non-critical transcription at scale.

Setup

pip install groq

Get your API key from console.groq.com (free tier available, generous rate limits). Set it as an environment variable - never hardcode it in source files.

export GROQ_API_KEY=your_key_here

# Or in .env (use python-dotenv to load it)
GROQ_API_KEY=your_key_here

Basic Chat Completion

import os
from groq import Groq

client = Groq(api_key=os.environ.get("GROQ_API_KEY"))

response = client.chat.completions.create(
    model="llama-3.3-70b-versatile",
    messages=[
        {"role": "system", "content": "You are a senior Python developer."},
        {"role": "user", "content": "Write a FastAPI endpoint that accepts JSON and validates it."},
    ],
    temperature=0.3,
    max_tokens=1024,
)

print(response.choices[0].message.content)
print(f"Tokens: {response.usage.total_tokens}")
print(f"Speed: {response.usage.completion_tokens / response.usage.total_time:.0f} t/s")

The response object is compatible with the OpenAI Python SDK structure. If you are migrating from OpenAI, replacing the client instantiation is often all that changes.

Streaming for Real-Time Output

For any UI that shows the model output as it generates, use streaming. Groq streaming is low-latency end to end - the time to first token is typically under 200ms for most models.

stream = client.chat.completions.create(
    model="llama-3.3-70b-versatile",
    messages=[{"role": "user", "content": "Explain transformers in depth."}],
    stream=True,
)

for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)

Audio Transcription with Groq Whisper

Groq hosts Whisper large-v3, which runs significantly faster on their hardware than on standard GPU instances. I use this in LingoScribe for Urdu and Pakistani Punjabi transcription. The language parameter forces the transcription language, which prevents Whisper from outputting in the wrong script when the audio is ambiguous.

import os
from groq import Groq

client = Groq(api_key=os.environ.get("GROQ_API_KEY"))

with open("audio.mp3", "rb") as audio_file:
    transcription = client.audio.transcriptions.create(
        file=audio_file,
        model="whisper-large-v3",
        language="ur",          # force Urdu / Perso-Arabic script
        response_format="verbose_json",
    )

# Access full segment data
for seg in transcription.segments:
    print(f"[{seg.start:.1f}s - {seg.end:.1f}s] {seg.text.strip()}")

The verbose_json format returns per-segment timestamps, which is essential for building synchronized transcription UIs. For a straightforward transcription with no timestamp data, use response_format='text' instead.

Using Groq with LangChain

The langchain-groq package gives you a drop-in LangChain chat model. Replace ChatOpenAI or ChatAnthropic with ChatGroq and your existing chains and agents work immediately.

pip install langchain-groq

from langchain_groq import ChatGroq
from langchain_core.messages import HumanMessage, SystemMessage

llm = ChatGroq(
    model="llama-3.3-70b-versatile",
    temperature=0,
    groq_api_key=os.environ.get("GROQ_API_KEY"),
)

messages = [
    SystemMessage(content="You extract structured data from text."),
    HumanMessage(content="Extract the name and email from: John Smith, john@example.com"),
]

response = llm.invoke(messages)
print(response.content)

ChatGroq also supports tool calling via bind_tools(), which makes it fully compatible with LangGraph agents. This is the exact setup I use for production agentic pipelines that need high-throughput inference.

Structured Output with JSON Mode

For classification, extraction, and any task that needs reliably parseable output, Groq supports JSON mode. Set response_format to json_object and include a JSON instruction in your system prompt.

response = client.chat.completions.create(
    model="llama-3.3-70b-versatile",
    messages=[
        {
            "role": "system",
            "content": "Extract data as JSON with keys: name, email, company.",
        },
        {"role": "user", "content": "Farhan, farhanwork0318@gmail.com, Datraxa"},
    ],
    response_format={"type": "json_object"},
    temperature=0,
)

import json
data = json.loads(response.choices[0].message.content)
print(data)  # {"name": "Farhan", "email": "farhanwork0318@gmail.com", "company": "Datraxa"}

When to Use Groq vs Other Providers

Use Groq when:

• -Speed is a product requirement - voice interfaces, live coding agents, real-time processing
• -You need cost-effective high-throughput inference at scale
• -You are using open-weight models (Llama 3.3, Mixtral, Gemma) and want the fastest inference
• -You need Whisper transcription faster than standard hosted Whisper

Do not use Groq when:

• -You need GPT-4o vision (Groq does not support image inputs currently)
• -You need Claude for long-context reasoning over 128k tokens
• -Your task needs the highest possible accuracy and the best closed-source models outperform Llama 3.3 for it

Rate Limits and Error Handling

Groq rate limits are per-model per-minute. On the free tier the limits are generous for development but you will hit them under production load. Use exponential backoff for rate limit errors and consider caching responses for repeated identical queries.

import time
from groq import RateLimitError

def call_with_retry(client, **kwargs):
    for attempt in range(5):
        try:
            return client.chat.completions.create(**kwargs)
        except RateLimitError:
            wait = 2 ** attempt
            print(f"Rate limited, retrying in {wait}s...")
            time.sleep(wait)
    raise Exception("Max retries exceeded")

Frequently Asked Questions

Is Groq production-ready?

Yes. Groq has been running production workloads for over a year at this point and the uptime has been excellent. The main constraint is model selection - you are limited to the open-weight models they support, not GPT-4 or Claude. If Llama 3.3 70B meets your accuracy bar, Groq is a solid production choice.

How much does Groq cost vs OpenAI?

Groq is roughly 20-50x cheaper per token than GPT-4o for comparable model sizes. Llama 3.3 70B on Groq costs around $0.05-0.09 per million tokens. GPT-4o costs $2.50-10 per million tokens depending on input/output. For high-volume use cases, the cost difference is significant.

Can I use Groq for function calling and tool use?

Yes. Groq supports OpenAI-compatible function calling on Llama 3.3 70B and Llama 3.1 8B. The syntax is identical to OpenAI tool calling. LangChain and LangGraph agents that use bind_tools() work out of the box with ChatGroq.