Groq

Overview

Groq is an AI inference company founded in 2016 by Jonathan Ross. Its core bet is custom silicon: the LPU (Language Processing Unit), described by the company as the first chip purpose-built for inference rather than training. The product most developers touch is GroqCloud, which exposes that hardware as an OpenAI-compatible API and web console. Groq's positioning is blunt: "fast, low cost inference."

Crucially, Groq does not build its own foundation models. It serves other people's open-weight models (Llama, Qwen, GPT-OSS, Kimi, Mistral, Whisper, and more) on LPU hardware, competing on speed and price. So in this directory it is a platform / inference layer, not an agent. Its honest autonomy level is assistant: by default it responds to a request and returns tokens. It gains agentic surface (tool use, MCP, the Compound feature) when a developer wires it into an application, but the orchestration and approvals stay in that application.

What it does

GroqCloud runs LLM, speech-to-text, text-to-speech, and image-to-text models on LPU chips. Key capabilities per the docs and pricing page:

• High-throughput inference on open models, marketed for fast token output and low, linear per-token cost.
• OpenAI-compatible API at https://api.groq.com/openai/v1, so existing OpenAI SDK code can be pointed at Groq with minimal changes.
• Function calling, structured outputs, and built-in tools (web search, website visits, code execution, Wolfram Alpha).
• Speech models: Whisper-family transcription and text-to-speech (including Orpheus) for real-time voice.
• Compound (agentic) systems plus remote tools and MCP, including Google Workspace connectors (Gmail, Calendar, Drive).
• Batch API for asynchronous jobs at a reported 50% discount versus on-demand.

Groq reports serving roughly 3 million developers and teams, with named customers including Dropbox, Vercel, Canva, Robinhood, and the McLaren F1 Team (per its homepage).

Integrations & setup

Because the API is OpenAI-compatible, integration is close to a drop-in: swap the base URL and key in an OpenAI SDK call, or use ecosystem libraries like LangChain and the Vercel AI SDK. The platform supports MCP and remote tools (with Google Workspace connectors) for tool-augmented inference. For regulated or air-gapped environments, Groq offers on-premises hardware (GroqRack, and the LPX rack introduced with Nvidia at GTC 2026).

Pricing

Groq uses usage-based, tokens-as-a-service pricing it describes as "linear and predictable" with no idle-infrastructure fees. Reported figures from its pricing page:

• Free tier to start (free API key).
• On-demand per-token pricing, e.g. Llama 3.1 8B at about $0.05 input / $0.08 output per million tokens, up to Llama 3.3 70B at about $0.59 / $0.79.
• Batch API at a reported ~50% discount for asynchronous workloads.
• Speech: text-to-speech around $22-40 per million characters; speech recognition around $0.04-0.11 per hour transcribed.
• Enterprise / on-prem custom pricing.

Exact prices and the model catalog change over time; check https://groq.com/pricing.

Best for / not for

Best for developers building latency-sensitive apps (chat, voice, agent backends) who want fast, cheap inference on open models behind a familiar OpenAI-style API, and teams that need an on-prem inference option.

Not for teams that need a managed, end-to-end agent out of the box (Groq gives you the engine, not the autopilot), or that require a proprietary frontier model. Buyers should also weigh the uncertainty introduced by the December 2025 Nvidia licensing deal, which is under regulatory scrutiny even though GroqCloud reportedly continues operating independently.

Alternatives

The closest comparisons are other fast / open-model inference providers: Together AI, Fireworks AI, Cerebras (another custom-silicon inference play), OpenRouter (a routing aggregator), and Replicate. Against general-purpose API providers like OpenAI, the tradeoff is open models and inference speed versus proprietary frontier models.