Function calling turns LLMs into production tools, but latency and error handling remain hard

Function calling has moved from experimental feature to production requirement. The capability lets LLMs invoke external tools (Salesforce CRUD operations, weather APIs, database queries) rather than just generate text. For enterprise architects, this means models can now orchestrate workflows, not just answer questions.

The production reality

Latency matters more than benchmarks suggest. Time-to-first-token (TTFT) compounds when models make sequential tool calls. A three-step workflow (query database, process result, update CRM) can easily hit 5-10 seconds even with optimized inference. Batching tool calls helps, but introduces complexity in error handling.

Error handling is where most implementations get messy. Network timeouts, API rate limits, and partial failures need retry logic that doesn't cascade into user-facing delays. Production systems require fallback strategies: what happens when the third tool call in a chain fails? Most agent frameworks punt this to developers.

What's actually shipping

Native function calling support varies widely. GPT-4, Claude, and Gemini handle it well. Open models like Llama and DeepSeek-R1 require more scaffolding. For enterprises prioritizing control and cost, open models win despite the integration tax.

Reasoning models (GPT-5.2, Claude Sonnet 4.5, o4-mini) add chain-of-thought before tool invocation. This improves accuracy on complex multi-step tasks but adds 20-40% latency overhead. The trade-off makes sense for high-value workflows, less so for simple API calls.

LlamaIndex and similar frameworks abstract the orchestration layer, but production deployments still need custom timeout handling, circuit breakers, and observability. The "new runtime era" referenced in recent discussions is less about model capabilities and more about the harnesses we build around them.

The enterprise calculus

For CTOs evaluating function calling systems: Start with narrow use cases where latency tolerance is high and error recovery is simple. Instrument everything. The technology works, but it's not yet fire-and-forget infrastructure.