Using Temporal to Optimize LLM Requests

When building AI-powered features, two operational concerns appear immediately: latency and cost. Calling an LLM third party service typically takes 1 to 3 seconds and incurs token costs on every request.

We recently tackled this while building AI features for Molita.co. Imagine 10 users asking the exact same question (for example, “What is Ikigai?”) at the same time. In a naive setup, your backend fires 10 simultaneous API calls, burns 10x the tokens, and risks hitting external rate limits (429 Too Many Requests).

This is where Query Coalescing (also known as the Single-Flight pattern) comes in.

What is Query Coalescing? Link to heading

Query coalescing ensures that if multiple callers request the exact same expensive operation while one is already in progress, the system executes the operation only once and returns the same result to all waiting callers.

“Simultaneous” does not mean within nanoseconds. It spans the entire 1–3 second window that the LLM needs to generate a response. Any request arriving while the first execution is RUNNING simply shares that single in-flight execution.

How It Works in Temporal Link to heading

Implementing single-flight across multiple web servers usually requires Redis distributed locks, pub/sub channels, and tricky race-condition cleanup.

Temporal handles this out of the box using three primitives:

1. Deterministic Workflow IDs Link to heading

Normalize the user’s input (trim whitespace, lowercase, strip trailing punctuation) and compute a deterministic hash:

import { createHash } from "node:crypto";

function getWorkflowId(query: string): string {
  const normalized = query.trim().toLowerCase().replace(/[?!.]*$/, "");
  const hash = createHash("sha256").update(normalized).digest("hex");
  return `ask-cache-${hash}`;
}

Note: This is an intentionally simple normalization example. In production, you should include the model name/version and prompt version in the hash to avoid cache pollution across model updates. You can also make normalization more robust with Unicode normalization (NFKC), collapsing redundant whitespace, or even using fast embeddings/semantic clustering to coalesce differently phrased questions with the same intent.

Because identical questions generate identical IDs, Temporal’s cluster-wide registry treats them as the exact same logical unit of work across all API servers.

2. In-Flight Coalescing & Conflict Policy Link to heading

When starting the workflow, configure Temporal’s conflict policy to attach to an active run instead of throwing an error:

const workflowId = getWorkflowId(query);

const handle = await client.workflow.start("askWorkflow", {
  taskQueue: "ask-queue",
  workflowId,
  args: [query],
  workflowIdConflictPolicy: "USE_EXISTING",
});

// Awaits the single execution whether started by this request or a peer
const result = await handle.result();

• Sub-Millisecond Concurrency: If two requests hit different API servers at the exact same millisecond, workflowIdConflictPolicy: "USE_EXISTING" ensures the second request silently reuses the first workflow without throwing a duplicate ID error.
• The 1–3 Second Window: If another request arrives 500ms later while the workflow status is RUNNING, calling handle.result() awaits until that active execution finishes.

3. Seamless Transition to Cache Link to heading

Once the LLM activity returns, the workflow transitions to COMPLETED.

Subsequent requests that arrive minutes or hours later simply read the completed execution history via handle.result() within the namespace retention window.

There is no need for a separate Redis cache—Temporal’s durable execution history is your cache.

Why Temporal Over In-Memory or Redis? Link to heading

The Takeaway Link to heading

At Molita.co, combining deterministic workflow IDs with workflowIdConflictPolicy: "USE_EXISTING" completely removed duplicate LLM calls during traffic spikes:

1. One LLM call is executed.
2. All waiting users receive the answer simultaneously.
3. Future callers receive cached results instantly.
4. Zero distributed lock plumbing required in application code.

Summary Link to heading