Skip to main content

Module 7 — Queue and concurrent load

A demo that answers one visitor in three seconds is a demo. A demo that gets fifty visitors from a link on a conference slide, all clicking at once on a single GPU, is a queueing problem — and if you have not thought about it, the answer is a spinner that never resolves, or worse, an out-of-memory error that crashes the process for everyone. This module explains the queue Gradio ships by default, the two knobs that shape its behavior, and how to sanity-check the numbers before the demo is public.

The queue is on by default, and here is why​

Since Gradio 4, every Interface and every Blocks app runs its callbacks through an internal queue. That was a change from earlier versions, where each user's request spawned a worker thread and the server hoped it had enough resources to serve them all in parallel. With a GPU model that takes 2 GB of VRAM per call, four concurrent users on an 8 GB card meant an immediate crash; the queue prevents that by design.

You can inspect the queue behavior with queue() on the Blocks or Interface object, and you can tune two dimensions.

import gradio as gr

def slow_task(prompt: str) -> str:
    import time
    time.sleep(2)
    return prompt.upper()

demo = gr.Interface(fn=slow_task, inputs="text", outputs="text")

demo.queue(
    default_concurrency_limit=2,  # at most 2 requests running in parallel
    max_size=25,                   # at most 25 users waiting in line
)
demo.launch()

Concurrency: what "2 in parallel" really means​

default_concurrency_limit=2 means Gradio will start at most two invocations of the callback simultaneously. If a third user submits, their request is queued and waits for one of the first two to finish. That number should reflect the resource the callback consumes, not a generic "how fast do I want it".

For a small text function that runs in a few milliseconds, default_concurrency_limit=10 costs nothing and reduces waiting to zero. For a language model that uses 8 GB of VRAM per batch, default_concurrency_limit=1 is often the honest answer: two calls at once would either crash on out-of-memory or halve each user's tokens-per-second. Better to serve one at a time cleanly than to fail intermittently under load. Modern GPUs and libraries can batch multiple sequences, in which case default_concurrency_limit=4 or 8 is achievable, but the safe default when you do not know is one.

Per-event limits override the default:

btn.click(fn=heavy_model, inputs=x, outputs=y, concurrency_limit=1)
btn_light.click(fn=cheap_task, inputs=x, outputs=y, concurrency_limit=8)

The first heavy call is serialized; the second cheap call runs eight at a time. That granularity is important on demos that mix a model call with lightweight helpers, so a queued user still gets instant feedback from the cheap paths.

Max size, or how to refuse politely​

max_size caps the queue itself. If the queue is full, additional requests are refused immediately with a clear "The queue is full" error rather than piling up until the server dies. Pick a number based on the average wait time you find acceptable.

The math is simple. If each call takes 3 seconds and concurrency_limit=1, the tenth person in the queue waits 30 seconds. The twentieth waits a full minute. Above that, most visitors will close the tab, so max_size between 15 and 30 is a sane range. For a conference demo, err on the low side: refusing the fiftieth click with a message tells the sender to try later, while a two-minute silent wait wastes their attention.

Request timeouts​

An individual call can also hang — a network stall to an external API, an unusually long prompt, a stuck generation. .click(...) and .submit(...) accept an api_open parameter and a wall-clock timeout on some backends; the more portable way to bound a single call is to use concurrency_id groups and to enforce your own timeout in Python:

import gradio as gr
import time
from concurrent.futures import ThreadPoolExecutor, TimeoutError

_executor = ThreadPoolExecutor(max_workers=4)

def bounded(fn, timeout):
    def wrapped(*args, **kwargs):
        future = _executor.submit(fn, *args, **kwargs)
        try:
            return future.result(timeout=timeout)
        except TimeoutError:
            future.cancel()
            raise gr.Error("The request took too long. Please try a shorter prompt.")
    return wrapped

Wrap bounded(chat_stream, timeout=30) around the callback and Gradio will show the error message to the user cleanly, without leaving the request hanging. gr.Error is the right way to raise a message that reaches the UI without a full traceback.

Measuring the demo under load​

Guessing is a bad way to size a queue. Ten minutes with a load-testing tool teaches more than a week of tuning. locust and k6 are the two mainstream options; the smaller hey tool suffices for a first read.

# Fire 50 requests with 5 concurrent virtual users to the Gradio API endpoint.
hey -n 50 -c 5 -m POST \
    -H "Content-Type: application/json" \
    -d '{"data":["a short prompt to summarize"]}' \
    http://127.0.0.1:7860/api/predict

The endpoint is derived from your event's function; open http://127.0.0.1:7860/?view=api in a browser to see the exact URL and payload shape for each function. The numbers to watch are the p50 and p95 response times (median and 95th percentile) and the error rate. On a healthy demo the p95 stays within twice the p50 as concurrent users grow up to default_concurrency_limit; beyond that, it rises linearly because you are timing the queue wait. That is expected, and it is what your max_size bounds.

Two profiles, two demos

A recruiter or a client wants a demo that never fails, with a queue that gracefully turns overflow away. A live audience during a talk wants a demo that answers within seconds. If your visit rate can burst, either provision more hardware or duplicate the Space (module 9) and put a load balancer in front — one Gradio process cannot be both.

Streaming, queues, and the numbers that count​

Streaming plays well with the queue: the callback holds its slot while it yields tokens, so users behind it wait for the current stream to finish. That is why the tokens-per-second target of module 5 matters twice — once for perceived speed, once for how many people you can serve per minute. A model that streams at 40 tokens per second on a good GPU delivers a 200-token answer in 5 seconds, so a concurrency_limit=1 serves 12 users per minute, or 720 per hour. Below those numbers, you have room; above, plan for more hardware.

In summary​

  • The queue is on by default; default_concurrency_limit and max_size are the two knobs, one per parallel workers, one per waiting-line capacity.
  • Set concurrency_limit=1 for heavy GPU callbacks unless you have measured that a higher value fits in memory and does not degrade tokens per second.
  • Cap the queue with max_size so overflow is refused politely instead of silently timing out.
  • Measure with hey, k6 or locust before going public: p50, p95 and error rate under the traffic pattern you actually expect.

Next module: the temporary share link, share=True, and what it exposes beyond your Wi-Fi.