Skip to main content

Module 2 — Text, image, audio and video components

Module 1 called the components "inputs" and "outputs" without saying much about the data they carry. That imprecision hides the most common source of bugs in a Gradio demo: the function expects one shape and the component delivers another, so the model returns nonsense on the first user click. This module maps each component to the exact Python object it hands over, and shows how to render results the way the user expects.

What the image component actually gives you​

gr.Image has a single argument that decides everything about the demo's contract with the function: type. It accepts three values, and each corresponds to a common way of consuming an image.

  • type="numpy" (the default) hands your function a NumPy array of shape (H, W, 3) and dtype uint8, with pixel values from 0 to 255 in RGB order. This is what OpenCV, scikit-image and most classical pipelines expect — with one caveat: OpenCV reads BGR from disk, so if you copy-paste code from an OpenCV tutorial, remember to flip the channels or you will get a blue-tinted world.
  • type="pil" hands over a PIL.Image.Image, which is what Hugging Face pipeline calls and most torchvision.transforms chains want as input. That is the choice we made in module 1.
  • type="filepath" writes the image to a temporary file and hands over the path as a string. Use it for models that only know how to read from disk, like some ffmpeg-based or onnxruntime command-line tools; for anything else it is slower and less clean than the two options above.
import gradio as gr
import numpy as np

def sharpness_index(image: np.ndarray) -> float:
    # Laplacian variance is a coarse proxy for image sharpness.
    from scipy.ndimage import laplace
    gray = image.mean(axis=2)
    return float(laplace(gray).var())

demo = gr.Interface(
    fn=sharpness_index,
    inputs=gr.Image(type="numpy", label="Upload a photo"),
    outputs=gr.Number(label="Sharpness index"),
)

The most frequent trap is a function written for PIL receiving a NumPy array, or the opposite. The traceback is loud but the fix is trivial: set type explicitly instead of relying on the default. Never leave it implicit in a demo you plan to share.

Ranked probabilities with gr.Label​

For classification, the output component that produces the recognizable horizontal-bar layout is gr.Label. The contract is symmetrical to the image side: give the function a dictionary mapping labels to probabilities, and Gradio sorts them, formats them as percentages, and cuts the list at num_top_classes.

import gradio as gr
import numpy as np
from transformers import pipeline

classifier = pipeline("image-classification", model="google/vit-base-patch16-224")

def classify(image):
    predictions = classifier(image)
    return {p["label"]: float(p["score"]) for p in predictions}

gr.Interface(
    fn=classify,
    inputs=gr.Image(type="pil"),
    outputs=gr.Label(num_top_classes=3),
).launch()

The probabilities do not need to sum to one for Gradio to render them; the component only sorts. That flexibility matters when your model outputs raw logits or independent binary scores. Two rules keep the display honest, though: probabilities should be float, never numpy.float32, or JSON serialization can complain, and the total should be at most 1 for a mutually exclusive classifier — otherwise a user reading the bars will silently misinterpret the model.

Audio: array, sample rate, or file path​

gr.Audio follows the same type idea, with an extra dimension because a sound has a sampling rate.

  • type="numpy" (the default for the input side) hands the function a tuple (sample_rate: int, audio: np.ndarray). The array is either 1-D for mono or 2-D of shape (samples, channels) for stereo. This is what you feed to a classifier trained on log-Mel spectrograms.
  • type="filepath" hands over a path to a temporary WAV file; use it for models that call librosa.load or ffmpeg themselves.

On the output side, returning (sample_rate, np.ndarray) renders an audio player with a waveform, and returning a file path plays that file. Whisper and other speech-to-text models return text, which pairs naturally with gr.Textbox as the output.

import gradio as gr
from transformers import pipeline

asr = pipeline("automatic-speech-recognition", model="openai/whisper-small")

def transcribe(audio):
    if audio is None:
        return ""
    sample_rate, waveform = audio
    result = asr({"sampling_rate": sample_rate, "raw": waveform.astype("float32")})
    return result["text"]

demo = gr.Interface(
    fn=transcribe,
    inputs=gr.Audio(sources=["microphone", "upload"], type="numpy", label="Speak or upload"),
    outputs=gr.Textbox(label="Transcript", lines=6),
    title="Audio transcription",
    description="Whisper turns speech into text; module 3 will feed this output into a summarizer.",
    allow_flagging="never",
)
demo.launch()

Three details in this snippet earn a mention. Setting sources=["microphone", "upload"] gives the user both ways of providing audio, which is what people expect on a demo. Whisper wants a float32 waveform, so the cast is mandatory: without it the model runs on int16 values and produces gibberish. And the if audio is None guard exists because a user who clicks submit before recording anything sends None, which would otherwise crash the demo — a small robustness gesture that pays off every day in front of a live audience.

Video, and a quick tour of the other components​

gr.Video delivers a file path to a temporary MP4 by default. Reading it with imageio or decord gives you frames as NumPy arrays if you need them. For real-time streaming a webcam, add streaming=True, but expect much more code — that use case belongs in a specialized recipe.

The other components you will meet are almost self-describing: gr.Number, gr.Slider, gr.Checkbox, gr.Radio, gr.Dropdown on the input side; gr.JSON, gr.Dataframe, gr.HTML, gr.Markdown, gr.Gallery on the output side. All follow the same principle: pick the widget that matches the natural Python object, and the demo reads itself.

In summary​

  • gr.Image returns a NumPy array by default, but choose type="pil" or type="filepath" explicitly — do not rely on the default.
  • gr.Label expects a dictionary of label to probability; float values, at most one summed to keep the bars honest.
  • gr.Audio gives a tuple (sample_rate, waveform); cast to float32 before feeding Whisper and guard against None.
  • Every component maps to a natural Python type: pick the widget so the function's signature reads like the model's contract.

Next module: composing these components into a full pipeline with gr.Blocks, where transcription flows into summarization on a single page.