Module 10 — Project: a language model demo

Nine modules of mechanisms. This module assembles them into a complete, published assistant that a colleague can open in their browser this afternoon. The design decisions are the ones we have discussed one at a time; the code below shows them working together, and the last section covers the real-world questions that only surface once a demo has traffic — feedback capture, cost, and the limits to make explicit.

The full app

The project is a streaming chat assistant with a system prompt, a temperature slider, three prefilled examples, a queue with concurrency_limit=1, a rate-bound timeout, and a Flag button that writes every flagged exchange to a CSV. It runs on a Hugging Face Space with an OpenAI-compatible backend, but the same code drops onto Ollama by swapping the base URL.

import os
import csv
import time
from datetime import datetime
from pathlib import Path

import gradio as gr
from openai import OpenAI

# ---------------------------------------------------------------------------
# Configuration read from environment. Repository secrets on the Space
# fill in OPENAI_API_KEY; the model name is set at build time.
# ---------------------------------------------------------------------------
API_KEY = os.environ["OPENAI_API_KEY"]
BASE_URL = os.environ.get("OPENAI_BASE_URL", "https://api.openai.com/v1")
MODEL = os.environ.get("MODEL", "gpt-4o-mini")

client = OpenAI(api_key=API_KEY, base_url=BASE_URL)

# ---------------------------------------------------------------------------
# Feedback log. Written to a local CSV; on a Space, mount a persistent
# volume in Settings, otherwise the log is lost on the next rebuild.
# ---------------------------------------------------------------------------
FEEDBACK_PATH = Path("feedback.csv")
if not FEEDBACK_PATH.exists():
    FEEDBACK_PATH.write_text("timestamp,rating,system,user,assistant\n", encoding="utf-8")


def log_feedback(rating: str, system_prompt: str, user: str, assistant: str) -> None:
    with FEEDBACK_PATH.open("a", newline="", encoding="utf-8") as fh:
        writer = csv.writer(fh)
        writer.writerow([datetime.utcnow().isoformat(), rating, system_prompt, user, assistant])


# ---------------------------------------------------------------------------
# Chat callback. Streams the response, with a wall-clock timeout to keep
# the queue moving. All Gradio-friendly errors go through gr.Error.
# ---------------------------------------------------------------------------
def chat_stream(message: str, history: list[dict], system_prompt: str, temperature: float):
    if not message.strip():
        raise gr.Error("Empty message. Type a question first.")

    messages = [{"role": "system", "content": system_prompt}]
    messages.extend(history)
    messages.append({"role": "user", "content": message})

    try:
        stream = client.chat.completions.create(
            model=MODEL,
            messages=messages,
            temperature=temperature,
            stream=True,
            timeout=30,
        )
    except Exception as exc:
        raise gr.Error(f"The backend is unavailable ({exc.__class__.__name__}). Please retry.")

    reply = ""
    started = time.monotonic()
    for chunk in stream:
        delta = chunk.choices[0].delta.content or ""
        reply += delta
        yield reply
        if time.monotonic() - started > 45:
            reply += "\n\n[Cut off after 45 s.]"
            yield reply
            return


# ---------------------------------------------------------------------------
# Layout. ChatInterface for the main experience, a separate Blocks tab
# for the feedback form so the log grows on real, curated exchanges.
# ---------------------------------------------------------------------------
DEFAULT_SYSTEM = (
    "You are a concise technical assistant. Answer in at most three sentences, "
    "and refuse politely when the request is outside programming, data or ML."
)

with gr.Blocks(title="Streaming assistant", theme=gr.themes.Soft()) as demo:
    gr.Markdown(
        "# Streaming assistant\n"
        "A minimal, opinionated chat demo built from the ten modules of course 39. "
        "Read the tips before submitting your first question."
    )

    chat = gr.ChatInterface(
        fn=chat_stream,
        additional_inputs=[
            gr.Textbox(value=DEFAULT_SYSTEM, label="System prompt", lines=3),
            gr.Slider(0.0, 1.5, value=0.4, step=0.05, label="Temperature"),
        ],
        examples=[
            "Explain gradient descent in three sentences.",
            "What is a common misuse of accuracy on an imbalanced dataset?",
            "Refactor for clarity: def f(x): return sum(i*i for i in x)/len(x)",
        ],
        cache_examples=False,
        description="Streaming answers, at most 45 seconds per turn.",
    )

    with gr.Accordion("Rate the last exchange", open=False):
        rating = gr.Radio(["good", "neutral", "bad"], label="Your rating", value="neutral")
        submit = gr.Button("Send feedback", variant="secondary")
        status = gr.Markdown()

        def send_feedback(rating_value, chat_history, system_prompt, temperature):
            # chat_history is a list of dicts in modern Gradio. Grab the last exchange.
            last_user = ""
            last_assistant = ""
            for turn in reversed(chat_history):
                if turn["role"] == "assistant" and not last_assistant:
                    last_assistant = turn["content"]
                elif turn["role"] == "user" and last_assistant and not last_user:
                    last_user = turn["content"]
                    break
            log_feedback(rating_value, system_prompt, last_user, last_assistant)
            return "Feedback recorded. Thank you."

        submit.click(
            fn=send_feedback,
            inputs=[rating, chat.chatbot, chat.additional_inputs[0], chat.additional_inputs[1]],
            outputs=status,
        )

demo.queue(default_concurrency_limit=1, max_size=25)
demo.launch()

Every part of this file points back to a module. The generator with a cumulative yield is module 5. The system prompt as an additional_input is module 4. The three examples are module 6. The queue and the wall-clock timeout are module 7. The reading of OPENAI_API_KEY from the environment and the deployment on a Space are module 9. And the entire skeleton — Blocks, a Chat, an Accordion — is the composition of module 3.

Files to accompany the app

The Space needs three companions in the repo. Get all four right and the demo deploys on the next git push.

app.py                # the file above
requirements.txt      # gradio>=4.36, openai>=1.30
README.md             # SDK frontmatter and a description
.gitignore            # __pycache__, gradio_cached_examples/, .env, feedback.csv

The README.md frontmatter follows module 9's template. Set sdk: gradio, sdk_version: 4.44.0 (or whatever the current stable release is), and app_file: app.py.

Feedback capture and the loop that improves the demo

The Accordion opens a lightweight feedback form. The point is not the code, which is trivial, but the routine: at the end of every day of demo traffic, open feedback.csv, look at the "bad" entries, read the exchanges, and adjust the system prompt or the examples accordingly. A demo without a feedback path drifts silently; a demo with one gets better week after week without a single new line of code.

On a Space, the feedback.csv sits in the container's temporary filesystem and vanishes on the next build. Two options: enable a persistent volume in Settings → Storage (paid), or POST the feedback to a small external service — a Google Sheet through a webhook, a Supabase table, a self-hosted Postgres. The right answer depends on whether the feedback is regulated (customer data) or not (public evaluation).

What you cannot log

Never log a chat that contains identifiable user data unless you have a documented consent flow. In practice: put a one-line notice above the demo — "Conversations may be recorded to improve the model" — that a reviewer can read before typing. It costs nothing to display, and it is often the difference between "we can use this data" and "we have to delete it".

Costs and limits, honestly

A hosted-API demo costs are proportional to tokens. Rough orders of magnitude at the time of writing: a small model like gpt-4o-mini bills a few tens of cents per million tokens; a larger model like gpt-4o costs an order of magnitude more. A 200-token answer with a 300-token history is roughly USD 0.0002 on the small model, USD 0.003 on the larger one. A hundred users a day at ten turns each puts the small model at USD 0.20 per day, the larger one at USD 3 — usable numbers.

A self-hosted demo on a Space's paid GPU is different: you pay for the hour of runtime, whether or not someone is using it. A T4 small at roughly USD 0.60 per hour equals USD 15 per day of continuous availability. That is only cheaper than the hosted API if your traffic is dense; below a few hundred requests per hour, the API is the cheaper answer.

The limits worth writing on the demo itself are equally boring and equally important: a model is not a knowledge base, its answers can be wrong, and the traffic to this demo is not free. Two sentences in the description spare you a class of misuse that comes down to visitors who trust the tool for something it was never built to do.

In summary

The full app composes what the nine previous modules taught: a chat, streaming, a system prompt, examples, a queue, timeouts, secrets, feedback.
Each design decision points back to one module — read the comments in the code to trace them.
Log feedback into a file or a service, and iterate on the demo weekly from the "bad" entries.
Publish the costs and the limits in the description: they save more misuse than any technical defense.

Final step: the recap and the 40-question exam.

The full app​

Files to accompany the app​

Feedback capture and the loop that improves the demo​

Costs and limits, honestly​

In summary​

The full app

Files to accompany the app

Feedback capture and the loop that improves the demo

Costs and limits, honestly

In summary