Skip to main content

Module 8 — Calling a model from the application

The dashboard has widgets, filters, layout, caches and state; it can now afford to make a real prediction. This module wires the two plausible sources of that prediction: a model loaded locally and a remote inference API. The choice is not neutral — it changes what the app must handle, and this module walks each of them through with the code the running example ends up shipping.

Two deployment shapes, two contracts​

Local model. The model file lives on disk next to the app. The process loads it once with @st.cache_resource (module 5) and calls .predict_proba(features) on every scoring request. Latency is in-process microseconds. The cost is that every replica of the app carries the full model in memory, and updating the model means redeploying the app.

Remote API. A separate service — Flask, FastAPI, TorchServe, a managed endpoint — holds the model behind an HTTP contract. The Streamlit app becomes a client. Multiple apps can share the same model, scaling is decoupled from the UI, and updating the model does not touch Streamlit. The cost is a network hop, a serialization layer and the failure modes of any HTTP call: timeouts, retries, 5xx.

Small internal teams start local, then graduate to a remote API once the model is used by more than one app or the training team owns its own release cycle. Course 40 covers building such an API; this module covers what the client side must do to consume it correctly.

The local path​

The two lines that matter live in module 5, and they belong at the top of the file so every callable in the module sees the same instance.

import streamlit as st
import joblib
import numpy as np

@st.cache_resource(show_spinner="Loading the churn model...")
def get_model():
    return joblib.load("models/churn_v3.joblib")

FEATURES = ["tenure", "monthly_charge", "contract_encoded", "payment_encoded"]

def score_local(features: dict) -> float:
    """Return the churn probability for one customer."""
    model = get_model()
    vector = np.array([[features[name] for name in FEATURES]])
    return float(model.predict_proba(vector)[0, 1])

Two habits carry over from the training code. Order the features explicitly with a FEATURES list — a dict-order dependency between training and inference has cost more than one production incident. And never mutate the loaded model at inference time; if you must augment it (a lookup table, a threshold), keep those alongside the model in a small dict returned by get_model(), not on the model object.

The remote API path​

For a remote model, the client speaks HTTP. requests is the standard, httpx a modern alternative with the same synchronous API. Both accept timeout= and both raise on network errors, which is exactly what you want.

import requests

INFERENCE_URL = st.secrets.get("INFERENCE_URL", "http://localhost:8000/predict")
INFERENCE_TOKEN = st.secrets.get("INFERENCE_TOKEN", "")

def score_remote(features: dict, timeout: float = 3.0) -> float:
    response = requests.post(
        INFERENCE_URL,
        json={"features": features},
        headers={"Authorization": f"Bearer {INFERENCE_TOKEN}"},
        timeout=timeout,
    )
    response.raise_for_status()
    return float(response.json()["probability"])

st.secrets is populated from .streamlit/secrets.toml in development and from the platform's secret manager in production — the mechanism is covered in module 10. Two habits worth taking on now. Always pass a timeout= (the default is no timeout, which will hang the browser tab indefinitely on a stuck server). And .raise_for_status() turns a 500 or a 401 into an exception you can catch, instead of letting the happy-path .json()["probability"] crash later with an unhelpful KeyError.

Error handling, the version the user sees​

The reason to handle errors well is that the sales team should never see a Python traceback. Three exceptions cover 95 % of what a remote call throws.

import requests

def score_with_feedback(features: dict) -> float | None:
    try:
        return score_remote(features, timeout=3.0)
    except requests.Timeout:
        st.error("The scoring service did not answer in time. Please try again.")
    except requests.HTTPError as e:
        if e.response.status_code == 401:
            st.error("Your session expired. Please refresh the page.")
        else:
            st.error(f"The scoring service returned an error ({e.response.status_code}).")
    except requests.RequestException:
        st.error("Cannot reach the scoring service. Is it running?")
    return None

Two design decisions in that helper. The function returns None on failure rather than re-raising, so the calling code stays a linear "score, check, display". And the messages talk in the user's terms ("session expired"), never in the framework's ("HTTP 401"). Log the original exception (logging.exception(...)) for the on-call engineer, but do not paste it into the UI.

Loading indicators and cancellation​

A synchronous HTTP call blocks the run. st.spinner is the minimum courtesy owed to the user during that block:

with st.spinner("Scoring the customer..."):
    probability = score_with_feedback(features)

if probability is not None:
    st.metric("Churn probability", f"{probability:.0%}", delta_color="inverse")

For a long batch — module 7's five-thousand-row upload — the spinner becomes a progress bar and the loop is split into chunks so the UI updates. Streamlit does not offer a "cancel" gesture on a running Python call; the closest is st.button("Stop") combined with a chunked loop that reads st.session_state.stop_requested between chunks:

if st.button("Stop"):
    st.session_state.stop_requested = True

for chunk in chunks(customers, size=200):
    if st.session_state.get("stop_requested"):
        st.warning("Stopped by the user.")
        break
    scored.extend(score_batch_remote(chunk))

stop_requested is per-session, so one user's cancellation does not affect anyone else. The mechanism is not free — checking it between chunks — but it is a real answer to "my export got stuck at 30 000 rows and I want to cancel".

Cache the API call, not the model​

Local models sit behind @st.cache_resource. Remote calls sit behind @st.cache_data, because you are caching a value, not an object.

@st.cache_data(ttl=60, show_spinner=False)
def score_cached(features_tuple: tuple) -> float:
    features = dict(zip(FEATURES, features_tuple))
    return score_remote(features)

probability = score_cached(tuple(features[name] for name in FEATURES))

The tuple conversion is deliberate: tuples are hashable and cheap to hash, dicts are not the natural cache key. A 60-second TTL keeps the cache from growing without a bound on a demo, and short enough that the next user of the same features re-hits the model within a normal freshness horizon. Do not cache a POST that has side effects — a "submit an order" endpoint has no business behind a cache.

Adding both paths to the running example​

The final version of the app uses a single environment switch, so a developer runs against a local model and production runs against the API without any code change.

import os

USE_REMOTE = os.getenv("USE_REMOTE_MODEL", "").lower() == "true"

def score(features: dict) -> float | None:
    if USE_REMOTE:
        return score_with_feedback(features)
    try:
        return score_local(features)
    except Exception:
        st.error("The local model failed. See the server logs.")
        return None

# In the Score tab, after submit:
with st.spinner("Scoring..."):
    probability = score(features)

if probability is not None:
    st.session_state.last_result = {"id": customer_id, "score": probability}
    st.metric(f"Churn probability for {customer_id}", f"{probability:.0%}",
              delta_color="inverse")

That switch is what lets the training team ship a new model with no Streamlit change: bump the API version and every dashboard picks it up.

A timeout is not optional

Without an explicit timeout=, requests.post waits for the OS default, which can be minutes. On a Streamlit page, that is a locked browser tab with a spinning circle and a user who reloads the page — pushing another request to the already-stuck server. Set a timeout on every network call, no exceptions.

In summary​

  • Local model: @st.cache_resource loads once, .predict_proba scores in microseconds; keep the feature order explicit.
  • Remote API: requests.post with timeout=, raise_for_status(), a Bearer token from st.secrets; return None on error, message the user in their terms.
  • Loading indicators: st.spinner for a single call, a progress bar for a batch; a stop button through st.session_state, per chunk.
  • Caching: @st.cache_data on the remote result with a short TTL, never on a POST that has side effects.

Next module: theme, appearance and usability, and the split into a multipage application.