Module 5 — pandas: Series, DataFrame and indexing

pandas is the tool a data practitioner spends most of their day in. Conceptually it is simple: a DataFrame is a two-dimensional NumPy array whose rows and columns carry labels. All the power — and the few traps — follow from that idea.

Series and DataFrame: the two objects

A Series is one column: NumPy values plus an index of labels. A DataFrame is a set of Series sharing the same index.

import pandas as pd

df = pd.read_csv("orders.csv", parse_dates=["date"])

df.shape        # (12480, 6) — rows, columns
df.dtypes       # each column's type — READ THIS SYSTEMATICALLY
df["amount"]    # one column → Series
df[["client", "amount"]]   # several columns → DataFrame

The first reflex on any unknown file fits in four commands:

df.head()       # the first 5 rows — what does the data look like?
df.info()       # types, non-nulls, memory — are columns typed correctly?
df.describe()   # numeric column statistics — magnitudes, extremes
df["country"].value_counts()   # a categorical's distribution — values, typos

These four commands surface most problems before they become bugs: the amount column read as text (because of an "N/A" or a decimal comma), the date still a string, the category spelled three different ways.

loc and iloc: selecting without ambiguity

pandas offers two explicit accessors, and the discipline of using them avoids most confusion:

df.loc[42, "amount"]            # by LABELS: index label 42, column "amount"
df.iloc[0, 3]                   # by POSITIONS: first row, fourth column

df.loc[df["country"] == "Morocco", ["client", "amount"]]   # filter + columns
df.iloc[:100]                   # the first 100 rows

The difference becomes critical as soon as the index is no longer 0, 1, 2…: after a sort or a filter, loc[5] refers to label 5 (which can be anywhere), iloc[5] to the sixth row. Simple rule: loc by default, iloc when you are thinking in positions.

Filtering: boolean masks, like NumPy

The module 4 mechanism applies as is, labels included:

large = df[df["amount"] > 1000]

target = df[(df["amount"] > 1000) & (df["country"] == "Canada")]   # mandatory parentheses

recent = df[df["date"] >= "2026-01-01"]        # direct date comparisons
europe = df[df["country"].isin(["France", "Belgium", "Switzerland"])]
no_email = df[df["email"].isna()]

isin replaces chains of |; isna()/notna() test for missing values. Proportions read directly: (df["amount"] > 1000).mean().

Creating and transforming columns

df["amount_incl_tax"] = df["amount"] * 1.15                  # vectorized
df["year"] = df["date"].dt.year                              # dates accessor
df["domain"] = df["email"].str.split("@").str[1]             # strings accessor
df["segment"] = np.where(df["amount"] > 1000, "premium", "standard")

The .dt (dates) and .str (strings) accessors vectorize operations that would otherwise require a loop. apply with a row-wise function exists, but it is the last resort: a hundred times slower than vectorized operations, justified only for genuinely irreducible logic.

The SettingWithCopyWarning

Modifying the result of a filter (sub_df = df[mask] then sub_df["x"] = …) triggers pandas' most famous warning: pandas does not guarantee whether you are modifying a copy or the original table. The two gestures that avoid it: an explicit .copy() when you want an independent subset, and df.loc[mask, "x"] = value when you want to modify the original. It is the direct consequence of the views/references seen in modules 2 and 4.

Sorting, renaming, reindexing

df.sort_values("amount", ascending=False)          # simple sort
df.sort_values(["country", "amount"], ascending=[True, False])   # multiple sort
df.rename(columns={"amt": "amount"})               # readable renaming
df.reset_index(drop=True)                          # index reset to 0..n-1 after filter/sort

An important architectural point: these methods return a new DataFrame rather than modifying the original. The idiomatic style chains the transformations, each step remaining inspectable:

top_clients = (
    df[df["date"] >= "2026-01-01"]
    .groupby("client")["amount"].sum()
    .sort_values(ascending=False)
    .head(20)
)

This chain — filter, aggregate, sort, head — is the canonical sentence of pandas analysis; the groupby driving it is the subject of module 7.

Key takeaways

• A DataFrame = a NumPy array + row labels (index) and column labels; a Series = one column.
• On every new file: head, info, describe, value_counts — four commands that surface wrong types, missing values and dirty categories.
• loc by labels, iloc by positions; boolean masks with parentheses, isin, isna.
• New columns through vectorized operations and the .dt / .str accessors; apply as a last resort.
• .copy() or df.loc[mask, col] = … against the SettingWithCopyWarning; chained transformations, each step inspectable.

Next module: systematic cleaning — missing values, duplicates and types — the step that decides the reliability of everything downstream.