Skip to main content

Module 2 — Data structures and comprehensions

Four built-in structures carry all the preparation work in Python: the list, the dictionary, the tuple and the set. Knowing which one to pick — and how to turn one into another in a single line — is the skill that separates laborious code from fluent code. It is also the mental gymnastics that will make pandas feel natural.

The list: the all-purpose sequence​

Ordered, mutable, heterogeneous if needed (but do not do that without a reason):

scores = [0.91, 0.87, 0.94, 0.79]
scores.append(0.88)          # append at the end
scores[0]                    # first element
scores[-1]                   # last one
len(scores), sum(scores)     # 5, 4.39
sorted(scores, reverse=True) # new sorted list

Two methods people confuse: sorted(list) returns a sorted copy; list.sort() sorts in place and returns None — the source of a classic bug (result = list.sort() gives None).

Sorting by key is used constantly:

models = [("baseline", 0.71), ("forest", 0.86), ("boosting", 0.89)]
best = max(models, key=lambda m: m[1])   # ('boosting', 0.89)

The dictionary: keys to values​

Python's most important structure — pandas DataFrames are conceptually an extension of it. Key access in constant time:

config = {"model": "xgboost", "depth": 6, "threshold": 0.5}
config["depth"]               # 6 — KeyError if absent
config.get("cache", False)    # False — default if absent
config["threshold"] = 0.4     # update

The three traversals:

for key in config:                    # the keys
for value in config.values():         # the values
for key, value in config.items():     # both — the most common

The counting idiom, ubiquitous in data exploration:

counts = {}
for category in category_column:
    counts[category] = counts.get(category, 0) + 1
# or, in one line: from collections import Counter ; Counter(category_column)

Tuple and set: the two specialists​

The tuple is an immutable sequence: what is frozen cannot be modified by accident. Use it for groups of values that belong together — coordinates, (name, score) pairs — and for multiple return values:

def evaluate(y_true, y_pred):
    return precision, recall        # returns a tuple

precision, recall = evaluate(yt, yp)  # unpacking

The set stores unique, unordered elements with instant membership testing. Two everyday uses in data work:

duplicates = len(ids) - len(set(ids))            # count duplicates

expected_columns = {"id", "amount", "date"}
missing = expected_columns - set(df_columns)      # set difference

The x in a_set test runs in constant time, versus a full scan for x in a_list — over millions of elements the difference is measured in minutes.

Comprehensions: transform in one line​

The list comprehension is Python's most idiomatic construct: transform and filter a sequence without an explicit loop.

# transform
amounts_incl_tax = [a * 1.15 for a in amounts]

# filter
valid = [x for x in measures if x is not None]

# both
error_logs = [l.strip() for l in lines if "ERROR" in l]

The general pattern: [expression for element in sequence if condition]. It exists in dictionary and set flavors:

price_by_id = {p["id"]: p["price"] for p in products}   # dict
extensions = {f.split(".")[-1] for f in files}           # set
The readability limit

A comprehension beats the equivalent loop as long as it fits readably on one line, with at most one if. A comprehension nested three levels deep is a loop disguised as a riddle: write the loop. The criterion is a colleague rereading it, not concision.

This way of thinking — "apply this expression to every element, keep the ones that pass the filter" — is exactly the mindset of NumPy and pandas. [a * 1.15 for a in amounts] will become amounts * 1.15 in module 4; the filter will become a boolean mask. Comprehensions are the school of vectorization.

Copies and references: the structural trap​

Python variables are references. Assignment does not copy:

a = [1, 2, 3]
b = a          # b points to THE SAME list
b.append(4)
a              # [1, 2, 3, 4] — surprise

To actually copy: b = a.copy() (shallow) or copy.deepcopy(a) for nested structures. This behavior will resurface in pandas, where the view/copy distinction on DataFrames is behind a famous warning (SettingWithCopyWarning, module 6).

Corollary: never use a list as a parameter's default value (def f(x, acc=[])) — the list is created once and shared across calls. The correct idiom: acc=None then if acc is None: acc = [].

Key takeaways​

  • List for ordered sequences, dictionary for key → value mappings, tuple for frozen groups, set for uniqueness and fast membership.
  • sorted() returns, .sort() mutates in place; sorting by key with key=.
  • Comprehensions transform and filter in one line — and train the vectorization reflex; beyond one if, go back to the loop.
  • Variables are references: .copy() to copy, never a mutable as a parameter default.

Next module: functions and code organization — moving from the throwaway script to the reusable module.