Skip to main content

Module 3 — Data on Cloud Storage and BigQuery

Vertex AI does not store data of its own — every dataset the platform reads sits either in Cloud Storage (files) or BigQuery (tables). This module lays out how the fraud-detection data is organised on both, what a Vertex "managed dataset" adds on top, and how to read a query's price tag before you run it.

The bucket layout that survives contact with reality​

A messy bucket becomes a permanent tax on every future project: nobody dares delete anything, and every job embeds a path that turns out to be wrong. Two conventions save this from happening.

One bucket per environment and region, named after both: gs://fraud-detection-dev-vertex-eu, gs://fraud-detection-prd-vertex-eu. Vertex is regional, and a bucket in the wrong region charges egress every time a training job reads it.

A stable prefix scheme inside, mirroring the ML lifecycle rather than a random author's mental model:

gs://fraud-detection-dev-vertex-eu/
    data/raw/                       inputs pulled from source systems
    data/processed/                 cleaned parquet, partitioned by date
    training/{experiment_id}/       inputs snapshotted for one run
    artifacts/{model_id}/{version}/ model files uploaded to the registry
    pipelines/{pipeline_run_id}/    KFP artifacts of module 9
    tmp/                            everything with a 7-day lifecycle rule

The tmp/ directory with a lifecycle rule at seven days is one of the highest-leverage two lines of configuration in the whole course:

{
  "lifecycle": {
    "rule": [{
      "action": {"type": "Delete"},
      "condition": {"age": 7, "matchesPrefix": ["tmp/"]}
    }]
  }
}

Applied with gcloud storage buckets update, it deletes intermediate junk automatically. Bucket size stops growing linearly with experiment count.

BigQuery: the table you actually train on​

The raw transactions land in BigQuery, one row per payment. The table is partitioned by day and clustered by merchant — two design choices that determine every query's scan size.

CREATE TABLE `fraud-detection-dev.raw.transactions`
PARTITION BY DATE(created_at)
CLUSTER BY merchant_id
AS
SELECT
  transaction_id,
  amount,
  currency,
  merchant_id,
  merchant_category,
  device_type,
  hour_of_day,
  days_since_signup,
  is_fraud,
  created_at
FROM `source.raw.card_transactions`;

A day of data is one partition; a query on a single week scans seven partitions. Without partitioning, every query scans the whole table — 40 GB, roughly $0.20 per query at BigQuery on-demand pricing ($5 per TB). That is cheap once, expensive when a notebook fires the same query fifty times an afternoon.

The true dollar cost of a query — read it before you run​

Every BigQuery query returns a bytes billed figure, and the console shows it in the top-right of the query editor before you press Run. Read it. A 40 GB scan costs $0.20; a 4 TB scan costs $20. That gap is one absent-minded SELECT * away.

from google.cloud import bigquery

client = bigquery.Client(location="europe-west1")

QUERY = """
    SELECT is_fraud, COUNT(*) AS n
    FROM `fraud-detection-dev.raw.transactions`
    WHERE _PARTITIONDATE BETWEEN '2026-06-01' AND '2026-06-30'
    GROUP BY is_fraud
"""

# Dry-run: no bytes read, only an estimate returned
job_config = bigquery.QueryJobConfig(dry_run=True, use_query_cache=False)
dry = client.query(QUERY, job_config=job_config)
gb  = dry.total_bytes_processed / 1e9
print(f"This query will scan about {gb:.2f} GB, roughly ${gb * 0.005:.4f}.")

A dry run costs nothing and returns the scan size the paid run will pay for. Wrapping every non-trivial notebook query in a dry-run helper is the single habit that keeps a monthly bill from surprising anyone.

Three small changes cut typical scans by 10× or more:

  • Project only the columns you need. SELECT * on the transactions table reads every column, including large device metadata blobs.
  • Filter on the partition key (_PARTITIONDATE or DATE(created_at)) as early as possible in the query.
  • Cache the intermediate. For a slice you plan to re-use, write once to a parquet file in gs://.../data/processed/ and read from there.

Cloud Storage: the file format that fits training​

Vertex training containers read from GCS by URL (gs://...). Formats matter: parquet is the pragmatic default for tabular ML on Vertex. Columnar, compressed, typed, and pandas / Spark / BigQuery all read it out of the box.

df = bq.query(TRAINING_QUERY).to_dataframe()
df.to_parquet(
    "gs://fraud-detection-dev-vertex-eu/data/processed/2026-06.parquet",
    index=False,
)

For image or text data, keep the files themselves in GCS and reference them from a manifest (JSONL, one line per example, with a gs:// URL). Reading tens of thousands of small files from GCS during training is slower than reading one large TFRecord or sharded parquet — a common trap when scaling from a laptop to Vertex.

Vertex managed datasets: what they add, what they do not​

A managed dataset is a Vertex resource that wraps your GCS or BigQuery data with:

  • A type (tabular, image classification, text, video) — mostly relevant for AutoML.
  • A split scheme (train / validation / test with fixed fractions or an explicit column).
  • A catalogued entry in the Vertex console with a stable resource name.
from google.cloud import aiplatform

dataset = aiplatform.TabularDataset.create(
    display_name="fraud-2026-06-processed",
    bq_source="bq://fraud-detection-dev.raw.transactions",
)

You do not need a managed dataset to run a CustomTrainingJob. A path to a parquet file in GCS is enough, and it is what the red thread will use in module 4. Managed datasets pay for themselves when running AutoML, when non-technical users need to see the dataset in the console, or when you want lineage from dataset to model to appear automatically in metadata. They add nothing for a pure CustomTrainingJob on parquet.

The training-slice export, done properly​

The training job of module 4 will read a single, immutable parquet snapshot. Exporting that snapshot from BigQuery, once, is the last thing this module does.

EXPORT_QUERY = """
    SELECT
      amount, merchant_category, device_type,
      hour_of_day, days_since_signup,
      SAFE_DIVIDE(amount, avg_amount_30d) AS amount_vs_avg,
      is_fraud
    FROM `fraud-detection-dev.raw.transactions`
    WHERE _PARTITIONDATE BETWEEN '2026-06-01' AND '2026-06-30'
"""

# Materialise as a table, then export to parquet - one billed scan, one export
tmp_table = "fraud-detection-dev.tmp.training_2026_06"
client.query(f"CREATE OR REPLACE TABLE `{tmp_table}` AS {EXPORT_QUERY}").result()

extract = client.extract_table(
    tmp_table,
    "gs://fraud-detection-dev-vertex-eu/training/exp-042/2026-06-*.parquet",
    job_config=bigquery.ExtractJobConfig(destination_format="PARQUET"),
    location="europe-west1",
)
extract.result()

Two properties matter. First, the export shard pattern 2026-06-*.parquet produces multiple files, which parallelise reads later. Second, the snapshot is immutable: even if raw.transactions receives corrections tomorrow, the file at training/exp-042/ is exactly what the model was trained on. Reproducibility hinges on that separation.

to_dataframe() on the full training query

Calling .to_dataframe() on the full training query pulls every row into the notebook's RAM. On the 40 GB table that is a memory error, or a killed kernel, or a very large notebook auto-save. Use the BigQuery-to-parquet path above and let the training job stream the file.

In summary​

  • A bucket per environment, a stable prefix scheme and a lifecycle rule on tmp/ turn Cloud Storage from a source of clutter into a boring commodity.
  • Design BigQuery tables with partitioning (typically by date) and clustering (by a natural query key); every query then scans a small share of the table.
  • Read the bytes billed before every non-trivial query, either from the console or via a dry run; project the columns you need and filter on the partition key.
  • Vertex managed datasets add value for AutoML and lineage — for pure CustomTrainingJob on tabular data, a parquet snapshot in GCS is simpler and reproducible.

Next module: the training container itself — prebuilt versus custom, CustomTrainingJob, machine types and accelerators, and where the trained model actually lands.