Module 3 — Data assets and datastores

The compute from Module 2 is ready to run the demand-forecast training, but it does not yet know where the sales data lives, nor how to prove it is allowed to read it. Azure ML separates those two questions cleanly: datastores answer "where", data assets answer "which version, in what shape", and identity-based access answers "under whose credentials".

Datastores: connection metadata, not data

A datastore is a reference to a storage location — a Blob container, an ADLS Gen2 filesystem, a File share, or a SQL database. It stores the account name, the container name, and the authentication method. It does not copy the data. When a job reads from a datastore, Azure ML mounts or downloads the underlying storage at runtime.

Every workspace ships with two datastores by default: workspaceblobstore and workspacefilestore, both backed by the storage account created in Module 1. These are fine for scratch and small experiments; production data lives in its own account.

For the forecast project, the store point-of-sale export lands nightly as Parquet in a Blob container sales/ of a dedicated storage account. Register it as a datastore once:

az ml datastore create --file datastore-sales.yml

# datastore-sales.yml
$schema: https://azuremlschemas.azureedge.net/latest/azureBlob.schema.json
name: sales_blob
type: azure_blob
description: Nightly POS exports from the point-of-sale team
account_name: stforecastdatalake
container_name: sales
credentials:
  # empty on purpose: use the workspace's managed identity, see below

The three types of data asset

A data asset is a named, versioned pointer to data. Azure ML v2 recognizes three types:

Type	What it wraps	When to pick it
uri_file	A single file, referenced by URI	A CSV of test predictions, a JSON config
uri_folder	A folder tree, referenced by URI	The full two years of Parquet sales files, all in one path
mltable	A folder plus an MLTable YAML that describes columns, types and transformations	Tabular data used by AutoML, or data with a schema you want to freeze

Registering an asset creates a version, and versions are immutable. Version 1 always points at the exact snapshot it was registered against; a run that used version 1 last month can be re-executed today with the same bytes on the same rows.

For the forecast project, the two years of weekly sales are a uri_folder because the training code reads a folder of Parquet parts. AutoML (Module 6) will use an MLTable on the same data to expose it as a typed table.

az ml data create \
  --name sales-2y \
  --version 1 \
  --path azureml://datastores/sales_blob/paths/history/ \
  --type uri_folder \
  --tags rows=~104M source=pos-nightly

The azureml:// URI pattern is the workspace-internal syntax: it hides the storage account name and lets code reference the asset by name and version.

MLTable: schema-locked tabular data

An MLTable is a folder that contains an MLTable YAML file next to the data files. The YAML describes how to load them and, optionally, apply lightweight transformations:

# ./sales-mltable/MLTable
$schema: https://azuremlschemas.azureedge.net/latest/MLTable.schema.json
type: mltable
paths:
  - pattern: ./*.parquet
transformations:
  - read_parquet
  - convert_column_types:
      - columns: sale_date
        column_type: datetime
      - columns: [store_id, sku_id]
        column_type: int
  - drop_columns: [raw_receipt_id]

Two properties are worth the ceremony. The schema travels with the data, so AutoML and downstream jobs read the same columns with the same types across environments. And the transformations run inside the data loading, before the training code sees a row, which removes an entire class of "the notebook works but the pipeline does not" bugs.

Identity-based access, the boring default that saves the day

The safest way to give a job access to data is to grant its managed identity the right role on the storage account and leave the datastore's credentials block empty. The alternative — embedding an account key in the datastore — works, but rotates poorly, leaves a copy of the key in the workspace, and lights up every security scanner.

# grant the compute cluster read access on the storage account
az role assignment create \
  --assignee-object-id <managed-identity-object-id> \
  --role "Storage Blob Data Reader" \
  --scope /subscriptions/<sub>/resourceGroups/rg-forecast-prod/providers/Microsoft.Storage/storageAccounts/stforecastdatalake

Trace back to Module 1: the job runs under the cluster's managed identity, so this is what needs the role — not your user, not the workspace, not a service principal.

Reading the asset from training code

Inside train.py, the data path is injected by Azure ML as a plain local path (mount or download, transparent to you):

import argparse, pandas as pd, glob, os

parser = argparse.ArgumentParser()
parser.add_argument("--sales", type=str)
args = parser.parse_args()

files = sorted(glob.glob(os.path.join(args.sales, "*.parquet")))
df = pd.concat(pd.read_parquet(f) for f in files)

No storage account name, no SAS token, no connection string. The job spec (Module 5) will bind --sales to azureml:sales-2y:1, and Azure ML resolves the rest.

The one thing you must not do

Do not pip install azure-storage-blob and read the container by URL from your training code. It works locally under your user credentials and fails in the cluster under the managed identity, wasting a job's runtime to discover it. Read only through data-asset inputs.

Summary

• Datastore: named connection to a storage location. Data asset: named, versioned pointer to specific data.
• Three asset types: uri_file for a single file, uri_folder for a folder, mltable for schema-locked tabular data.
• Prefer identity-based access over stored credentials; grant the role to the compute's managed identity.
• Training code receives the asset as a local path; the URI resolution is handled by the runtime.

Next module: environments — freezing the Python packages the training and scoring code depend on, so a job that runs today still runs in six months.