Module 2 — Elasticsearch key concepts: cluster, node, index, shard, document

The kit is running, Kibana is answering. Before indexing a single article, Inès wants Sami to know how to read a cluster: how many nodes, how many indices, how many shards, what color, why. This module gives the minimum vocabulary that will make half of the error messages in the rest of the course self-evident.

The vocabulary, in the order you meet it

Elasticsearch exposes its concepts as successive nested layers. We take them top to bottom.

Cluster

A cluster is a group of one or more servers that declare themselves members of the same set. Ours is called veille — you can read it in docker-compose.yml under cluster.name=veille. A cluster has a name, a health (green, yellow or red), a version, and it collectively owns all the data.

Node

A node is an Elasticsearch process that runs on a server and has joined a cluster. The kit starts a single node, veille-es, in discovery.type=single-node mode. In production a cluster of three or more nodes is the norm, but a single node is more than enough to explore the concepts and to handle the 200,853 articles of the corpus.

Index

An index is the logical grouping of documents of the same kind — our News articles form the news index. An index exposes two configuration blocks: the settings (number of shards, number of replicas, analyzers) and the mapping (fields, types, options). An index then lives inside one or more physical shards.

Shard, replica

A shard is a Lucene partition of an index. The documents of an index are spread across its shards by hashing the _id. Each shard is a complete Lucene engine, independent, capable of hosting many documents and serving queries.

A replica is a copy of a primary shard, placed on another node. It serves two purposes: absorbing the loss of a node (high availability) and absorbing read traffic. Zero replicas only makes sense in a lab — a single node cannot host a replica of itself anyway. That is the kit's case.

Document

A document is a JSON object stored inside an index. It has an identifier _id (provided by you or generated), a _source (the JSON as you sent it) and version metadata (_seq_no, _primary_term). A News article is a document.

Mental mapping to the relational world

The following table helps, on the condition that you do not cling to it: Elasticsearch is not a relational database and these equivalences are approximate.

Elasticsearch	Relational (approximate)
Cluster	Database server
Node	Instance
Index	Table
Shard	Table partition
Document	Row
Field	Column
_id	Primary key
Mapping	Schema (CREATE TABLE)

No joins

There is no Elasticsearch equivalent of JOIN. The nested field (previewed in module 4) allows nested objects, but you do not link two indices by a foreign key. It is a choice: each shard must be able to answer on its own to stay fast. If you need relationships, that is Neo4j (modules 10 to 12).

Looking at the cluster

Open Kibana Dev Tools (Management → Dev Tools). Every query below pastes directly into the console on the left; the Ctrl-Enter shortcut (Cmd-Enter on macOS) runs it.

GET / — a node introduces itself

GET /

You get the node name (veille-es), the cluster name (veille), the Elasticsearch version (9.5.3), the Lucene version and the cluster UUID. This is the shortest version of "I'm alive, I'm answering, here's who I am".

GET /_cluster/health — the pulse

GET /_cluster/health

Typical output on our kit:

{
  "cluster_name": "veille",
  "status": "green",
  "number_of_nodes": 1,
  "number_of_data_nodes": 1,
  "active_primary_shards": 1,
  "active_shards": 1,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 0
}

The status field takes three values:

• green: all primary shards and all their replicas are assigned.
• yellow: all primaries are assigned, but at least one replica is missing.
• red: at least one primary shard is not assigned — some documents are unreachable.

The kit is green because the news index is created with number_of_replicas: 0: there is no replica to place, so nothing is missing. Many tutorials start with the default of one replica and show yellow — at which point Sami wonders what he broke. Answer: nothing, the single node simply cannot host a copy of its own primary shard.

GET /_cat/nodes?v — who makes up the cluster

./lab.sh es _cat/nodes?v

Or in Dev Tools:

GET /_cat/nodes?v

You see a single node, veille-es, its internal IP address, the percentage of memory and heap used, its load and its role (cdfhilmrstw: the node does everything at once — normal in single-node mode).

GET /_cat/indices?v — which indices exist

GET /_cat/indices?v

Before the import, the output only shows Kibana's system indices (.kibana_*, .security-*) prefixed with a dot. After ./lab.sh import-news, one more line shows up:

health status index    uuid       pri rep docs.count docs.deleted store.size pri.store.size
green  open   news     ...        1   0   200853     0            ...        ...

Reading: one primary shard (pri: 1), zero replicas (rep: 0), 200,853 documents, none deleted, a footprint of about one hundred forty megabytes.

GET /_cat/shards?v — where the shards live

GET /_cat/shards/news?v

index shard prirep state   docs   store ip         node
news  0     p      STARTED 200853 ...   172.20.0.3 veille-es

A single primary shard (p), started, hosting the 200,853 documents. prirep = r for a replica.

Create, describe, delete an index by hand

Before indexing News with the kit, let us bring a small demo index to life and then remove it: Léa wants to track her manual searches in a separate index.

Create with settings and mapping

PUT recherches_lea
{
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 0
  },
  "mappings": {
    "properties": {
      "sujet":     { "type": "keyword" },
      "requete":   { "type": "text" },
      "date":      { "type": "date", "format": "yyyy-MM-dd" },
      "resultats": { "type": "integer" }
    }
  }
}

Expected response:

{ "acknowledged": true, "shards_acknowledged": true, "index": "recherches_lea" }

Note what module 4 will unpack: sujet is a keyword (values compared exactly, usable in aggregations), requete is text (analyzed for search), date with an explicit format so the dynamic mapping does not guess.

9.x API, no _doc in the URL

Since Elasticsearch 7, named types have disappeared. You write PUT /recherches_lea with the mapping at the root; the old tutorials with PUT /recherches_lea/_doc/_mapping or include_type_name=true are deprecated and rejected in 9.x.

Describe

GET recherches_lea

Returns the settings (with default values added by Elasticsearch: creation_date, uuid, version), the mapping as stored, and the aliases (none here). To get only the mapping part:

GET recherches_lea/_mapping

Adjust a dynamic setting

Some settings are dynamic (changeable on the fly): number_of_replicas, refresh_interval. Others are static (fixed at creation): number_of_shards. Going from one shard to two shards requires a reindex.

PUT recherches_lea/_settings
{ "index": { "refresh_interval": "5s" } }

Module 3 will use this lever during the import: we set refresh_interval to 30s while loading 200,853 documents, then put it back to 1s. That is a visible doubling of indexing throughput.

Delete

DELETE recherches_lea

Response {"acknowledged": true}. The shard is dismounted, its Lucene files erased, the index disappears from _cat/indices.

Deletions

DELETE is irreversible cluster-side: no trash, no rollback. On a production cluster, it is wise to enable action.destructive_requires_name=true to forbid DELETE _all or DELETE *. The kit stays permissive so as not to hinder learning.

What a shard changes, what a replica changes

Two parameters, two effects not to be confused.

• More primary shards = the same index can be spread over more nodes, each shard receives fewer documents, indexing and search parallelize more. The cost: each shard consumes memory (buffers, Lucene structures) and coordinates its queries. Elastic's published rule of thumb is to keep each shard between ten and fifty gigabytes, and not exceed about twenty shards per gigabyte of heap.
• More replicas = more resilience and more read capacity, no benefit for indexing (on the contrary: each write is replicated). A replica only makes sense on a node other than the primary; two replicas only make sense from three nodes onwards.

On the 200,853 News documents, a single shard fits comfortably (total storage is about one hundred forty megabytes). In production, on billions of documents, you split into dozens of shards spread across several data nodes.

Java heap and disk, in one page

Elasticsearch runs on the JVM. Two tensions define it:

• The Java heap (-Xms1g -Xmx1g in our docker-compose.yml): half of the container's RAM, no more than thirty-one gigabytes in production (past that, the JVM changes its pointer mode and loses efficiency). You read the heap used in GET /_cat/nodes?v&h=name,heap.percent.
• The disk: Elasticsearch monitors free space and, by default, automatically flips indices to read-only when a threshold is hit (low, high, flood_stage). The kit disables this behavior (cluster.routing.allocation.disk.threshold_enabled=false) so it does not disrupt the lab. In production, you keep this mechanism and you add space.

One command to watch both at once:

GET /_cat/nodes?v&h=name,heap.percent,ram.percent,disk.used_percent

Try it 1 — Read a cluster

In Kibana Dev Tools, run GET /_cluster/health. Record status, number_of_nodes, active_primary_shards, active_shards, unassigned_shards. Then run GET /_cat/indices?v. How many indices are present? How many are not system indices (prefixed with a dot)?

Solution

Status green, one node, one or several primary shards assigned depending on the system indices already created by Kibana (.kibana_*, .security-*, .apm-* — the list evolves). Zero unassigned shards. Before the import, no user index: only system indices. After ./lab.sh import-news, one user index is added: news.

Try it 2 — Create a mini-index, then delete it

Create an index notes_veille with a single shard, zero replicas, two fields (titre as text, tag as keyword). Check that it shows up in _cat/indices, describe it, then delete it.

Solution

PUT notes_veille
{
  "settings": { "number_of_shards": 1, "number_of_replicas": 0 },
  "mappings": {
    "properties": {
      "titre": { "type": "text" },
      "tag":   { "type": "keyword" }
    }
  }
}

GET _cat/indices/notes_veille?v
GET notes_veille
DELETE notes_veille

Check that the second GET _cat/indices/notes_veille?v returns the index_not_found_exception error: the deletion is effective.

Try it 3 — Simulate a yellow, come back to green

Create an index with one replica and watch what happens on our single-node cluster. Then set replicas back to zero and re-check.

PUT test_yellow
{ "settings": { "number_of_shards": 1, "number_of_replicas": 1 } }

GET _cluster/health/test_yellow

What does status say? Change the number of replicas on the fly:

PUT test_yellow/_settings
{ "index": { "number_of_replicas": 0 } }

Re-check the health, then delete the index.

Solution

The first GET /_cluster/health/test_yellow returns "status": "yellow" with unassigned_shards: 1: the requested replica has nowhere to go (single node). After setting number_of_replicas to zero, this index's health returns to green immediately — Elasticsearch drops the assignment wait.

DELETE test_yellow

Remember the logic: yellow on a single-node cluster is not an outage, it is a configuration choice. On a cluster of three or more nodes, a persistent yellow deserves a real diagnostic (GET _cluster/allocation/explain).

Key takeaways

• Cluster → node → index → shard → document: five levels, in that order.
• An Elasticsearch index has settings (shards, replicas, refresh) and a mapping (fields, types).
• green = everything is assigned; yellow = a primary shard is there, a replica is missing; red = a primary is missing.
• The kit is green because the news index is configured with zero replicas — logical on a single-node cluster.
• A shard is a self-contained Lucene engine; you set the count at creation, you can change the replica count on the fly.
• _cat/indices, _cat/nodes, _cat/shards are the three most useful diagnostic commands: memorize them.
• In 9.x, you no longer put _doc in a mapping URL, no include_type_name, no string type: those syntaxes are rejected.

Troubleshooting

• GET /_cluster/health returns 401 in Dev Tools → the elastic user is no longer recognized; the .env password was changed after the first up. ./lab.sh reset then ./lab.sh up.
• GET /_cat/indices?v stays empty or times out → Elasticsearch has not finished starting; look at ./lab.sh logs elasticsearch and wait for [YELLOW] to [GREEN] on the system indices.
• illegal_argument_exception, mapper_parsing_exception when creating an index → a key in the mapping is misspelled (typo on properties) or an invalid type (string no longer exists in 9.x, replace with text or keyword).
• Status turns red after import-news → at least one primary shard could not initialize; ./lab.sh logs elasticsearch will hunt for a disk usage exceeded flood-stage watermark or translog corruption message, then ./lab.sh reset.

Further reading

• Elasticsearch 9 documentation — Nodes, shards, and replicas
• Elasticsearch 9 documentation — Cluster health API
• Elasticsearch 9 documentation — cat APIs
• Elasticsearch 9 documentation — Size your shards