Module 3 — Documents: CRUD, versions and bulk import with _bulk

The cluster answers, the concepts are in place. Sami can finally handle documents: create an article, update it, retrieve it, delete it, and understand how Elasticsearch avoids concurrent writes overwriting each other. Then, with Inès, he indexes the full corpus — 200,853 articles in under thirty seconds thanks to the _bulk API.

Create, read, update, delete a document

Every request in this module pastes into Kibana Dev Tools. We first practice on a dedicated mini-index so we do not mix things with the real corpus.

PUT bac_a_sable
{
  "settings": { "number_of_shards": 1, "number_of_replicas": 0 },
  "mappings": {
    "properties": {
      "titre":  { "type": "text" },
      "auteur": { "type": "keyword" },
      "vues":   { "type": "integer" },
      "date":   { "type": "date", "format": "yyyy-MM-dd" }
    }
  }
}

PUT with a chosen _id: idempotent

PUT bac_a_sable/_doc/1
{
  "titre":  "Change Is Here. Climate Change.",
  "auteur": "Inès",
  "vues":   0,
  "date":   "2026-09-09"
}

Typical response:

{
  "_index": "bac_a_sable",
  "_id": "1",
  "_version": 1,
  "result": "created",
  "_shards": { "total": 1, "successful": 1, "failed": 0 },
  "_seq_no": 0,
  "_primary_term": 1
}

Re-run the same request: result becomes updated, _version moves to 2, _seq_no increments. PUT /index/_doc/1 fully replaces the document; it is the idempotent write you want when you have a business key (article number, product reference).

POST without _id: Elasticsearch generates one

POST bac_a_sable/_doc
{
  "titre":  "Trump Abandons Commitment To 2-State Solution",
  "auteur": "Sami",
  "vues":   0,
  "date":   "2026-09-09"
}

_id is a base64 identifier of twenty characters (OFm5s5UB…). Useful when your documents have no business key; to avoid as soon as you risk re-importing, otherwise you will create duplicates on every run.

PUT _create: refuses if the _id already exists

PUT bac_a_sable/_create/1
{ "titre": "Doublon", "auteur": "Test", "vues": 0, "date": "2026-09-09" }

Returns a version_conflict_engine_exception error because 1 already exists. To use when you want the "definitely do not overwrite" guarantee.

GET a specific document

GET bac_a_sable/_doc/1

Returns the _source (the JSON you sent) plus the metadata _seq_no, _primary_term, _version, found: true. A missing _id returns found: false with a 404 HTTP status.

Two useful variants:

GET bac_a_sable/_source/1

Returns only the _source, without envelope.

GET bac_a_sable/_doc/1?_source_includes=titre,auteur

Retrieves a subset of the fields — handy when _source is large.

POST _update with doc: partial merge

POST bac_a_sable/_update/1
{
  "doc": {
    "vues": 12
  }
}

Only the vues field changes; the other fields stay. Elasticsearch re-reads the document, applies the patch, reindexes. result returns updated, or noop if the content was already the requested one.

POST _update with script: increment atomically

To increment views on every read without having to read then write:

POST bac_a_sable/_update/1
{
  "script": {
    "source": "ctx._source.vues += params.n",
    "lang":   "painless",
    "params": { "n": 1 }
  }
}

The painless language is sandboxed, compatible across versions, and shard-level scheduling makes the operation atomic. It is the clean way to aggregate counters without a race between requests.

DELETE

DELETE bac_a_sable/_doc/1

Marks the document as deleted. The shard physically cleans it up at the next Lucene merge. The docs.deleted counter in _cat/indices goes up, then drops again after a merge.

_mget: several documents in one request

GET bac_a_sable/_mget
{ "ids": ["1", "2", "3"] }

Returns a docs array with, for each id, found and optionally _source. Useful to reload twenty keys at once.

Optimistic versions: _seq_no and _primary_term

Karim wants to update an article from the API. Léa, in parallel, does the same from Kibana. How to prevent the slowest write from overwriting the freshest one? Elasticsearch exposes two pairs of values:

• _version: integer that increments on every write, informational;
• _seq_no + _primary_term: the true identity of a version, carried by the shard.

The optimistic control protocol fits in one sentence: read, then write by passing back the _seq_no and _primary_term you read. If another client wrote in between, your request is rejected with version_conflict_engine_exception. You re-read, redo your computation, rewrite.

GET bac_a_sable/_doc/1

Say in the response we get _seq_no: 5, _primary_term: 1. We update:

PUT bac_a_sable/_doc/1?if_seq_no=5&if_primary_term=1
{
  "titre":  "Titre revu",
  "auteur": "Inès",
  "vues":   99,
  "date":   "2026-09-09"
}

If the document has not moved, the request goes through. Otherwise, status 409: version_conflict_engine_exception. Replay the read, replay the write. This mechanism is what protects your counters and statuses without having to lock.

No multi-document transactions

Elasticsearch does not know ACID transactions across multiple documents. Each document is atomic in isolation; the consistency of several documents together is handled on the application side (idempotency, retry). For a true transactional model, Neo4j (module 12) or a relational database are the right tools.

The _bulk API: the NDJSON format

Sending 200,853 articles with 200,853 POSTs takes hours. _bulk accepts batches of thousands of actions in a single HTTP request.

The format is called NDJSON ("newline-delimited JSON"): one action line, one document line, newline, one action line, one document line, newline. Each line ends with \n, including the last one. No array, no comma between objects.

POST _bulk
{ "index": { "_index": "bac_a_sable", "_id": "10" } }
{ "titre": "Un article", "auteur": "Léa", "vues": 0, "date": "2026-09-09" }
{ "index": { "_index": "bac_a_sable", "_id": "11" } }
{ "titre": "Un autre", "auteur": "Karim", "vues": 0, "date": "2026-09-09" }
{ "delete": { "_index": "bac_a_sable", "_id": "10" } }
{ "update": { "_index": "bac_a_sable", "_id": "11" } }
{ "doc": { "vues": 42 } }

Four possible actions: index (replaces), create (fails if it exists), update (patch), delete (no document line). Elasticsearch returns an items array of the same length as the number of actions, with the status of each one.

The last line, a classic trap

A _bulk without a trailing \n is refused with The bulk request must be terminated by a newline. This is THE classic trap when you write _bulk by hand. In Kibana Dev Tools, the console adds the newline automatically; in a script, you have to force it.

Index 200,853 articles with ./lab.sh import-news

The kit wraps this whole mechanism into a single command. From the kit folder:

./lab.sh import-news        # macOS, Linux, WSL2, Git Bash
.\lab.ps1 import-news       # Windows PowerShell

Output observed on a standard workstation (Docker Desktop, 8 GB allocated, SSD disk):

[import-news] starting
[import-news] connecting to http://elasticsearch:9200 as elastic...
[import-news] cluster "veille" in state green
[import-news] dataset already present: /data/News_Category_Dataset_v2.json (83 MB), download skipped
[import-news] index "news" created with mapping news.json
[import-news] indexing in batches of 2000 documents
[import-news]   20000 documents indexed (3 s)
[import-news]   40000 documents indexed (6 s)
[import-news]   ...
[import-news]  200000 documents indexed (28 s)
[import-news] 200853 documents sent in 28 s
[import-news] check: _count = 200853 ✔
[import-news] Neo4j file written: /neo4j-import/news.csv (43 MB)
[import-news] done. In Kibana → Dev Tools: GET news/_count

Confirm in Dev Tools:

GET news/_count

{ "count": 200853, "_shards": { "total": 1, "successful": 1, "skipped": 0, "failed": 0 } }

One hundred forty megabytes of storage for 200,853 documents, indexed in under a minute. That is what a well-tuned _bulk delivers on a single-node lab cluster — on a properly sized production cluster, we speak in tens of thousands of documents per second per node.

Guided reading of importer/import_news.py

The script fits in about two hundred fifty lines of standard Python — not a single dependency to install on your machine, everything runs in a container. We walk through it function by function: each choice hides a lesson.

es(method, path, body=…): the in-house HTTP client

A single function handles every REST call. It carries Basic authentication, retries five times with exponential backoff on timeouts and on 429 Too Many Requests, and decodes the JSON response. Remember the lesson: when a service can return 429 under heavy load, a naive client crashes; retry with backoff is the bare minimum.

telecharger(): idempotent download

If /data/News_Category_Dataset_v2.json already exists and weighs more than ten megabytes, the function skips the download. Otherwise it downloads into a .part file and renames at the end (atomic move on the filesystem). An interruption leaves the incomplete .part, which will be overwritten on the next run; never a half-truncated News_Category_Dataset_v2.json.

creer_index(): a clean index on each run

If the news index exists, the function deletes it then recreates it with elasticsearch/mappings/news.json. An import is therefore idempotent in both directions: two successive runs give exactly the same state. This avoids the trap of a reimport that adds to the previous one and silently inflates the count.

Idempotency: why it is essential

An indexing pipeline that restarts after a failure must always land in the same state. By setting _id = row_number and deleting the index before recreating it, import-news is replayable at will without side effects. In production, we often prefer to reindex into a new index (news-2026-09-09), point an alias news at it (module 8), then delete the old one — same logic, without breaking reads.

envoyer_lot(lignes): the real _bulk

The body is a Python array of already-serialized strings, joined by \ns, encoded as UTF-8, terminated with a \n. The action line is the smallest possible: {"index": {"_index": "news", "_id": "42"}}. The document line is the raw JSON. The URL parameter ?filter_path=errors,items.*.error asks Elasticsearch not to return the 2,000 success lines — just errors: false when there is nothing to report, and the detail of the rejected documents otherwise. On 200,853 documents, this saves several megabytes of JSON to parse on the client side.

indexer(): batches of two thousand, CSV in parallel

Two thousand is a compromise published in the Elasticsearch documentation: a batch too small (100) loses the batching benefit, a batch too large (20,000) risks a 413 Request Entity Too Large rejection and hurts the coordinating node's memory. Two thousand is a balance point for short documents like the News articles.

In parallel with the _bulk, each row is also written into /neo4j-import/news.csv (a csv.writer in QUOTE_ALL mode). A single pass over the source file thus feeds both engines — this is the file Neo4j will read in module 11 with LOAD CSV.

verifier(attendu): a final _refresh, a _count

The script ends with:

POST /news/_refresh
PUT  /news/_settings   { "index": { "refresh_interval": "1s" } }
GET  /news/_count

The _refresh forces the opening of a new Lucene segment that is visible to search. The second call puts refresh_interval back to 1s — the mapping had set it to 30s to speed up the import. This is the second big performance lever: during a bulk import, every default 1s refresh creates a segment that Lucene will then have to merge. Spacing refreshes out to 30s reduces merge pressure and nearly doubles indexing throughput. Once the import is done, we put it back to 1s for near-real-time visibility on the query side.

_id = row number: the key to idempotent re-import

Article n° 42 of the source file is indexed with _id = "42" and written to news.csv with id = 42. Replaying the import produces the exact same 200,853 documents with the exact same identifiers. The Neo4j graph in module 11 will use this id as the uniqueness-constraint key: two engines, one identity.

Try it 1 — Full CRUD cycle

In Dev Tools, create a document in bac_a_sable with _id = 42, a title and an author of your choice. Read it. Update it with _update by adding a note field of value 5. Read it again. Delete it. Confirm its disappearance with an _mget.

Solution

PUT bac_a_sable/_doc/42
{ "titre": "Test CRUD", "auteur": "Sami", "vues": 0, "date": "2026-09-09" }

GET bac_a_sable/_doc/42

POST bac_a_sable/_update/42
{ "doc": { "note": 5 } }

GET bac_a_sable/_doc/42

DELETE bac_a_sable/_doc/42

GET bac_a_sable/_mget
{ "ids": ["42"] }

The last _mget returns docs[0].found = false. Confirm that the document count is what you expect with GET bac_a_sable/_count.

Try it 2 — Optimistic version and conflict

Create a document, record _seq_no and _primary_term. Write twice in a row with those same values. What happens on the second request?

Solution

PUT bac_a_sable/_doc/100
{ "titre": "V1", "auteur": "Léa", "vues": 0, "date": "2026-09-09" }

Record _seq_no and _primary_term.

PUT bac_a_sable/_doc/100?if_seq_no=<n>&if_primary_term=<t>
{ "titre": "V2", "auteur": "Léa", "vues": 1, "date": "2026-09-09" }

Success: the document moves to V2, _seq_no increments. Repeat the same command with the old if_seq_no and if_primary_term values: status 409, version_conflict_engine_exception. A message that reminds you to re-read before rewriting.

Try it 3 — Confirm the import

After ./lab.sh import-news, run these three requests in Dev Tools and read what they tell:

GET news/_count

GET news/_doc/1

GET news/_search
{
  "size": 1,
  "query": { "match": { "headline": "climate change" } }
}

Solution

_count returns exactly 200,853. GET news/_doc/1 returns the first article of the file, with its headline, short_description, category, authors, link, date fields. The _search on "climate change" returns 2,834 total results (hits.total.value); the first document has a high _score and its headline is "Change Is Here. Climate Change.". You just confirmed that the corpus is complete, that the mapping is active and that the search engine is answering.

Key takeaways

• PUT /index/_doc/<id> replaces, POST /index/_doc generates the _id, PUT /index/_create/<id> refuses to overwrite.
• POST _update accepts doc (partial patch) or script (atomic shard-side computation).
• Optimistic control combines _seq_no and _primary_term: replayable, lock-free.
• _bulk expects NDJSON: one action, one document, newline, trailing newline included.
• Two thousand documents per batch is a good balance for short documents.
• ?filter_path=errors,items.*.error shaves several megabytes off _bulk responses on large imports.
• Setting refresh_interval to 30s during the import then back to 1s doubles the throughput — the kit does it for you.
• _id = row_number makes the import idempotent and gives a stable identity shared with Neo4j.

Troubleshooting

• ./lab.sh import-news returns Elasticsearch is not started → the healthcheck is not green. ./lab.sh status, then ./lab.sh logs elasticsearch if the container is not healthy.
• _bulk returns 413 Request Entity Too Large → the batch is too large or your documents are unusually big; reduce the batch size or raise http.max_content_length (default 100 MB).
• version_conflict_engine_exception on a PUT _create → the _id already exists. That is the intended effect of _create — use PUT /index/_doc/<id> if you want to replace.
• _count returns fewer than 200,853 after an import → a _bulk ended with a silent error; re-run ./lab.sh import-news (idempotent), and if the problem persists, ./lab.sh doctor then ./lab.sh logs elasticsearch.

Further reading

• Elasticsearch 9 documentation — Document APIs (index, get, update, delete)
• Elasticsearch 9 documentation — Bulk API
• Elasticsearch 9 documentation — Optimistic concurrency control
• Elasticsearch 9 documentation — Tune for indexing speed