Module 15 — Diagnose: the twelve classic failures and their remedy

The Veille kit is running, the index is in place, the graph is loaded — then one day, Kibana displays "server is not ready yet", LOAD CSV can no longer find the file, or Elasticsearch refuses to authenticate. Inès ended up filing everything: twelve recurring failures, triggered by hand, read from the logs, repaired with a single command, verified.

Summary table

N	Exact symptom	Cause	Fix
1	failed to bind host port ... address already in use	Port 9200, 5601, 7474, or 7687 already taken	docker ps then docker stop <name>, or free the port
2	Container Exited (137), OOMKilled: true	Docker killed the process, memory exceeded	Raise Docker Desktop → Memory or mem_limit
3	max virtual memory areas vm.max_map_count [65530] is too low	Linux: kernel setting too low	sudo sysctl -w vm.max_map_count=262144
4	_cluster/health in yellow or red	Unassigned replicas, or missing primary shard	GET _cluster/allocation/explain then fix
5	cluster_block_exception, index read-only / allow delete (api)	Disk watermark reached	PUT news/_settings {"index.blocks.read_only_allow_delete": null}
6	security_exception ... unable to authenticate user [elastic]	ELASTIC_PASSWORD changed after volume creation	./lab.sh reset then ./lab.sh up
7	Kibana server is not ready yet	kibana_system or Elasticsearch not ready	./lab.sh logs setup then re-run ./lab.sh up
8	mapper_parsing_exception or mapper cannot be changed	Incompatible field type	_reindex into a new index with the correct mapping
9	Couldn't load the external resource at: file:/import/news.csv	File missing from neo4j/import/	./lab.sh import-news
10	CALL { ... } IN TRANSACTIONS can only be executed in an implicit transaction	Executed inside an explicit transaction	Prefix with :auto in Neo4j Browser
11	There is no procedure with the name apoc.periodic.iterate registered	APOC not loaded	Check NEO4J_PLUGINS, restart Neo4j
12	The client is unauthorized due to authentication failure (Neo4j)	Password changed in .env but volume already exists	./lab.sh reset then ./lab.sh up

The rest of the module goes through each row: how to trigger it deliberately in the kit (when it is safe), what you will read in the logs, the fix command, and how to verify it is really solved.

1. Port already in use

Trigger. Open a terminal and keep a fake server on 9200:

docker run --rm -p 9200:80 --name faux-serveur nginx

In a second terminal, run ./lab.sh up.

Exact message.

Error response from daemon: driver failed programming external connectivity on endpoint veille-es
(...): failed to bind host port for 0.0.0.0:9200:172.19.0.2:9200/tcp: address already in use

Cause. Another program (often an old Elasticsearch container, sometimes a local service) is already holding port 9200. Docker cannot map the same host port twice.

Fix.

docker ps
docker stop faux-serveur
./lab.sh doctor
./lab.sh up

On Linux and macOS, sudo lsof -iTCP:9200 -sTCP:LISTEN identifies the process if it is not a container. On Windows PowerShell, netstat -ano | findstr :9200 returns the PID.

Verify. ./lab.sh status must list the three containers as Up (healthy) with the ports 0.0.0.0:9200->9200/tcp facing veille-es.

2. Container killed for lack of memory (Exited (137))

Trigger. Reduce the RAM allocated to Docker Desktop below 4 GB (Settings → Resources → Memory). Restart Docker then run ./lab.sh up. On a constrained machine, temporarily add mem_limit: 1g on the elasticsearch service in docker-compose.yml.

Exact message.

docker ps -a

CONTAINER   IMAGE                                       STATUS
veille-es   docker.elastic.co/.../elasticsearch:9.5.3   Exited (137) 12 seconds ago

docker inspect -f '{{.State.OOMKilled}}' veille-es

true

137 equals 128 + 9: the kernel sent SIGKILL (signal 9) to the container, almost always because the memory limit was hit. OOMKilled: true confirms.

Cause. The Elasticsearch process (1 GB heap + JVM overhead + page cache) exceeded the container's or Docker Desktop's memory limit.

Fix. Raise Docker Desktop → Settings → Resources → Memory to 6 GB (8 GB if you add OpenSearch). On Linux via WSL2, edit %UserProfile%\.wslconfig:

[wsl2]
memory=8GB

then wsl --shutdown and reopen Docker Desktop. Finally:

./lab.sh down
./lab.sh up

Verify. docker stats --no-stream must show veille-es around 1.3 GB resident (the heap plus the buffers). ./lab.sh logs elasticsearch --tail=20 must end with started.

3. vm.max_map_count too low (native Linux)

Trigger. On a Linux machine without Docker Desktop (Docker Engine directly):

sudo sysctl -w vm.max_map_count=65530
./lab.sh reset
./lab.sh up

Exact message in ./lab.sh logs elasticsearch:

ERROR: [1] bootstrap checks failed
[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

Cause. Elasticsearch uses mmap intensively for Lucene indices. The Linux kernel limits the number of mappable memory areas to 65,530 by default — not enough. Docker Desktop sets the value on its own inside its internal WSL2 distribution; on native Linux, it's up to the administrator.

Fix.

sudo sysctl -w vm.max_map_count=262144
echo 'vm.max_map_count=262144' | sudo tee /etc/sysctl.d/99-elasticsearch.conf
./lab.sh up

The second line makes the setting permanent across reboots.

Verify.

sysctl vm.max_map_count
./lab.sh logs elasticsearch --tail=20

The displayed value is 262144 and the log ends with started.

4. Cluster yellow or red

Trigger. On a single-node cluster (the kit), create an index with a replica — impossible to assign:

PUT test_yellow
{
  "settings": { "number_of_shards": 1, "number_of_replicas": 1 }
}
GET _cluster/health

Exact message.

{
  "cluster_name": "veille",
  "status": "yellow",
  "unassigned_shards": 1,
  ...
}

GET _cluster/allocation/explain details the reason:

"explanation": "the shard cannot be allocated to the same node on which a copy of the shard already exists"

Cause. yellow = primary shard present, replica missing (often: not enough nodes). red = primary shard itself missing (node down, disk corrupted). The news corpus is configured with number_of_replicas: 0 to stay green on a single node.

Fix. On a single-node cluster, bring replicas back to zero:

PUT test_yellow/_settings
{
  "index": { "number_of_replicas": 0 }
}

For a real red, GET _cluster/allocation/explain gives the exact reason: disk full, missing node, corruption. The remedy depends on the diagnosis — the command, however, is always the same first step.

Verify. GET _cluster/health returns "status": "green", "unassigned_shards": 0.

5. Index in read-only mode (disk watermark)

Trigger. The kit disables the threshold for the lab. Simulate the consequence by hand:

PUT news/_settings
{
  "index.blocks.read_only_allow_delete": true
}
POST news/_doc
{ "headline": "test" }

Exact message.

{
  "error": {
    "type": "cluster_block_exception",
    "reason": "index [news] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"
  },
  "status": 429
}

Cause. In production, Elasticsearch watches the disk. At 95% used (the flood_stage threshold), it automatically adds the read_only_allow_delete block on every index to prevent corruption. Once set, this block does not go away on its own even if the disk drops back down — this is intentional.

Fix.

PUT news/_settings
{
  "index.blocks.read_only_allow_delete": null
}

null removes the setting. The block falls immediately, writes resume. In production, first make room on the disk (delete old indices, grow the volume) before lifting the block.

Verify. Replay POST news/_doc {"headline":"test"}: the response is {"result": "created"} and no longer cluster_block_exception.

6. 401 after changing ELASTIC_PASSWORD

Trigger. Edit .env, replace ELASTIC_PASSWORD=veille2026 with ELASTIC_PASSWORD=nouveau2026, then:

./lab.sh down
./lab.sh up

Exact message in the first call from Kibana (./lab.sh logs kibana):

[error][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes.
security_exception: [security_exception] Reason: unable to authenticate user [elastic] for REST request [/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip]

Cause. The elastic user's password is written into the es-data volume at the very first startup. Once written, the ELASTIC_PASSWORD environment variable no longer modifies it: Elasticsearch reads it only during the initialization of a new node. ./lab.sh down keeps the volumes; the stored password stays the old one, the one you typed in .env is the new one, they no longer match.

Fix. Two options.

Option A (fast, without starting over) — update the password via the API, authenticating with the old one:

docker exec veille-es curl -s -u elastic:veille2026 \
  -H 'Content-Type: application/json' \
  -X POST http://localhost:9200/_security/user/elastic/_password \
  -d '{"password":"nouveau2026"}'

Option B (radical) — start over from scratch:

./lab.sh reset
./lab.sh up

Verify.

./lab.sh es _cluster/health

The JSON response arrives instead of a security_exception.

7. Kibana: "Kibana server is not ready yet"

Trigger. Interrupt Elasticsearch while Kibana starts:

./lab.sh down
docker compose up -d kibana   # without elasticsearch upstream

Then open http://localhost:5601.

Exact message in the browser:

Kibana server is not ready yet

In ./lab.sh logs kibana:

[error][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes.

Cause. Kibana does not hand back control until it has talked to Elasticsearch using kibana_system credentials. Three possible reasons: Elasticsearch not started, setup failed (so kibana_system has the old password), or Kibana started before the others. The kit's healthchecks normally protect against this case; docker compose up -d kibana on its own bypasses dependencies.

Fix.

./lab.sh logs setup
./lab.sh down
./lab.sh up

./lab.sh up respects the order: elasticsearch → setup → kibana → neo4j, waiting service_healthy between each step. If ./lab.sh logs setup shows a failure (password out of sync, Elasticsearch service dead), a ./lab.sh reset followed by ./lab.sh up puts everything back in order.

Verify. curl -s http://localhost:5601/api/status | grep 'available' returns "level":"available".

8. Mapping conflict (mapper_parsing_exception, mapper cannot be changed)

Trigger.

PUT test_mapping
{
  "mappings": {
    "properties": {
      "date": { "type": "date", "format": "yyyy-MM-dd" }
    }
  }
}
POST test_mapping/_doc
{ "date": "hier" }

Then, second case:

PUT test_mapping/_mapping
{
  "properties": {
    "date": { "type": "text" }
  }
}

Exact messages.

{"error":{"type":"mapper_parsing_exception","reason":"failed to parse field [date] of type [date] in document with id ..."}}

{"error":{"type":"illegal_argument_exception","reason":"mapper [date] cannot be changed from type [date] to [text]"}}

Cause. Elasticsearch freezes a field's type as soon as it is first indexed. You can add a field, never change its type. Moving from a date to a text (or from a text to a keyword) requires creating a new index with the correct mapping and copying documents over with _reindex.

Fix.

PUT test_mapping_v2
{
  "mappings": {
    "properties": {
      "date": { "type": "text" }
    }
  }
}

POST _reindex
{
  "source": { "index": "test_mapping" },
  "dest":   { "index": "test_mapping_v2" }
}

Then switch readers and writers over to test_mapping_v2, ideally through an alias (POST _aliases), and delete the old one.

Verify.

GET test_mapping_v2/_mapping

The date field is indeed of type text in the response.

9. LOAD CSV: "Couldn't load the external resource"

Trigger.

./lab.sh reset
./lab.sh up
./lab.sh cypher 11-charger-news.cypher

Without ./lab.sh import-news first, the file neo4j/import/news.csv does not exist.

Exact message in the cypher-shell output:

Couldn't load the external resource at: file:/import/news.csv

Cause. LOAD CSV WITH HEADERS FROM 'file:///news.csv' looks for the file inside the /import directory of the Neo4j container — a directory mounted on ./neo4j/import/ of the host. The news.csv file is only written by ./lab.sh import-news, which first downloads the corpus, indexes it into Elasticsearch, then produces the CSV for Neo4j.

Fix.

./lab.sh import-news
./lab.sh cypher 11-charger-news.cypher

Verify.

ls neo4j/import/

news.csv (about 25 MB) is present, and apoc.meta.stats() in cypher-shell counts the 200,853 articles, 41 categories, and 23,082 authors.

10. CALL { ... } IN TRANSACTIONS in an explicit transaction

Trigger. Open Neo4j Browser (http://localhost:7474) and run, without the :auto prefix:

LOAD CSV WITH HEADERS FROM 'file:///news.csv' AS ligne
CALL {
  WITH ligne
  MERGE (a:Article {id: toInteger(ligne.id)})
} IN TRANSACTIONS OF 1000 ROWS

Exact message.

A query with 'CALL { ... } IN TRANSACTIONS' can only be executed in an implicit transaction, but tried to execute in an explicit transaction.

Cause. Neo4j Browser wraps every query by default in an explicit transaction (BEGIN/COMMIT) to offer the cancel button. But CALL { } IN TRANSACTIONS — the syntax that handles heavy batched loads — needs an implicit transaction, the one the server creates automatically for each submitted command. The two modes are incompatible.

Fix. Prefix the query with :auto in Neo4j Browser:

:auto
LOAD CSV WITH HEADERS FROM 'file:///news.csv' AS ligne
CALL {
  WITH ligne
  MERGE (a:Article {id: toInteger(ligne.id)})
} IN TRANSACTIONS OF 1000 ROWS

:auto is a Browser command that switches the query to implicit mode. From cypher-shell, ./lab.sh cypher 11-charger-news.cypher does the same job with no prefix: the file is executed in implicit mode by default.

Verify. The query completes and apoc.meta.stats() returns the expected article count.

11. Unknown APOC procedure

Trigger. Temporarily remove the plugin:

docker exec veille-neo4j sh -c 'rm -f /plugins/apoc-*.jar'
docker restart veille-neo4j

Then, in cypher-shell:

CALL apoc.periodic.iterate(
  "MATCH (a:Article) RETURN a",
  "SET a.vu = true",
  {batchSize: 1000}
);

Exact message.

There is no procedure with the name `apoc.periodic.iterate` registered for this database instance.
Please ensure you've spelled the procedure name correctly and that the procedure is properly deployed.

Cause. APOC is an external procedures library. It is not included by default in Neo4j Community: the kit downloads it at first startup via NEO4J_PLUGINS=["apoc"] and keeps it in the neo4j-plugins volume. If the volume is empty, corrupted, or if the download failed at first up (network blocked), the apoc.* procedures are not registered.

Fix.

./lab.sh logs neo4j | grep -i apoc
./lab.sh down
./lab.sh up

The log shows Loading APOC plugins at startup. If the line is missing, check that docker-compose.yml still contains NEO4J_PLUGINS=["apoc"] and that the neo4j-plugins volume is declared. As a last resort, ./lab.sh reset then ./lab.sh up forces a fresh download.

Verify.

CALL apoc.help("apoc.periodic.iterate");

The procedure is listed with its signature.

12. Neo4j password changed in .env but refused

Trigger. Edit .env, replace NEO4J_PASSWORD=veille2026 with NEO4J_PASSWORD=nouveau2026, then:

./lab.sh down
./lab.sh up

Try to log in from Neo4j Browser with nouveau2026.

Exact message.

The client is unauthorized due to authentication failure.

In ./lab.sh logs neo4j:

[o.n.k.a.p.SecurityLogFilter] Failed authentication attempt for 'neo4j' from ...

Cause. Symmetric with case 6. Neo4j writes the password hash into the neo4j-data volume at first startup. NEO4J_AUTH=neo4j/${NEO4J_PASSWORD} is only read at that moment; afterwards, the password is the one from the volume, not the one from .env.

Fix. Two options.

Option A — change the password via cypher-shell while authenticating with the old one:

docker exec -it veille-neo4j cypher-shell -u neo4j -p veille2026 \
  "ALTER USER neo4j SET PASSWORD 'nouveau2026' CHANGE NOT REQUIRED;"

Option B — start over from scratch:

./lab.sh reset
./lab.sh up

The neo4j-data volume disappears, the NEO4J_AUTH variable is re-read at first startup.

Verify.

docker exec veille-neo4j cypher-shell -u neo4j -p nouveau2026 \
  "RETURN 1 AS ok;"

The ok column returns 1 without an authentication error.

General five-step method

When the failure is none of the twelve above, apply the same sequence:

1. Read the status. ./lab.sh status — which containers are Up (healthy), which have Exited (code)? The code already gives the essentials (0 = normal, 137 = OOM, 143 = SIGTERM, 1 = application error).
2. Read the logs of the concerned service. ./lab.sh logs <service> --tail=200. The exact error message is almost always in the last few lines. Copy it word for word.
3. Isolate the service. Test each brick separately: ./lab.sh es _cluster/health, then Kibana (/api/status), then Neo4j (cypher-shell "RETURN 1;"). The first one that fails is the source.
4. Reset at the right level. Three levels: docker restart <container>, then ./lab.sh down / up (keeps the volumes), then ./lab.sh reset / up (wipes everything, last resort). Never apply 3 without having tried 1 and 2.
5. Ask for help with the right information. Copy and paste in order: the command typed, the output of ./lab.sh status, the last twenty lines of ./lab.sh logs <service>, ./lab.sh doctor, your OS, and docker version --format '{{.Server.Version}}'. These five blocks let anyone reproduce your situation in thirty seconds.

The log is right

The only job of a log is to tell the truth. Every time you are tempted to "restart to see", read the last twenty lines first. Nine times out of ten, the cause is spelled out — often with the fix command.

Try it

Try it 1 — Reproduce and read

Trigger failure 5 (read-only index) on a test index, write in a panne5.txt file: the exact returned message, the fix command, the command that verifies the fix.

Solution

PUT test_ro
{"settings":{"index":{"number_of_shards":1,"number_of_replicas":0}}}

PUT test_ro/_settings
{"index.blocks.read_only_allow_delete": true}

POST test_ro/_doc
{"x": 1}

Message:

cluster_block_exception, index [test_ro] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];

Fix:

PUT test_ro/_settings
{"index.blocks.read_only_allow_delete": null}

Verify:

POST test_ro/_doc
{"x": 2}

Response: {"result": "created"}. The block is lifted.

Try it 2 — Cold diagnosis

A colleague shows you his screen: Kibana displays "Kibana server is not ready yet" for the last three minutes. He has already restarted his browser three times. Write the exact sequence of three commands to run, in order, without guessing, to identify the cause.

Solution

./lab.sh status
./lab.sh logs setup --tail=50
./lab.sh logs elasticsearch --tail=50

status tells whether veille-es and veille-setup are in good shape; logs setup reveals a failure in syncing the kibana_system password; logs elasticsearch reveals a deeper issue (heap saturated, watermark, port). If all three pass, a ./lab.sh logs kibana --tail=50 closes the diagnosis. The rule: never restart before reading.

Try it 3 — The message to paste

Draft, as if you were opening a question on an internal help channel, the perfect message for failure 12 (Neo4j password refused after change). Five blocks, in the order of step 5 of the general method.

Solution

Hi, Neo4j Browser refuses authentication ever since I changed the password in .env.

Command typed:
  docker exec veille-neo4j cypher-shell -u neo4j -p nouveau2026 "RETURN 1;"

./lab.sh status:
  veille-es       Up 8 minutes (healthy)   0.0.0.0:9200->9200/tcp
  veille-kibana   Up 7 minutes (healthy)   0.0.0.0:5601->5601/tcp
  veille-neo4j    Up 8 minutes (healthy)   0.0.0.0:7474->7474/tcp, 0.0.0.0:7687->7687/tcp

./lab.sh logs neo4j --tail=20:
  ...
  [o.n.k.a.p.SecurityLogFilter] Failed authentication attempt for 'neo4j' from 172.19.0.5
  ...

./lab.sh doctor: all OK, ports free, memory 8 GB.

OS: Windows 11 + WSL2 Ubuntu, Docker Desktop 29.0.2.

Thirty seconds later, whoever is helping knows it is failure 12 and points to ./lab.sh reset.

Key takeaways

• Twelve failures cover nearly every blocker in the kit: port, memory, vm.max_map_count, cluster color, watermark, Elasticsearch password, Kibana not ready, frozen mapping, LOAD CSV, :auto, APOC, Neo4j password.
• Five of them are fixed with a single command; two (passwords 6 and 12) require ./lab.sh reset because credentials live in the volume from the first startup.
• A container's exit code already gives the essentials: 137 = OOM, 143 = SIGTERM, 1 = application error.
• The general method fits in five steps — status, logs, isolate, graduated reset, informed help — and works for failures that are not in the table too.
• ./lab.sh doctor detects failures 1 (ports), 2 (memory), 3 (vm.max_map_count) in advance; run it first.
• The log is always right: the last twenty lines of ./lab.sh logs <service> almost always contain the exact cause.
• A good help message (command, status, logs, doctor, OS) saves thirty minutes for the helper and the asker.

Troubleshooting

This section points back to the summary table at the top: find the exact symptom in the "Exact symptom" column, follow the row. If your error message is not there, apply the general five-step method and copy-paste the five blocks to someone on the team. Nine times out of ten, the answer arrives before you have finished writing.

Further reading

• Elasticsearch — Cluster allocation explain
• Elasticsearch — Fix a red or yellow cluster status
• Elasticsearch — Fix common cluster issues (watermark, mapping)
• Neo4j — Cypher CALL { } IN TRANSACTIONS