Module 14 — Security, users, backups, and day-to-day operations
The Veille engine is up, indexed, and queried; Inès now wants two guarantees before exposing anything to a client. First, that Karim's API cannot modify an article by accident. Then, that a hardware incident won't wipe out six months of aggregations saved in Kibana. This module sets up the roles, the API keys, the snapshots, and the monitoring routine that keep a cluster healthy over time.
What the kit already secures
Open docker-compose.yml from the kit and read the elasticsearch block: security is enabled by default in the lab, which many earlier tutorials skip. Four lines are enough to grasp the cluster's state.
- xpack.security.enabled=true
- xpack.security.http.ssl.enabled=false
- xpack.security.transport.ssl.enabled=false
- ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
Three consequences. Every request is authenticated: without credentials, Elasticsearch answers 401 security_exception. The superuser password is the one from the .env file (veille2026 by default), set at first startup and burned into the es-data volume. HTTP travels in clear text: exchanges between veille-kibana, veille-python, and veille-es are not encrypted, which is acceptable inside an isolated Docker network but unacceptable the moment you expose port 9200 to the outside.
The veille-setup service completes the picture: it waits for Elasticsearch to be healthy, then calls the /_security/user/kibana_system/_password API to align the internal password of the system user with KIBANA_PASSWORD. You never have to copy a token by hand between the two services.
The kit favors readability: a curl or a ./lab.sh es works with no certificate. In production, switch to HTTPS. Elastic ships a utility, elasticsearch-certutil (inside the image, bin/elasticsearch-certutil ca then cert), that generates the authority and the node certificates; then enable xpack.security.http.ssl.enabled=true and mount the files into the container. This course stays on HTTP; the switch procedure is documented on elastic.co and takes an hour to set up.
Create a read-only role and user for Karim
Karim's API only needs to read the news index: never write, never delete, never touch the mapping. We build a dedicated role, then a user who carries that role. Open Kibana Dev Tools.
POST _security/role/veille_lecture
{
"cluster": ["monitor"],
"indices": [
{
"names": ["news"],
"privileges": ["read", "view_index_metadata"]
}
]
}
read allows _search, _count, _msearch, _mget; view_index_metadata allows fetching the mapping and the settings, essential for a client that wants to know the name of the time field. monitor at the cluster level allows _cluster/health: the application's liveness probe will not have to re-authenticate as elastic.
POST _security/user/api_karim
{
"password": "karim2026",
"roles": ["veille_lecture"],
"full_name": "API Veille (Karim)",
"email": "karim@veille.example"
}
Elasticsearch replies {"created": true}. Verify immediately with two calls from a terminal, inside the veille-es container which ships curl:
docker exec veille-es curl -s -u api_karim:karim2026 \
http://localhost:9200/news/_count
Expected output:
{"count":200853,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0}}
The read passes. Now test that a write is refused:
docker exec veille-es curl -s -u api_karim:karim2026 \
-H 'Content-Type: application/json' \
-X PUT http://localhost:9200/news/_doc/999999 \
-d '{"headline":"attempt","date":"2026-09-09"}'
Expected output:
{"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:data/write/index] is unauthorized for user [api_karim] with effective roles [veille_lecture] on indices [news]"}],"status":403}}
403 security_exception: the role does exactly what we asked it to. That is the first testing reflex whenever you create a user — check what passes and what must be refused.
GET _security/role/veille_lecture and GET _security/user/api_karim return the complete JSON definition. GET _security/_query/user lists every user with pagination. In Kibana, Management → Stack Management → Security → Users / Roles offers the same thing with the mouse.
API keys for machine authentication
A password works, but every service that uses it must know the full password, and revoking it cleanly means redeploying everything. API keys solve the problem: each service receives its own key, with a role attached and an expiration date. You revoke one key without touching the rest.
POST _security/api_key
{
"name": "api-karim-lecture",
"expiration": "90d",
"role_descriptors": {
"veille_lecture": {
"cluster": ["monitor"],
"indices": [
{
"names": ["news"],
"privileges": ["read", "view_index_metadata"]
}
]
}
}
}
Elasticsearch replies with an object that has three important fields: id, api_key, and encoded. The encoded field already contains the concatenation id:api_key Base64-encoded, ready to drop into the HTTP header.
{
"id" : "V0cU5oQBz2ExampleId",
"name" : "api-karim-lecture",
"expiration" : 1770000000000,
"api_key" : "abcDEF...",
"encoded" : "VjBjVTVvUUJ6MkV4YW1wbGVJZDphYmNERUY..."
}
The application calls Elasticsearch by pasting the encoded value after ApiKey :
docker exec veille-es curl -s \
-H "Authorization: ApiKey VjBjVTVvUUJ6MkV4YW1wbGVJZDphYmNERUY..." \
http://localhost:9200/news/_search?size=1
Two commands useful every day. GET _security/api_key?owner=true lists the keys you have created with their expiration date. DELETE _security/api_key with a body {"ids": ["V0cU5oQBz2ExampleId"]} revokes a compromised key immediately.
Kibana: Spaces at a glance
Kibana ships Spaces — the equivalent of siloed folders for saved objects: dashboards, Data Views, Lens visualizations. Access them via Management → Stack Management → Spaces. The Veille team can create a "Léa" space that only contains client dashboards, and a "Sami" space reserved for operations views. A Kibana role can restrict a user to one or several spaces with distinct access levels (read on the client space, all on the internal space). Fine-grained Kibana RBAC goes beyond this module; remember it exists and is free under the basic license.
Back up Elasticsearch with an fs repository
Elasticsearch stores data incrementally in a snapshot kept inside a repository. The simplest repository is of type fs — a plain local directory. Two rules to know before you start.
- The directory must be declared to the node through the
path.repoconfiguration. Without it,PUT _snapshot/…refuses the creation. - The directory must be accessible from every node in a multi-node cluster. On our single-node cluster, a local Docker volume is enough.
Add the repository to the container
Edit docker-compose.yml, in the elasticsearch block, add one environment line and the matching volume:
services:
elasticsearch:
environment:
# ... existing lines ...
- path.repo=/usr/share/elasticsearch/snapshots
volumes:
- es-data:/usr/share/elasticsearch/data
- es-snapshots:/usr/share/elasticsearch/snapshots
volumes:
es-data:
es-snapshots:
# ... rest unchanged ...
Restart Elasticsearch without wiping data:
./lab.sh down
./lab.sh up
./lab.sh reset would delete es-data — you would lose the news index. down then up keeps the volumes.
down/up and not restartA restart of a container does not replay the Compose configuration, so path.repo would not be picked up. ./lab.sh down then ./lab.sh up recreates the container with the new parameters while keeping the existing volumes.
Create the repository and a first snapshot
From Kibana Dev Tools:
PUT _snapshot/veille_repo
{
"type": "fs",
"settings": {
"location": "/usr/share/elasticsearch/snapshots",
"compress": true
}
}
Expected response:
{"acknowledged":true}
Launch the first snapshot, synchronously so you see the result immediately:
PUT _snapshot/veille_repo/snap1?wait_for_completion=true
{
"indices": "news",
"include_global_state": false
}
On the full News corpus, the backup takes about ten seconds and returns:
{
"snapshot": {
"snapshot": "snap1",
"state": "SUCCESS",
"indices": ["news"],
"shards": {"total": 1, "failed": 0, "successful": 1}
}
}
Two control commands. GET _snapshot/veille_repo/_all lists snapshots. GET _snapshot/veille_repo/snap1/_status details the bytes copied and the duration.
Restore into a renamed index
You rarely restore on top of an existing index: too dangerous. Elasticsearch offers a projection with rename_pattern and rename_replacement that renames indices on the fly during the restore.
POST _snapshot/veille_repo/snap1/_restore
{
"indices": "news",
"rename_pattern": "news",
"rename_replacement": "news_restaure",
"include_global_state": false
}
Check:
GET _cat/indices/news*?v
Expected output:
health status index uuid ... docs.count store.size
green open news ... 200853 ...
green open news_restaure ... 200853 ...
Two indices side by side, same documents, same mapping. You can compare, validate, then switch an alias over to news_restaure with _aliases (seen in module 8) without ever cutting the service.
Back up Neo4j Community
Neo4j Community ships a native command, neo4j-admin database dump, that writes a binary file. The database must be stopped for this edition; the Enterprise edition provides an online dump, but not this one.
docker exec veille-neo4j cypher-shell -u neo4j -p veille2026 \
-d system "STOP DATABASE neo4j;"
docker exec veille-neo4j neo4j-admin database dump neo4j \
--to-path=/dumps
STOP DATABASE puts the database offline, dump writes neo4j.dump into /dumps. So that this directory is reachable from the host machine, mount it as a volume in docker-compose.yml:
services:
neo4j:
volumes:
# ... existing lines ...
- ./neo4j/dumps:/dumps
(After editing: ./lab.sh down then ./lab.sh up, as for Elasticsearch.) Restart the database:
docker exec veille-neo4j cypher-shell -u neo4j -p veille2026 \
-d system "START DATABASE neo4j;"
Restoration goes through neo4j-admin database load neo4j --from-path=/dumps --overwrite-destination=true, database stopped as well.
Neo4j Community users
The community edition knows users, not fine-grained roles. You can create an account and change its password, but every account stays at the same privilege level as long as it lives in the system database. Open cypher-shell on the system database:
./lab.sh cypher-shell
Then:
:use system
CREATE USER karim SET PASSWORD 'karim2026' CHANGE NOT REQUIRED;
ALTER USER karim SET PASSWORD 'karim2026-b' CHANGE NOT REQUIRED;
SHOW USERS;
Expected output:
+---------------------------------------------------------------+
| user | roles | passwordChangeRequired | suspended |
+---------------------------------------------------------------+
| "karim" | ["PUBLIC"] | FALSE | FALSE |
| "neo4j" | ["admin"] | FALSE | FALSE |
+---------------------------------------------------------------+
Granting Karim a role allowed only to read certain labels or traverse certain relationships requires GRANT MATCH { … } ON GRAPH … TO role, which belongs to the Enterprise edition. On Community, the only possible separation is "admin" (created at startup) versus "PUBLIC" (full default access). For real production isolation, plan for an Enterprise switch or move the authorization logic into the application API.
Daily monitoring
Three commands make up Sami's morning dashboard. Each fits on one line in Dev Tools.
GET _cat/nodes?v&h=name,heap.percent,ram.percent,cpu,disk.used_percent
Expected output:
name heap.percent ram.percent cpu disk.used_percent
veille-es 27 82 3 41.7
heap.percent must stay below 75% in steady state. Above 85% for long, garbage collection slows queries down; you must raise the heap (ES_JAVA_OPTS variable in docker-compose.yml) or reduce the load. disk.used_percent must stay below 85%: above, Elasticsearch automatically applies the flood_stage threshold and switches indices to read-only (the kit disables this threshold for the lab, but in production it is active).
GET _cat/indices?v
Expected output:
health status index pri rep docs.count store.size
green open news 1 0 200853 130mb
Three values to watch per index: health (green = ok, yellow = missing replicas, red = missing primary shard, case handled in module 15), docs.count (moves when you write), store.size (grows over time; if an index keeps ballooning, consider ILM).
GET _nodes/stats/jvm
Detailed JVM output: mem.heap_used_in_bytes, gc.collectors.young.collection_time_in_millis. A young GC time that climbs is the first sign of an under-sized heap.
Host-side, two commands complete the picture:
docker stats --no-stream
./lab.sh status
./lab.sh logs elasticsearch --tail=50
docker stats shows CPU, memory, network from Docker's point of view — useful to spot a container consuming more than its limit (mem_limit: 2g in our compose). ./lab.sh status summarizes the state of the containers, ./lab.sh logs tails a service's logs.
Password best practices
Three rules learned from previous cohorts, spelled out in env.example:
- Letters and digits only in
.env. A!,$, or@will be interpreted by the shell whenlab.shexports the variables, and the password received by Elasticsearch will not be the one written in the file. - Reasonable minimum length — twelve characters in production; the kit password (
veille2026) is a pedagogical convention, to be changed before any real use. ./lab.sh resetrequired after any change. Passwords are written into thees-datavolume at first startup (forelastic) and intoneo4j-data(forneo4j). A plaindown/updoes not pick them up again; you must wipe the volumes withreset, which also deletes thenewsindex — remember to take a snapshot first.
Try it
Try it 1 — A role for Léa (analyst)
Create a veille_analyse role that lets Léa read the news index and create, read, and modify her own Kibana objects. Then create a user lea with this role and check that she can search in Dev Tools but cannot delete the news index.
Solution
Kibana defines a named privilege over the whole analytics feature (kibana-.kibana) that composes on the role side. The simplest path is to create the role via the interface Stack Management → Roles → Create role by adding the Kibana Analytics: All privilege. From Dev Tools, you can create an Elasticsearch role with the same index privileges:
POST _security/role/veille_analyse
{
"cluster": ["monitor"],
"indices": [
{ "names": ["news"], "privileges": ["read", "view_index_metadata"] }
]
}
Then the user:
POST _security/user/lea
{
"password": "lea2026abcd",
"roles": ["veille_analyse", "kibana_admin"],
"full_name": "Léa Bernier"
}
The built-in kibana_admin role covers the Kibana side. Negative test:
docker exec veille-es curl -s -u lea:lea2026abcd \
-X DELETE http://localhost:9200/news
Expected:
{"error":{"type":"security_exception","reason":"action [indices:admin/delete] is unauthorized ..."},"status":403}
Try it 2 — Snapshot before, restore after
Launch the snapshot snap_avant of the single news index, delete three documents (DELETE news/_doc/1, 2, 3), verify that _count dropped, then restore only into news_restaure and compare the two counts.
Solution
PUT _snapshot/veille_repo/snap_avant?wait_for_completion=true
{ "indices": "news", "include_global_state": false }
DELETE news/_doc/1
DELETE news/_doc/2
DELETE news/_doc/3
GET news/_count
Response: {"count": 200850}.
POST _snapshot/veille_repo/snap_avant/_restore
{
"indices": "news",
"rename_pattern": "news",
"rename_replacement": "news_restaure",
"include_global_state": false
}
GET news_restaure/_count
Response: {"count": 200853}. The restore correctly recovered the deleted documents without overwriting the production index. All that remains is to switch an alias to news_restaure or to reinject the three missing documents with a filtered _reindex.
Try it 3 — The morning dashboard
Write the three-request sequence that Sami runs every morning to verify that Veille is healthy. It must fit in three Dev Tools blocks and cover: cluster health, main index, node resources.
Solution
GET _cluster/health
GET _cat/indices/news?v
GET _cat/nodes?v&h=name,heap.percent,ram.percent,cpu,disk.used_percent
Three lines of clear output: status, docs.count, heap.percent. Sami can save these three requests in the Dev Tools history (they stay there) and replay them with Ctrl+Enter.
Key takeaways
- The kit enables
xpack.security.enabled=trueand leaves HTTP in clear text: it's a lab choice; in production, enable TLS withelasticsearch-certutil. - Create one role per use case (
veille_lecturefor the API), then a user or an API key per service; always test what must pass and what must be refused. - API keys (
POST _security/api_key) are used with the headerAuthorization: ApiKey <encoded>; you revoke them per key, without touching the rest. - Elasticsearch snapshots require
path.repodeclared in the node configuration, then aPUT _snapshot/...repository; the renamed restore (rename_pattern/rename_replacement) avoids overwriting a live index. - Neo4j Community backs up with
neo4j-admin database dumpwhile the database is stopped; fine-grained RBAC stays Enterprise. - Daily monitoring:
_cat/nodes?v&h=...,_cat/indices?v,_nodes/stats/jvm,docker stats,./lab.sh status, and./lab.sh logs. - Passwords in
.env: letters and digits only; any change requires./lab.sh reset(think about the snapshot first).
Troubleshooting
PUT _snapshot/veille_reporeturnsrepository_verification_exception→path.repois not declared in the container or the volume is not mounted. Checkdocker-compose.yml, then./lab.sh downand./lab.sh up.neo4j-admin database dumpanswers "database is not offline" → the database was not stopped. Opencypher-shellonsystemand runSTOP DATABASE neo4j;before thedump.curl -u api_karim:...returns401 security_exception→ the password was changed in Dev Tools without updating the client, or was copied with a trailing space. RunPOST _security/user/api_karim/_passwordagain to realign it.- After changing
ELASTIC_PASSWORDin.env,./lab.sh uprefuses authentication → the password is already written into thees-datavolume../lab.sh resetthen./lab.sh up— after a snapshot if you want to keepnews.