Skip to main content

Module 1 — Why a search engine and a graph database? Install the Veille kit

At Veille, a young Montréal media-monitoring startup, Inès set the day-one rule: "before we write any code, we have to understand why the PostgreSQL of version 1 gave up on us at fifty thousand articles." Sami just joined; this module shows him why the rest of the product will be written with Elasticsearch and Neo4j, then installs the Docker kit that will accompany the course's sixteen modules.

Why a relational database is no longer enough​

The first version of Veille's engine looked for an article with SELECT * FROM articles WHERE headline LIKE '%climate change%'. That works up to a few thousand rows. Past fifty thousand, three problems show up at once.

  • No relevance. SQL returns every matching row in disk order. "Climate Change Is Real" and "How I Learned to Change My Diet in a Changing Climate" come out at the same rank. The user has no idea what to read first.
  • No tolerance. LIKE '%climat change%' returns zero results. One typo and the article disappears. Singulars, plurals, uppercase, accents and variants ("climate", "climatic", "climates") each require their own clause.
  • Slow, very slow. A LIKE '%…%' cannot use a B-tree index: PostgreSQL rescans the whole table on every search. On 200,000 articles, each query costs several seconds.
The useful counter-example

PostgreSQL's full-text extensions (tsvector, pg_trgm) solve part of the problem and are sometimes enough. This course is not against them: it just shows that once search becomes the product — relevance, autocomplete, aggregations, dashboards, typo tolerance, multilingual — a dedicated engine costs less to maintain than a stack of extensions.

The idea behind a search engine: the inverted index​

Elasticsearch never searches inside the articles. It searches inside a table that, for each term of the corpus, lists the documents that contain it. This is called an inverted index.

On our headlines, the analysis pipeline cuts "Climate Change Is Here" into tokens [climate, change, is, here], throws out the stop words (is, here), lowercases them, applies an English stemmer (changing → change) and stores:

climate  → docs [42, 137, 501, 1729, ...]
change   → docs [42, 501, 833, 1729, ...]

Searching for "climate change" then reduces to intersecting two sorted lists: an immediate operation, independent of the corpus size. On top of that the engine adds a relevance score (BM25) that lifts documents where the terms are rarer and more frequent in the document — hence "Climate Change Is Real" ahead of "How I Learned to Change My Diet".

The inverted index is also what makes typo tolerance, autocomplete, category analysis and Kibana dashboards possible. Module 4 takes the analysis pipeline apart; modules 5 and 6 exploit the index from the query side.

When a graph beats a join​

The relational database is perfect for counting articles by category. It gets sluggish as soon as you want to follow relationships across several hops: "which authors wrote about the same topics as a given author?", "what is the shortest chain between two topics through shared articles?", "which article should we recommend to a reader who liked this one?". Each question needs another join, and each one multiplies the intermediate rows.

A graph database stores relationships directly as first-class objects. (:Article)-[:WRITTEN_BY]->(:Author)-[:WRITTEN_BY]-(:Article)-[:PUBLISHED_IN]->(:Category) is traversed by following pointers, without materializing an intermediate table. A path of length three stays immediate even on a graph of two hundred thousand articles, twenty-three thousand authors and forty-one categories — which module 12 demonstrates live on the News corpus.

The Veille kit at a glance​

The entire course fits inside a single folder, one docker-compose.yml file and two scripts, lab.sh for macOS, Linux, WSL2 and Git Bash, lab.ps1 for Windows PowerShell. The rule is simple: you never type a pip install, never an apt, never a brew. Everything you need — Elasticsearch, Kibana, Neo4j with APOC, the corpus importer, a Python container with the official clients, and OpenSearch for the module 9 comparison — runs in Docker.

Download and open the folder​

Download the kit (42-elasticsearch-neo4j.zip, 30 KB), unzip it, open a terminal inside it. On macOS, Linux and WSL2, make the scripts executable once with chmod +x lab.sh doctor.sh. The folder looks like this:

42-elasticsearch-neo4j/
  docker-compose.yml
  env.example
  lab.sh          lab.ps1
  doctor.sh       doctor.ps1
  elasticsearch/mappings/news.json
  importer/import_news.py
  neo4j/cypher/   neo4j/import/
  python/         data/
Windows without WSL2

lab.ps1 mirrors lab.sh identically in PowerShell. Every command in the course will be given as ./lab.sh <subcommand>; the Windows equivalent is .\lab.ps1 <subcommand>. The PowerShell line will be reminded once per module.

Check the machine with doctor​

The first command to run is the diagnostic. It checks Docker, Docker Compose v2, allocated memory, ports 9200, 5601, 7474, 7687, vm.max_map_count, disk space and access to the image registry docker.elastic.co.

./lab.sh doctor            # macOS, Linux, WSL2, Git Bash
.\lab.ps1 doctor           # Windows PowerShell

A passing run looks like this:

Veille kit — diagnostic

[OK] Docker 29.0.2 — daemon reachable
[OK] Docker Compose v2.36.1
[OK] memory allocated to Docker: 8 GB
[OK] port 9200 free
[OK] port 5601 free
[OK] port 7474 free
[OK] port 7687 free
[OK] vm.max_map_count = 262144
[OK] free disk space: 42 GB
[OK] docker.elastic.co reachable (the first ./lab.sh up will download ≈ 3 GB)

Ready. Run: ./lab.sh up

A typical failure and its fix:

[KO] port 9200 already used by another program
      → Linux/macOS: sudo lsof -iTCP:9200 -sTCP:LISTEN
        Windows: netstat -ano | findstr :9200
        — stop that program (often an old container: docker ps)
[KO] memory allocated to Docker: 3 GB — insufficient (4 GB minimum, 6 GB recommended)
      → Docker Desktop → Settings → Resources → Memory
        WSL2: %UserProfile%\.wslconfig → [wsl2] memory=8GB, then wsl --shutdown

Each [KO] is followed by the exact command that fixes it. You should never be stuck for more than thirty seconds on a prerequisite: that is the course's first commitment.

Windows and WSL2

Docker Desktop sets vm.max_map_count automatically in its internal WSL2 distribution. If you installed Docker Engine in a separate WSL2 distribution, doctor asks you to export it manually.

Start the platform with up​

./lab.sh up

The command starts elasticsearch, waits until it is healthy, launches the ephemeral setup service that sets the kibana_system password, then starts kibana and neo4j. The first startup takes about four minutes: image downloads (Elasticsearch 9.5.3 weighs a bit more than one gigabyte, Neo4j 5.26 around six hundred megabytes). Later startups run in two minutes forty on a standard machine.

At the end, the output shows:

NAME              STATUS                    PORTS
veille-es         Up 2 minutes (healthy)    0.0.0.0:9200->9200/tcp
veille-kibana     Up 1 minute (healthy)     0.0.0.0:5601->5601/tcp
veille-neo4j      Up 2 minutes (healthy)    0.0.0.0:7474->7474/tcp, 0.0.0.0:7687->7687/tcp

Access
  Elasticsearch  http://localhost:9200   (elastic / veille2026)
  Kibana         http://localhost:5601   (elastic / veille2026)
  Neo4j Browser  http://localhost:7474   (neo4j / veille2026)  bolt://localhost:7687

Connect to Kibana and Neo4j Browser​

Open http://localhost:5601 in a browser. The user is elastic, the password veille2026. Kibana loads in English, takes ten seconds or so for its first page. Navigate to Management → Dev Tools: this is the console we will use for every query in the course.

Type this first query in Dev Tools and click the green arrow (shortcut Ctrl-Enter):

GET /

You should see the Elasticsearch node answer with its name, its version, the cluster name and a welcome message. This is your first successful API call.

Now open http://localhost:7474. Neo4j Browser asks for a URI: leave bolt://localhost:7687, user neo4j, password veille2026. Once connected, type in the top bar:

CALL dbms.components() YIELD name, versions, edition

The result shows Neo4j Kernel, version 5.26.x, edition community. The graph is empty for now; module 10 will populate it.

Query Elasticsearch without Kibana​

For quick command-line checks, ./lab.sh es <path> runs an authenticated GET:

./lab.sh es _cat/indices?v
./lab.sh es _cluster/health?pretty
./lab.sh es _cat/nodes?v

This command is reserved for GET calls without a body. Every other operation (POST, PUT, DELETE, _search with a JSON body) goes through the Kibana Dev Tools console. You avoid the escape-quote trap in the terminal that way.

Stop and restart​

./lab.sh down          # stops everything, keeps the data
./lab.sh reset         # stops everything AND deletes the data (back to zero)
./lab.sh status        # state, URLs and credentials
./lab.sh logs kibana   # follow the logs of one service

down is the normal end-of-day gesture: the Docker volumes keep the data, the next up starts back up in three minutes. reset is the safety-net gesture: containers deleted, volumes wiped, neo4j/import/news.csv removed, and the next up starts from a completely fresh state. The downloaded file data/News_Category_Dataset_v2.json is kept to avoid an eighty-megabyte download.

One container, one role​

The kit starts six main containers. Understanding who does what makes the logs readable.

ContainerRole
veille-esthe Elasticsearch 9.5.3 node: single-node cluster, security enabled, HTTP without TLS, heap fixed at 1 GB.
veille-setupephemeral service that, after veille-es, sets the internal kibana_system password and exits.
veille-kibanaKibana 9.5.3 in English, wired to veille-es, with Dev Tools, Discover, Lens and dashboards.
veille-neo4jNeo4j 5.26 community with the APOC plugin preinstalled; mounts neo4j/import and neo4j/cypher.
veille-importercontainer of the tools profile, on demand: downloads the corpus, creates the news index, indexes with _bulk.
veille-pythoncontainer of the tools profile: Python 3 with the official elasticsearch and neo4j clients preinstalled.

A seventh and an eighth container, veille-opensearch and veille-os-dashboards, sleep under the opensearch profile. They only start when you run ./lab.sh opensearch-up in module 9, on ports 9201 and 5602 so they do not conflict with Elasticsearch and Kibana.

What the kit does for you​

The kit silently handles everything that cost previous cohorts hours of pain:

  • The kibana_system password is set by the setup service via the /_security/user/kibana_system/_password API. You never have to copy a token by hand between Elasticsearch and Kibana.
  • The Java heap (-Xms1g -Xmx1g) is written in docker-compose.yml, not in .env. The quotation-mark trap that makes the heap look ignored disappears.
  • The disk threshold is disabled (cluster.routing.allocation.disk.threshold_enabled=false): a disk at ninety-five percent will not flip your index to read-only mid-lab.
  • The healthchecks wait for each service to actually answer before ./lab.sh up returns. You never open Kibana while Elasticsearch is still starting up.
  • APOC is added to Neo4j via NEO4J_PLUGINS=["apoc"] and kept in a neo4j-plugins volume: downloaded once, then available even offline.
  • A Kibana encryption key longer than thirty-two characters ships in env.example to avoid the red banner "Kibana requires a value for xpack.encryptedSavedObjects.encryptionKey".
Editing env.example

The file env.example is copied to .env on the first run. You can change the passwords there (letters and digits only to avoid escape issues in the shell) or set NEWS_LIMIT=20000 for a lightning import on a small machine. After editing .env, you must run ./lab.sh reset then ./lab.sh up: the passwords are written into the volume on the first startup.

Machine requirements​

Four gigabytes of RAM allocated to Docker are enough for Elasticsearch, Kibana and Neo4j. Six gigabytes are needed to add OpenSearch in parallel in module 9. You need about ten gigabytes of disk space (three for the images, two for the data), and outbound access to docker.elastic.co and registry-1.docker.io on the first startup. None of these constraints is checked by hand: ./lab.sh doctor does it for you.

Try it 1 — Diagnose and start​

Run ./lab.sh doctor, fix each [KO] by following the arrow, then run ./lab.sh up. Record, in a journal.md file, how long your first up took.

Solution

On a standard machine with Docker Desktop, doctor returns in three seconds. The first up takes between three and five minutes (image downloads included); later runs are around two minutes forty. If your up exceeds eight minutes, open another terminal and run ./lab.sh logs elasticsearch to see what is happening.

Try it 2 — Check three times​

In Kibana Dev Tools, execute:

GET /
GET /_cluster/health
GET /_cat/nodes?v

Record the cluster name, its color (green or yellow) and the node name. Then, in a terminal, run the same last query with ./lab.sh es _cat/nodes?v. Confirm that the output is identical.

Solution

The cluster name is veille, the color green (we will see why in module 2), the node name veille-es. The command ./lab.sh es _cat/nodes?v returns exactly the same tabular output: it is the same HTTP API called with the same credentials, once via Kibana, once via curl inside the veille-es container.

Try it 3 — Shut down cleanly, restart​

Run ./lab.sh down, wait for the three containers to disappear (docker ps), then run ./lab.sh up again. Time the second startup. Finally, try ./lab.sh reset — the volumes disappear — and re-run ./lab.sh up: you will see the difference.

Solution

After down then up, the startup drops to around two minutes forty: the images are cached, the heap is already provisioned, Elasticsearch comes up faster. After reset then up, the time climbs back to three or four minutes because the kibana_system password has to be re-set and Kibana recreates its system indices. That is normal.

Key takeaways​

  • A LIKE '%…%' in SQL is not a search engine: no relevance, no tolerance, and slow past a few thousand rows.
  • Elasticsearch searches via an inverted index: "which documents contain this term" rather than "which terms does this document contain".
  • A graph database makes variable-depth traversals immediate that a relational database pays for with multiple joins.
  • The Veille kit fits inside a single folder and is driven with three commands: doctor, up, import-news.
  • ./lab.sh doctor diagnoses the machine and gives the exact fix command for each problem.
  • Kibana Dev Tools (http://localhost:5601, elastic / veille2026) is the course's reference console; ./lab.sh es <path> covers command-line GETs.
  • down keeps the data, reset starts from scratch: those are your two end-of-day gestures.

Troubleshooting​

  • ./lab.sh up stays stuck on "waiting for Kibana" → Kibana sometimes takes more than three minutes on its first startup on a slow disk. Check with ./lab.sh logs kibana: if you see Kibana is now available, be patient; otherwise, look at the red message.
  • [KO] port 9200 already used by another program → an old Elasticsearch container is still running. docker ps to spot it, docker stop <name> to stop it, then ./lab.sh doctor.
  • Kibana shows "Kibana server is not ready yet" → the setup service has not finished. Wait thirty seconds or check ./lab.sh logs setup; as a last resort, ./lab.sh reset then ./lab.sh up.
  • vm.max_map_count too low on native Linux → sudo sysctl -w vm.max_map_count=262144 fixes the current session; to make it permanent, ./lab.sh doctor gives you the command to add in /etc/sysctl.d/99-elasticsearch.conf.

Further reading​