Skip to main content

Module 16 — Project: the complete Veille engine, search and recommendation

Inès has booked the little glass room at Veille's HQ and written three verbs in blue marker on the board: "find, analyze, recommend." This module is the contract Sami signs with her: assemble the fifteen previous modules into an engine that answers these three verbs on the 200,853 articles of the News corpus, and know how to demo it to a prospect in five minutes without ever leaving the kit.

One-page specification

The product targets three client profiles — a political communications agency, a wellness magazine, a travel agency. It fits inside the kit and can be demonstrated on a laptop with 6 GB of RAM allocated to Docker. Nothing to install locally; every operation goes through ./lab.sh, Kibana Dev Tools, Neo4j Browser, and the veille-python container.

What must be running at the end:

  • Elasticsearch 9.5.3 with the news index (200,853 documents, one shard, zero replicas, the kit's mapping).
  • Kibana with the news Data View and the Veille — vue d'ensemble dashboard (four visualizations, one control per category).
  • Neo4j 5.26 Community with the News graph (200,853 :Article, 41 :Categorie, 23,082 :Auteur) and its three uniqueness constraints.
  • An extended python/veille.py script that combines Elasticsearch and Neo4j.

What must be versioned inside a livraison-projet/ folder (next to the kit, to be zipped at the end of the project):

livraison-projet/
├── README.md # demo how-to, 5 min on the clock
├── es/
│ ├── mapping-news.json # annotated copy of the mapping
│ └── requetes-personas.md # 15 commented queries (5 per persona)
├── kibana/
│ └── veille.ndjson # dashboard + data view export
├── neo4j/
│ └── recommandations.cypher # 5 recommendation queries
└── python/
└── veille.py # extended script (category/period filter + explanation)

The three personas

Each persona drives five queries in step 2, one entry in the dashboard of step 3, and one line of the demo in step 7.

  • Client A — Political communications agency. Follows the POLITICS articles (32,739 in total) between 2016 and 2017, wants to identify the most prolific authors on the presidential campaign and export a monthly timeline. Known target authors: Lee Moran (264 POLITICS articles), Ed Mazza, Ron Dicker.
  • Client B — Wellness magazine. Covers WELLNESS (17,827) and HEALTHY LIVING. Requires autocomplete on titles (the completion component from module 8) and typo tolerance ("yoga" found even when the user types "yaga"). Expected keyword base: yoga, meditation, mindfulness, sleep, diet.
  • Client C — Travel agency. Follows TRAVEL (9,887 articles) month by month to align campaigns with seasonality. Wants to spot summer and end-of-year peaks between 2012 and 2018.

The seven steps

Step 1 — The news index with a justified mapping

Deliverable. livraison-projet/es/mapping-news.json copied from kits/42-elasticsearch-neo4j/elasticsearch/mappings/news.json, plus a section of the README.md that justifies every field.

Success criterion.

./lab.sh es news/_count
{"count":200853,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0}}
./lab.sh es news/_mapping

The output must show headline as text with the titre_en analyzer and two sub-fields (raw as keyword, suggest as completion), category as keyword, date as date with format yyyy-MM-dd, authors as text with a raw sub-field.

Starting point. The kit's mapping already covers every need; the work is to write the justification. For each field, one sentence: "category is keyword to allow exact aggregation over the 41 values and filter-context filtering (module 4)", "headline.suggest is a completion for persona B's autocomplete (module 8)".

Pitfall. Redefining headline without the suggest sub-field breaks the entire B step. A completion cannot be added to an existing mapping: it requires a full _reindex (module 4). Check before you start that the sub-field is present.

Step 2 — Five search queries per persona

Deliverable. livraison-projet/es/requetes-personas.md, 15 queries grouped by persona, each with the expected count and a sentence of business context.

Success criterion. All queries paste-and-play in Kibana Dev Tools and return numbers consistent with the other personas.

Starting point. Three founding examples to build on.

Persona A — a pertinence search on the 2016-2017 campaign:

GET news/_search
{
"size": 5,
"query": {
"bool": {
"must": [ { "match": { "headline": "trump election" } } ],
"filter": [
{ "term": { "category": "POLITICS" } },
{ "range": { "date": { "gte": "2016-01-01", "lte": "2017-12-31" } } }
]
}
},
"_source": ["headline", "date", "authors"],
"highlight": {
"fields": { "headline": { "pre_tags": ["<mark>"], "post_tags": ["</mark>"] } }
}
}

This bool pattern with must (free text, scored) and filter (binary criteria, cached) is the one that must recur in everything Veille does (module 5). Persona A's five queries: this search, a monthly date_histogram on POLITICS 2016-2017, a terms on authors.raw filtered on POLITICS, a term on authors.raw = "Lee Moran" with a sub-aggregation by category, and a bool variant that excludes Russia with must_not.

Persona B — autocomplete on headline.suggest:

GET news/_search
{
"suggest": {
"titre_suggest": {
"prefix": "yog",
"completion": {
"field": "headline.suggest",
"size": 5,
"skip_duplicates": true
}
}
}
}

Typo tolerance:

GET news/_search
{
"size": 5,
"query": {
"match": {
"headline": { "query": "meditaton mindfullness", "fuzziness": "AUTO" }
}
}
}

Persona B's five queries: those two, a multi_match on "healthy diet" (headline^3 and short_description), a terms on category filtered on WELLNESS and HEALTHY LIVING (result: 17,827 + your HEALTHY LIVING count may differ slightly), a top_hits per category returning the most recent title (module 6).

Persona C — monthly seasonality:

GET news/_search
{
"size": 0,
"query": { "term": { "category": "TRAVEL" } },
"aggs": {
"par_mois": {
"date_histogram": {
"field": "date",
"calendar_interval": "month",
"format": "yyyy-MM"
}
}
}
}

The TRAVEL corpus totals 9,887 documents spread from 2012-01-28 to 2018-05-26; the histogram must produce about 77 monthly buckets, with a visible summer peak every year (your figure may vary slightly month to month). The four other persona C queries: a terms on TRAVEL authors, a bool with should on summer or vacation, a category × date_histogram sub-aggregation with calendar_interval: year to compare TRAVEL and STYLE & BEAUTY (9,649), a fixed_interval: 90d variant to reason in quarters.

Pitfall. Using term instead of match on headline returns zero results with no error (the field is analyzed, module 5, Try it 3). Forgetting size: 0 on aggregations pulls 10 documents back for nothing.

Step 3 — The four-visualization Kibana dashboard

Deliverable. livraison-projet/kibana/veille.ndjson obtained through the "with related objects" export.

Success criterion. On a clean workstation, importing the .ndjson rebuilds the news Data View, the four visualizations, and the Veille — vue d'ensemble dashboard. The metric shows 200,853, the first bar bucket shows POLITICS 32,739, the curve covers 2012-2018.

Starting point. Module 7 details the clicks. Key reminders: set the time range to absolute 2012-01-01 → 2018-06-01 before anything; create each visualization in the library before adding it to the dashboard; add an Options list control on category so the client can filter without leaving the dashboard.

Pitfall. Forgetting to tick "Include related objects" on export means the news Data View isn't carried along, and the import on the next workstation fails with an unclear message ("unable to find data view").

Save the time range

Tick "Store time with dashboard" when you save the dashboard. Otherwise, every reopening starts on "~last 15 minutes" and the client sees an empty dashboard.

Step 4 — The News graph loaded and verified

Deliverable. The complete graph in Neo4j 5.26 Community, with its three uniqueness constraints and its date index.

Success criterion. Two commands suffice after a ./lab.sh up:

./lab.sh import-news
./lab.sh cypher 11-charger-news.cypher

Then, in Neo4j Browser or ./lab.sh cypher-shell:

CALL apoc.meta.stats() YIELD labels, relTypesCount
RETURN labels, relTypesCount;
labels: { "Article": 200853, "Categorie": 41, "Auteur": 23082 }
relTypesCount: { "PUBLIE_DANS": 200853, "ECRIT_PAR": <your count may differ slightly> }
SHOW CONSTRAINTS;

Must list article_id, categorie_nom, auteur_nom as UNIQUENESS, plus the article_date index.

Starting point. The 11-charger-news.cypher script from the kit is idempotent thanks to MERGE. Rerunning it doubles no node (module 11).

Pitfall. Running ./lab.sh cypher 11-charger-news.cypher without having done ./lab.sh import-news produces "Couldn't load the external resource: file:///news.csv": the CSV has not yet been written into neo4j/import/ (module 15).

Step 5 — Five Cypher recommendation queries

Deliverable. livraison-projet/neo4j/recommandations.cypher.

Success criterion. Each query runs in under one second on the full graph. PROFILE must show a plan that starts with a NodeUniqueIndexSeek, never an AllNodesScan (module 12).

Starting point. Five patterns to write.

Query 1 — Other recent articles by the same author, starting from a seed article:

MATCH (:Article {id: $id})-[:ECRIT_PAR]->(au:Auteur)<-[:ECRIT_PAR]-(reco:Article)
WHERE reco.id <> $id AND au.nom <> 'Reuters'
RETURN reco.titre AS titre, au.nom AS via, reco.date AS date
ORDER BY reco.date DESC LIMIT 5;

Query 2 — Nearby authors via shared categories, two-hop pattern:

MATCH (moi:Auteur {nom: $nom})<-[:ECRIT_PAR]-(:Article)-[:PUBLIE_DANS]->(c:Categorie)
<-[:PUBLIE_DANS]-(:Article)-[:ECRIT_PAR]->(voisin:Auteur)
WHERE voisin <> moi AND voisin.nom <> 'Reuters'
WITH voisin, count(DISTINCT c) AS communes
RETURN voisin.nom, communes ORDER BY communes DESC LIMIT 10;

Query 3 — Shortest path between two authors (bounds are mandatory):

MATCH chemin = shortestPath(
(a:Auteur {nom: $de})-[*..6]-(b:Auteur {nom: $vers})
)
RETURN [n IN nodes(chemin) | coalesce(n.nom, n.titre)] AS etapes,
length(chemin) AS profondeur;

Query 4 — An author's top categories:

MATCH (:Auteur {nom: $nom})<-[:ECRIT_PAR]-(:Article)-[:PUBLIE_DANS]->(c:Categorie)
RETURN c.nom AS categorie, count(*) AS articles
ORDER BY articles DESC LIMIT 5;

For Lee Moran, expected result (see brief): COMEDY 779, WEIRD NEWS 391, ENTERTAINMENT 387, POLITICS 264, SPORTS 101.

Query 5 — Cross-recommendation from a category and a date:

MATCH (c:Categorie {nom: $cat})<-[:PUBLIE_DANS]-(a:Article)-[:ECRIT_PAR]->(au:Auteur)
WITH au, count(a) AS n WHERE n >= 3 AND au.nom <> 'Reuters'
MATCH (au)<-[:ECRIT_PAR]-(reco:Article)
WHERE reco.date >= date($depuis)
RETURN reco.titre AS titre, au.nom AS via, reco.date AS date
ORDER BY reco.date DESC LIMIT 10;

Pitfall. Forgetting au.nom <> 'Reuters' floods the results: the agency signs 4,954 articles by itself, saturating every recommendation (module 13).

Step 6 — The extended python/veille.py script

Deliverable. livraison-projet/python/veille.py with three extensions compared to the kit version:

  1. Category and period filters (command-line arguments).
  2. Top authors for the query (Elasticsearch aggregation in the same pass).
  3. Recommendation explanation: each recommended article mentions the common author it was found through.

Success criterion. The full command runs in under five seconds on the freshly started kit:

./lab.sh python veille.py "climate change" POLITICS 2016-01-01 2017-12-31

Expected output (scores and recommendations may vary slightly depending on BM25):

Elasticsearch 9.5.3 — cluster « veille »

3 articles for « climate change » (POLITICS, 2016-01-01 → 2017-12-31):
[15.44] 2017-06-01 Trump Just Pulled Out Of The Paris Agreement — Kate Sheppard (POLITICS)
...

Top authors on the selection:
Kate Sheppard, Karen Feridun, Reuters

5 recommendations (common author shown):
2018-04-19 The E.P.A. Rolls Back A Regulation On Coal Ash — via Kate Sheppard (ENVIRONMENT)
...

Starting point. Restart from the rechercher() and recommander() functions of module 13 and extend rechercher():

def rechercher(es, texte, categorie=None, du=None, au=None, taille=5):
filtres = []
if categorie:
filtres.append({"term": {"category": categorie}})
if du or au:
borne = {}
if du: borne["gte"] = du
if au: borne["lte"] = au
filtres.append({"range": {"date": borne}})
rep = es.search(
index="news",
size=taille,
query={
"bool": {
"must": [{"multi_match": {"query": texte,
"fields": ["headline^3", "short_description"]}}],
"filter": filtres,
}
},
aggs={"auteurs": {"terms": {"field": "authors.raw", "size": 3}}},
source=["headline", "authors", "category", "date"],
)
top = [b["key"] for b in rep["aggregations"]["auteurs"]["buckets"]]
articles = [{"id": int(h["_id"]), "score": round(h["_score"], 2), **h["_source"]}
for h in rep["hits"]["hits"]]
return articles, top

The recommander() function now receives RETURN reco.titre AS titre, au.nom AS via, c.nom AS categorie, reco.date AS date and the display follows.

Pitfall. Hard-coding the arguments breaks the demo. Go through sys.argv (or argparse for comfort) and provide default values for quick tests.

Step 7 — The five-minute demonstration

Deliverable. livraison-projet/README.md that captures the minute-by-minute script (the "Demonstration" section below).

Success criterion. Replay the demo against a stopwatch, land at 4 min 45 s ± 15 s. No live tab-hunting.

Starting point. Five minutes before showtime, open four tabs in this order: Kibana Dev Tools with queries preloaded, the Veille — vue d'ensemble Kibana dashboard, Neo4j Browser connected to bolt://localhost:7687, a terminal placed inside the kit (cd kits/42-elasticsearch-neo4j).

Pitfall. Running ./lab.sh up or ./lab.sh import-news live wastes two or three minutes. Both commands must be launched behind the scenes, before the client arrives.

Bonus "Try it" step — The multi-source "climate" timeline

Deliver a single report livraison-projet/rapport-climat.md that proves the four tools hold together on the same subject: a counter (match headline: "climate change" → 2,834), an annual timeline (date_histogram calendar_interval: year, seven buckets 2012-2018), the three most active authors on the topic (filtered terms), a capture of the dashboard filtered via KQL on headline: "climate change", and five articles recommended by the Python recommander() function.

Solution

A mini Python script that composes the four bricks:

"""Rapport climat — Elasticsearch pour trouver, Neo4j pour recommander."""
from veille import rechercher, recommander
from elasticsearch import Elasticsearch
from neo4j import GraphDatabase
import os

es = Elasticsearch(os.environ["ES_URL"],
basic_auth=(os.environ["ES_USER"], os.environ["ES_PASSWORD"]))
pilote = GraphDatabase.driver(os.environ["NEO4J_URI"],
auth=(os.environ["NEO4J_USER"], os.environ["NEO4J_PASSWORD"]))

articles, top = rechercher(es, "climate change", taille=5)
with pilote.session() as session:
top_auteurs = session.run(
"MATCH (au:Auteur)<-[:ECRIT_PAR]-(a:Article) "
"WHERE a.titre CONTAINS 'limate' "
"RETURN au.nom AS nom, count(a) AS n "
"ORDER BY n DESC LIMIT 3"
).data()
recos = recommander(session, [a["id"] for a in articles])
pilote.close()

print("# Rapport climat\n")
print("- Compteur `match headline: \"climate change\"` : 2 834 articles.")
print("- Chronologie annuelle : 7 buckets (2012 → 2018).")
print("- Top auteurs sur le sujet climat :")
for r in top_auteurs:
print(f" - {r['nom']} : {r['n']} articles")
print("- Recommandations :")
for r in recos:
print(f" - {r['date']} {r['titre']} — via {r['via']} ({r['categorie']})")

Saved as python/rapport_climat.py, it runs with ./lab.sh python rapport_climat.py. The livraison-projet/rapport-climat.md file captures the output and adds the screenshot of the filtered dashboard, taken from Kibana's Share → Copy as image button.

100-point grading rubric

CriterionPointsHow to verify
Step 1 — Justified mapping10./lab.sh es news/_mapping shows headline as text + sub-fields raw and suggest, category as keyword, date as date. The README explains each choice in one sentence.
Step 2 — 15 persona queries20requetes-personas.md contains 15 executable json blocks; every query cites the expected count and the associated business role.
Step 3 — Kibana dashboard15Import of veille.ndjson on a clean workstation recomposes the Data View, the 4 visualizations, and the category control; metric = 200,853, top bucket = POLITICS 32,739.
Step 4 — Verified News graph10CALL apoc.meta.stats() returns 200,853 Article, 41 Categorie, 23,082 Auteur; SHOW CONSTRAINTS lists the three uniqueness constraints and the date index.
Step 5 — 5 Cypher recommendation queries15Each query runs in < 1 s (PROFILE without AllNodesScan); the 5 patterns cover common author, shared categories, shortest path, top categories, cross-recommendation.
Step 6 — Extended Python script15./lab.sh python veille.py "climate change" POLITICS 2016-01-01 2017-12-31 returns articles + top authors + recommendations in < 5 s, with the via <author> mention.
Step 7 — Five-minute demo10Replayed in 4 min 45 s ± 15 s, 4 tabs prepared, no live tab or command hunting.
Diagnostic quality (bonus)5The README.md lists the 3 most likely failures observed during the assembly and points to module 15 for each.
Total100

Indicative scale: 90 and above, ready to demo at a real client. 75 to 89, ready internally but one axis remains to consolidate — most often step 3 (the time range) or step 5 (a query that scans instead of seeks). Below 75, redo step 2 (queries) and step 7 (demo) first, they carry most of the perceived value.

Variants

Three tracks to adapt the project without redoing it.

  • OpenSearch variant (engine comparison). ./lab.sh opensearch-up starts OpenSearch 3.8.0 on http://localhost:9201 with no authentication. Reimport the corpus into that index (adapt a Python script to point at 9201) and replay the 15 persona queries. Note which ones work identically (basic Query DSL: match, bool, range), which ones return a slightly different ranking (the scoring formula has diverged since 2021), and which ones do not exist on the OpenSearch side (ES|QL). The result frames Veille's engine choice (module 9). ./lab.sh opensearch-down shuts it down cleanly.
  • Another corpus variant. A Veille client arrives with his own CSV (an internal RSS feed, 50,000 titles, or a CMS export). Adapt the mapping: keep the five fields headline, short_description, category, date, authors; rename them if the columns differ. importer/import_news.py provides the _bulk pattern. Replaying personas A, B, C on that corpus proves that Veille's queries stay robust outside the HuffPost dataset.
  • Enriched graph variant with keywords. Extend the Neo4j model to capture keywords from short_description:
LOAD CSV WITH HEADERS FROM 'file:///news.csv' AS ligne
CALL {
WITH ligne
MATCH (a:Article {id: toInteger(ligne.id)})
UNWIND [mot IN split(toLower(coalesce(ligne.short_description, '')), ' ')
WHERE size(mot) > 4
AND NOT mot IN ['about', 'their', 'which', 'there', 'other', 'would']] AS mot
MERGE (m:MotCle {nom: mot})
MERGE (a)-[:MENTIONNE]->(m)
} IN TRANSACTIONS OF 2000 ROWS;

This enriched graph unlocks a sixth recommendation query: "articles that share at least three keywords with the seed article." It enters the rubric as a bonus query, provided a MotCle.nom IS UNIQUE constraint is added before the load.

Frequent mistakes

Seven mistakes an end-of-course project almost systematically piles up, with the pointer to module 15 for the detailed analysis.

  • hits.total.value: 0 on a term against headline. The field is text with a stemmer; term looks for the raw value (module 5, Try it 3). Switch to match, or target headline.raw if you want exact matching.
  • Couldn't load the external resource: file:///news.csv. The importer was not launched, or the file was deleted by a ./lab.sh reset. Replay ./lab.sh import-news before ./lab.sh cypher 11-charger-news.cypher (module 15, case #9).
  • can only be executed in an implicit transaction on CALL { } IN TRANSACTIONS. In Neo4j Browser, prefix with :auto. From ./lab.sh cypher <file>, no prefix is required — the kit's cypher-shell already runs in implicit mode (module 15, case #10).
  • security_exception 401 after an incomplete reset. The Elasticsearch password lives in the volume; changing ELASTIC_PASSWORD in .env without going through ./lab.sh reset has no effect. A proper reset then up puts the credentials back in sync (module 15, case #6).
  • The Kibana dashboard is empty after import. The time range is stuck on "~last 15 minutes". Switch to absolute 2012-01-01 → 2018-06-01 and save the range with the dashboard (module 7, pitfall number 1; module 15 for the "invisible data" variant).
  • Fielddata is disabled on text fields by default on a terms aggregation over headline. Always aggregate on the keyword sub-field (headline.raw, authors.raw); never re-enable fielddata, which blows up the heap (module 6, Troubleshooting).
  • The Cypher recommendation drags. PROFILE the query; if the plan starts with AllNodesScan, the uniqueness constraint is missing or the parameter does not match any indexed property. Reload 11-charger-news.cypher, then check with SHOW CONSTRAINTS that the three constraints are indeed there (module 15, case #11).

Scripted five-minute demonstration

To be replayed in front of a prospect. Four tabs prepared upstream, in this order: terminal cd'd into the kit, Kibana Dev Tools with queries preloaded, Kibana Veille — vue d'ensemble dashboard, Neo4j Browser connected.

  • 0:00 → 0:45 — The index and the corpus. Window: terminal. Type ./lab.sh status then ./lab.sh es news/_count. Line: "We index 200,853 press articles in thirty seconds on a 6 GB machine. Here is the state: three healthy containers, an index at 200,853 documents."
  • 0:45 → 1:45 — Relevance-based search. Window: Kibana Dev Tools. Replay the persona A bool query (climate change + POLITICS + 2016-2017 + highlight). Line: "BM25 pushes titles where the words appear in the title over those where they appear in the summary; highlight highlights the match. The query fits in ten lines of JSON and it is cacheable."
  • 1:45 → 2:45 — The dashboard. Window: Kibana dashboard. Click POLITICS in the Options list control: the four panels filter simultaneously. Line: "The same index powers the search bar and the analytics. Léa hands this dashboard to clients: they click, they compare, they export to CSV."
  • 2:45 → 3:45 — The graph. Window: Neo4j Browser. Paste the "authors close to Lee Moran through shared categories" query. Switch the display to Graph. Line: "Elasticsearch finds an article, Neo4j goes up to the author, back down to other articles by neighboring authors. Three lines of Cypher where SQL would need four recursive joins."
  • 3:45 → 4:45 — The Python assembly. Window: terminal. Run ./lab.sh python veille.py "climate change" POLITICS 2016-01-01 2017-12-31. Line: "Veille's API calls this script. Five relevant results, five recommendations, all in under five seconds, with the via <author> mention that explains why each article is recommended."
  • 4:45 → 5:00 — The wrap-up. Window: dashboard, filter cleared. Line: "The engine fits in one folder, deploys in three commands, tears down in ten seconds with ./lab.sh reset. We install it at a prospect's site in half an hour."
Pacing and breathing

Do not speak while a command is running: use the three seconds of the Cypher query to let the graph draw itself. The silence gives weight to the result.

Key takeaways

  • The project assembles seven deliverables versioned inside livraison-projet/; each one is verifiable by a command.
  • Three personas fix the expected queries: chronological POLITICS, WELLNESS with autocomplete, seasonal TRAVEL.
  • The justified mapping is the foundation: everything that follows depends on headline as text with raw + suggest, category as keyword, date as date.
  • The 100-point rubric values the demo (10 points) as much as the Cypher queries (15 points); the client's perceived value goes through the staging.
  • The OpenSearch variant puts module 9 into practice and settles the engine choice for Veille.
  • The extended python/veille.py script proves the two engines combine in thirty lines without stepping on each other.
  • The five-minute demo can be repeated: stopwatch in hand, four tabs prepared, one line per window.

Troubleshooting

  • The dashboard is empty → the time range stayed relative; switch to absolute 2012-01-01 → 2018-06-01 and re-save with the dashboard (module 7, module 15 for the variant).
  • A Cypher recommendation query dragsPROFILE reveals an AllNodesScan: reload 11-charger-news.cypher and check with SHOW CONSTRAINTS (module 15, case #11).
  • ./lab.sh python veille.py returns ImportError → the script is run on the host instead of the container; always go through ./lab.sh python …, never python veille.py directly (module 13).
  • The demo blocks on ./lab.sh import-news live → run the import five minutes before showtime; never reimport during the demo, it takes 28 seconds that drop the room's energy (module 15 for the full diagnosis).

Further reading