Skip to main content

Recap and final exam

Sixteen modules to move from LIKE '%word%' to a complete search and recommendation engine: Veille indexes 200,853 articles, retrieves them by relevance and despite typos, analyzes them in Kibana, links 23,082 authors to 41 categories in Neo4j, and recommends what to read next from a Python script. Here is the course condensed, the threads that run through it, and the checklist of an engine ready to demo to a client.

The course at a glance​

ModuleThe essential point
1. Veille kitdoctor before up; up waits for every service to answer; reset returns to a clean state. Kibana on 5601, Neo4j Browser on 7474, everything in veille2026.
2. Core conceptsAn index is split into shards, each shard can have replicas; a single node stays green with zero replicas; _cat and _cluster/health are read first.
3. Documents and _bulkPUT creates or replaces, POST _update merges, _seq_no/_primary_term guard against overwrites; _bulk = NDJSON, two lines per document, batches of 1,000 to 5,000.
4. Mappingtext is analyzed for search, keyword is exact for filtering, sorting, and aggregating; dynamic mapping guesses, explicit mapping decides; changing a type = _reindex.
5. Query DSLmatch for text, term for exact, range for dates; bool combines must, should, filter, must_not; filter context computes no score and is cached.
6. Aggregationssize: 0; terms, date_histogram with calendar_interval, cardinality, stats, top_hits; sub-aggregations answer crossed questions (categories per year).
7. KibanaData View with time field date and range 2012-2018; Discover + KQL to explore; Lens to visualize; a dashboard exports as .ndjson.
8. Advanced searchcompletion on headline.suggest (max_input_length 120), fuzziness: AUTO, phrase suggester; ES|QL chains FROM, WHERE, STATS … BY, SORT; aliases enable reindexing without downtime.
9. OpenSearchFork of Elasticsearch 7.10 under Apache 2.0 since 2021; same base queries, different security and plugins; choose by license, host, and features, not by fashion.
10. Neo4j and CypherNodes, labels, directed and typed relationships, properties; MATCH (a)-[:REL]->(b) reads the graph the way you draw it; MERGE over CREATE for replayable scripts.
11. Model and loadWhat you filter or traverse becomes a node, what you display stays a property; uniqueness constraints first; LOAD CSV + CALL { } IN TRANSACTIONS (with :auto in Browser).
12. Advanced CypherPaths *1..3, shortestPath, OPTIONAL MATCH, WITH/collect/UNWIND, CASE; neighborhood recommendation fits in one query; PROFILE reveals missing indices and cartesian products.
13. Python./lab.sh python runs your scripts with the official clients; Elasticsearch(url, basic_auth) and GraphDatabase.driver(uri, auth); veille.py combines search and recommendation.
14. Security and operationsUsers and roles _security, API keys, fs snapshots with path.repo; Neo4j Community handles users but no fine-grained roles; _cat/nodes and docker stats for monitoring.
15. DiagnosisTwelve failures, one method: status, logs, isolate, reset, ask for help with the right information.
16. Veille projectJustified mapping, five queries per persona, four-visualization dashboard, verified graph, five recommendation queries, combined script, five-minute demo.

The threads running through the course​

The mapping decides everything that comes after. A category in text does not aggregate cleanly, a title in keyword cannot be searched, a date as a string cannot be sorted. Module 4 sets the rule, modules 5 to 8 live by it: every query that "finds nothing" or every aggregation that "returns bits of words" is fixed in the mapping, therefore by a _reindex, never by twisting the query.

Searching and filtering are two different operations. The query context computes a score and is expensive; the filter context answers yes or no and is cached. Putting category and date in filter and keeping headline in must is not a stylistic quirk: it is what keeps the Veille engine fast under load. Kibana applies the same separation between the KQL bar and the pinned filters.

The graph answers questions that joins make unreadable. Who writes in the same categories as Lee Moran? Which recent articles are by the authors from that result? Which chain links two authors? In SQL, every extra depth adds a join; in Cypher, *1..3 is enough. Elasticsearch finds the entry point, Neo4j explores around it: module 13 combines them in the same script, module 16 turns them into a product.

A failure must be read before it is fixed. Exit code 137, cluster_block_exception, "Couldn't load the external resource", "can only be executed in an implicit transaction": every message from module 15 corresponds to a single cause and a single kit command. The reflex to acquire is not to restart at random but to read ./lab.sh status and then ./lab.sh logs <service>.

Checklist of a Veille engine ready to demo​

  • ./lab.sh doctor fully green, ./lab.sh status shows three healthy containers.
  • GET news/_count returns 200,853 and GET news/_mapping shows category as keyword, headline as text with raw and suggest, date as date.
  • One bool query per persona, with headline in must and category + date in filter, each one commented (what it searches for, why that score).
  • A terms aggregation on category and a monthly date_histogram, whose numbers match the Kibana dashboard.
  • A news Data View with time field date, a four-visualization dashboard exported as .ndjson in the project folder.
  • An autocomplete query on headline.suggest and a typo-tolerant search with fuzziness: AUTO.
  • CALL apoc.meta.stats() returns 200,853 Article, 41 Categorie, 23,082 Auteur; SHOW CONSTRAINTS lists the three uniqueness constraints.
  • Five Cypher recommendation queries, including one by author neighborhood and one by path, each passed through PROFILE without a cartesian product.
  • ./lab.sh python veille.py <topic> returns results and recommendations in under five seconds, with a category and period filter.
  • A read-only Elasticsearch user for Karim's API and a restorable snapshot of the news index.
  • The twelve-failure table reread, and the certainty of knowing what to do in front of each.

The final exam​

The exam contains 40 questions covering the sixteen modules: kit and concepts, documents and _bulk, mapping and analyzers, Query DSL and relevance, aggregations, Kibana, autocomplete and ES|QL, OpenSearch, Cypher and modeling, LOAD CSV, advanced Cypher, Python, security and backups, diagnosis, project.

Several questions present situations to diagnose: which field type to choose for a category you want to aggregate; why a match query on a keyword returns nothing; what a yellow cluster on a single node means; how to read a cluster_block_exception; why LOAD CSV cannot find news.csv; what the difference is between MERGE and CREATE when replaying a script; which engine to pick for which business question. It is judgement that is assessed — naming the cause, choosing the fitting query or command — not the recitation of a vocabulary.

On success, your certificate of completion is issued immediately; its number is verifiable by any third party on the platform.

Before you start

Go back through the checklist above. For every line, ask yourself "would I know how to write the query or the command that verifies it?". If you can write a bool with must and filter, a date_histogram, a uniqueness constraint followed by a batched LOAD CSV, and a neighborhood recommendation query, you are ready. Good luck!

Final exam

Ready to validate this course?

40 questions drawn at random from the course bank · passing score 70% · verifiable PDF certificate issued immediately on success.

Start the exam

You need to be signed in to your InSkillML account with an active subscription. You can also start the exam from My courses.