Skip to main content

Module 5 — Search: Query DSL, bool, filters and relevance

The news index holds the 200,853 articles imported in module 3, and the mapping laid down in module 4 gives them the right analyzers. Sami receives his first order from Inès: make the Veille search bar as precise as a human operator, with relevance, filters, highlighting and deep pagination. This module is the toolbox that answers that order.

Two contexts, two compasses​

Every clause of a Query DSL request is evaluated either in query context or in filter context. The difference is not cosmetic: it decides relevance and performance.

  • The query context answers "how much does this document match my question?" and computes a _score. That is what you want for free text typed by a reader (headline, short_description).
  • The filter context answers "yes or no, does this document pass?" without computing a score. Elasticsearch caches the result in a bitset: the second execution is nearly free. That is what you want for exact criteria (category, date range, field existence).

The Veille rule: as soon as a criterion is a binary choice, it goes into filter; free text stays in must or should.

Filter reflex

If you do not plan to sort by relevance on a criterion (category, date, authors.raw), put it in filter. You gain speed and you make the score readable.

match: the analyzed query​

match is the base full-text query. The passed text is analyzed with the same analyzer as the field, then each term is looked up in the inverted index. The score combines BM25 and term frequency.

GET news/_search
{
  "query": {
    "match": { "headline": "climate change" }
  },
  "size": 3,
  "_source": ["headline", "date", "category"]
}

Response: hits.total.value = 2,834, the first document is "Change Is Here. Climate Change.". The titre_en analyzer (standard + lowercase + asciifolding + English stop + English stemmer) turns climate change into two terms climat and chang, which catches the variants changed, changing, changes.

By default match accepts any document that contains at least one of the terms (or operator). To require all terms:

GET news/_search
{
  "query": {
    "match": {
      "headline": { "query": "climate change", "operator": "and" }
    }
  }
}

To search for the exact phrase, in order:

GET news/_search
{
  "query": {
    "match_phrase": { "headline": "climate change" }
  }
}

The count drops significantly: we only keep the headlines where the two words follow each other (your number may differ slightly depending on the analyzer version).

multi_match: search across several fields​

A word won in the headline weighs more than a word in the body. You tell Elasticsearch so by boosting the field with ^3.

GET news/_search
{
  "query": {
    "multi_match": {
      "query": "climate change",
      "fields": ["headline^3", "short_description"],
      "type": "best_fields"
    }
  }
}

best_fields (default) picks the best field for each document. most_fields adds them, cross_fields treats the fields as a single text, phrase searches for the phrase. For Veille, best_fields with a boost on headline gives natural results.

term, terms, range, exists​

term looks up an exact, unanalyzed value. On a text field it is almost always a mistake; on a keyword, it is the right key.

GET news/_search
{
  "query": {
    "term": { "category": "POLITICS" }
  }
}

Result: 32,739 documents (the largest category). For several values:

GET news/_search
{
  "query": {
    "terms": { "category": ["POLITICS", "WELLNESS", "TRAVEL"] }
  }
}

Expected total: 32,739 + 17,827 + 9,887 = 60,453.

range filters on a range. On the date field (format yyyy-MM-dd):

GET news/_search
{
  "query": {
    "range": {
      "date": { "gte": "2017-01-01", "lt": "2018-01-01" }
    }
  },
  "size": 0
}

exists selects documents that have a field (useful after a partial import):

GET news/_search
{
  "query": { "exists": { "field": "authors" } },
  "size": 0
}

prefix and wildcard: handle with care​

prefix searches for the beginning of a term on a keyword. It is acceptable if the prefix is longer than two characters and if the field is not huge.

GET news/_search
{
  "query": { "prefix": { "authors.raw": "Lee " } }
}

wildcard with * at the start of the pattern forces a full scan of the terms: it is the clause that brings a cluster to its knees.

Leading wildcard

{"wildcard": {"headline.raw": "*trump*"}} scans every term of the field. On news that is still tolerable, on a customer index of twenty million documents it is an incident. Prefer match on the text field or autocomplete (module 8).

bool: composing a query​

bool is the Swiss Army knife of the Query DSL. It combines four lists of clauses:

  • must: all these clauses must match, query context (score).
  • should: these clauses may match; if one matches, the score goes up.
  • filter: all these clauses must match, filter context (no score, cached).
  • must_not: none of these clauses may match, filter context.

Example: POLITICS articles that talk about Trump since 2016, without mentioning "Russia" in the headline.

GET news/_search
{
  "query": {
    "bool": {
      "must":   [ { "match": { "headline": "trump" } } ],
      "filter": [
        { "term":  { "category": "POLITICS" } },
        { "range": { "date": { "gte": "2016-01-01" } } }
      ],
      "must_not": [ { "match_phrase": { "headline": "Russia" } } ],
      "should":   [ { "match": { "short_description": "immigration" } } ]
    }
  },
  "size": 5
}

Reading this request top to bottom gives a clear sentence: "I want trump in relevance, in POLITICS, from 2016 on, without Russia, with a bonus if the summary talks about immigration". It is the template to reuse for almost any Veille search.

The score, _score and BM25 in intuition​

Elasticsearch ranks results by decreasing _score. The default formula is BM25 (Best Matching 25), an evolution of TF-IDF. Three ideas are enough to read it:

  1. TF: the more the term appears in the document, the higher the score goes, with saturation (the tenth trump weighs almost nothing more than the third).
  2. IDF: the rarer the term in the corpus, the more valuable it is (Trump weighs more than the).
  3. Length: a short headline that contains the term is scored better than a long article where it is drowned.

To see the calculation in detail on a specific document, add "explain": true:

GET news/_search
{
  "explain": true,
  "query": { "match": { "headline": "climate change" } },
  "size": 1
}

Each hits[i]._explanation gives the arithmetic decomposition of the score. It is indispensable when a result surprises you.

explain in production

explain: true is costly — reserve it for debugging. In production, prefer to log the query and replay it in Dev Tools when a customer disputes a ranking.

highlight: surface what matched​

For the Veille UI, Léa wants to see in the results what matched.

GET news/_search
{
  "query": { "match": { "headline": "climate change" } },
  "highlight": {
    "fields": {
      "headline": { "pre_tags": ["<mark>"], "post_tags": ["</mark>"] }
    }
  },
  "size": 3
}

Every hit receives a highlight.headline object with HTML fragments ready to display. Karim injects them as-is into the React component of the results bar.

Pagination: from/size and search_after​

For the first pages, from and size are enough:

GET news/_search
{
  "from": 0,
  "size": 20,
  "query": { "match_all": {} },
  "sort": [ { "date": "desc" }, { "_id": "asc" } ]
}

Elasticsearch refuses from + size > 10,000 by default (index.max_result_window parameter). Beyond that, deep pagination uses search_after: you pass back the sort values of the last hit received to restart just after, without memory cost on the cluster side.

GET news/_search
{
  "size": 20,
  "query": { "match_all": {} },
  "sort": [ { "date": "desc" }, { "_id": "asc" } ],
  "search_after": ["2018-05-26", "199999"]
}

Two rules: the sort field must be stable (a date is rarely enough, add _id), and the request must be identical from one call to the next.

_source: reduce network payload​

The API returns the whole document by default. On Veille, Karim only needs headline, date, category for the list:

GET news/_search
{
  "_source": ["headline", "date", "category"],
  "query": { "term": { "category": "TRAVEL" } },
  "size": 20
}

_source: false outright removes the content (useful for counting or for an internal top_hits).

Try it 1 — How many POLITICS articles mention "election"?​

Write the query in a single Dev Tools command and read the counter in hits.total.value.

Solution
GET news/_search
{
  "size": 0,
  "query": {
    "bool": {
      "must":   [ { "match": { "headline": "election" } } ],
      "filter": [ { "term": { "category": "POLITICS" } } ]
    }
  }
}

must carries the free text (therefore the score), filter holds the category without polluting the relevance. size: 0 avoids fetching documents when you only want the count (your number may differ slightly).

Try it 2 — TRAVEL articles from 2017, sorted from most recent to oldest, with headline highlighted​

Return only headline, date and 10 documents.

Solution
GET news/_search
{
  "size": 10,
  "_source": ["headline", "date"],
  "query": {
    "bool": {
      "filter": [
        { "term":  { "category": "TRAVEL" } },
        { "range": { "date": { "gte": "2017-01-01", "lt": "2018-01-01" } } }
      ]
    }
  },
  "sort": [ { "date": "desc" }, { "_id": "asc" } ],
  "highlight": {
    "fields": { "headline": { "pre_tags": ["<mark>"], "post_tags": ["</mark>"] } }
  }
}

Both criteria go into filter (no need for a score). The stable sort adds _id as a second criterion, which lets you chain on with search_after for the next page.

Try it 3 — Fix a query that returns nothing​

Sami writes this and gets no result. Why, and how do you fix it?

GET news/_search
{
  "query": {
    "term": { "headline": "Trump" }
  }
}
Solution

term looks up the exact, unanalyzed value. But headline is a text field with the titre_en analyzer that lowercases everything and applies a stemmer: the term in the index is neither Trump nor trump but trump (root). term with Trump fails silently. Two possible fixes:

GET news/_search
{ "query": { "match": { "headline": "Trump" } } }

or, if you really want an exact comparison on the raw value:

GET news/_search
{ "query": { "term": { "headline.raw": "Trump Wins" } } }

Rule: match on a text, term on a keyword.

Key takeaways​

  • Query context for the score, filter context for binary criteria and the cache.
  • match analyzes the text, term takes the raw value; use .raw for the exact match on a text.
  • bool structures the query: must for meaning, filter for constraint, must_not for exclusion, should for the bonus.
  • _score follows BM25: saturated frequency, valued rarity, penalized length; "explain": true reveals the calculation.
  • highlight returns HTML fragments ready to display, _source limits what is fetched.
  • from/size up to 10,000 results, search_after with a stable sort beyond that.
  • Avoid wildcard at the start of a pattern; prefer autocomplete (module 8).

Troubleshooting​

  • "search_context_missing_exception" when paginating with scroll → the scroll API is reserved for reindexing; use search_after for user results.
  • hits.total.value: 0 unexpectedly on a term over text → the field is text: switch to match, or target the keyword sub-field (for example headline.raw).
  • "too_many_clauses" on a giant terms → the value of indices.query.bool.max_clause_count is exceeded, split the query; on the kit, restart with ./lab.sh down && ./lab.sh up if the parameter was touched.
  • Puzzling result → add "explain": true then replay the query on a specific document with GET news/_explain/<id> to read the decomposition of the score.

Further reading​