Skip to main content

Autocompletion, typo tolerance, ES|QL and advanced search

Karim has to deliver the Veille product's search bar: it suggests headlines as you type, it forgives typos, and it opens the door to a more readable language than Query DSL when an analyst wants to aggregate quickly. Léa, on her side, wants to flip a mapping in production without cutting service. This module bundles four workstreams: completion on headline.suggest, fuzziness: AUTO, ES|QL, aliases and zero-downtime reindexing.

Autocompletion: the completion suggester​

Mapping reminder (module 4): headline has a headline.suggest sub-field of type completion with max_input_length: 120. This special type is not a classic inverted index — it is a finite-state transducer (FST) that matches a prefix instantly against every indexed term. The trade-off: it stays in RAM, and it is expensive to write.

Why max_input_length: 120​

The default for completion is 50 characters. On the HuffPost corpus, that truncates one headline out of two: "The 20 Best Vegan Recipes You'll Actually Want To Cook This Weekend" (66 characters) would be cut down to "The 20 Best Vegan Recipes You'll Actually Want To". A user who types "weekend" would find nothing. Bumping the value to 120 covers the whole News corpus while keeping a reasonable FST.

The default value is a silent trap

completion returns no error on truncated headlines: it simply indexes them up to the max length. The day the autocompletion "misses" obvious suggestions on long headlines, max_input_length is the first parameter to check with GET news/_mapping.

Query _search with suggest​

In Kibana Dev Tools:

GET news/_search
{
  "_source": false,
  "suggest": {
    "titles": {
      "prefix": "trum",
      "completion": {
        "field": "headline.suggest",
        "size": 5,
        "skip_duplicates": true
      }
    }
  }
}

Expected response (extract from options):

'Truman Show' Delusion: Believing Your Life Is A Reality TV Show
Trump Abandons Commitment To 2-State Solution In Press Conference With Netanyahu
Trump Signs Order Ordering Federal Agencies To Cut Two Regulations For Every New One
Truman Capote's Ashes Sold For $43,750 At Auction
Truman Show Syndrome, Or When People Think Their Life Is A TV Show

Three things to take away from this request:

  • The prefix trum matches both Trump and Truman: completion looks at the beginning of the term, not at its semantics.
  • skip_duplicates: true avoids two identical headlines in the suggestions (useful when the same story was published twice).
  • _source: false removes the classic hits; we only want the suggest.titles key. It slims down the response.

The response time is on the order of a fraction of a millisecond on this corpus — that is the FST promise.

search_as_you_type as a preview​

An alternative to completion: the search_as_you_type type, which automatically creates sub-fields ._2gram, ._3gram, ._index_prefix. It forgives mid-word typos and matches across several fields at once, at the price of a heavier index. For the Veille bar, completion is enough; keep search_as_you_type in mind for multilingual corpora or when you want to "search as you type" rather than "suggest an exact headline".

Typo tolerance: fuzziness: AUTO​

A reader searching for "climat chnage" (the "n" before the "a") should still find "Climate Change". That is the job of fuzziness — the Damerau-Levenshtein edit distance — that match accepts directly.

GET news/_search
{
  "query": {
    "match": {
      "headline": {
        "query": "climat chnage",
        "fuzziness": "AUTO"
      }
    }
  },
  "size": 3,
  "_source": ["headline"]
}

Expected response: several thousand results (your number may vary slightly depending on tokens), headed by headlines that contain "climate change". The AUTO value applies a smart rule: 0 edits allowed for a term of 1 to 2 characters, 1 for 3 to 5 characters, 2 for 6 characters and above. It is the setting to keep by default.

phrase suggester: "Did you mean"​

When the typo spans several words, a dedicated suggester generates the most likely corrected phrase:

GET news/_search
{
  "suggest": {
    "correction": {
      "text": "climat chnage",
      "phrase": {
        "field": "headline",
        "size": 3,
        "gram_size": 3,
        "direct_generator": [
          { "field": "headline", "suggest_mode": "always" }
        ]
      }
    }
  }
}

Response (extract):

climate change    (high score)
climate changes
climat change

The phrase suggester uses a language model on n-grams of the field to rank corrections by plausibility. This is what search engines use behind the classic "Did you mean: …".

Fuzzy yes, but not everywhere

fuzziness: AUTO on a massive terms or on a short prefix ("a", "le") becomes slow and pollutes the results. Reserve it for the main field (headline) and for searches of two terms or more. On other fields, stick to strict match.

match_phrase with slop: the flexible phrase​

match_phrase looks for the expression in order and stuck together. slop allows Elasticsearch to accept a few words between the terms (or an inversion) while respecting the intent of the phrase.

GET news/_search
{
  "query": {
    "match_phrase": {
      "headline": {
        "query": "climate change",
        "slop": 2
      }
    }
  },
  "size": 3
}

With slop: 0 (the default), the request matches only "climate change" stuck together. With slop: 2, it also matches "climate is changing", "change in climate" or "climate rapid change". The score drops as the number of moves required grows. This is exactly what the main Veille bar wants: a little slack around an expression, without going for a fully loose match.

multi_match with weighting​

A headline carries more meaning than a summary: we boost headline over short_description.

GET news/_search
{
  "query": {
    "multi_match": {
      "query": "climate change",
      "fields": ["headline^3", "short_description"],
      "type": "best_fields",
      "fuzziness": "AUTO"
    }
  },
  "size": 5,
  "_source": ["headline", "category"]
}

The ^3 suffix multiplies the headline field's contribution to the score by three. Combined with fuzziness: AUTO, this is the "robust" search-bar request: it forgives typos, it favors headlines, it spans several fields. This is what Karim wires into the first version of the API.

ES|QL: Elasticsearch's pipe language​

ES|QL (Elasticsearch Query Language) is a piped language (à la Splunk / SQL) that appeared in Elasticsearch 8 and stabilized in 9. It complements Query DSL: where DSL is declarative JSON tuned for weighted search, ES|QL chains FROM, WHERE, STATS, SORT, LIMIT in a single readable line, tuned for analysis.

Three ways to run it:

  • In Kibana Dev Tools with the POST _query endpoint ({"query": "..."}).
  • In Discover (Kibana) by switching the language selector from KQL to ES|QL.
  • From Python (module 13) via the official elasticsearch client.

Example 1 — Count articles per category​

POST _query
{
  "query": "FROM news | STATS n = COUNT(*) BY category | SORT n DESC | LIMIT 10"
}

Response (first lines of values):

POLITICS         32739
WELLNESS         17827
ENTERTAINMENT    16058
TRAVEL            9887
STYLE & BEAUTY    9649

One line, one pipeline, a directly tabular result. In classic Query DSL, the same thing required size: 0, a terms with field: category and a sort, plus some JSON noise.

Example 2 — Filter, aggregate, sort​

POST _query
{
  "query": "FROM news | WHERE category == \"POLITICS\" | STATS n = COUNT(*) BY category | SORT n DESC | LIMIT 10"
}

Result: POLITICS 32,739. We keep the STATS ... BY category structure here to show the syntax; on a single group, a plain STATS n = COUNT(*) is enough.

Example 3 — Top authors over a date range​

POST _query
{
  "query": "FROM news | WHERE date >= \"2017-01-01\" AND date < \"2018-01-01\" | STATS articles = COUNT(*) BY authors.raw | SORT articles DESC | LIMIT 5"
}

Expected result (your number may vary slightly):

Reuters       1 900+
Lee Moran       600+
Ed Mazza        500+
Ron Dicker      450+
Cole Delbyck    350+

The pipe stays readable even when you stack several filters, which is precisely ES|QL's selling point for Léa (she writes about thirty analysis requests per week).

Example 4 — Extract a year and pivot​

POST _query
{
  "query": "FROM news | EVAL year = DATE_EXTRACT(\"year\", date) | STATS n = COUNT(*) BY year, category | SORT year ASC, n DESC | LIMIT 20"
}

EVAL creates a calculated field (year) on each row, STATS ... BY year, category does a cross aggregation. Result: the dominant categories per year, from 2012 to 2018. That is the request you type in thirty seconds to feed a chart.

Query DSL, KQL and ES|QL: when to use which​

CriterionQuery DSLKQLES|QL
FormatDeclarative JSONCompact string (category : "POLITICS" and headline : trump)Pipeline FROM ... | ...
WhereREST, client code, Dev ToolsDiscover, Lens, Alerting (Kibana)Dev Tools, Discover, client code
Relevance, score, _scoreYes (BM25, explain)Yes (thin layer over DSL)Not designed for it (tabular result)
Complex aggregationsYes (verbose)NoYes, very readable
Composite filters (bool, must_not)YesYesYes, with WHERE
Calculated fieldsruntime_mappingsNoYes, EVAL
Target audienceAPI developersKibana usersAnalysts, data engineers
Outputhits with _scorehitscolumns / values table

The Veille rule:

  • Query DSL for anything the API serves to a client: search bar, autocompletion, relevance-ranked results. That is module 5.
  • KQL for quick exploration in Discover and to define filters in a Lens dashboard. That is module 7.
  • ES|QL for anything that looks like SQL analysis on the data: pivoted aggregations, period-to-period comparisons, calculated fields. That is here.

The three coexist: a single dashboard can display a Lens driven by KQL, feed an ES|QL panel and trigger an alert defined in Query DSL.

Index aliases and zero-downtime reindexing​

Sami wants to add a resume field to news, or change the headline analyzer. As we saw in module 4, we do not modify a type in place: we have to create a new index. But the product API queries GET news/_search — if we rename the index, we break the client.

The classic workaround: an alias. An alias is a logical name that points to one or more physical indices, and switches atomically.

Step 1 — Rename the current index​

If news is already a physical index, the migration starts with: "make news an alias that points to a real news_v1".

POST _reindex
{
  "source": { "index": "news" },
  "dest":   { "index": "news_v1" }
}

Then delete news (warning: no readers for two seconds) and create the alias:

DELETE news
POST _aliases
{
  "actions": [
    { "add": { "index": "news_v1", "alias": "news" } }
  ]
}

In Veille production, we plan for this from the start: the first index is called news_v1 and news is an alias to it. Module 3 could have done this; we will do it as soon as we ship a real service.

Step 2 — Create news_v2 with the new mapping​

PUT news_v2
{
  "settings": { "number_of_shards": 1, "number_of_replicas": 0 },
  "mappings": {
    "properties": {
      "headline":          { "type": "text", "analyzer": "titre_en" },
      "short_description": { "type": "text", "analyzer": "titre_en" },
      "category":          { "type": "keyword" },
      "authors":           { "type": "text", "fields": { "raw": { "type": "keyword", "ignore_above": 256 } } },
      "link":              { "type": "keyword", "index": false },
      "date":              { "type": "date", "format": "yyyy-MM-dd" },
      "resume":            { "type": "text", "analyzer": "titre_en" }
    }
  }
}

Step 3 — Reindex​

POST _reindex
{
  "source": { "index": "news_v1" },
  "dest":   { "index": "news_v2" }
}

On 200,853 articles, count a few dozen seconds.

Step 4 — Atomic alias switch​

POST _aliases
{
  "actions": [
    { "remove": { "index": "news_v1", "alias": "news" } },
    { "add":    { "index": "news_v2", "alias": "news" } }
  ]
}

Both actions are applied together: not a millisecond without an alias, no client request rejected. This is the maneuver we repeat on every mapping evolution.

One alias per use case

Nothing forces you to have a single alias. news can point to news_v2 for writes and general reads, and news_recent can be another alias filtered on the last 30 days (with filter in the add action). Two logical views, one physical index.

ILM: a conceptual preview​

On a static corpus of 200,000 articles, a single index is enough. On a stream — application logs, real-time news wires, telemetry traces — every day brings new volumes that it is absurd to keep "hot" indefinitely. ILM (Index Lifecycle Management) automates the lifecycle of indices in four phases:

PhaseRoleWhat you typically do there
hotActive writes and readsOne primary shard, refresh_interval: 1s, heavy indexing
warmNo more writes, frequent readsforcemerge to compact, replicas can be reduced
coldRare reads, storage optimizedNo indexing, often mounted in the frozen tier
deleteDeletionDELETE of the index once it passes N days

Moving from one phase to the next triggers on criteria (size, age, document count). You define a policy (PUT _ilm/policy/veille_logs), attach it to an index template (news-*), and Elasticsearch does the rest. The News corpus does not have this need; keep the principle in mind for the day Veille indexes application logs.

Try it​

Try it 1 — Suggest on a prefix and count unique proposals​

Query headline.suggest on the prefix climat, ask for 10 suggestions without duplicates, and deduce how many distinct headlines start with this prefix (up to the 10 returned).

Solution
GET news/_search
{
  "_source": false,
  "suggest": {
    "titles": {
      "prefix": "climat",
      "completion": {
        "field": "headline.suggest",
        "size": 10,
        "skip_duplicates": true
      }
    }
  }
}

The response lists up to 10 distinct headlines starting with "Climat…" (skip_duplicates: true removes strict duplicates). If you want more, bump size — beware, this value also drops low-score proposals beyond it.

Try it 2 — Intentional typo​

Compare the number of results for a strict match versus a match with fuzziness: AUTO on the query "climat chnage". Explain the difference.

Solution
GET news/_count
{ "query": { "match": { "headline": "climat chnage" } } }

GET news/_count
{
  "query": {
    "match": {
      "headline": { "query": "climat chnage", "fuzziness": "AUTO" }
    }
  }
}

The first request returns very few results: chnage is almost never a real term. The second returns thousands: fuzziness: AUTO allows one edit on chnage (a two-letter transposition), which becomes change, and one edit on climat which matches climate (after stemming, they already fall onto the same climat term). Compare with "Change Is Here. Climate Change." at the top of the hits.

Try it 3 — ES|QL: monthly evolution of a category​

Write an ES|QL request that counts POLITICS articles per month between 2016-01-01 and 2018-05-26, sorted from oldest to newest.

Solution
POST _query
{
  "query": "FROM news | WHERE category == \"POLITICS\" AND date >= \"2016-01-01\" AND date <= \"2018-05-26\" | EVAL month = DATE_TRUNC(1 month, date) | STATS n = COUNT(*) BY month | SORT month ASC"
}

DATE_TRUNC(1 month, date) truncates every date to the first day of its month; STATS ... BY month aggregates. The output is a (month, n) table ready to chart. In Query DSL, the same thing would use a date_histogram with calendar_interval: month — see module 6.

Key takeaways​

  • completion on headline.suggest responds in milliseconds to a prefix; max_input_length: 120 avoids the silent truncation of the default value (50).
  • fuzziness: AUTO catches one or two typos depending on term length; keep it for the main full-text fields.
  • The phrase suggester generates the most likely correction of a multi-word request, to display as "Did you mean: …".
  • match_phrase with slop finds an expression even when a few words or an inversion sneak in.
  • ES|QL (FROM ... | WHERE ... | STATS ... BY ... | SORT | LIMIT) is the pipe to prefer for analysis; Query DSL still rules for weighted search.
  • An index alias switches atomically from news_v1 to news_v2: this is the key to zero-downtime reindexing.
  • ILM automates the hot → warm → cold → delete lifecycle for corpora that grow over time; useless on news, essential the moment you index logs.

Troubleshooting​

  • suggest returns an empty list on a prefix that is present in the headlines → the field is not of type completion, or the headline was truncated to 50 characters (default). Check GET news/_mapping then ./lab.sh import-news to reindex with the kit's mapping.
  • unknown query [fuzziness] on a term request → fuzziness applies to full-text clauses (match, multi_match), not to term. Switch to match.
  • POST _query returns unknown function [DATE_TRUNC] or similar → the ES|QL function does not exist in 9.5.3 in this form. Check elastic.co/docs/reference/query-languages/esql/esql-functions-operators and adapt (DATE_EXTRACT, BUCKET).
  • POST _aliases returns index_not_found_exception → the index cited in remove does not exist (already deleted) or the one in add does not either (not yet created). Run GET _cat/indices?v then relaunch the switch in the right order: create news_v2, reindex, then POST _aliases with the two actions together.

Further reading​