Module 4 — Mapping, text versus keyword, analyzers

The corpus is loaded, _count returns 200,853, and Sami already wants to run his first match on "climate change". Inès stops him: before searching, you have to understand why Elasticsearch stored each field the way it did. That is the job of the mapping — the schema of the documents — and of its analyzers, which decide how a headline becomes a sequence of searchable terms. This module opens elasticsearch/mappings/news.json and comments on each line.

The dynamic mapping: what Elasticsearch guesses on its own

If you create an index without a mapping, Elasticsearch builds one on the fly from the first indexed document. Let us try in Kibana Dev Tools:

POST devine/_doc
{
  "titre":  "Change Is Here. Climate Change.",
  "vues":   42,
  "publie": true,
  "date":   "2018-05-26"
}

Then read what it guessed:

GET devine/_mapping

Result (excerpt):

{
  "devine": {
    "mappings": {
      "properties": {
        "titre": {
          "type": "text",
          "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }
        },
        "vues":   { "type": "long" },
        "publie": { "type": "boolean" },
        "date":   { "type": "date" }
      }
    }
  }
}

Three rules to remember from this guesswork:

1. Any string becomes a text field with an automatic keyword sub-field of 256 characters (ignore_above: 256). This gives you the flexibility of full-text plus the exactness of a keyword, at the cost of doubling the storage on that field.
2. Whole numbers become long (64 bits), not integer. On a view counter that will never exceed two billion, that is waste.
3. A string in ISO 8601 format is recognized as date. A string in 26/05/2018 format is not — it lands as text.

The trap of guessed dates

The dynamic mapping recognizes a few date formats (ISO 8601, RFC 1123…) but not all. A dataset with local dates dd/MM/yyyy will end up as text, which makes range and date_histogram impossible. The Veille rule: as soon as a field holds a date, we write its type and its format by hand.

That is precisely why the kit ships elasticsearch/mappings/news.json, applied before any _bulk.

The real mapping of the news index, field by field

Let us open elasticsearch/mappings/news.json — this is the file that ./lab.sh import-news sends via PUT /news.

{
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 0,
    "refresh_interval": "30s",
    "analysis": {
      "analyzer": {
        "titre_en": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["lowercase", "asciifolding", "stop_en", "stem_en"]
        }
      },
      "filter": {
        "stop_en": { "type": "stop", "stopwords": "_english_" },
        "stem_en": { "type": "stemmer", "language": "english" }
      }
    }
  },
  "mappings": {
    "properties": {
      "headline": {
        "type": "text",
        "analyzer": "titre_en",
        "fields": {
          "raw":     { "type": "keyword", "ignore_above": 512 },
          "suggest": { "type": "completion", "max_input_length": 120 }
        }
      },
      "short_description": {
        "type": "text",
        "analyzer": "titre_en"
      },
      "category": { "type": "keyword" },
      "authors": {
        "type": "text",
        "fields": {
          "raw": { "type": "keyword", "ignore_above": 256 }
        }
      },
      "link": { "type": "keyword", "index": false },
      "date": { "type": "date", "format": "yyyy-MM-dd" }
    }
  }
}

Let us see what each of these choices says.

headline: text with the titre_en analyzer, plus two sub-fields

The headline is what Sami searches in full text. It is therefore text — analyzed, split into terms, searchable with match. The analyzer attached to it, titre_en, is defined just above in settings.analysis.analyzer. An analyzer is a three-stage chain: a character filter (none here), a tokenizer (standard, which splits on punctuation and whitespace) and a list of token filters:

• lowercase: everything to lowercase, so "Trump" and "trump" land on the same term.
• asciifolding: folds accents and non-ASCII characters (café → cafe, Beyoncé → beyonce, naïve → naive). This is what lets a reader typing "Beyonce" without an accent still find "Beyoncé".
• stop_en: removes English stop words (the, a, of…) defined by the _english_ list.
• stem_en: reduces words to their root. changing, changed, changes all become chang; climate becomes climat. That is the stemming, which brings 2,834 results for "climate change" instead of the strict fraction of occurrences.

headline has two sub-fields, declared with fields. This is the multi-field, the technique that stores several views of the same content:

• headline.raw of type keyword (ignore_above: 512): the raw string, unanalyzed, truncated past 512 characters. This is the view used to aggregate, sort or run an exact term on the headline.
• headline.suggest of type completion (max_input_length: 120): a special structure (FST, a finite-state transducer) that enables prefix autocomplete. The max_input_length: 120 parameter allows 120-character titles; the default value for completion is 50, which would have truncated half the HuffPost headlines and broken autocomplete on long articles (module 8).

short_description: text with the same analyzer

No sub-field: we do not need to sort or aggregate on the summaries. titre_en does the same job as on the headlines, with the same benefits (stemming, asciifolding).

category: keyword

The 41 categories (POLITICS, WELLNESS, ENTERTAINMENT…) are fixed labels. We want to aggregate, filter and sort on them, never search in full text. keyword is the only type that fulfills the three missions: it stores the raw value in an index optimized for exact matches and aggregations in doc_values.

GET news/_search
{
  "size": 0,
  "aggs": { "cats": { "terms": { "field": "category", "size": 5 } } }
}

Expected result:

{ "aggregations": { "cats": { "buckets": [
  { "key": "POLITICS",       "doc_count": 32739 },
  { "key": "WELLNESS",       "doc_count": 17827 },
  { "key": "ENTERTAINMENT",  "doc_count": 16058 },
  { "key": "TRAVEL",         "doc_count": 9887  },
  { "key": "STYLE & BEAUTY", "doc_count": 9649  }
] } } }

If category had been text, these values would have been analyzed: "STYLE & BEAUTY" would have become two terms style and beauty, the aggregation would have counted each one separately, and alphabetical sort would have been impossible. Remember the rule: keyword for what you classify, text for what you search.

authors: text and authors.raw as keyword

The authors field is two-headed, and it is no accident. A string like "Lee Moran and Ron Dicker" must serve two contradictory uses:

• Search for all articles by an author in full text: match authors: "Lee Moran". There we want the analyzer that splits on spaces and lowercases.
• Count how many articles each exact author signed: terms field: authors.raw. There we want the raw string, without analysis.

The top of the authors, with authors.raw:

GET news/_search
{
  "size": 0,
  "aggs": { "top": { "terms": { "field": "authors.raw", "size": 5 } } }
}

Expected result:

Reuters       4 954
Lee Moran     2 433
Ron Dicker    1 915
Ed Mazza      1 328
Cole Delbyck  1 145

The multi-field costs a bit of storage (each headline is indexed twice) but it is a profitable investment for just about any field where you hesitate between search and grouping.

link: keyword with index: false

An article has a link to the original HuffPost article. We display it to the user, we never search inside it. The right configuration is keyword + index: false: Elasticsearch stores the value in _source and returns it in hits, but maintains no inverted index on it. We save space and memory, at the price of a single constraint: no way to run a term, a match or an aggregation on link. Here, that is exactly the intent.

GET news/_search
{
  "query": { "term": { "link": "https://www.huffingtonpost.com/entry/..." } }
}

Returns a search_phase_execution_exception error: "Cannot search on field [link] since it is not indexed.". The message is explicit: we read the field, we do not query it.

date: date in yyyy-MM-dd format

date with format: yyyy-MM-dd accepts only strings like 2018-05-26. Elasticsearch stores the value in milliseconds since the Unix epoch, which makes range and date_histogram (module 6) very fast. The News corpus covers from 2012-01-28 to 2018-05-26; any date outside this format would be rejected at indexing with a mapper_parsing_exception.

The titre_en analyzer at work: the _analyze API

_analyze is the magnifier: it shows, word by word, how an analyzer transforms a text before it enters the inverted index. Sami can test even before indexing.

POST news/_analyze
{
  "field": "headline",
  "text": "Change Is Here. Climate Change."
}

Response (excerpt, keeping only the tokens):

chang  |  here  |  climat  |  chang

Only four terms. Is, . disappear (stop_en drops is, punctuation is eaten by the tokenizer). Change and changing would produce the same term chang thanks to stem_en. Here stays as is: it is not a stop word in English.

Let us compare with Elasticsearch's default standard analyzer:

POST news/_analyze
{
  "analyzer": "standard",
  "text": "Change Is Here. Climate Change."
}

change | is | here | climate | change

The standard keeps is, does not stem. A match on "climate changing" with standard would return zero results because no headline contains exactly changing. With titre_en, it returns more than 2,800: the magic of the stemmer.

asciifolding in action

POST news/_analyze
{
  "field": "headline",
  "text": "Beyoncé, Céline Dion et François"
}

beyonc  |  celin  |  dion  |  francoi

The accents disappear, then the stemmer operates (beyonce → beyonc, celine → celin). A reader who types "beyonce" without an accent types the same indexed key: they find the article. It is an indispensable gift for an engine that indexes English but is read by francophone users.

Always test with _analyze

Before creating a mapping in production, run five to ten _analyze calls on representative examples of your data. You will immediately see if your headlines are properly split and normalized. That is five minutes that avoid a mapping redesign three months later.

The field types to know

The news mapping uses five types. Elasticsearch offers about thirty; the essentials fit on a single page:

Type	Use	Example in news
text	Strings searchable in full text, analyzed	headline, short_description, authors
keyword	Exact strings: filter, sort, aggregate	category, headline.raw, link
date	Instants, flexible format	date (yyyy-MM-dd)
integer, long, short, byte	Integers on 32 / 64 / 16 / 8 bits	not used here
float, double, half_float, scaled_float	Decimals	not used
boolean	true/false	not used
completion	Prefix autocomplete (module 8)	headline.suggest
object	Nested JSON object, flat-indexed	default on objects
nested	Nested object to index independently	good to know as an overview
geo_point, geo_shape	Coordinates and polygons	not in this course

nested deserves a note: when you index an array of objects ([{ "auteur": "A", "role": "principal" }, { "auteur": "B", "role": "invité" }]), Elasticsearch flattens by default, which breaks the A/principal and B/invité associations (a query "author A and role invité" would wrongly return the document). nested preserves the structure at the cost of a slightly heavier query (nested query). On news we do not need it — the authors are a simple string.

Changing a mapping = _reindex

Here is the classic mistake. Sami accidentally creates an index where headline is declared keyword instead of text, realizes he can no longer run a clean match, and wants to "fix" the mapping:

PUT news_v0
{
  "mappings": {
    "properties": {
      "headline": { "type": "keyword" }
    }
  }
}

PUT news_v0/_mapping
{
  "properties": {
    "headline": { "type": "text" }
  }
}

The second call fails:

illegal_argument_exception : mapper [headline] cannot be changed from type [keyword] to [text]

You do not change the type of an existing field. You can add new properties, add multi-fields (fields) on an existing text, change some accessory parameters (ignore_above), but never change the base type. The right procedure fits in three steps:

1. Create a new index news_v1 with the correct mapping.
2. Reindex the data with the _reindex API.
3. Swap an alias from news_v0 to news_v1 (module 8) so client queries do not change.

POST _reindex
{
  "source": { "index": "news_v0" },
  "dest":   { "index": "news_v1" }
}

_reindex works internally like a search + _bulk. On 200,853 articles, expect a few dozen seconds with the kit. You can launch the command asynchronously with ?wait_for_completion=false and follow progress via GET _tasks.

Reindexing is copying, not transforming

_reindex does not modify documents in place. It re-reads them from the source, sends them to the dest. If the new mapping is stricter (for example a date with a precise format), a document that does not match the format will be rejected on write. _reindex returns a report with updated, created, failures: read it before believing everything went through.

Index templates: at a glance

An index template automatically applies a mapping and settings to new indices whose names match a pattern. If Veille indexes tomorrow news-2026-09-09, news-2026-09-10… a template news-* avoids repeating the mapping. You define them with PUT _index_template/veille_news; they are the backbone of rolling-index architectures. We will run into them again in module 8 when we talk about ILM and alias swaps.

Try it 1 — Prove the stemmer's effect

Compare the number of results returned by a match on "climate change" then on "climates changed". Without the stemmer, these two queries would give different counts. Explain why they give the same one.

Solution

GET news/_count
{ "query": { "match": { "headline": "climate change" } } }

GET news/_count
{ "query": { "match": { "headline": "climates changed" } } }

Both queries return exactly the same count, around 2,834. The titre_en analyzer applies stem_en: climate and climates both reduce to climat, change and changed to chang. The terms looked up in the inverted index are therefore identical in both cases. Check with POST news/_analyze { "field": "headline", "text": "climates changed" }: you will see the climat and chang tokens.

Try it 2 — Choose between text and keyword

For each of these fields in a future livres index, say whether you would choose text, keyword, both (multi-field) or keyword with index: false. Justify in a single sentence.

1. titre (free string, searched in full text, displayed)
2. isbn (13-digit identifier, never searched in full text)
3. couverture_url (image URL, displayed, never queried)
4. auteur (free string, both searched and aggregated)
5. genre (one value among 30, filtered and aggregated)

Solution

1. titre: text with the language-appropriate analyzer. Add titre.raw as keyword if you want to sort alphabetically or aggregate.
2. isbn: keyword. It is not a number to add up; it is an exact key to filter and join on.
3. couverture_url: keyword with index: false. We store it in _source, we never query it.
4. auteur: multi-field — text (search) + auteur.raw keyword (aggregation), as in news.
5. genre: keyword. Filter and aggregate, that is all. A text would split "Science-fiction" into two terms.

Try it 3 — Reproduce the "mapper cannot be changed from type [text] to [keyword]" error

Create an index piege, index a document with a titre field that the dynamic mapping classifies as text, then try to force titre to keyword. Read the exact error message.

Solution

POST piege/_doc
{ "titre": "Un premier document" }

PUT piege/_mapping
{ "properties": { "titre": { "type": "keyword" } } }

Response:

illegal_argument_exception : mapper [titre] cannot be changed from type [text] to [keyword]

The clean fix: create piege_v1 with the right mapping and POST _reindex { "source": {"index": "piege"}, "dest": {"index": "piege_v1"} }. That is exactly what you would do on news in production, with an alias on top so readers are not cut off (module 8).

Key takeaways

• The dynamic mapping turns every string into text + keyword sub-field of 256: convenient, but we do not let it decide in production.
• text for what you search in full text, keyword for what you filter, sort or aggregate; the multi-field field.raw marries the two.
• The kit's titre_en analyzer (standard + lowercase + asciifolding + stop_en + stem_en) is the reason a "climate change" search returns 2,834 results and not just a few dozen.
• _analyze shows, before you even index, how a text will be split — it is the reflex verification tool.
• index: false on a keyword saves storage and memory when you only display the value.
• headline.suggest is a completion with max_input_length: 120 (the default 50 would truncate half the HuffPost headlines).
• You do not change the type of a field: you create a new index with the right mapping and use _reindex (with an alias in production).

Troubleshooting

• GET news/_mapping shows a missing field → the importer ran into a document without that field. Check with GET news/_search { "query": { "exists": { "field": "authors" } } } how many documents carry it, then ./lab.sh import-news if the count is abnormal.
• illegal_argument_exception : mapper ... cannot be changed → an attempt to change an existing type. Create a new index and use POST _reindex.
• mapper_parsing_exception on a date → the value does not match the declared format (yyyy-MM-dd). Look at the offending document in the logs, then fix it at the source or broaden the format ("format": "yyyy-MM-dd||yyyy/MM/dd").
• A match returns zero results on a headline that is clearly present → the analyzer is not doing what you think. Run POST news/_analyze { "field": "headline", "text": "your text" } to see the actually-indexed tokens.

Further reading

• Elasticsearch 9 documentation — Mapping and field types
• Elasticsearch 9 documentation — Analyzers, tokenizers and token filters
• Elasticsearch 9 documentation — _reindex and data migration
• Elasticsearch 9 documentation — Multi-fields