Aggregations: count, group, follow across time

Module 5 knew how to retrieve specific articles; here we ask the corpus for numbers and trends. Léa is preparing the Veille quarterly review: how many articles per category, how does the climate topic climb year over year, who writes the most? Elasticsearch aggregations answer these questions in a single request, without ever returning documents.

Two reflexes that change everything

size: 0 tells Elasticsearch to bring back no document: we only want the aggregations block. Forgetting size triggers 10 useless hits on top of the computation.

Aggregations live under the aggs key (alias aggregations). Every aggregation carries a name you choose, a type (terms, date_histogram…), parameters, and optionally sub-aggregations in its own aggs.

terms: count by value

terms groups documents by the exact value of a keyword field. It is the most-used aggregation.

GET news/_search
{
  "size": 0,
  "aggs": {
    "by_category": {
      "terms": { "field": "category", "size": 5 }
    }
  }
}

Expected result, buckets at the top:

• POLITICS — 32,739
• WELLNESS — 17,827
• ENTERTAINMENT — 16,058
• TRAVEL — 9,887
• STYLE & BEAUTY — 9,649

The corpus total is 200,853 documents; the sum of the first five buckets therefore equals 86,160, or 43% of the corpus. Elasticsearch also returns doc_count_error_upper_bound (a potential error margin tied to shard slicing; zero here since the index has a single shard) and sum_other_doc_count (the rest).

For authors, we target the keyword sub-field authors.raw:

GET news/_search
{
  "size": 0,
  "aggs": {
    "top_authors": {
      "terms": { "field": "authors.raw", "size": 5 }
    }
  }
}

The first five buckets:

• Reuters — 4,954
• Lee Moran — 2,433
• Ron Dicker — 1,915
• Ed Mazza — 1,328
• Cole Delbyck — 1,145

terms on a text field

{"terms": {"field": "headline"}} fails or is expensive: headline is analyzed (titre_en) and has no fielddata by default. Always aggregate on a keyword (here headline.raw, category, authors.raw).

date_histogram: follow time

date_histogram slices the time axis into regular intervals. Since Elasticsearch 8, use calendar_interval (calendar-aligned: year, month, week, day) or fixed_interval (a fixed duration in hours or minutes). The old interval is removed.

GET news/_search
{
  "size": 0,
  "aggs": {
    "by_year": {
      "date_histogram": {
        "field": "date",
        "calendar_interval": "year",
        "format": "yyyy"
      }
    }
  }
}

The corpus runs from 2012-01-28 to 2018-05-26, so we see seven buckets (2012 to 2018). Empty buckets are omitted by default; add "min_doc_count": 0 with extended_bounds for a complete axis.

For a monthly step:

GET news/_search
{
  "size": 0,
  "aggs": {
    "by_month": {
      "date_histogram": {
        "field": "date",
        "calendar_interval": "month",
        "format": "yyyy-MM"
      }
    }
  }
}

(your number may vary slightly from one bucket to another depending on the dates).

range: intervals à la carte

Where date_histogram slices regularly, range defines named brackets.

GET news/_search
{
  "size": 0,
  "aggs": {
    "before_after_2016": {
      "range": {
        "field": "date",
        "ranges": [
          { "to":   "2016-01-01", "key": "2012-2015" },
          { "from": "2016-01-01", "key": "2016-2018" }
        ]
      }
    }
  }
}

cardinality: count distinct values

How many different authors in the corpus?

GET news/_search
{
  "size": 0,
  "aggs": {
    "n_authors": {
      "cardinality": { "field": "authors.raw" }
    }
  }
}

cardinality uses the HyperLogLog++ algorithm: fast and memory-bounded, but approximate. The precision_threshold parameter (default 3,000, maximum 40,000) tunes the precision/memory trade-off — below the threshold, the count is almost always exact; above it, the average relative error stays under 1 to 2%.

Exact numbers on a small set

On news, 41 exact categories fit easily within the default precision. For a perfectly exact count on a very large set, go through composite (see below) and count buckets client-side.

avg, stats, min, max

Classic metrics apply to numeric and date fields. news has no business numeric field, but we can look at the oldest and newest dates:

GET news/_search
{
  "size": 0,
  "aggs": {
    "date_bounds": { "stats": { "field": "date" } }
  }
}

The response contains min = 2012-01-28, max = 2018-05-26 (with readable min_as_string / max_as_string variants).

top_hits: a sample inside each bucket

top_hits is a sub-aggregation that returns N representative documents per parent bucket. Ideal for "the most recent headline of each category".

GET news/_search
{
  "size": 0,
  "aggs": {
    "by_category": {
      "terms": { "field": "category", "size": 5 },
      "aggs": {
        "latest_headline": {
          "top_hits": {
            "size": 1,
            "sort": [ { "date": "desc" } ],
            "_source": ["headline", "date"]
          }
        }
      }
    }
  }
}

Sub-aggregations: the power that counts

Nesting is what sets aggregations apart from a simple GROUP BY. Léa's question: how many articles per category, year by year, on POLITICS and WELLNESS?

GET news/_search
{
  "size": 0,
  "query": {
    "terms": { "category": ["POLITICS", "WELLNESS"] }
  },
  "aggs": {
    "by_category": {
      "terms": { "field": "category", "size": 2 },
      "aggs": {
        "by_year": {
          "date_histogram": {
            "field": "date",
            "calendar_interval": "year",
            "format": "yyyy"
          }
        }
      }
    }
  }
}

Every parent bucket (POLITICS, WELLNESS) carries its own by_year. The total for by_category.POLITICS is 32,739; that of by_category.WELLNESS is 17,827; everything fits into a single request.

filter and filters in aggregation

filter (singular) restricts the calculation to a sub-set without changing the global query.

GET news/_search
{
  "size": 0,
  "aggs": {
    "recent": {
      "filter": { "range": { "date": { "gte": "2017-01-01" } } },
      "aggs": {
        "by_category": { "terms": { "field": "category", "size": 5 } }
      }
    }
  }
}

filters (plural) produces N named buckets in a single pass, a bit like range but for free-form criteria.

GET news/_search
{
  "size": 0,
  "aggs": {
    "topics": {
      "filters": {
        "filters": {
          "climate":  { "match": { "headline": "climate change" } },
          "election": { "match": { "headline": "election" } },
          "health":   { "match": { "headline": "health" } }
        }
      }
    }
  }
}

The climate bucket contains 2,834 documents (see module 5).

composite: paginate an aggregation

terms with size: 10000 is an anti-pattern: memory climbs and nothing guarantees you get every key back. composite paginates cleanly, with an after_key equivalent to search_after.

GET news/_search
{
  "size": 0,
  "aggs": {
    "all_categories": {
      "composite": {
        "size": 100,
        "sources": [
          { "cat": { "terms": { "field": "category" } } }
        ]
      }
    }
  }
}

The response contains an after_key; you re-run passing it under "after": { "cat": "..." } until exhaustion. On news, two calls are enough to cover the 41 exact categories.

Order and size: three pitfalls

• terms returns buckets sorted by doc_count descending by default. To sort by key, "order": {"_key": "asc"}; by a sub-metric, "order": {"metric_name": "desc"}.
• The default size is 10. A top 20 requires "size": 20.
• Across shards, terms asks each shard for shard_size keys (default size * 1.5 + 10) then merges. doc_count_error_upper_bound measures the uncertainty. On news (a single shard), the margin is zero.

Reading an aggregation response

The structure is regular:

• hits.total.value: number of documents matched by the query.
• aggregations.<name>.buckets: list of buckets.
• Each bucket: key (the value), doc_count (the count), and sub-aggregations under their name.
• For a date_histogram: key (epoch ms timestamp), key_as_string (formatted string).

Writing a small paragraph that walks each field on day one lets you effortlessly read any response later on.

Try it 1 — Top 3 categories in 2017

Write a request that returns the three most frequent categories, restricted to articles published in 2017. No document data is needed.

Solution

GET news/_search
{
  "size": 0,
  "query": {
    "bool": {
      "filter": [ { "range": { "date": { "gte": "2017-01-01", "lt": "2018-01-01" } } } ]
    }
  },
  "aggs": {
    "top3_2017": {
      "terms": { "field": "category", "size": 3 }
    }
  }
}

The query narrows the analysis field to 2017 (as a filter, without score). The terms aggregation then works on this sub-corpus. Compare with the global ranking to see whether POLITICS still leads WELLNESS that year (your number may differ slightly).

Try it 2 — Number of Lee Moran articles per category

The brief states that Lee Moran wrote 2,433 articles in total. How many in COMEDY, WEIRD NEWS, ENTERTAINMENT, POLITICS?

Solution

GET news/_search
{
  "size": 0,
  "query": {
    "term": { "authors.raw": "Lee Moran" }
  },
  "aggs": {
    "by_category": {
      "terms": { "field": "category", "size": 5 }
    }
  }
}

Expected buckets: COMEDY 779, WEIRD NEWS 391, ENTERTAINMENT 387, POLITICS 264, SPORTS 101. The sum (1,922) is lower than 2,433: the remainder spreads across the 36 other categories. Pass "size": 41 to see it all.

Try it 3 — Distinct authors per year

Is the corpus getting more contributive over time? Return a year → distinct author count (approx.).

Solution

GET news/_search
{
  "size": 0,
  "aggs": {
    "by_year": {
      "date_histogram": {
        "field": "date",
        "calendar_interval": "year",
        "format": "yyyy"
      },
      "aggs": {
        "distinct_authors": {
          "cardinality": { "field": "authors.raw", "precision_threshold": 5000 }
        }
      }
    }
  }
}

Every by_year bucket carries a cardinality sub-aggregation. precision_threshold is raised to keep good accuracy on the 23,082 distinct authors listed in the corpus.

Key takeaways

• size: 0 isolates aggregations; useless hits go away.
• terms groups by keyword, date_histogram groups by calendar_interval or fixed_interval.
• cardinality is an approximate HyperLogLog++: tune precision_threshold for the accuracy you want.
• Sub-aggregations reproduce the computation per parent bucket: categories by year, authors by category.
• filter / filters restrict an aggregation without touching the query; composite paginates with after_key.
• Read the response in order: buckets[].key, buckets[].doc_count, buckets[].<sub_agg>.

Troubleshooting

• "Fielddata is disabled on text fields by default" on a terms over headline → aggregate on the keyword sub-field (headline.raw, authors.raw); do not re-enable fielddata.
• doc_count_error_upper_bound is non-zero and the top moves → raise shard_size or size; on news (a single shard) the error is zero.
• date_histogram returns 0 bucket → check that the field is truly date with ./lab.sh es news/_mapping?filter_path=**.date; otherwise fix the mapping (module 4).
• "Trying to create too many buckets" → cap size or switch to composite with size: 100 and pagination.

Further reading

• Aggregations — overview
• terms aggregation
• date_histogram aggregation
• composite aggregation