# Production-Grade Log Aggregation: Architecture and Engineering of the ELK Stack

## Blog Details

- **Author**: Naveen R.
- **Date**: September 20, 2026
- **Tags**: elasticsearch, log aggregation, elk stack, distributed systems, observability
- **Read Time**: 20 mins

## Introduction

When your application fleet grows from dozens to thousands of instances, each emitting thousands of log lines per second, centralized logging stops being a convenience and becomes a technical necessity. A production-grade log aggregation and search system must ingest terabytes of data daily, index it for sub-second queries, manage storage costs through intelligent tiering, and maintain years of historical data without breaking the budget.

This post examines the architecture of systems like the ELK Stack (Elasticsearch, Logstash, Kibana), dissecting how they handle the full lifecycle of log data: from ingestion under backpressure, through indexing strategies that enable fast full-text search, to storage tiering that reduces costs by 60-80%, and finally to the query execution path that delivers results in milliseconds.

We'll explore the engineering trade-offs at each layer, backed by performance benchmarks and resource requirements drawn from production deployments processing billions of log events daily.

![High level architecture of a log aggregation system where application and host agents ship logs to an ingestion gateway that enqueues them into a Kafka buffer, an indexer writes to a log index, and a query service serves search from that index to the dashboard UI.](https://d5osvdbc8um23.cloudfront.net/static-asset/blog_images/system-design-log-aggregation/01-high-level-architecture.png)

![Scalable log architecture where fleet agents feed an ingest tier that partitions into Kafka by tenant, an indexer fleet writes hot shards that age out to warm shards and then a cold archive, and distributed queries scatter-gather across hot and warm shards.](https://d5osvdbc8um23.cloudfront.net/static-asset/blog_images/system-design-log-aggregation/02-scalable-architecture.png)

## Ingestion Pipeline and Backpressure

![Ingestion pipeline where a log shipper batches and compresses to a collector that enqueues into a durable buffer, and when buffer lag grows a backpressure policy throttles the shipper, while a parse and enrich worker feeds structured events to the indexer.](https://d5osvdbc8um23.cloudfront.net/static-asset/blog_images/system-design-log-aggregation/03-ingestion-pipeline.png)

### Collection Layer Architecture

The ingestion pipeline begins with lightweight agents deployed across your infrastructure. Filebeat, the most resource-efficient option, consumes just 10-50 MB of RAM and 5-15% CPU overhead per host. Logstash, while more feature-rich with its filter plugins, requires 1-4 GB of RAM per instance. This resource difference matters when you're deploying to thousands of hosts.

Modern collection agents can push 10,000-100,000 events per second per node, but raw throughput means nothing without backpressure handling. When downstream systems slow down or fail, you need buffering and flow control to prevent data loss.

### Buffering Strategies

An example configuration for Filebeat might look like this:

```yaml
queue.mem:
  events: 4096
  flush.min_events: 2048
  flush.timeout: 1s
```

This creates an in-memory buffer of 4,096 events, flushing when either 2,048 events accumulate or one second elapses. The buffer provides a cushion during brief downstream slowdowns, but it's bounded by available memory and doesn't survive process restarts.

For production systems requiring durability, a message queue like Kafka sits between collection and indexing. Kafka adds 50-200ms of latency but provides critical capabilities: persistent buffering that survives restarts, horizontal scaling through partitioning, and complete decoupling of ingestion from indexing rates.

### Backpressure Propagation

When Elasticsearch slows down (perhaps due to heavy query load or a garbage collection pause), the ingestion pipeline must react. Without proper backpressure handling, you face three bad outcomes: unbounded memory growth leading to out-of-memory crashes, dropped log events, or cascading failures as buffers fill throughout the system.

Kafka's consumer group protocol provides natural backpressure. When Logstash can't keep up with the ingestion rate, it simply stops pulling messages from Kafka. The messages remain durably stored in Kafka's log, waiting for processing capacity to become available. This pattern trades increased latency (messages sit in the queue longer) for guaranteed delivery and system stability.

The typical latency budget for log ingestion is under 1 second from source to searchable index under normal conditions. With a Kafka buffer in the path, you might see 2-5 seconds during peak load, but you maintain zero data loss.

### Batch Processing for Throughput

Elasticsearch's bulk API is critical for achieving high throughput. Individual document indexing might achieve 1,000-5,000 documents per second, while bulk indexing reaches 50,000-100,000 documents per second per node, a 5-10x improvement.

The batch size involves trade-offs. Larger batches improve throughput but increase latency and memory pressure. An example Logstash output configuration might look like this:

```ruby
output {
  elasticsearch {
    hosts => ["es-cluster:9200"]
    index => "logs-%{+YYYY.MM.dd}"
    pipeline => "log-enrichment"
  }
}
```

Logstash internally batches documents before sending them to Elasticsearch, balancing throughput with latency. The framework handles retries with exponential backoff when the cluster returns 429 (Too Many Requests) responses, implementing backpressure at the application level.

## Inverted Index Architecture

![Indexing strategy where parsed events pass through an index strategy router that sends free-text search to a full-text inverted index and cheap label filters to a label and stream index, both written into immutable segments that a background merger compacts.](https://d5osvdbc8um23.cloudfront.net/static-asset/blog_images/system-design-log-aggregation/04-indexing-strategy.png)

### How Inverted Indices Enable Fast Search

Elasticsearch's query performance, delivering results in 10-50ms for simple term queries across billions of documents, comes from its inverted index structure. Understanding this data structure is essential for designing efficient log schemas and queries.

An inverted index maps each unique term to the list of documents containing it. For the log message "Failed to connect to database db-prod-01", the inverted index creates entries:

```
"failed" -> [doc_123, doc_456, doc_789]
"connect" -> [doc_123, doc_234, doc_567]
"database" -> [doc_123, doc_345, doc_678]
"db-prod-01" -> [doc_123]
```

When you search for "database connection", Elasticsearch looks up both terms in the index, retrieves their document lists, and computes the intersection. This operation is fast because the posting lists (document IDs for each term) are sorted and compressed.

### Text Analysis and Tokenization

Before building the inverted index, Elasticsearch must analyze text fields: lowercase them, split on whitespace and punctuation, remove stop words, and potentially apply stemming. This analysis happens at both index time and query time, and they must match for searches to work correctly.

An example analyzer configuration might look like this:

```json
{
  "analysis": {
    "analyzer": {
      "log_analyzer": {
        "type": "custom",
        "tokenizer": "standard",
        "filter": ["lowercase", "stop"]
      }
    }
  }
}
```

The choice of analyzer significantly impacts both storage size and query behavior. The standard analyzer works well for human-readable logs, but structured data like UUIDs, IP addresses, or stack traces often need keyword fields (no analysis) or specialized tokenizers.

### Field Types and Index Strategies

Not every field needs full-text search capability. Elasticsearch supports multiple field types with different indexing strategies:

**Text fields** receive full analysis and inverted indexing, enabling phrase queries and relevance scoring. They're essential for log messages and error descriptions.

**Keyword fields** are indexed as single tokens without analysis. They're perfect for structured data: host names, service names, user IDs, or status codes. Keyword fields support exact matching and aggregations but not partial matching.

**Numeric and date fields** use specialized index structures (BKD trees) optimized for range queries. When you filter logs by timestamp or status code, these structures deliver results faster than inverted indices could.

An example mapping that combines these types might look like this:

```json
{
  "mappings": {
    "properties": {
      "timestamp": { "type": "date" },
      "level": { "type": "keyword" },
      "service": { "type": "keyword" },
      "message": { "type": "text", "analyzer": "log_analyzer" },
      "response_time": { "type": "integer" }
    }
  }
}
```

This schema enables fast filtering on timestamp, level, and service (using optimized structures), while supporting full-text search on the message field.

### Index Size and Shard Planning

The inverted index, along with document storage and metadata, determines your cluster's storage requirements. With standard compression algorithms, you can expect a 10:1 to 20:1 compression ratio compared to raw log data.

Elasticsearch divides each index into shards, which are independent Lucene indices. The recommended shard size is 30-50 GB for optimal performance. Proper shard sizing can improve performance by 30-40% compared to poorly sized shards.

More important than the size limit is the count limit. Performance degrades significantly when a cluster has more than 1000 shards per node. Each shard consumes memory for its segment metadata, and operations like cluster state updates and search coordination scale with total shard count.

For a system ingesting 1 TB of raw logs daily (compressing to 50-100 GB), a time-based indexing strategy makes sense. An example index template might look like this:

```json
{
  "index_patterns": ["logs-*"],
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 1,
    "refresh_interval": "30s"
  }
}
```

Daily indices of 50-100 GB with 3 shards yield 16-33 GB per shard, well within the optimal range. The 30-second refresh interval (instead of the default 1 second) can improve indexing throughput by 30-50%, trading slightly increased latency for better performance.

## Hot, Warm, and Cold Storage Tiering

![Storage tiering where the indexer writes recent data to a hot SSD tier, and a lifecycle manager rolls indices over and demotes them from hot to a warm HDD tier and then to a cold object store, deleting anything past its retention window.](https://d5osvdbc8um23.cloudfront.net/static-asset/blog_images/system-design-log-aggregation/05-storage-tiering.png)

### The Economics of Tiered Storage

Storage costs dominate the total cost of ownership for log aggregation systems. Hot SSD storage runs $0.10-0.20 per GB per month, while cold storage on HDD or object storage costs $0.01-0.03 per GB per month. For a system storing 50 TB of logs, moving 80% of that data to cold storage saves $4,000-8,500 monthly.

The hot-warm-cold architecture recognizes that log access patterns are heavily skewed toward recent data. Queries predominantly target logs from the last few days or weeks, while older logs are accessed infrequently for compliance, auditing, or investigating long-term trends.

### Hot Tier: Recent, Frequently Queried Data

Hot nodes run on high-performance hardware: SSDs with 10,000+ IOPS, 64+ GB of RAM (50% allocated to Elasticsearch's JVM heap, maximum 31 GB, with the remainder for filesystem cache), and 8-16 CPU cores. They hold indices from the last 7-30 days, depending on your query patterns and retention requirements.

The refresh interval on hot indices is typically 1-5 seconds, making newly ingested logs searchable almost immediately. This near-real-time visibility is critical for operational use cases: debugging production incidents, monitoring alert conditions, or tracking user activity.

### Warm Tier: Older, Less Frequently Accessed Data

Warm nodes use medium-performance hardware: SSDs for reasonable query performance but with less RAM (32 GB) and fewer CPU cores. They hold indices from 30-90 days old, data that's still regularly queried but with less urgency.

When an index ages out of the hot tier, Elasticsearch can shrink it, reducing the shard count to match the decreased query load. An example shrink operation might look like this:

```json
POST /logs-2024-01-15/_shrink/logs-2024-01-15-shrunk
{
  "settings": {
    "index.number_of_shards": 1,
    "index.number_of_replicas": 1
  }
}
```

This consolidation reduces cluster overhead (fewer shards to manage) and can improve query performance by reducing the number of shards each query must search.

### Cold Tier: Archive Storage

Cold nodes use low-cost storage: spinning disks or object storage like Amazon S3. They hold indices older than 90 days, accessed rarely for compliance queries or historical analysis. Query latency on cold storage is significantly higher (seconds instead of milliseconds), but the cost savings are substantial.

Elasticsearch's searchable snapshots feature allows indices to be stored entirely in object storage, with only a small cache on local disk. This reduces hot storage requirements dramatically while keeping the data searchable (albeit slowly).

### Index Lifecycle Management

Managing the transition between tiers manually is error-prone and tedious. Index Lifecycle Management (ILM) automates the process based on policies you define. An example ILM policy might look like this:

```json
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "50GB",
            "max_age": "1d"
          }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "shrink": { "number_of_shards": 1 },
          "forcemerge": { "max_num_segments": 1 }
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": {
          "searchable_snapshot": {
            "snapshot_repository": "cold_repository"
          }
        }
      },
      "delete": {
        "min_age": "365d",
        "actions": { "delete": {} }
      }
    }
  }
}
```

This policy rolls over indices daily or when they reach 50 GB, moves them to warm storage after 7 days (shrinking and force-merging to optimize storage), transitions to cold storage after 30 days, and deletes them after one year.

The forcemerge operation consolidates Lucene segments, reducing storage overhead and improving query performance. It's CPU-intensive, so it's performed during the transition to warm storage when the index is no longer being written to.

### Cost Analysis Example

Consider a system ingesting 1 TB of raw logs daily, compressing to 100 GB on disk. Over 365 days, that's 36.5 TB of storage. With an all-hot architecture at $0.15/GB/month, monthly storage costs would be $5,475.

With tiered storage (7 days hot, 23 days warm, 335 days cold):

- Hot: 700 GB at $0.15/GB = $105
- Warm: 2,300 GB at $0.08/GB = $184  
- Cold: 33,500 GB at $0.02/GB = $670
- Total: $959/month

The tiered approach saves $4,516 monthly (82% reduction), demonstrating why this architecture is standard for production log systems.

## Retention Policies and Rollups

### Time-Based Retention

The simplest retention strategy is time-based deletion: keep logs for N days, then delete them. Time-based indexing (daily or weekly indices) makes this efficient. Instead of running expensive delete-by-query operations, you simply drop entire indices.

An example deletion policy might look like this:

```json
{
  "delete": {
    "min_age": "90d",
    "actions": {
      "delete": {}
    }
  }
}
```

This is part of the ILM policy discussed earlier. When an index reaches 90 days old, Elasticsearch deletes it entirely. The operation is fast (just metadata updates) and frees storage immediately.

### Rollups for Long-Term Trend Analysis

Sometimes you need to retain log data for years, but storing every individual log event is prohibitively expensive. Rollups solve this by aggregating older data into summary statistics, dramatically reducing storage requirements while preserving the ability to analyze trends.

For example, instead of storing individual HTTP request logs from a year ago, you might store hourly aggregations: request count, average response time, error rate, and top endpoints. This loses the ability to search individual requests but enables trend analysis at a fraction of the storage cost.

An example rollup configuration might look like this:

```json
{
  "groups": {
    "date_histogram": {
      "field": "timestamp",
      "interval": "1h"
    },
    "terms": {
      "fields": ["service", "endpoint", "status_code"]
    }
  },
  "metrics": [
    {
      "field": "response_time",
      "metrics": ["avg", "max", "min"]
    },
    {
      "field": "request_count",
      "metrics": ["sum"]
    }
  ]
}
```

This configuration creates hourly summaries grouped by service, endpoint, and status code, calculating average, max, and min response times along with total request counts.

The storage savings from rollups depend on your data's cardinality and the aggregation interval. Higher cardinality (more unique combinations of service, endpoint, and status code) yields less compression. Longer intervals (daily instead of hourly) compress more but lose granularity.

### Selective Retention by Log Level

Not all logs have equal value. DEBUG logs are useful during development but rarely needed in production after a few days. ERROR logs might need retention for months or years for compliance and trend analysis.

You can implement selective retention by routing different log levels to separate indices with different ILM policies. An example Logstash configuration might look like this:

```ruby
output {
  if [level] == "ERROR" or [level] == "FATAL" {
    elasticsearch {
      index => "logs-error-%{+YYYY.MM.dd}"
    }
  } else {
    elasticsearch {
      index => "logs-debug-%{+YYYY.MM.dd}"
    }
  }
}
```

Then apply different retention policies: 365 days for error logs, 30 days for debug logs. This keeps critical data while reducing storage costs for high-volume, low-value logs.

## The Full-Text Query Path

![Full-text query path where the query UI sends a time-bounded query to a planner that checks a result cache and fans out to hot and warm shards, a scatter-gather merger combines the shard results, and ranked results return to the user.](https://d5osvdbc8um23.cloudfront.net/static-asset/blog_images/system-design-log-aggregation/06-query-path.png)

### Query Coordination and Execution

When you submit a search query to Elasticsearch, it enters a two-phase distributed execution process. Understanding this flow is essential for optimizing query performance and troubleshooting slow queries.

The coordinating node (whichever node receives the request) determines which shards contain relevant data. For a query with a timestamp filter, only shards covering that time range participate. This is why time-based indexing improves query performance: it enables shard pruning, reducing the amount of data to search.

### Query Phase

In the query phase, the coordinating node broadcasts the query to one copy of each relevant shard (primary or replica). Each shard executes the query against its local inverted index, scoring and ranking documents.

For a simple term query like `level:ERROR AND service:auth`, each shard:

1. Looks up "ERROR" in the inverted index for the level field
2. Looks up "auth" in the inverted index for the service field  
3. Computes the intersection of the two posting lists
4. Scores each matching document (for term queries, often just a constant score)
5. Returns the top N document IDs and scores to the coordinator

This phase is fast (10-50ms for simple queries) because inverted indices are optimized for these operations.

### Fetch Phase

The query phase returns only document IDs and scores, not the full document content. In the fetch phase, the coordinating node identifies which shards hold the actual documents for the top results and requests them.

For a query requesting the top 100 results across 10 shards, each shard might return 100 candidate document IDs (1,000 total). The coordinator merges and sorts these by score, then fetches the actual top 100 documents from the appropriate shards.

This two-phase approach minimizes network traffic. Transferring 1,000 document IDs and scores is cheap; transferring 1,000 full log documents would be expensive.

### Query Types and Performance Characteristics

Different query types have vastly different performance profiles:

**Term queries** (exact matches on keyword fields) are fastest, typically 10-50ms. They use the inverted index directly without complex scoring.

**Phrase queries** (exact phrase matches) are slower because they must verify that terms appear in the correct order with no intervening terms. Expect 50-200ms.

**Wildcard and regex queries** can be 10-100x slower than term queries because they can't use the inverted index efficiently. A query like `message:*exception*` must scan many terms in the index.

**Aggregation queries** compute statistics across matching documents: counts, averages, percentiles, or top values. Performance depends on cardinality and data volume. Expect 100-500ms for aggregations over millions of documents.

### Query Optimization Strategies

Several techniques can dramatically improve query performance:

**Filter context instead of query context**: Filters don't compute relevance scores and are cached. An example query might look like this:

```json
{
  "query": {
    "bool": {
      "filter": [
        { "term": { "level": "ERROR" } },
        { "range": { "timestamp": { "gte": "now-1h" } } }
      ]
    }
  }
}
```

This uses filter context for both conditions. Elasticsearch caches the filter results, so repeated queries with the same filters are much faster (cache hit ratio target: >80%).

**Field data types matter**: Searching on keyword fields is faster than text fields. If you're doing exact matching (service names, user IDs), use keyword type.

**Limit result size**: Fetching 1,000 documents is 10x slower than fetching 100. Use pagination with search_after for deep pagination instead of from/size, which becomes expensive for large offsets.

**Use index patterns**: Instead of searching all indices, target specific date ranges: `logs-2024-01-*` instead of `logs-*`. This enables shard pruning.

### Caching Layers

Elasticsearch employs multiple caches to accelerate repeated queries:

**Query cache** stores the results of filter clauses. When you repeatedly filter by `level:ERROR`, the second query retrieves results from cache (sub-millisecond response time).

**Field data cache** stores field values in memory for aggregations and sorting. This cache is heap-resident and must be managed carefully to avoid out-of-memory errors.

**Filesystem cache** (outside the JVM heap) caches recently accessed index segments. This is why Elasticsearch recommends allocating only 50% of RAM to the JVM heap: the other 50% serves as filesystem cache, dramatically improving performance for recently accessed data.

Target cache hit ratios above 80%. Lower hit ratios indicate your working set doesn't fit in cache, suggesting you need more memory or should adjust your query patterns.

### Monitoring Query Performance

Elasticsearch exposes detailed query performance metrics through the _profile API. An example profiled query might look like this:

```json
{
  "profile": true,
  "query": {
    "bool": {
      "filter": [
        { "term": { "level": "ERROR" } },
        { "range": { "timestamp": { "gte": "now-1h" } } }
      ]
    }
  }
}
```

The response breaks down time spent in each query phase, showing which clauses are expensive. This is invaluable for optimizing slow queries.

Key metrics to monitor cluster-wide:

- **Search latency**: P50, P95, P99 percentiles. Alert if P95 exceeds 1 second.
- **Query throughput**: Queries per second. Decreasing throughput with constant load indicates resource exhaustion.
- **Query rejection rate**: When thread pools fill up, Elasticsearch rejects queries. Alert if rejection rate exceeds 1%.
- **Cache hit rates**: For query cache and field data cache. Target >80%.

## Resource Requirements and Cluster Sizing

### Hardware Specifications

Production Elasticsearch nodes require substantial resources. The recommended minimum per node:

- **CPU**: 8-16 cores
- **RAM**: 32-64 GB (50% for JVM heap, maximum 31 GB)
- **Storage**: SSD required, 10,000+ IOPS recommended  
- **Network**: 10 Gbps for high-throughput clusters

These aren't arbitrary numbers. The CPU count determines how many concurrent queries and indexing operations the node can handle. RAM determines both JVM heap size (for in-memory data structures) and filesystem cache size (for recently accessed index segments). SSD IOPS prevent disk I/O from becoming a bottleneck (which accounts for 60-70% of performance issues in under-provisioned clusters). Network bandwidth matters when nodes are transferring large volumes of data during indexing or query coordination.

### JVM Heap Sizing

The JVM heap size is the most critical tuning parameter. Too small, and you'll see constant garbage collection pauses. Too large, and GC pauses can last 10+ seconds, making the cluster unresponsive.

The rule: allocate 50% of available RAM to the heap, with a maximum of 31 GB. The 31 GB limit exists because the JVM uses compressed object pointers (compressed oops) below this threshold, significantly reducing memory overhead. A 31 GB heap provides more usable memory than a 32 GB heap due to this compression.

The remaining 50% of RAM serves as filesystem cache, caching recently accessed index segments. This cache is critical for query performance, especially for hot data.

### Cluster Topology

Small deployments (100 GB/day) can run on a 3-node cluster, with each node serving all roles: master-eligible, data, and ingest. Estimated cost: $700-1,400/month including storage.

Medium deployments (1 TB/day) need 10+ nodes with role separation. An example topology might include:

- 3 dedicated master nodes (lightweight, 4 CPU, 8 GB RAM)
- 6 hot data nodes (high-performance, 16 CPU, 64 GB RAM, SSD)
- 3 warm data nodes (medium-performance, 8 CPU, 32 GB RAM, SSD)

Estimated cost: $4,500-9,000/month.

Large deployments (10 TB/day) require 50+ nodes with multiple tiers. Estimated cost: $30,000-75,000/month.

### Scaling Patterns

Elasticsearch scales near