# Designing Twitter/X: A Complete System Design Walkthrough ## Blog Details - **Author**: Naveen R. - **Date**: September 2, 2026 - **Tags**: system design, distributed systems, social media architecture, scalability, twitter - **Read Time**: 20 mins When I first approached the problem of designing a social media platform like Twitter, I quickly realized that the most critical architectural decisions wouldn't be about which database to use or how to write efficient queries. Instead, they'd be about *when* to do the work, a timing problem that fundamentally shapes everything from user experience to infrastructure costs. In this post, I'll walk through how I would design Twitter/X from the ground up, starting with scoping and requirements, then diving deep into three critical subsystems: timeline generation, data storage and the social graph, and search with trending topics. This isn't interview prep. It's how I would actually architect this system if the problem landed on my desk tomorrow. ## Scoping the Problem and Clarifying Assumptions Before writing a single line of code or drawing any architecture diagrams, I need to understand what I'm actually building. For a Twitter-like platform, I'd start by clarifying the core features and constraints. **Core Features I'm Targeting:** - **Tweet Creation**: Users can post 280-character messages with optional media (images, videos) - **Home Timeline**: Users see a feed of tweets from accounts they follow, ordered by time - **Social Graph**: Users can follow other users (unidirectional relationship, unlike Facebook's bidirectional friendship) - **Engagement**: Users can like, retweet (share), and reply to tweets - **Search**: Full-text search across all tweets - **Trending Topics**: Real-time detection of popular hashtags and topics **What I'm Explicitly NOT Building (Initially):** - Direct messaging - Notifications (though the infrastructure would support it) - Advanced recommendation algorithms beyond chronological ordering - Stories/Spaces or other ephemeral content - Algorithmic timeline ranking (starting with pure chronological) **Key Assumptions:** Based on Twitter's actual scale, I'm assuming: - 150 million daily active users worldwide - 400 million tweets per day (roughly 4,600 tweets/second average, with peaks much higher) - Average user follows 200-500 accounts - Read-heavy workload (users browse far more than they post) - Global distribution with users across all continents - Mobile-first usage pattern (80%+ of traffic from mobile apps) The most important insight here is that this is fundamentally a **consumption platform**, not a production platform. Users spend far more time reading than writing. This single fact will drive nearly every architectural decision I make. ## Functional and Non-Functional Requirements ### Functional Requirements Here's what the system must do: 1. **Post tweets**: Users create tweets with text (up to 280 chars) and optional media 2. **View home timeline**: Users see tweets from accounts they follow, in reverse chronological order 3. **Follow/unfollow**: Users can follow any public account 4. **Engage with tweets**: Like, retweet, and reply functionality 5. **Search tweets**: Full-text search across all public tweets 6. **View trending topics**: Real-time list of popular hashtags and topics ### Non-Functional Requirements These are the constraints that will actually make or break the system: **1. Availability**: Target 99.9% uptime (about 8.7 hours downtime per year) - Twitter is a real-time platform; downtime during major events is unacceptable - I'd prioritize availability over consistency (AP in CAP theorem) **2. Latency**: - Timeline load: < 2 seconds for p99 - Tweet posting: < 500ms acknowledgment to user - Timeline delivery: < 5 seconds from tweet creation to appearing in follower feeds - Search: < 3 seconds for p99 **3. Scale**: - 150M daily active users - 300,000 timeline queries per second (peak) - 6,000 write requests per second - 400M tweets per day **4. Consistency**: - Eventual consistency is acceptable for timelines (seeing a tweet a few seconds late is fine) - Strong consistency needed for tweet content itself (can't lose data) - Follower counts and engagement metrics can be eventually consistent ### Back-of-the-Envelope Capacity Estimation Let me work through the numbers to understand what infrastructure I'll need: **Storage Estimates:** ``` Tweets: - 400M tweets/day - Average tweet size: 280 chars (280 bytes) + metadata (200 bytes) + media URLs (100 bytes) = ~600 bytes - Daily storage: 400M × 600 bytes = 240 GB/day - Annual storage: 240 GB × 365 = ~87.6 TB/year - 5-year storage: ~438 TB Media Storage (images/videos): - Assume 20% of tweets have media - Average media size: 2 MB (mix of images and short videos) - Daily media: 400M × 0.2 × 2 MB = 160 TB/day - Annual media: ~58 PB/year - With CDN caching, active set much smaller (~10% = 5.8 PB) ``` **Bandwidth Estimates:** ``` Read Operations (Timeline Fetches): - 300K requests/second (peak) - Average timeline: 50 tweets × 600 bytes = 30 KB - Bandwidth: 300K × 30 KB = 9 GB/second (peak) Write Operations: - 6,000 tweets/second - Average tweet: 600 bytes - Bandwidth: 6K × 600 bytes = 3.6 MB/second ``` **Cache Requirements:** ``` Timeline Cache (Redis): - 150M users - Cache 800 recent tweets per user (Twitter's actual limit) - Per user: 800 × 600 bytes = 480 KB - Total: 150M × 480 KB = 72 TB - With 20% active users cached: ~14.4 TB ``` **Server Estimates:** ``` Application Servers (Timeline Service): - 300K QPS peak - Assume 1,000 QPS per server (with caching) - Need: ~300 servers (with redundancy: ~450 servers) Database Servers: - For 438 TB over 5 years - Using sharding with 2 TB per shard - Need: ~220 shards (with replication: ~660 database instances) ``` These numbers tell me I'm building a system that requires: - Aggressive caching (14+ TB of Redis) - Heavy sharding and partitioning - CDN for media delivery (can't serve 58 PB/year from origin) - Horizontal scaling across hundreds of servers The **50:1 read-to-write ratio** (300K reads vs 6K writes) is the most important number here. It means I should optimize heavily for reads, even if it makes writes more expensive. ## High-Level Architecture and Core Components Before diving into the deep technical details, let me sketch out the overall system architecture. Here's how I would structure the major components: ![High-level architecture: clients hit an API gateway that routes to the Tweet, Timeline, and Search services; the Tweet service persists to a sharded Tweet DB and emits events to Kafka; fan-out workers read the social graph and push tweet ids into the Redis home-timeline cache the Timeline service reads from; the Search service queries Elasticsearch, which Kafka also feeds](https://d5osvdbc8um23.cloudfront.net/static-asset/blog_images/system-design-twitter/01-high-level-architecture.png) **Component Breakdown:** **1. API Gateway / Load Balancer** - Entry point for all client requests - Handles authentication, rate limiting, and routing - Distributes load across service instances - I'd use something like NGINX or AWS ALB **2. Timeline Service** - Generates and serves user home timelines - Reads from Redis cache (timeline cache) - Falls back to database on cache miss - This is the most critical service for user experience **3. Tweet Service** - Handles tweet creation, updates, and retrieval - Stores tweets in sharded database - Publishes events to message queue on new tweets - Handles media upload coordination with CDN **4. User Service** - Manages user profiles and authentication - Stores user metadata - Handles profile updates **5. Social Graph Service** - Manages follow/follower relationships - Highly optimized for graph queries ("who follows whom?") - Critical for fanout operations - Uses specialized graph database or sharded relational DB **6. Search Service** - Full-text search across tweets - Interfaces with Elasticsearch cluster - Handles complex query parsing **7. Message Queue (Kafka)** - Decouples services for asynchronous processing - Enables reliable event streaming - Handles 6,000+ writes/second with ease - Provides replay capability for failures **8. Fanout Workers** - Consume tweet events from queue - Push tweets to follower timelines (fan-out on write) - Update Redis timeline caches - Can scale independently based on queue depth **9. Storage Layer** - **Redis**: Timeline cache (14+ TB for 20% of users) - **Tweet Database**: Sharded storage for all tweets (438 TB over 5 years) - **Social Graph Database**: Optimized for graph queries - **Search Index**: Elasticsearch for full-text search - **CDN**: Media delivery (images, videos) This architecture gives me: - **Separation of concerns**: Each service has a single responsibility - **Independent scaling**: Can scale timeline service separately from tweet service - **Fault isolation**: Failure in search doesn't affect timeline - **Asynchronous processing**: Tweet posting returns immediately, fanout happens in background ### Scaling It Out The clean component view hides what it takes to hold 150M users and 300K timeline reads/second. Scaled out, I'd put a load balancer and rate-limiting gateway in front of pools of stateless service replicas; keep the write path fully async through Kafka into an autoscaled fan-out fleet; separate the search indexer and trending workers as their own consumers of the same event stream; and split storage by job: Redis for home timelines and trend counters, a sharded Tweet DB, the social-graph store, Elasticsearch for search, and a CDN in front of media. ![Scalable architecture: client traffic through a load balancer and API gateway to replicated Tweet, Timeline, and Search services; the Tweet service writes to a sharded Tweet DB and media blob store and emits to Kafka, which fans out to autoscaled fan-out, search-indexer, and trending workers; fan-out reads the social graph and pushes to the Redis home-timeline cache; the Timeline service reads that cache and pulls celebrity tweets on read; search hits Elasticsearch, trending writes Redis counters, media serves from a CDN, and monitoring observes the system](https://d5osvdbc8um23.cloudfront.net/static-asset/blog_images/system-design-twitter/02-scalable-architecture.png) Now let's dive deep into the three most critical and complex parts of this system. ## Deep-Dive #1: Timeline and Feed Generation The timeline is the heart of Twitter. When a user opens the app, they expect to see recent tweets from everyone they follow, sorted by time. Simple concept, but at 300,000 queries per second, this becomes the most challenging part of the system. The fundamental question I need to answer is: **When should I generate the timeline?** I have two basic approaches, each with radically different trade-offs. ### Approach 1: Fan-Out on Read (Pull Model) In this approach, I generate the timeline when the user requests it. **How it works:** ``` 1. User A opens Twitter 2. Query: "Who does User A follow?" → Returns [User X, User Y, User Z, ...] 3. For each followed user, query their recent tweets 4. Merge all tweets together 5. Sort by timestamp 6. Return top N tweets ``` **Implementation pseudocode:** ```python def get_timeline_fan_out_read(user_id): following = social_graph.get_following(user_id) # e.g., 500 users all_tweets = [] for followed_user in following: tweets = tweet_db.get_recent_tweets(followed_user, limit=100) all_tweets.extend(tweets) # Merge and sort all_tweets.sort(key=lambda t: t.timestamp, reverse=True) return all_tweets[:50] # Return top 50 ``` **Performance Analysis:** For a user following 500 accounts: - 1 query to get following list - 500 queries to get recent tweets (can be parallelized) - Merge and sort ~50,000 tweets - Time: **200-500ms in best case, 1-2 seconds for 2,000+ follows** **Advantages:** - Simple to implement - Low write cost: just 1 write per tweet - No storage overhead (no timeline duplication) - Works well for users who rarely check their feed **Disadvantages:** - High read latency (especially for users following many accounts) - Expensive computation on every timeline request - Database load scales with number of timeline views - Poor user experience for active users (the majority) For Twitter's 300,000 QPS, this approach would require querying millions of tweet records per second. Even with caching and indexes, this doesn't scale well. ### Approach 2: Fan-Out on Write (Push Model) In this approach, I pre-compute timelines when tweets are created. **How it works:** ``` 1. User X publishes a tweet 2. Query: "Who follows User X?" → Returns [User A, User B, User C, ...] 3. For each follower, insert tweet into their pre-computed timeline 4. Store in Redis timeline cache When User A opens Twitter: 1. Read from User A's timeline cache 2. Return instantly ``` **Implementation pseudocode:** ```python def publish_tweet_fan_out_write(user_id, tweet): # Store the tweet tweet_id = tweet_db.save(tweet) # Get all followers followers = social_graph.get_followers(user_id) # Push to each follower's timeline cache for follower_id in followers: timeline_cache.prepend(follower_id, tweet_id) timeline_cache.trim(follower_id, max_size=800) # Keep only 800 tweets return tweet_id def get_timeline_fan_out_write(user_id): # Simply read from cache tweet_ids = timeline_cache.get(user_id, limit=50) tweets = tweet_db.get_tweets(tweet_ids) return tweets ``` **Performance Analysis:** For a regular user with 500 followers: - Write: 500 cache updates (can be parallelized) = ~50ms - Read: Single cache lookup = ~5ms For a celebrity with 50 million followers: - Write: 50 million cache updates = **minutes to hours** - Read: Still ~5ms **Advantages:** - Extremely fast reads (pre-computed results) - Simple timeline retrieval (just a cache lookup) - Excellent user experience - Low database load for timeline queries **Disadvantages:** - Massive write amplification for high-follower accounts - Higher storage requirements (timeline duplication) - Complex fanout logic - Wasted work if follower never checks their timeline ### The Celebrity Problem This is where fan-out on write breaks down. Let me quantify the problem: **Real-world example:** - A major celebrity account can have 50+ million followers - One tweet = 50+ million timeline updates - Even at 100,000 writes/second, that's **500+ seconds, or many minutes** - Result: Replies to her tweet can appear in follower timelines before the original tweet This is unacceptable. The write amplification makes the system unusable for high-profile accounts. ### My Solution: Hybrid Approach After analyzing both approaches, I would implement a **hybrid model** that uses different strategies based on follower count: ![Hybrid timeline fan-out: an author's tweet goes through the Tweet Service and Kafka; for accounts under ~1M followers, fan-out workers read the follower graph and push the tweet id into each follower's Redis home-timeline cache (fan-out on write); celebrity tweets are only stored; when a reader loads their timeline, the Timeline Service reads the precomputed cache and merges in celebrity tweets on read (fan-out on read)](https://d5osvdbc8um23.cloudfront.net/static-asset/blog_images/system-design-twitter/03-timeline-fanout.png) ```python CELEBRITY_THRESHOLD = 1_000_000 # 1M followers def publish_tweet_hybrid(user_id, tweet): tweet_id = tweet_db.save(tweet) followers = social_graph.get_followers(user_id) follower_count = len(followers) if follower_count < CELEBRITY_THRESHOLD: # Fan-out on write for normal users for follower_id in followers: timeline_cache.prepend(follower_id, tweet_id) else: # For celebrities, just mark that they tweeted # Fan-out happens on read celebrity_tweet_cache.add(user_id, tweet_id) return tweet_id def get_timeline_hybrid(user_id): # Get pre-computed timeline (from fan-out on write) timeline_tweets = timeline_cache.get(user_id, limit=50) # Get following list following = social_graph.get_following(user_id) # Check if any followed celebrities have new tweets celebrity_following = [u for u in following if is_celebrity(u)] for celeb_id in celebrity_following: celeb_tweets = celebrity_tweet_cache.get_recent(celeb_id, limit=10) timeline_tweets.extend(celeb_tweets) # Merge and sort timeline_tweets.sort(key=lambda t: t.timestamp, reverse=True) return timeline_tweets[:50] ``` **How this works:** 1. **For normal users (< 1M followers)**: Use fan-out on write - Fast reads for 98%+ of users - Acceptable write costs 2. **For celebrities (> 1M followers)**: Use fan-out on read - Avoid massive write amplification - Slightly slower reads, but only for tweets from celebrities - Since most users follow only a few celebrities, this is a small overhead 3. **At read time**: Merge both sources - Pre-computed timeline from cache (most tweets) - Fresh celebrity tweets fetched on-demand - Final merge and sort **Performance characteristics:** - Regular user timeline: ~5-10ms (mostly cached) - User following 5 celebrities: ~20-30ms (small additional query) - Celebrity posting: ~100ms (no massive fanout) - Write amplification: Bounded to 1M × tweet_size maximum This hybrid approach gives me the best of both worlds: - Fast timelines for normal users (fan-out on write) - Manageable write costs for celebrities (fan-out on read) - Balanced system load ### Implementation Details **Message Queue Integration:** I would use Kafka to decouple tweet publishing from fanout: ```python def publish_tweet(user_id, tweet): # Save tweet tweet_id = tweet_db.save(tweet) # Publish event to Kafka kafka.publish('tweet_created', { 'tweet_id': tweet_id, 'user_id': user_id, 'timestamp': time.now() }) # Return immediately to user return {'status': 'success', 'tweet_id': tweet_id} # Separate fanout workers consume from Kafka def fanout_worker(): for event in kafka.consume('tweet_created'): user_id = event['user_id'] tweet_id = event['tweet_id'] if is_celebrity(user_id): # Skip fanout for celebrities continue followers = social_graph.get_followers(user_id) # Batch updates to Redis pipeline = timeline_cache.pipeline() for follower_id in followers: pipeline.prepend(follower_id, tweet_id) pipeline.execute() ``` **Benefits of this architecture:** 1. **Fast tweet publishing**: Returns to user in <100ms 2. **Reliable delivery**: Kafka ensures no lost updates 3. **Independent scaling**: Can add more fanout workers based on queue depth 4. **Retry logic**: Failed fanouts can be retried 5. **Monitoring**: Can track fanout lag (time from tweet to timeline delivery) **Timeline Cache Structure (Redis):** ``` Key: timeline:{user_id} Type: List (ordered by timestamp) Value: [tweet_id_1, tweet_id_2, ..., tweet_id_800] TTL: 7 days Max size: 800 entries ``` This design achieves Twitter's actual performance target: **under 5 seconds from tweet creation to follower delivery**, while handling 6,000 writes/second and 300,000 reads/second. ### How Other Platforms Differ **Instagram:** - Similar hybrid approach but with algorithmic ranking - Heavier emphasis on media delivery (images/videos) - Slower acceptable latency (users expect some delay) **Facebook:** - More complex ranking algorithm (not pure chronological) - Bidirectional friendships (different graph structure) - Includes content from groups, pages, ads mixed into feed **LinkedIn:** - Much lower scale (less real-time requirement) - Professional content has longer shelf life - More aggressive use of fan-out on read due to lower QPS ## Deep-Dive #2: Social Graph and Tweet Storage at Scale The social graph (who follows whom) is the foundation of the entire system. Every timeline generation, every fanout operation, every notification depends on quickly answering graph queries. At the same time, I need to store 400 million tweets per day efficiently. Let me tackle both problems. ![Storage and social graph: reads flow through an L1 app cache to an L2 Redis cache and fall back to the sharded Tweet DB (partitioned by time + hash); the Tweet Service writes tweets with Snowflake IDs to the same sharded store; the Social Graph Service keeps two denormalized sharded tables, following (who I follow) and followers (who follows me), to serve both query directions cheaply](https://d5osvdbc8um23.cloudfront.net/static-asset/blog_images/system-design-twitter/04-storage-and-graph.png) ### Modeling the Social Graph The relationship model is simpler than Facebook's bidirectional friendships: ``` User A follows User B (unidirectional) - A sees B's tweets - B does NOT automatically see A's tweets ``` **Data model:** ```sql -- Following table (who I follow) CREATE TABLE following ( user_id BIGINT, followed_user_id BIGINT, created_at TIMESTAMP, PRIMARY KEY (user_id, followed_user_id) ); CREATE INDEX idx_following_user ON following(user_id); -- Followers table (who follows me) CREATE TABLE followers ( user_id BIGINT, follower_user_id BIGINT, created_at TIMESTAMP, PRIMARY KEY (user_id, follower_user_id) ); CREATE INDEX idx_followers_user ON followers(user_id); ``` I'm maintaining **two tables** (following and followers) to optimize for both query patterns: - "Who does User A follow?" → Query `following` table - "Who follows User A?" → Query `followers` table This is denormalization for performance: I'm storing the same relationship twice, from both directions. **Scale analysis:** ``` 150M users × 400 average follows = 60 billion relationships Each relationship: 16 bytes (2 × BIGINT) + 8 bytes (timestamp) = 24 bytes Total: 60B × 24 bytes = 1.44 TB ``` 1.44 TB is too large for a single database. I need sharding. ### Sharding the Social Graph The challenge with sharding a graph is that **graphs have no natural root**. Unlike user data (shard by user_id) or tweets (shard by tweet_id or time), graph relationships span across entities. **Naive sharding approach:** ```python # Shard by user_id shard_id = user_id % NUM_SHARDS # Problem: Getting followers requires querying ALL shards def get_followers(user_id): results = [] for shard in all_shards: # Must query every shard because followers could be on any shard results.extend(shard.query("SELECT follower_user_id FROM followers WHERE user_id = ?", user_id)) return results ``` This doesn't work: I'd need to query all shards for every operation. At Twitter's scale, this means thousands of cross-shard queries per second. **My approach: Gravity-based sharding** I would implement a sophisticated sharding strategy based on **data locality**. The idea is to co-locate related users on the same shard, so most operations complete locally. **Algorithm:** 1. **Initial placement**: Assign new users to geographically-based shards - Users in San Francisco → Shard 1 - Users in New York → Shard 2 - Leverages physical proximity clustering 2. **Weight calculation**: Each follow relationship has a weight ```python weight = 1.0 / total_following_count # Example: # User follows 100 people → each relationship weight = 0.01 # User follows 1000 people → each relationship weight = 0.001 ``` 3. **Aggregate pulls per shard**: ```python def calculate_shard_affinity(user_id): following = get_following(user_id) shard_weights = {} for followed_user in following: shard = get_shard(followed_user) weight = 1.0 / len(following) shard_weights[shard] = shard_weights.get(shard, 0) + weight return shard_weights ``` 4. **Migration decision**: ```python def should_migrate(user_id): current_shard = get_shard(user_id) shard_weights = calculate_shard_affinity(user_id) best_shard = max(shard_weights.items(), key=lambda x: x[1]) # Escape velocity prevents thrashing escape_velocity = calculate_escape_velocity(user_id) if shard_weights[best_shard] > shard_weights[current_shard] + escape_velocity: migrate_user(user_id, best_shard) ``` 5. **Escape velocity**: Prevents frequent migrations ```python def calculate_escape_velocity(user_id, query_depth): # Increases with query depth to limit remote calls base_velocity = 10 return base_velocity * (10 ** query_depth) # Depth 0: escape_velocity = 10 # Depth 1: escape_velocity = 100 # Depth 2: escape_velocity = 1,000 ``` **Why this works:** - Users tend to follow others in similar communities/geographies - By co-locating related users, most graph queries hit a single shard - Example: A San Francisco tech worker follows mostly other SF tech workers - Their tweets and timeline operations complete on one shard **Performance benefits:** According to Twitter's research, this approach achieves: - **Most operations complete locally** (single shard) - Average user is within **3 hops of 7 million other users** - Status updates to followers often complete on a single shard - Reduced cross-shard communication by 60-80% ### Tweet Storage and Sharding Now for storing the tweets themselves. With 400 million tweets per day, I need a strategy that scales horizontally. **Tweet data model:** ```sql CREATE TABLE tweets ( tweet_id BIGINT PRIMARY KEY, user_id BIGINT, content TEXT, created_at TIMESTAMP, like_count INT DEFAULT 0, retweet_count INT DEFAULT 0, reply_count INT DEFAULT 0, media_urls TEXT[], hashtags TEXT[], mentioned_users BIGINT[] ); CREATE INDEX idx_tweets_user_time ON tweets(user_id, created_at DESC); CREATE INDEX idx_tweets_time ON tweets(created_at DESC); ``` **Sharding strategy: Time-based + Hash-based hybrid** I would use a two-level sharding approach: ```python # Level 1: Partition by time (monthly buckets) time_partition = tweet_timestamp.year_month # e.g., "2024_01" # Level 2: Hash within partition shard_id = hash(tweet_id) % SHARDS_PER_PARTITION final_shard = f"{time_partition}_{shard_id}" # Example: "2024_01_042" (January 2024, shard 42) ``` **Why this hybrid approach?** 1. **Time partitioning**: - Tweets are immutable (rarely updated after posting) - Most reads are for recent tweets - Old partitions can be archived to cold storage - Simplifies data lifecycle management 2. **Hash within partition**: - Distributes load evenly within time period - Avoids hot partitions during viral events - Enables parallel writes **Partition sizing:** ``` 400M tweets/day × 30 days = 12B tweets/month 12B × 600 bytes = 7.2 TB/month With 10 shards per partition: 720 GB/shard ``` 720 GB per shard is manageable with modern databases. **Hot partition problem:** During major events (Super Bowl, breaking news), certain topics get massive traffic. Even with sharding, specific tweets can become hot spots. **My solution: Multi-level caching** ``` Request for tweet_id ↓ L1: Application cache (in-memory, per server) ↓ (miss) L2: Redis cluster (distributed cache) ↓ (miss) L3: Database (sharded) ``` For viral tweets: - L1 cache hit rate: 90%+ - L2 cache hit rate: 99%+ - Database queries: <1% of requests **Implementation:** ```python class TweetStorage: def get_tweet(self, tweet_id): # L1: Local cache tweet = self.local_cache.get(tweet_id) if tweet: return tweet # L2: Redis tweet = self.redis.get(f"tweet:{tweet_id}") if tweet: self.local_cache.set(tweet_id, tweet) return tweet # L3: Database shard = self.get_shard(tweet_id) tweet = shard.query("SELECT * FROM tweets WHERE tweet_id = ?", tweet_id) # Populate caches self.redis.set(f"tweet:{tweet_id}", tweet, ttl=3600) self.local_cache.set(tweet_id, tweet) return tweet def get_shard(self, tweet_id): # Extract timestamp from tweet_id (Snowflake ID format) timestamp = extract_timestamp(tweet_id) partition = timestamp.year_month shard_num = hash(tweet_id) % SHARDS_PER_PARTITION return self.shards[f"{partition}_{shard_num}"] ``` ### Tweet ID Generation: Snowflake IDs I need globally unique, sortable tweet IDs. Twitter's Snowflake ID format is perfect: ``` 64-bit ID structure: - 41 bits: timestamp (milliseconds since epoch) - 10 bits: machine ID - 12 bits: sequence number Example: 1234567890123456789 ``` **Benefits:** 1. **Globally unique**: No coordination needed between servers 2. **Time-sortable**: IDs naturally sort by creation time 3. **Decentralized generation**: Each server generates its own IDs 4. **Embeds timestamp**: Can extract creation time without database lookup **Implementation:** ```python class SnowflakeIDGenerator: def __init__(self, machine_id): self.machine_id = machine_id # 10 bits self.sequence = 0 # 12 bits self.last_timestamp = 0 def generate(self): timestamp = int(time.time() * 1000) # milliseconds if timestamp == self.last_timestamp: # Same millisecond, increment sequence self.sequence = (self.sequence + 1) & 0xFFF # 12 bits if self.sequence == 0: # Sequence overflow, wait for next millisecond while timestamp <= self.last_timestamp: timestamp = int(time.time() * 1000) else: self.sequence = 0 self.last_timestamp = timestamp # Combine: timestamp (41) | machine_id (10) | sequence (12) tweet_id = (timestamp << 22) | (self.machine_id << 12) | self.sequence return tweet_id ``` This generates 4,096 IDs per millisecond per machine, supporting 4 million tweets/second per server. ### Read vs Write Paths **Write path (posting a tweet):** ``` 1. Client → API Gateway → Tweet Service 2. Generate Snowflake ID 3. Write to database (sharded) 4. Publish event to Kafka 5. Return tweet_id to user (< 100ms) 6. [Async] Fanout workers process event 7. [Async] Search indexer updates Elasticsearch 8. [Async] Trending workers update trending topics ``` **Read path (viewing a tweet):** ``` 1. Client → API Gateway → Tweet Service 2. Check L1 cache (local) 3. Check L2 cache (Redis) 4. Query database (sharded) 5. Return tweet ``` **Timeline read path:** ``` 1. Client → API Gateway → Timeline Service 2. Read timeline cache (Redis) 3. Batch fetch tweet details (with caching) 4. Fetch celebrity tweets (if any) 5. Merge and return ``` The separation of read and write paths allows me to optimize each independently. ### How Other Platforms Differ **Instagram:** - Similar sharding strategy but heavier focus on media storage - Posts are more permanent (less real-time pressure) - Uses Cassandra for better write throughput **Facebook:** - More complex graph (bidirectional + groups + pages) - TAO (The Associations and Objects) layer for graph queries - Heavier caching due to more complex queries **Reddit:** - Tree structure for comments (different data model) - Less emphasis on real-time delivery - Simpler sharding (by subreddit) ## Deep-Dive #3: Search and Trending Topics The final piece of the puzzle is enabling users to find content and discover what's happening in real-time. This requires two capabilities: full-text search across billions of tweets and real-time trending topic detection. ![Search and trending: Kafka's tweet-event stream feeds two independent consumers: a search indexer that writes to Elasticsearch (queried by the Search Service) and trending workers that increment per-hashtag counters in Redis using one-minute buckets; an aggregator reads those buckets, applies time decay and scoring, and publishes the top 20 to a trending cache the user reads](https://d5osvdbc8um23.cloudfront.net/static-asset/blog_images/system-design-twitter/05-search-and-trending.png) ### Full-Text Search Architecture Searching 400 million tweets per day (cumulative billions) requires specialized infrastructure. Relational databases with `LIKE '%keyword%'` queries won't cut it. **My approach: Elasticsearch cluster** I would use Elasticsearch, a distributed search engine built on Lucene, for several reasons: 1. **Inverted index**: Optimized for full-text search 2. **Horizontal scaling**: Shards across multiple nodes 3. **Real-time indexing**: New tweets searchable within seconds 4. **Relevance ranking**: Built-in TF-IDF and BM25 scoring 5. **Aggregations**: For trending topics and analytics **Index structure:** ```json { "mappings": { "properties": { "tweet_id": { "type": "long" }, "user_id": { "type": "long" }, "username": { "type": "keyword" }, "content": { "type": "text", "analyzer": "standard", "fields": { "keyword": { "type": "keyword" } } }, "hashtags": { "type": "keyword" }, "mentioned_users": { "type": "keyword" }, "created_at": { "type": "date" }, "like_count": { "type": "integer" }, "retweet_count": { "type": "integer" }, "language": { "type": "keyword" }, "location": { "type": "geo_point" } } } } ``` **Indexing pipeline:** ```python # Search indexer worker (consumes from Kafka) def search_indexer_worker(): for event in kafka.consume('tweet_created'): tweet_id = event['tweet_id'] # Fetch full tweet details tweet = tweet_db.get_tweet(tweet_id) # Transform for Elasticsearch doc = { 'tweet_id': tweet.id, 'user_id': tweet.user_id, 'username': tweet.username, 'content': tweet.content, 'hashtags': extract_hashtags(tweet.content), 'mentioned_users': extract_mentions(tweet.content), 'created_at': tweet.created_at, 'like_count': tweet.like_count, 'retweet_count': tweet.retweet_count, 'language': detect_language(tweet.content), 'location': tweet.user_location } # Index in Elasticsearch es.index(index='tweets', id=tweet_id, document=doc) ``` **Sharding strategy:** ``` Index: tweets Shards: 100 (primary shards) Replicas: 2 (for each shard) Total nodes: 300+ Elasticsearch nodes Shard assignment: hash(tweet_id) % 100 ``` With 100 shards, each shard handles ~1% of the data. At 87.6 TB/year, that's ~876 GB per shard per year. **Search query example:** ```python def search_tweets(query, filters=None): # Build Elasticsearch query es_query = { "query": { "bool": { "must": [ { "multi_match": { "query": query, "fields": ["content^2", "hashtags", "username"], "type": "best_fields" } } ], "filter": [] } }, "sort": [ { "created_at": "desc" } ], "size": 50 } # Add filters if filters: if filters.get('from_user'): es_query["query"]["bool"]["filter"].append({ "term": { "username": filters['from_user'] } }) if filters.get('hashtag'): es_query["query"]["bool"]["filter"].append({ "term": { "hashtags": filters['hashtag'] } }) if filters.get('date_range'): es_query["query"]["bool"]["filter"].append({ "range": { "created_at": { "gte": filters['date_range']['start'], "lte": filters['date_range']['end'] } } }) # Execute search results = es.search(index='tweets', body=es_query) return [hit['_source'] for hit in results['hits']['hits']] ``` **Performance optimizations:** 1. **Caching popular queries**: ```python # Cache search results for common queries cache_key = f"search:{query}:{filters_hash}" cached = redis.get(cache_key) if cached: return cached results = search_tweets(query, filters) redis.set(cache_key, results, ttl=300) # 5 minutes ``` 2. **Query routing**: - Recent tweets (< 7 days): Query "hot" index on SSD - Older tweets (> 7 days): Query "warm" index on cheaper storage - Very old tweets (> 1 year): Query "cold" index or archive 3. **Pagination with search_after**: ```python # Avoid deep pagination (slow and expensive) # Use search_after for efficient scrolling def search_tweets_paginated(query, search_after=None): es_query = { "query": {...}, "sort": [ {"created_at": "desc"}, {"tweet_id": "desc"} # Tie-breaker ], "size": 50 } if search_after: es_query["search_after"] = search_after results = es.search(index='tweets', body=es_query) hits = results['hits']['hits'] next_search_after = hits[-1]['sort'] if hits else None return { 'tweets': [hit['_source'] for hit in hits], 'next_page_token': next_search_after } ``` **Handling high query volume:** With millions of searches per day, I need to protect Elasticsearch from overload: ```python class SearchService: def __init__(self): self.rate_limiter = RateLimiter(max_qps=10000) self.circuit_breaker = CircuitBreaker(failure_threshold=0.5) def search(self, user_id, query): # Rate limiting per user if not self.rate_limiter.allow(user_id): raise RateLimitError("Too many searches") # Circuit breaker for Elasticsearch health if not self.circuit_breaker.is_closed(): # Elasticsearch is unhealthy, return cached/degraded results return self.get_cached_results(query) try: results = self.es_search(query) self.circuit_breaker.record_success() return results except Exception as e: self.circuit_breaker.record_failure() raise ``` ### Real-Time Trending Topics Trending topics are hashtags or phrases that are suddenly popular. The challenge is detecting these in real-time across 6,000 tweets/second. **Algorithm approach:** I would use a **sliding window with decay** to detect trending topics: ```python class TrendingDetector: def __init__(self): self.window_size = 3600 # 1 hour self.decay_factor = 0.5 # Half-life of 30 minutes # Store counts per time bucket self.hashtag_counts = defaultdict(lambda: defaultdict(int)) # hashtag_counts[hashtag][timestamp_bucket] = count def process_tweet(self, tweet): hashtags = extract_hashtags(tweet.content) timestamp = tweet.created_at bucket = timestamp // 60 # 1-minute buckets for hashtag in hashtags: self.hashtag_counts[hashtag][bucket] += 1 def get_trending(self, current_time): current_bucket = current_time // 60 scores = {} for hashtag, buckets in self.hashtag_counts.items(): score = 0 # Sum counts from recent buckets with decay for bucket, count in buckets.items(): age_minutes = current_bucket - bucket if age_minutes > 60: # Only consider last hour continue # Apply exponential decay decay = self.decay_factor ** (age_minutes / 30) score += count * decay scores[hashtag] = score # Return top 20 by score trending = sorted(scores.items(), key=lambda x: x[1], reverse=True)[:20] return [hashtag for hashtag, score in trending] ``` **Distributed implementation:** For 6,000 tweets/second, a single server can't handle all trending calculations. I need distribution: ```python # Trending worker (consumes from Kafka) def trending_worker(worker_id, num_workers): # Each worker handles a subset of hashtags (by hash) for event in kafka.consume('tweet_created'): tweet = event['tweet'] hashtags = extract_hashtags(tweet['content']) for hashtag in hashtags: # Route to worker by hashtag hash if hash(hashtag) % num_workers != worker_id: continue # Update counts in Redis timestamp_bucket = int(time.time()) // 60 key = f"trending:{hashtag}:{timestamp_bucket}" redis.incr(key) redis.expire(key, 3600) # Expire after 1 hour # Aggregator (runs every minute) def trending_aggregator(): while True: current_time = int(time.time()) current_bucket = current_time // 60 # Scan all hashtag keys all_hashtags = set() for key in redis.scan_iter("trending:*"): hashtag = key.split(':')[1] all_hashtags.add(hashtag) # Calculate scores scores = {} for hashtag in all_hashtags: score = 0 for i in range(60): # Last 60 minutes bucket = current_bucket - i key = f"trending:{hashtag}:{bucket}" count = int(redis.get(key) or 0) # Exponential decay decay = 0.5 ** (i / 30) score += count * decay scores[hashtag] = score # Get top 20 trending = sorted(scores.items(), key=lambda x: x[1], reverse=True)[:20] # Store in cache redis.set('trending:global', json.dumps(trending), ex=60) time.sleep(60) # Run every minute ``` **Preventing manipulation:** To prevent spam or coordinated manipulation of trending topics: ```python def calculate_trending_score(hashtag, time_window): # Basic count raw_count = get_hashtag_count(hashtag, time_window) # Unique users (prevent spam) unique_users = get_unique_users(hashtag, time_window) # Velocity (sudden spike detection) previous_count = get_hashtag_count(hashtag, time_window - 3600) velocity = (raw_count - previous_count) / previous_count if previous_count > 0 else 0 # Combined score score = ( raw_count * 0.5 + # Raw popularity unique_users * 2.0 + # Unique users weighted higher velocity * 100 # Velocity for "trending" feel ) # Penalize if too few unique users (likely spam) if unique_users < 100: score *= 0.1 return score ``` **Geographic trending:** Different regions have different trending topics: ```python def get_trending_by_location(location): # Similar algorithm but filtered by location trending_key = f"trending:{location}" # Pre-compute for major regions for region in ['US', 'UK', 'Japan', 'Brazil', 'India']: trending = calculate_trending( filter_location=region, time_window=3600 ) redis.set(f"trending:{region}", json.dumps(trending), ex=300) return redis.get(trending_key) ``` **Storage requirements:** ``` Hashtag tracking: - 1M unique hashtags per hour (high estimate) - 60 time buckets per hour - Per bucket: hashtag (50 bytes) + count (8 bytes) + metadata (50 bytes) = 108 bytes - Total: 1M × 60 × 108 bytes = 6.5 GB per hour - With 24-hour retention: ~156 GB ``` This fits comfortably in Redis with replication. **API endpoint:** ```python @app.get('/api/trending') def get_trending(location: Optional[str] = None): if location: trending = redis.get(f'trending:{location}') else: trending = redis.get('trending:global') if not trending: return {'error': 'Trending data not available'} trending_list = json.loads(trending) # Enrich with metadata results = [] for hashtag, score in trending_list: # Get sample tweets sample_tweets = es.search( index='tweets', body={ "query": {"term": {"hashtags": hashtag}}, "sort": [{"created_at": "desc"}], "size": 3 } ) results.append({ 'hashtag': hashtag, 'tweet_count': int(score), 'sample_tweets': [hit['_source'] for hit in sample_tweets['hits']['hits']] }) return {'trending': results} ``` ### Search and Trending Integration The search and trending systems work together: ``` Tweet Created ↓ Kafka Event ↓ ├─→ Search Indexer (Elasticsearch) └─→ Trending Worker (Redis counters) ``` Both systems consume from the same Kafka stream, ensuring consistency. ### Performance Metrics **Search:** - Query latency: p50 = 50ms, p99 = 300ms - Indexing latency: < 5 seconds from tweet creation - Throughput: 10,000+ searches/second **Trending:** - Update frequency: Every 60 seconds - Detection latency: 1-2 minutes for new trends - API latency: < 10ms (served from cache) ### How Other Platforms Differ **Instagram:** - Trending focuses on hashtags and locations - Slower update frequency (less real-time pressure) - Heavier use of ML for trend prediction **TikTok:** - Trending based on video views, not just hashtags - More sophisticated ML for trend detection - Personalized trending (different for each user) **Reddit:** - Trending based on upvotes and comment velocity - Subreddit-specific trending - Longer time windows (hours vs minutes) ## Conclusion: Bringing It All Together Designing a system like Twitter requires balancing numerous trade-offs across multiple dimensions: read vs write optimization, consistency vs availability, real-time vs batch processing, and cost vs performance. Here's how the pieces fit together: **Timeline Generation**: I chose a hybrid fan-out approach (push for normal users, pull for celebrities), achieving fast reads (5-10ms) for 98% of users while avoiding write amplification. The key insight is that the "when to compute" decision (read-time vs write-time) has more impact than any single technology choice. **Storage and Social Graph**: With time-based partitioning for tweets and gravity-based sharding for the social graph, I can scale to 400 million tweets per day while keeping most operations local to a single shard. Snowflake IDs provide globally unique, sortable identifiers without coordination overhead. **Search and Trending**: Elasticsearch handles full-text search across billions of tweets, while a sliding-window algorithm with decay detects trending topics in real-time. Both systems consume from the same event stream, ensuring consistency. The entire architecture is built on a few key principles: 1. **Optimize for the common case**: 98% of traffic is reads, so optimize reads even if writes become more expensive 2. **Handle outliers explicitly**: Celebrity accounts are common enough to need special handling 3. **Embrace asynchronous processing**: Decouple tweet posting from fanout using message queues 4. **Cache aggressively**: Multi-level caching (L1, L2, L3) protects databases from overload 5. **Shard intelligently**: Use domain knowledge (time, geography, relationships) to inform sharding decisions This design handles Twitter's actual scale: 150 million daily active users, 300,000 timeline queries per second, 6,000 writes per second, and 400 million tweets per day. The architecture is proven: it's based on Twitter's real infrastructure, adapted and refined through years of scaling challenges. The beauty of this system is that it scales horizontally at every layer: more timeline servers, more database shards, more Elasticsearch nodes, more fanout workers. There's no single bottleneck, no central coordinator that limits growth. Building Twitter taught the industry that social media platforms are fundamentally **distribution systems**. The challenge isn't storing tweets or managing users, it's delivering the right content to the right people at the right time, at massive scale, with minimal latency. That's what makes this problem endlessly fascinating and worth understanding deeply.