High-Concurrency Reliable Service Architecture#

When building high-concurrency, high-availability Internet services, architectural design is no longer just about “making it run.” It must achieve low latency, high throughput, scalability, and strong fault tolerance amid massive request volumes, limited resources, and complex business call chains. The following provides a systematic breakdown based on a typical BFF + microservices + cloud-native infrastructure architecture.

1. Architecture: Layering and Separation#

💡 Design principle: separate the API layer from the RPC layer. The BFF handles data aggregation, protocol translation, and business orchestration, while the RPC layer focuses on core atomic capabilities—consistent with the single-responsibility principle and high reusability.

2. Key Component Analysis and Considerations#

2.1 BFF + Microservice RPC: Split and Push Hot Logic Downstream#

Background: in a traditional monolithic architecture, a single API endpoint may call multiple databases and perform heavy computation; once concurrency rises, thread pools are quickly exhausted.

Solution:

• The BFF is stateless and lightweight, doing only aggregation and trimming
• Core logic is pushed down into user-rpc / social-rpc
• The RPC layer scales independently

⚠️ Caveats:

• The BFF is not a “universal proxy”; avoid complex transactions or slow queries in the BFF, or it will become the concurrency bottleneck
• The RPC layer should use batch processing and parallel calls (e.g., /batch_user_info) to reduce BFF ↔ RPC network overhead
• The BFF should remain stateless; session state should be externalized to Redis to enable horizontal scaling
• Set a timeout budget for the BFF: when aggregating multiple downstream calls, the total timeout must not exceed what upstream can tolerate, so tail latency does not drag down the whole

// The BFF aggregates multiple RPC calls in parallel with an overall timeout
func GetUserProfile(ctx context.Context, uid int64) (*Profile, error) {
    ctx, cancel := context.WithTimeout(ctx, 200*time.Millisecond)
    defer cancel()

    var wg sync.WaitGroup
    var user *User
    var social *Social
    var err1, err2 error

    wg.Add(2)
    go func() { defer wg.Done(); user, err1 = userRPC.Get(ctx, uid) }()
    go func() { defer wg.Done(); social, err2 = socialRPC.Get(ctx, uid) }()
    wg.Wait()

    if err1 != nil || err2 != nil {
        return nil, firstError(err1, err2)
    }
    return merge(user, social), nil
}

2.2 Etcd: Service Discovery and Configuration (Not State Storage)#

• RPC service address registration
• Dynamic configuration delivery (rate-limit thresholds, feature switches)
• Paired with local caching + long polling to reduce read pressure on etcd
• Etcd is not a database; do not store business state in it. It only answers “who is where” and “what the configuration is.” Once write volume grows too large, it will drag down the entire service-discovery chain.

sequenceDiagram
    participant RPC as RPC Service
    participant Etcd as etcd
    participant BFF as BFF Caller
    RPC->>Etcd: Register address + heartbeat
    BFF->>Etcd: Watch service list
    Etcd-->>BFF: Push changes
    Note over BFF: Cache the service list locally,<br/>avoid reading etcd on every call

2.3 Redis: The Key Buffer Layer for High-Concurrency Reads and Writes#

Core uses:

• Hot data cache (user info, relationship chains)
• Distributed rate limiting (token bucket, leaky bucket)
• Distributed locks (prevent duplicate writes)
• Counters (likes, online users)

flowchart TB
    Req["Read request"] --> L1["Local cache"]
    L1 -->|miss| L2["Redis"]
    L2 -->|miss| DB["Database"]
    DB --> Backfill["Backfill cache"]
    Backfill --> L2
    L2 --> L1

Common bottlenecks and optimizations:

2.4 Message Queues: Peak Shaving, Valley Filling, and Decoupling Non-Real-Time Paths#

Typical scenarios:

• IM message fan-out: send one message → MQ → asynchronously push to multiple recipients
• Logs/audit: asynchronously write to ES (ELK)
• Social feed push: when a user has many followers, writes cannot be synchronous; the MQ must absorb the peak

flowchart LR
    P["Producer"] --> MQ["Message queue"]
    MQ --> C1["Consumer 1"]
    MQ --> C2["Consumer 2"]
    MQ --> C3["Consumer 3"]
    C1 --> S1["Write DB / push / audit"]
    C2 --> S2["Write DB / push / audit"]
    C3 --> S3["Write DB / push / audit"]

Caveats:

• Clarify which paths must be reliable (orders, IM history) → MQ + persistence
• Which paths may tolerate loss (like notifications) → RabbitMQ for higher throughput
• MQ backlog must be monitored; otherwise, under high concurrency it becomes a hidden “slow path”
• Consumers must be idempotent, because MQ typically guarantees at-least-once delivery
• Set up dead-letter queues to handle repeatedly failing messages and avoid blocking the main queue

2.5 WebSocket: Long Connections and Concurrency Bottlenecks#

Background:

• Per-machine long-connection limits
• Excessive memory usage
• Broadcast storms

Optimization strategies:

• Use connection migration (at publish time, locate which WebSocket node a user is connected to via etcd)
• Message push does not go from the WebSocket node directly to the DB; instead it goes through MQ + a routing table
• Make the long-connection service itself stateless (session in Redis, or local memory + consistent hashing)
• WebSocket nodes only hold connections; they do not decide whom to push to. Routing decisions are delegated to an independent routing service. Only when the nodes themselves are stateless can they scale horizontally

flowchart LR
    Sender["Sender"] --> MQ["MQ"]
    MQ --> Router["Routing service<br/>looks up which node a user is on"]
    Router --> WS1["WebSocket node 1"]
    Router --> WS2["WebSocket node 2"]
    Router --> WS3["WebSocket node 3"]
    WS1 --> U1["User A"]
    WS2 --> U2["User B"]
    WS3 --> U3["User C"]

3. Core Design Patterns for High Concurrency#

On top of component selection, what truly supports high concurrency is a set of reusable design patterns. The following three patterns recur throughout this architecture.

3.1 Eventual Consistency vs. Strong Consistency#

flowchart TB
    Q{"Can the business tolerate brief inconsistency?"}
    Q -->|No| Strong["Strong consistency<br/>distributed lock + transaction"]
    Q -->|Yes| Eventual["Eventual consistency<br/>local message table + MQ + reconciliation"]
    Strong --> Cost1["High latency, low throughput"]
    Eventual --> Cost2["Low latency, high throughput"]

💡 Core principle: prefer eventual consistency on high-concurrency paths. Reserve strong consistency for truly non-negotiable scenarios (such as payments).

3.2 Cache–Database Consistency Solutions#

Cache consistency with the database under high concurrency is a classic hard problem. This architecture uses a cache-aside + asynchronous cleanup pattern:

Write request:
  1. Update the DB
  2. Send an MQ message (recording the updated key)
  3. Return success immediately

Asynchronous cleanup:
  1. The consumer pulls messages from the MQ
  2. Deletes the corresponding key in Redis
  3. The next read request triggers a cache rebuild

sequenceDiagram
    participant C as Client
    participant S as Service
    participant DB as Database
    participant MQ as Message queue
    participant R as Redis

    C->>S: Write request
    S->>DB: Update data
    S->>MQ: Send invalidation message
    S-->>C: Return success
    MQ->>R: Asynchronously delete cache key
    Note over R: Next read request triggers cache rebuild

Why not update the cache?#

Updating the cache directly under high concurrency introduces severe race conditions. Consider the following scenario:

T1: Request A reads the cache (miss) and reads X=100 from the DB
T2: Request B updates the DB, changing X to 200
T3: Request B updates the cache, setting X=200
T4: Request A updates the cache, setting X=100 (overwriting Request B's update)

Result: cache X=100, DB X=200. Consistency is broken and cannot recover automatically.

Even with a “delete cache first” strategy, the classic read-write concurrency problem remains:

T1: Request A reads the cache (miss) and reads X=100 from the DB
T2: Request B updates the DB, changing X to 200
T3: Request B deletes the cache (key removed)
T4: Request A updates the cache, setting X=100 (the cache holds the stale value again)

Does the problem still exist? — This is the classic read-write concurrency problem, which requires delayed double deletion or version numbers to solve.

sequenceDiagram
    participant A as Request A
    participant B as Request B
    participant DB as Database
    participant R as Redis

    A->>R: Read cache (miss)
    A->>DB: Read X=100
    B->>DB: Update X=200
    B->>R: Delete cache
    A->>R: Write back X=100 (stale)
    Note over R: Cache and DB are inconsistent

A safer approach: version numbers / timestamps#

// Values stored in the cache carry a version number
type CachedValue struct {
    Value   interface{} `json:"value"`
    Version int64       `json:"version"`
}

// On update: write the DB first, then delete the cache, and compare version numbers before writing
func UpdateWithVersion(ctx context.Context, key string, newValue interface{}) error {
    newVersion := time.Now().UnixNano()

    // 1. Update the DB (with version)
    err := db.Exec("UPDATE users SET name = ?, version = ? WHERE id = ?", newValue, newVersion, uid)
    if err != nil {
        return err
    }

    // 2. Delete the cache (so the next read rebuilds it)
    return redis.Del(key)
}

// On read: read from the cache and validate the version
func GetWithVersion(ctx context.Context, key string) (interface{}, error) {
    // 1. Read from the cache
    cached, err := redis.Get(key)
    if err == nil {
        var val CachedValue
        json.Unmarshal(cached, &val)
        return val.Value, nil
    }

    // 2. Cache miss: read from the DB
    var value interface{}
    var version int64
    db.QueryRow("SELECT name, version FROM users WHERE id = ?", uid).Scan(&value, &version)

    // 3. Write back to the cache (with version)
    cachedVal := CachedValue{Value: value, Version: version}
    redis.Set(key, cachedVal, 3600)

    return value, nil
}

💡 Delayed double deletion: after updating the DB, delete the cache once, then delete it again a few hundred milliseconds later to cover the read-write concurrency window. The trade-off is nondeterministic delay; suitable for scenarios with higher consistency requirements.

3.3 The Trio: Rate Limiting, Circuit Breaking, and Degradation#

flowchart TB
    Req["Request"] --> Limit{"Passes rate limit?"}
    Limit -->|No| Reject["Reject"]
    Limit -->|Yes| Circuit{"Circuit breaker state?"}
    Circuit -->|Open| Degrade["Degraded response"]
    Circuit -->|Closed| Call["Call downstream"]
    Call -->|Failure rate above threshold| Open["Circuit opens"]
    Call -->|Success| OK["Return result"]

🔧 Configuration dynamization: rate-limit thresholds and circuit-breaker switches are delivered via etcd—no service restart required. ⚠️ The essence of degradation is trade-off: degradation is not “fallback on error,” but proactively giving up secondary features to preserve the core path. When designing a degradation plan, first ask “which features can be temporarily sacrificed,” not “what to return when an error occurs.”

Asynchrony and Backpressure#

flowchart LR
    In["Request surge"] --> Queue["Queue buffer"]
    Queue --> Worker["Consumers process at their own capacity"]
    Worker -->|Backlog above threshold| Backpressure["Backpressure signal"]
    Backpressure --> In2["Upstream slows down or rejects"]

💡 Backpressure vs. rate limiting: rate limiting is “rejection at the entrance”; backpressure is “downstream feeding back to upstream to slow down.” The former protects yourself; the latter protects the entire chain.

4. Typical Incidents and Postmortems#

4.1 A Redis Big Key Triggering an Avalanche#

Symptom: at 20:00 one evening during peak hours, the P99 latency of the user-api interface spiked from 50 ms to 8 s, and some requests timed out.

Root cause analysis:

• User relationship chains were stored in Redis hashes; one top influencer had 20 million followers
• The hgetall command pulled all follower IDs at once, overflowing network buffers
• Other keys on the same Redis shard were also blocked

flowchart TB
    BigKey["Big key: 20 million followers"] --> HGetAll["hgetall pulls everything"]
    HGetAll --> Buffer["Network buffer overflow"]
    Buffer --> Block["Other keys on the same shard blocked"]
    Block --> Avalanche["Interface P99 latency spikes"]

Fixes:

• Split big keys: shard the data to avoid oversized single keys
• Ban hgetall; use hscan for batched reads
• Add local caching for hot data to reduce Redis pressure
• Establish a big-key inspection mechanism to detect issues early

4.2 MQ Backlog Causing a “Slow-Path Contagion”#

Symptom: IM messages were sent successfully, but recipients received the push 10 minutes late.

Root cause analysis:

• After pulling messages from Kafka, consumers needed to write to MySQL to build indexes
• A release introduced a slow SQL statement (missing index), and per-message processing time ballooned from 10 ms to 2 s
• Producer write rate stayed the same while consumption rate plummeted; the backlog quickly reached millions of messages

flowchart LR
    SlowSQL["Slow SQL: 10ms -> 2s"] --> SlowConsume["Consumption rate plummets"]
    Producer["Production rate unchanged"] --> SlowConsume
    SlowConsume --> Lag["Millions of messages backlogged"]
    Lag --> Delay["Push delayed by 10 minutes"]

Improvements:

• Add slow-SQL monitoring and alerts on the consumer side
• Set a backlog threshold; auto-scale consumers when it is exceeded
• Dead-letter queue as a safety net to avoid blocking by poison messages
• SQL review and load testing before every release

5. Capacity Planning and Load Testing#

High-concurrency capability is not “obtained naturally after launch”; it is calculated during the design phase and squeezed out through load testing before launch.

flowchart LR
    Base["Baseline load test"] --> Model["Single-machine capacity model"]
    Model --> Plan["Capacity plan"]
    Plan --> Full["End-to-end load test"]
    Full --> Bottleneck["Bottleneck identification"]
    Bottleneck --> Scale["Scale out / optimize"]
    Scale --> Chaos["Chaos engineering validation"]

💡 The goal of load testing is not “to reach the highest QPS,” but “to find the inflection point and the bottleneck.” Knowing at what pressure the system starts to degrade is more valuable than knowing the peak.

6. Summary#

What truly determines high-concurrency capability is often not component choice, but:

• ✅ Whether every core path has undergone capacity evaluation and load testing
• ✅ Whether degradation (returning cache/default data after rate limiting) and circuit breaking are designed in
• ✅ Whether cross-microservice transactions are avoided (distributed transactions are extremely costly under high concurrency)
• ✅ Whether “read-write separation + eventual consistency” is accepted
• ✅ Whether asynchrony, batching, and backpressure are the default tools
• ✅ Whether observability is in place to locate failures quickly

🎯 Core idea: high concurrency is not “solved by adding machines.” It means assuming from day one that every component will fail and every path will slow down, then turning uncertainty into controllable system behavior through splitting, asynchrony, caching, rate limiting, degradation, and observability.