Redis Lua Script & SETNX: High-Performance Rate Limiting & Quota Alerting for APIs

To protect API servers from denial-of-service (DDoS) attacks and manage infrastructure costs—especially when APIs integrate billed third-party services like the Google Maps API—implementing a robust traffic control system is essential.

In this article, we will explore how Gogoduk—a Vietnam mapping API platform—designs and optimizes its real-time API rate limiting and daily quota alerting systems using Go and Redis. We will dive deep into avoiding race conditions, stopping memory leaks with Redis Lua Scripts, and deduplicating notifications using SETNX.

Why APIs Require Rate Limiting & Daily Quotas

A production-ready API gateway must enforce traffic control at two distinct granularities:

1. Rate Limiting (Throttle): Restricting requests over a short window (e.g., maximum 60 requests per minute) to deter spam bots and prevent server overload.
2. Daily Quota (Daily Cap): Restricting total requests over a 24-hour cycle (e.g., 5,000 requests per day) based on the subscription tier of the user.

At Gogoduk, every incoming API request must check both limits in under a few milliseconds before proceeding. Because of this strict requirement, Redis—with its sub-millisecond, in-memory data structures—is the ideal engine for tracking request volumes.

The Memory Leak Hazard of Sequential Redis Commands

The most intuitive way to build a rate limiter (e.g., maximum 60 requests per minute) in Go is to execute standard Redis commands sequentially:

// WARNING: This pattern is prone to memory leaks!
count, err := rdb.Incr(ctx, minKey).Result()
if count == 1 {
    rdb.Expire(ctx, minKey, 60 * time.Second)
}

Why is this sequential pattern dangerous?

This code is not atomic. It requires two separate network calls between the application and Redis.

What happens if the application server crashes, loses connection, or gets terminated immediately after the Incr command succeeds, but before the Expire command is sent? The key minKey will remain in Redis's memory indefinitely with no Time To Live (TTL).

In high-traffic systems handling millions of unique IP addresses or user keys daily, these "orphan" keys will pile up in RAM. Over time, this leads to a severe memory leak, eventually causing the Redis server to run out of RAM and crash.

Achieving Atomicity with Redis Lua Scripts

To eliminate this vulnerability, the INCR and EXPIRE operations must run as a single atomic unit. Redis supports execution of Lua scripts natively. Since Redis is single-threaded, a Lua script runs to completion without any other commands interrupting it.

Here is the Lua script utilized in Gogoduk's rate limiter middleware:

local current = redis.call("INCR", KEYS[1])
if current == 1 then
  redis.call("EXPIRE", KEYS[1], ARGV[1])
end
return current

How it works:

1. The Go application sends the Lua script with the target key minKey (KEYS[1]) and the TTL 60 (ARGV[1]) to Redis.
2. Redis increments the key. If the returned count is 1 (indicating that the key was just initialized for this minute bucket), it immediately sets the TTL.
3. The entire sequence is executed atomically inside Redis. Even if the Go server crashes immediately after sending, the key is guaranteed to have a TTL and will self-delete.
4. Network Optimization: We reduce network Round Trip Time (RTT) from two round-trips to one.

In Go, we can load and run this script seamlessly:

var incrScript = redis.NewScript(`
local current = redis.call("INCR", KEYS[1])
if current == 1 then
  redis.call("EXPIRE", KEYS[1], ARGV[1])
end
return current
`)

// Execute inside Fiber/Gin middleware
minCount, err := incrScript.Run(ctx, rdb, []string{minKey}, 60).Int64()

The Challenge of Quota Alerting Under Concurrency

Beyond blocking requests that exceed limits, Gogoduk needs to warn users via webhooks or emails when they hit 80% or 100% of their daily quota, allowing them to upgrade their plans proactively.

The Concurrency Spikes Dilemma

Assume a user has a daily allowance of 10,000 requests. When the 10,000th request arrives, the client might be sending 100 concurrent requests at that exact second.

Without synchronization, all 100 concurrent requests would check the database, realize the limit has been crossed, and trigger 100 separate warning emails simultaneously. This hammers the email delivery service and results in a poor user experience (spamming the user's inbox).

Preventing Duplicate Notifications Using SETNX

To guarantee that a notification is sent exactly once per user per percentage threshold per day, Gogoduk employs a lightweight distributed lock pattern using the Redis SETNX (Set if Not Exists) command.

We define a temporary deduplication flag key following this schema: quota_alert:<userID>:<percent>:<date_string>.

Here is the implementation structure in Go:

func maybeFireQuotaAlert(ctx context.Context, rdb *redis.Client, userID string, percent int, used, dailyLimit int64, now time.Time) {
    dayStamp := now.UTC().Format("2006-01-02")
    flagKey := fmt.Sprintf("quota_alert:%s:%d:%s", userID, percent, dayStamp)
    
    // Set flag with 30-hour TTL to account for timezone boundaries safely
    ok, err := rdb.SetNX(ctx, flagKey, "1", 30 * time.Hour).Result()
    if err != nil || !ok {
        // If SETNX returns false, the flag is already set (alert already triggered today)
        // We silently abort to prevent duplicate emails
        return
    }

    // If SETNX returns true, dispatch the notification asynchronously
    go postQuotaAlert(userID, percent, used, dailyLimit)
}

Why this design works:

1. Atomic SETNX: Only a single request among the concurrent waves will succeed in setting the flag key in Redis and receive true. All other concurrent requests receive false and exit immediately.
2. Asynchronous Execution: Sending emails or dispatching external HTTP requests is slow (often taking 1-3 seconds). Running postQuotaAlert inside a background Goroutine prevents blocking the critical request path, returning a response to the client immediately.
3. Automatic Expiration: The flag key has a 30-hour TTL. Once the day passes, the key expires, preparing the system for the next day's quota checks without manual database cleanup.

Conclusion and Design Takeaways

When building high-performance rate limiting systems with Redis, keep these three engineering principles in mind:

1. Enforce Atomicity: Use Lua scripts for read-then-write groups of commands (such as incrementing counters and assigning expirations) to prevent memory leaks during network disruptions.
2. Deduplicate with SETNX: SETNX is an extremely lightweight way to prevent duplicate execution of heavy side effects (like email notifications or logging) under highly concurrent traffic.
3. Perform Side Effects Asynchronously: Keep the main request thread clear of slow external network actions. Offload alerts and logging to background routines to maintain low API latency.

To learn more about optimizing client integrations with map APIs, read our checklist on Optimizing Vietnam Address Autocomplete for Checkout UX.