From: "Vaidas Pilkauskas via GitGitGadget" <gitgitgadget@gmail•com>
To: git@vger•kernel.org
Cc: Vaidas Pilkauskas <vaidas.pilkauskas@shopify•com>,
Vaidas Pilkauskas <vaidas.pilkauskas@shopify•com>
Subject: [PATCH 1/3] http: add support for HTTP 429 rate limit retries
Date: Wed, 26 Nov 2025 12:30:25 +0000 [thread overview]
Message-ID: <ae0087cd1c7fbb6b748d6767b476c1bd1a19996f.1764160227.git.gitgitgadget@gmail.com> (raw)
In-Reply-To: <pull.2008.git.1764160227.gitgitgadget@gmail.com>
From: Vaidas Pilkauskas <vaidas.pilkauskas@shopify•com>
Add retry logic for HTTP 429 (Too Many Requests) responses to handle
server-side rate limiting gracefully. When Git's HTTP client receives
a 429 response, it can now automatically retry the request after an
appropriate delay, respecting the server's rate limits.
The implementation supports the RFC-compliant Retry-After header in
both delay-seconds (integer) and HTTP-date (RFC 2822) formats. If a
past date is provided, Git retries immediately without waiting.
Retry behavior is controlled by three new configuration options:
* http.maxRetries: Maximum number of retry attempts (default: 0,
meaning retries are disabled by default). Users must explicitly
opt-in to retry behavior.
* http.retryAfter: Default delay in seconds when the server doesn't
provide a Retry-After header (default: -1, meaning fail if no
header is provided). This serves as a fallback mechanism.
* http.maxRetryTime: Maximum delay in seconds for a single retry
(default: 300). If the server requests a delay exceeding this
limit, Git fails immediately rather than waiting. This prevents
indefinite blocking on unreasonable server requests.
All three options can be overridden via environment variables:
GIT_HTTP_MAX_RETRIES, GIT_HTTP_RETRY_AFTER, and
GIT_HTTP_MAX_RETRY_TIME.
The retry logic implements a fail-fast approach: if any delay
(whether from server header or configuration) exceeds maxRetryTime,
Git fails immediately with a clear error message rather than capping
the delay. This provides better visibility into rate limiting issues.
The implementation includes extensive test coverage for basic retry
behavior, Retry-After header formats (integer and HTTP-date),
configuration combinations, maxRetryTime limits, invalid header
handling, environment variable overrides, and edge cases.
Signed-off-by: Vaidas Pilkauskas <vaidas.pilkauskas@shopify•com>
---
Documentation/config/http.adoc | 24 ++
http-push.c | 8 +
http-walker.c | 5 +
http.c | 149 +++++++++++-
http.h | 2 +
remote-curl.c | 4 +
t/meson.build | 1 +
t/t5584-http-429-retry.sh | 429 +++++++++++++++++++++++++++++++++
8 files changed, 618 insertions(+), 4 deletions(-)
create mode 100755 t/t5584-http-429-retry.sh
diff --git a/Documentation/config/http.adoc b/Documentation/config/http.adoc
index 9da5c298cc..9e3c888df4 100644
--- a/Documentation/config/http.adoc
+++ b/Documentation/config/http.adoc
@@ -315,6 +315,30 @@ http.keepAliveCount::
unset, curl's default value is used. Can be overridden by the
`GIT_HTTP_KEEPALIVE_COUNT` environment variable.
+http.retryAfter::
+ Default wait time in seconds before retrying when a server returns
+ HTTP 429 (Too Many Requests) without a Retry-After header. If set
+ to -1 (the default), Git will fail immediately when encountering
+ a 429 response without a Retry-After header. When a Retry-After
+ header is present, its value takes precedence over this setting.
+ Can be overridden by the `GIT_HTTP_RETRY_AFTER` environment variable.
+ See also `http.maxRetries` and `http.maxRetryTime`.
+
+http.maxRetries::
+ Maximum number of times to retry after receiving HTTP 429 (Too Many
+ Requests) responses. Set to 0 (the default) to disable retries.
+ Can be overridden by the `GIT_HTTP_MAX_RETRIES` environment variable.
+ See also `http.retryAfter` and `http.maxRetryTime`.
+
+http.maxRetryTime::
+ Maximum time in seconds to wait for a single retry attempt when
+ handling HTTP 429 (Too Many Requests) responses. If the server
+ requests a delay (via Retry-After header) or if `http.retryAfter`
+ is configured with a value that exceeds this maximum, Git will fail
+ immediately rather than waiting. Default is 300 seconds (5 minutes).
+ Can be overridden by the `GIT_HTTP_MAX_RETRY_TIME` environment
+ variable. See also `http.retryAfter` and `http.maxRetries`.
+
http.noEPSV::
A boolean which disables using of EPSV ftp command by curl.
This can be helpful with some "poor" ftp servers which don't
diff --git a/http-push.c b/http-push.c
index d86ce77119..a602a302ec 100644
--- a/http-push.c
+++ b/http-push.c
@@ -716,6 +716,10 @@ static int fetch_indices(void)
case HTTP_MISSING_TARGET:
ret = 0;
break;
+ case HTTP_RATE_LIMITED:
+ error("rate limited by '%s', please try again later", repo->url);
+ ret = -1;
+ break;
default:
ret = -1;
}
@@ -1548,6 +1552,10 @@ static int remote_exists(const char *path)
case HTTP_MISSING_TARGET:
ret = 0;
break;
+ case HTTP_RATE_LIMITED:
+ error("rate limited by '%s', please try again later", url);
+ ret = -1;
+ break;
case HTTP_ERROR:
error("unable to access '%s': %s", url, curl_errorstr);
/* fallthrough */
diff --git a/http-walker.c b/http-walker.c
index e886e64866..9f06f47de1 100644
--- a/http-walker.c
+++ b/http-walker.c
@@ -414,6 +414,11 @@ static int fetch_indices(struct walker *walker, struct alt_base *repo)
repo->got_indices = 1;
ret = 0;
break;
+ case HTTP_RATE_LIMITED:
+ error("rate limited by '%s', please try again later", repo->base);
+ repo->got_indices = 0;
+ ret = -1;
+ break;
default:
repo->got_indices = 0;
ret = -1;
diff --git a/http.c b/http.c
index 41f850db16..212805cad5 100644
--- a/http.c
+++ b/http.c
@@ -22,6 +22,7 @@
#include "object-file.h"
#include "odb.h"
#include "tempfile.h"
+#include "date.h"
static struct trace_key trace_curl = TRACE_KEY_INIT(CURL);
static int trace_curl_data = 1;
@@ -149,6 +150,14 @@ static char *cached_accept_language;
static char *http_ssl_backend;
static int http_schannel_check_revoke = 1;
+
+/* Retry configuration */
+static long http_retry_after = -1; /* Default retry-after in seconds when header is missing (-1 means not set, exit with 128) */
+static long http_max_retries = 0; /* Maximum number of retry attempts (0 means retries are disabled) */
+static long http_max_retry_time = 300; /* Maximum time to wait for a single retry (default 5 minutes) */
+
+/* Store retry_after value from 429 responses for retry logic (-1 = not set, 0 = retry immediately, >0 = delay in seconds) */
+static long last_retry_after = -1;
/*
* With the backend being set to `schannel`, setting sslCAinfo would override
* the Certificate Store in cURL v7.60.0 and later, which is not what we want
@@ -209,13 +218,14 @@ static inline int is_hdr_continuation(const char *ptr, const size_t size)
return size && (*ptr == ' ' || *ptr == '\t');
}
-static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p UNUSED)
+static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)
{
size_t size = eltsize * nmemb;
struct strvec *values = &http_auth.wwwauth_headers;
struct strbuf buf = STRBUF_INIT;
const char *val;
size_t val_len;
+ struct active_request_slot *slot = (struct active_request_slot *)p;
/*
* Header lines may not come NULL-terminated from libcurl so we must
@@ -257,6 +267,47 @@ static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p UN
goto exit;
}
+ /* Parse Retry-After header for rate limiting */
+ if (skip_iprefix_mem(ptr, size, "retry-after:", &val, &val_len)) {
+ strbuf_add(&buf, val, val_len);
+ strbuf_trim(&buf);
+
+ if (slot && slot->results) {
+ /* Parse the retry-after value (delay-seconds or HTTP-date) */
+ char *endptr;
+ long retry_after;
+
+ errno = 0;
+ retry_after = strtol(buf.buf, &endptr, 10);
+
+ /* Check if it's a valid integer (delay-seconds format) */
+ if (endptr != buf.buf && *endptr == '\0' &&
+ errno != ERANGE && retry_after > 0) {
+ slot->results->retry_after = retry_after;
+ } else {
+ /* Try parsing as HTTP-date format */
+ timestamp_t timestamp;
+ int offset;
+ if (!parse_date_basic(buf.buf, ×tamp, &offset)) {
+ /* Successfully parsed as date, calculate delay from now */
+ timestamp_t now = time(NULL);
+ if (timestamp > now) {
+ slot->results->retry_after = (long)(timestamp - now);
+ } else {
+ /* Past date means retry immediately */
+ slot->results->retry_after = 0;
+ }
+ } else {
+ /* Failed to parse as either delay-seconds or HTTP-date */
+ warning(_("unable to parse Retry-After header value: '%s'"), buf.buf);
+ }
+ }
+ }
+
+ http_auth.header_is_last_match = 1;
+ goto exit;
+ }
+
/*
* This line could be a continuation of the previously matched header
* field. If this is the case then we should append this value to the
@@ -575,6 +626,21 @@ static int http_options(const char *var, const char *value,
return 0;
}
+ if (!strcmp("http.retryafter", var)) {
+ http_retry_after = git_config_int(var, value, ctx->kvi);
+ return 0;
+ }
+
+ if (!strcmp("http.maxretries", var)) {
+ http_max_retries = git_config_int(var, value, ctx->kvi);
+ return 0;
+ }
+
+ if (!strcmp("http.maxretrytime", var)) {
+ http_max_retry_time = git_config_int(var, value, ctx->kvi);
+ return 0;
+ }
+
/* Fall back on the default ones */
return git_default_config(var, value, ctx, data);
}
@@ -1422,6 +1488,10 @@ void http_init(struct remote *remote, const char *url, int proactive_auth)
set_long_from_env(&curl_tcp_keepintvl, "GIT_TCP_KEEPINTVL");
set_long_from_env(&curl_tcp_keepcnt, "GIT_TCP_KEEPCNT");
+ set_long_from_env(&http_retry_after, "GIT_HTTP_RETRY_AFTER");
+ set_long_from_env(&http_max_retries, "GIT_HTTP_MAX_RETRIES");
+ set_long_from_env(&http_max_retry_time, "GIT_HTTP_MAX_RETRY_TIME");
+
curl_default = get_curl_handle();
}
@@ -1871,6 +1941,10 @@ static int handle_curl_result(struct slot_results *results)
}
return HTTP_REAUTH;
}
+ } else if (results->http_code == 429) {
+ /* Store the retry_after value for use in retry logic */
+ last_retry_after = results->retry_after;
+ return HTTP_RATE_LIMITED;
} else {
if (results->http_connectcode == 407)
credential_reject(the_repository, &proxy_auth);
@@ -1886,6 +1960,8 @@ int run_one_slot(struct active_request_slot *slot,
struct slot_results *results)
{
slot->results = results;
+ /* Initialize retry_after to -1 (not set) */
+ results->retry_after = -1;
if (!start_active_slot(slot)) {
xsnprintf(curl_errorstr, sizeof(curl_errorstr),
"failed to start HTTP request");
@@ -2149,6 +2225,7 @@ static int http_request(const char *url,
}
curl_easy_setopt(slot->curl, CURLOPT_HEADERFUNCTION, fwrite_wwwauth);
+ curl_easy_setopt(slot->curl, CURLOPT_HEADERDATA, slot);
accept_language = http_get_accept_language_header();
@@ -2253,19 +2330,36 @@ static int update_url_from_redirect(struct strbuf *base,
return 1;
}
+/*
+ * Sleep for the specified number of seconds before retrying.
+ */
+static void sleep_for_retry(long retry_after)
+{
+ if (retry_after > 0) {
+ unsigned int remaining;
+ warning(_("rate limited, waiting %ld seconds before retry"), retry_after);
+ remaining = sleep(retry_after);
+ while (remaining > 0) {
+ /* Sleep was interrupted, continue sleeping */
+ remaining = sleep(remaining);
+ }
+ }
+}
+
static int http_request_reauth(const char *url,
void *result, int target,
struct http_get_options *options)
{
int i = 3;
int ret;
+ int rate_limit_retries = http_max_retries;
if (always_auth_proactively())
credential_fill(the_repository, &http_auth, 1);
ret = http_request(url, result, target, options);
- if (ret != HTTP_OK && ret != HTTP_REAUTH)
+ if (ret != HTTP_OK && ret != HTTP_REAUTH && ret != HTTP_RATE_LIMITED)
return ret;
if (options && options->effective_url && options->base_url) {
@@ -2276,7 +2370,7 @@ static int http_request_reauth(const char *url,
}
}
- while (ret == HTTP_REAUTH && --i) {
+ while ((ret == HTTP_REAUTH || ret == HTTP_RATE_LIMITED) && --i) {
/*
* The previous request may have put cruft into our output stream; we
* should clear it out before making our next request.
@@ -2302,7 +2396,54 @@ static int http_request_reauth(const char *url,
BUG("Unknown http_request target");
}
- credential_fill(the_repository, &http_auth, 1);
+ if (ret == HTTP_RATE_LIMITED) {
+ /* Handle rate limiting with retry logic */
+ int retry_attempt = http_max_retries - rate_limit_retries + 1;
+
+ if (rate_limit_retries <= 0) {
+ /* Retries are disabled or exhausted */
+ if (http_max_retries > 0) {
+ error(_("too many rate limit retries, giving up"));
+ }
+ return HTTP_ERROR;
+ }
+
+ /* Decrement retries counter */
+ rate_limit_retries--;
+
+ /* Use the stored retry_after value or configured default */
+ if (last_retry_after >= 0) {
+ /* Check if retry delay exceeds maximum allowed */
+ if (last_retry_after > http_max_retry_time) {
+ error(_("rate limited (HTTP 429) requested %ld second delay, "
+ "exceeds http.maxRetryTime of %ld seconds"),
+ last_retry_after, http_max_retry_time);
+ last_retry_after = -1; /* Reset after use */
+ return HTTP_ERROR;
+ }
+ sleep_for_retry(last_retry_after);
+ last_retry_after = -1; /* Reset after use */
+ } else {
+ /* No Retry-After header provided */
+ if (http_retry_after < 0) {
+ /* Not configured - exit with error */
+ error(_("rate limited (HTTP 429) and no Retry-After header provided. "
+ "Configure http.retryAfter or set GIT_HTTP_RETRY_AFTER."));
+ return HTTP_ERROR;
+ }
+ /* Check if configured default exceeds maximum allowed */
+ if (http_retry_after > http_max_retry_time) {
+ error(_("configured http.retryAfter (%ld seconds) exceeds "
+ "http.maxRetryTime (%ld seconds)"),
+ http_retry_after, http_max_retry_time);
+ return HTTP_ERROR;
+ }
+ /* Use configured default retry-after value */
+ sleep_for_retry(http_retry_after);
+ }
+ } else if (ret == HTTP_REAUTH) {
+ credential_fill(the_repository, &http_auth, 1);
+ }
ret = http_request(url, result, target, options);
}
diff --git a/http.h b/http.h
index f9d4593404..eb40456450 100644
--- a/http.h
+++ b/http.h
@@ -20,6 +20,7 @@ struct slot_results {
long http_code;
long auth_avail;
long http_connectcode;
+ long retry_after;
};
struct active_request_slot {
@@ -167,6 +168,7 @@ struct http_get_options {
#define HTTP_REAUTH 4
#define HTTP_NOAUTH 5
#define HTTP_NOMATCHPUBLICKEY 6
+#define HTTP_RATE_LIMITED 7
/*
* Requests a URL and stores the result in a strbuf.
diff --git a/remote-curl.c b/remote-curl.c
index 69f919454a..5959461cd3 100644
--- a/remote-curl.c
+++ b/remote-curl.c
@@ -529,6 +529,10 @@ static struct discovery *discover_refs(const char *service, int for_push)
show_http_message(&type, &charset, &buffer);
die(_("unable to access '%s' with http.pinnedPubkey configuration: %s"),
transport_anonymize_url(url.buf), curl_errorstr);
+ case HTTP_RATE_LIMITED:
+ show_http_message(&type, &charset, &buffer);
+ die(_("rate limited by '%s', please try again later"),
+ transport_anonymize_url(url.buf));
default:
show_http_message(&type, &charset, &buffer);
die(_("unable to access '%s': %s"),
diff --git a/t/meson.build b/t/meson.build
index dc43d69636..98bd6949e6 100644
--- a/t/meson.build
+++ b/t/meson.build
@@ -698,6 +698,7 @@ integration_tests = [
't5581-http-curl-verbose.sh',
't5582-fetch-negative-refspec.sh',
't5583-push-branches.sh',
+ 't5584-http-429-retry.sh',
't5600-clone-fail-cleanup.sh',
't5601-clone.sh',
't5602-clone-remote-exec.sh',
diff --git a/t/t5584-http-429-retry.sh b/t/t5584-http-429-retry.sh
new file mode 100755
index 0000000000..8bcc382763
--- /dev/null
+++ b/t/t5584-http-429-retry.sh
@@ -0,0 +1,429 @@
+#!/bin/sh
+
+test_description='test HTTP 429 Too Many Requests retry logic'
+
+. ./test-lib.sh
+
+. "$TEST_DIRECTORY"/lib-httpd.sh
+
+start_httpd
+
+test_expect_success 'setup test repository' '
+ test_commit initial &&
+ git clone --bare . "$HTTPD_DOCUMENT_ROOT_PATH/repo.git" &&
+ git --git-dir="$HTTPD_DOCUMENT_ROOT_PATH/repo.git" config http.receivepack true
+'
+
+test_expect_success 'HTTP 429 with retries disabled (maxRetries=0) fails immediately' '
+ write_script "$HTTPD_ROOT_PATH/one-time-script" <<-\EOF &&
+ printf "Status: 429 Too Many Requests\r\n"
+ printf "Retry-After: 1\r\n"
+ printf "Content-Type: text/plain\r\n"
+ printf "\r\n"
+ printf "Rate limited\n"
+ cat "$1" >/dev/null
+ EOF
+
+ # Set maxRetries to 0 (disabled)
+ test_config http.maxRetries 0 &&
+ test_config http.retryAfter 1 &&
+
+ # Should fail immediately without any retry attempt
+ test_must_fail git ls-remote "$HTTPD_URL/one_time_script/repo.git" 2>err &&
+
+ # Verify no retry happened (no "waiting" message in stderr)
+ ! grep -i "waiting.*retry" err &&
+
+ # The one-time script will be consumed on first request (not a retry)
+ test_path_is_missing "$HTTPD_ROOT_PATH/one-time-script"
+'
+
+test_expect_success 'HTTP 429 permanent should fail after max retries' '
+ # Install a permanent error script to prove retries are limited
+ write_script "$HTTPD_ROOT_PATH/http-429-permanent.sh" <<-\EOF &&
+ printf "Status: 429 Too Many Requests\r\n"
+ printf "Retry-After: 1\r\n"
+ printf "Content-Type: text/plain\r\n"
+ printf "\r\n"
+ printf "Permanently rate limited\n"
+ EOF
+
+ # Enable retries with a limit
+ test_config http.maxRetries 2 &&
+
+ # Git should retry but eventually fail when 429 persists
+ test_must_fail git ls-remote "$HTTPD_URL/error/http-429-permanent.sh/repo.git" 2>err
+'
+
+test_expect_success 'HTTP 429 with Retry-After is retried and succeeds' '
+ # Create a one-time script that returns 429 with Retry-After header
+ # on the first request. Subsequent requests will succeed.
+ # This contrasts with the permanent 429 above - proving retry works
+ write_script "$HTTPD_ROOT_PATH/one-time-script" <<-\EOF &&
+ # Return HTTP 429 response instead of git response
+ printf "Status: 429 Too Many Requests\r\n"
+ printf "Retry-After: 1\r\n"
+ printf "Content-Type: text/plain\r\n"
+ printf "\r\n"
+ printf "Rate limited - please retry after 1 second\n"
+ # Output something different from input so the script gets removed
+ cat "$1" >/dev/null
+ EOF
+
+ # Enable retries
+ test_config http.maxRetries 3 &&
+
+ # Git should retry after receiving 429 and eventually succeed
+ git ls-remote "$HTTPD_URL/one_time_script/repo.git" >output 2>err &&
+ test_grep "refs/heads/" output &&
+
+ # The one-time script should have been consumed (proving retry happened)
+ test_path_is_missing "$HTTPD_ROOT_PATH/one-time-script"
+'
+
+test_expect_success 'HTTP 429 without Retry-After uses configured default' '
+ write_script "$HTTPD_ROOT_PATH/one-time-script" <<-\EOF &&
+ printf "Status: 429 Too Many Requests\r\n"
+ printf "Content-Type: text/plain\r\n"
+ printf "\r\n"
+ printf "Rate limited - no retry info\n"
+ cat "$1" >/dev/null
+ EOF
+
+ # Enable retries and configure default delay
+ test_config http.maxRetries 3 &&
+ test_config http.retryAfter 1 &&
+
+ # Git should retry using configured default and succeed
+ git ls-remote "$HTTPD_URL/one_time_script/repo.git" >output 2>err &&
+ test_grep "refs/heads/" output &&
+ test_path_is_missing "$HTTPD_ROOT_PATH/one-time-script"
+'
+
+test_expect_success 'HTTP 429 retry delays are respected' '
+ write_script "$HTTPD_ROOT_PATH/one-time-script" <<-\EOF &&
+ printf "Status: 429 Too Many Requests\r\n"
+ printf "Retry-After: 2\r\n"
+ printf "Content-Type: text/plain\r\n"
+ printf "\r\n"
+ printf "Rate limited\n"
+ cat "$1" >/dev/null
+ EOF
+
+ # Enable retries
+ test_config http.maxRetries 3 &&
+
+ # Time the operation - it should take at least 2 seconds due to retry delay
+ start=$(date +%s) &&
+ git ls-remote "$HTTPD_URL/one_time_script/repo.git" >output 2>err &&
+ end=$(date +%s) &&
+ duration=$((end - start)) &&
+
+ # Verify it took at least 2 seconds (allowing some tolerance)
+ test "$duration" -ge 1 &&
+ test_grep "refs/heads/" output &&
+ test_path_is_missing "$HTTPD_ROOT_PATH/one-time-script"
+'
+
+test_expect_success 'HTTP 429 fails immediately if Retry-After exceeds http.maxRetryTime' '
+ write_script "$HTTPD_ROOT_PATH/one-time-script" <<-\EOF &&
+ printf "Status: 429 Too Many Requests\r\n"
+ printf "Retry-After: 100\r\n"
+ printf "Content-Type: text/plain\r\n"
+ printf "\r\n"
+ printf "Rate limited with long delay\n"
+ cat "$1" >/dev/null
+ EOF
+
+ # Configure max retry time to 3 seconds (much less than requested 100)
+ test_config http.maxRetries 3 &&
+ test_config http.maxRetryTime 3 &&
+
+ # Should fail immediately without waiting
+ start=$(date +%s) &&
+ test_must_fail git ls-remote "$HTTPD_URL/one_time_script/repo.git" 2>err &&
+ end=$(date +%s) &&
+ duration=$((end - start)) &&
+
+ # Should fail quickly (less than 2 seconds, no 100 second wait)
+ test "$duration" -lt 2 &&
+ test_grep "exceeds http.maxRetryTime" err &&
+
+ # The one-time script will be consumed on first request
+ test_path_is_missing "$HTTPD_ROOT_PATH/one-time-script"
+'
+
+test_expect_success 'HTTP 429 fails if configured http.retryAfter exceeds http.maxRetryTime' '
+ # Test misconfiguration: retryAfter > maxRetryTime
+ write_script "$HTTPD_ROOT_PATH/one-time-script" <<-\EOF &&
+ printf "Status: 429 Too Many Requests\r\n"
+ printf "Content-Type: text/plain\r\n"
+ printf "\r\n"
+ printf "Rate limited without header\n"
+ cat "$1" >/dev/null
+ EOF
+
+ # Configure retryAfter larger than maxRetryTime
+ test_config http.maxRetries 3 &&
+ test_config http.retryAfter 100 &&
+ test_config http.maxRetryTime 5 &&
+
+ # Should fail immediately with configuration error
+ start=$(date +%s) &&
+ test_must_fail git ls-remote "$HTTPD_URL/one_time_script/repo.git" 2>err &&
+ end=$(date +%s) &&
+ duration=$((end - start)) &&
+
+ # Should fail quickly
+ test "$duration" -lt 2 &&
+ test_grep "configured http.retryAfter.*exceeds.*http.maxRetryTime" err
+'
+
+test_expect_success 'HTTP 429 with Retry-After HTTP-date format' '
+ # Test HTTP-date format (RFC 2822) in Retry-After header
+ # Generate a date 2 seconds in the future
+ future_date=$(TZ=GMT date -d "+2 seconds" "+%a, %d %b %Y %H:%M:%S GMT" 2>/dev/null || \
+ TZ=GMT date -v+2S "+%a, %d %b %Y %H:%M:%S GMT" 2>/dev/null || \
+ echo "skip") &&
+
+ if test "$future_date" = "skip"
+ then
+ skip_all="date command does not support required format" &&
+ test_done
+ fi &&
+
+ write_script "$HTTPD_ROOT_PATH/one-time-script" <<-EOF &&
+ printf "Status: 429 Too Many Requests\\r\\n"
+ printf "Retry-After: $future_date\\r\\n"
+ printf "Content-Type: text/plain\\r\\n"
+ printf "\\r\\n"
+ printf "Rate limited with HTTP-date\\n"
+ cat "\$1" >/dev/null
+ EOF
+
+ # Enable retries
+ test_config http.maxRetries 3 &&
+
+ # Git should parse the HTTP-date and retry after the delay
+ start=$(date +%s) &&
+ git ls-remote "$HTTPD_URL/one_time_script/repo.git" >output 2>err &&
+ end=$(date +%s) &&
+ duration=$((end - start)) &&
+
+ # Should take at least 1 second (allowing tolerance for processing time)
+ test "$duration" -ge 1 &&
+ test_grep "refs/heads/" output &&
+ test_path_is_missing "$HTTPD_ROOT_PATH/one-time-script"
+'
+
+test_expect_success 'HTTP 429 with HTTP-date exceeding maxRetryTime fails immediately' '
+ # Generate a date 200 seconds in the future
+ future_date=$(TZ=GMT date -d "+200 seconds" "+%a, %d %b %Y %H:%M:%S GMT" 2>/dev/null || \
+ TZ=GMT date -v+200S "+%a, %d %b %Y %H:%M:%S GMT" 2>/dev/null || \
+ echo "skip") &&
+
+ if test "$future_date" = "skip"
+ then
+ skip_all="date command does not support required format" &&
+ test_done
+ fi &&
+
+ write_script "$HTTPD_ROOT_PATH/one-time-script" <<-EOF &&
+ printf "Status: 429 Too Many Requests\\r\\n"
+ printf "Retry-After: $future_date\\r\\n"
+ printf "Content-Type: text/plain\\r\\n"
+ printf "\\r\\n"
+ printf "Rate limited with long HTTP-date\\n"
+ cat "\$1" >/dev/null
+ EOF
+
+ # Configure max retry time much less than the 200 second delay
+ test_config http.maxRetries 3 &&
+ test_config http.maxRetryTime 10 &&
+
+ # Should fail immediately without waiting 200 seconds
+ start=$(date +%s) &&
+ test_must_fail git ls-remote "$HTTPD_URL/one_time_script/repo.git" 2>err &&
+ end=$(date +%s) &&
+ duration=$((end - start)) &&
+
+ # Should fail quickly (not wait 200 seconds)
+ test "$duration" -lt 2 &&
+ test_grep "exceeds http.maxRetryTime" err &&
+
+ # The one-time script will be consumed on first request
+ test_path_is_missing "$HTTPD_ROOT_PATH/one-time-script"
+'
+
+test_expect_success 'HTTP 429 with past HTTP-date should not wait' '
+ past_date=$(TZ=GMT date -d "-10 seconds" "+%a, %d %b %Y %H:%M:%S GMT" 2>/dev/null || \
+ TZ=GMT date -v-10S "+%a, %d %b %Y %H:%M:%S GMT" 2>/dev/null || \
+ echo "skip") &&
+
+ if test "$past_date" = "skip"
+ then
+ skip_all="date command does not support required format" &&
+ test_done
+ fi &&
+
+ write_script "$HTTPD_ROOT_PATH/one-time-script" <<-EOF &&
+ printf "Status: 429 Too Many Requests\\r\\n"
+ printf "Retry-After: $past_date\\r\\n"
+ printf "Content-Type: text/plain\\r\\n"
+ printf "\\r\\n"
+ printf "Rate limited with past date\\n"
+ cat "\$1" >/dev/null
+ EOF
+
+ # Enable retries
+ test_config http.maxRetries 3 &&
+
+ # Git should retry immediately without waiting
+ start=$(date +%s) &&
+ git ls-remote "$HTTPD_URL/one_time_script/repo.git" >output 2>err &&
+ end=$(date +%s) &&
+ duration=$((end - start)) &&
+
+ # Should complete quickly (less than 2 seconds)
+ test "$duration" -lt 2 &&
+ test_grep "refs/heads/" output &&
+ test_path_is_missing "$HTTPD_ROOT_PATH/one-time-script"
+'
+
+test_expect_success 'HTTP 429 with invalid Retry-After format uses configured default' '
+ write_script "$HTTPD_ROOT_PATH/one-time-script" <<-\EOF &&
+ printf "Status: 429 Too Many Requests\r\n"
+ printf "Retry-After: invalid-format-123abc\r\n"
+ printf "Content-Type: text/plain\r\n"
+ printf "\r\n"
+ printf "Rate limited with malformed header\n"
+ cat "$1" >/dev/null
+ EOF
+
+ # Configure default retry-after
+ test_config http.maxRetries 3 &&
+ test_config http.retryAfter 1 &&
+
+ # Should use configured default (1 second) since header is invalid
+ start=$(date +%s) &&
+ git ls-remote "$HTTPD_URL/one_time_script/repo.git" >output 2>err &&
+ end=$(date +%s) &&
+ duration=$((end - start)) &&
+
+ # Should take at least 1 second (the configured default)
+ test "$duration" -ge 1 &&
+ test_grep "refs/heads/" output &&
+ test_grep "waiting.*retry" err &&
+ test_path_is_missing "$HTTPD_ROOT_PATH/one-time-script"
+'
+
+test_expect_success 'HTTP 429 will not be retried without config' '
+ # Default config means http.maxRetries=0 (retries disabled)
+ # When 429 is received, it should fail immediately without retry
+ write_script "$HTTPD_ROOT_PATH/one-time-script" <<-\EOF &&
+ printf "Status: 429 Too Many Requests\r\n"
+ printf "Retry-After: 1\r\n"
+ printf "Content-Type: text/plain\r\n"
+ printf "\r\n"
+ printf "Rate limited\n"
+ cat "$1" >/dev/null
+ EOF
+
+ # Do NOT configure anything - use defaults (http.maxRetries defaults to 0)
+
+ # Should fail immediately without retry
+ test_must_fail git ls-remote "$HTTPD_URL/one_time_script/repo.git" 2>err &&
+
+ # Verify no retry happened (no "waiting" message)
+ ! grep -i "waiting.*retry" err &&
+
+ # Should get 429 error
+ test_grep "429" err &&
+
+ # The one-time script should be consumed on first request
+ test_path_is_missing "$HTTPD_ROOT_PATH/one-time-script"
+'
+
+test_expect_success 'GIT_HTTP_RETRY_AFTER overrides http.retryAfter config' '
+ write_script "$HTTPD_ROOT_PATH/one-time-script" <<-\EOF &&
+ printf "Status: 429 Too Many Requests\r\n"
+ printf "Content-Type: text/plain\r\n"
+ printf "\r\n"
+ printf "Rate limited - no Retry-After header\n"
+ cat "$1" >/dev/null
+ EOF
+
+ # Configure retryAfter to 10 seconds
+ test_config http.maxRetries 3 &&
+ test_config http.retryAfter 10 &&
+
+ # Override with environment variable to 1 second
+ start=$(date +%s) &&
+ GIT_HTTP_RETRY_AFTER=1 git ls-remote "$HTTPD_URL/one_time_script/repo.git" >output 2>err &&
+ end=$(date +%s) &&
+ duration=$((end - start)) &&
+
+ # Should use env var (1 second), not config (10 seconds)
+ test "$duration" -ge 1 &&
+ test "$duration" -lt 5 &&
+ test_grep "refs/heads/" output &&
+ test_grep "waiting.*retry" err &&
+ test_path_is_missing "$HTTPD_ROOT_PATH/one-time-script"
+'
+
+test_expect_success 'GIT_HTTP_MAX_RETRIES overrides http.maxRetries config' '
+ write_script "$HTTPD_ROOT_PATH/one-time-script" <<-\EOF &&
+ printf "Status: 429 Too Many Requests\r\n"
+ printf "Retry-After: 1\r\n"
+ printf "Content-Type: text/plain\r\n"
+ printf "\r\n"
+ printf "Rate limited\n"
+ cat "$1" >/dev/null
+ EOF
+
+ # Configure maxRetries to 0 (disabled)
+ test_config http.maxRetries 0 &&
+ test_config http.retryAfter 1 &&
+
+ # Override with environment variable to enable retries
+ GIT_HTTP_MAX_RETRIES=3 git ls-remote "$HTTPD_URL/one_time_script/repo.git" >output 2>err &&
+
+ # Should retry (env var enables it despite config saying disabled)
+ test_grep "refs/heads/" output &&
+ test_grep "waiting.*retry" err &&
+ test_path_is_missing "$HTTPD_ROOT_PATH/one-time-script"
+'
+
+test_expect_success 'GIT_HTTP_MAX_RETRY_TIME overrides http.maxRetryTime config' '
+ write_script "$HTTPD_ROOT_PATH/one-time-script" <<-\EOF &&
+ printf "Status: 429 Too Many Requests\r\n"
+ printf "Retry-After: 50\r\n"
+ printf "Content-Type: text/plain\r\n"
+ printf "\r\n"
+ printf "Rate limited with long delay\n"
+ cat "$1" >/dev/null
+ EOF
+
+ # Configure maxRetryTime to 100 seconds (would accept 50 second delay)
+ test_config http.maxRetries 3 &&
+ test_config http.maxRetryTime 100 &&
+
+ # Override with environment variable to 10 seconds (should reject 50 second delay)
+ start=$(date +%s) &&
+ test_must_fail env GIT_HTTP_MAX_RETRY_TIME=10 \
+ git ls-remote "$HTTPD_URL/one_time_script/repo.git" 2>err &&
+ end=$(date +%s) &&
+ duration=$((end - start)) &&
+
+ # Should fail quickly (not wait 50 seconds) because env var limits to 10
+ test "$duration" -lt 5 &&
+ test_grep "exceeds http.maxRetryTime" err &&
+ test_path_is_missing "$HTTPD_ROOT_PATH/one-time-script"
+'
+
+test_expect_success 'verify normal repository access still works' '
+ git ls-remote "$HTTPD_URL/smart/repo.git" >output &&
+ test_grep "refs/heads/" output
+'
+
+test_done
--
gitgitgadget
next prev parent reply other threads:[~2025-11-26 12:30 UTC|newest]
Thread overview: 49+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-11-26 12:30 [PATCH 0/3] http: add support for HTTP 429 rate limit retries Vaidas Pilkauskas via GitGitGadget
2025-11-26 12:30 ` Vaidas Pilkauskas via GitGitGadget [this message]
2025-12-09 23:15 ` [PATCH 1/3] " Taylor Blau
2025-12-12 12:36 ` Vaidas Pilkauskas
2025-11-26 12:30 ` [PATCH 2/3] remote-curl: fix memory leak in show_http_message() Vaidas Pilkauskas via GitGitGadget
2025-12-09 23:52 ` Taylor Blau
2025-11-26 12:30 ` [PATCH 3/3] http: add trace2 logging for retry operations Vaidas Pilkauskas via GitGitGadget
2025-12-18 14:44 ` [PATCH v2 0/2] http: add support for HTTP 429 rate limit retries Vaidas Pilkauskas via GitGitGadget
2025-12-18 14:44 ` [PATCH v2 1/2] " Vaidas Pilkauskas via GitGitGadget
2026-02-11 1:05 ` Taylor Blau
2026-02-11 9:13 ` Jeff King
2026-02-13 13:41 ` Vaidas Pilkauskas
2026-02-15 9:13 ` Jeff King
2026-02-13 13:30 ` Vaidas Pilkauskas
2025-12-18 14:44 ` [PATCH v2 2/2] http: add trace2 logging for retry operations Vaidas Pilkauskas via GitGitGadget
2026-02-11 1:06 ` Taylor Blau
2026-02-17 11:08 ` [PATCH v3 0/3] http: add support for HTTP 429 rate limit retries Vaidas Pilkauskas via GitGitGadget
2026-02-17 11:08 ` [PATCH v3 1/3] strbuf: fix incorrect alloc size in strbuf_reencode() Vaidas Pilkauskas via GitGitGadget
2026-02-17 20:51 ` Junio C Hamano
2026-02-18 13:43 ` Vaidas Pilkauskas
2026-02-17 11:08 ` [PATCH v3 2/3] remote-curl: introduce show_http_message_fatal() helper Vaidas Pilkauskas via GitGitGadget
2026-02-17 11:08 ` [PATCH v3 3/3] http: add support for HTTP 429 rate limit retries Vaidas Pilkauskas via GitGitGadget
2026-02-18 14:09 ` [PATCH v4 0/5] " Vaidas Pilkauskas via GitGitGadget
2026-02-18 14:09 ` [PATCH v4 1/5] strbuf: pass correct alloc to strbuf_attach() in strbuf_reencode() Vaidas Pilkauskas via GitGitGadget
2026-02-18 14:09 ` [PATCH v4 2/5] strbuf_attach: fix all call sites to pass correct alloc Vaidas Pilkauskas via GitGitGadget
2026-02-20 22:55 ` Junio C Hamano
2026-02-23 12:49 ` Vaidas Pilkauskas
2026-02-18 14:09 ` [PATCH v4 3/5] strbuf: replace strbuf_grow() in strbuf_attach() with BUG() check Vaidas Pilkauskas via GitGitGadget
2026-02-18 14:09 ` [PATCH v4 4/5] remote-curl: introduce show_http_message_fatal() helper Vaidas Pilkauskas via GitGitGadget
2026-02-18 14:09 ` [PATCH v4 5/5] http: add support for HTTP 429 rate limit retries Vaidas Pilkauskas via GitGitGadget
2026-02-23 14:20 ` [PATCH v5 0/4] " Vaidas Pilkauskas via GitGitGadget
2026-02-23 14:20 ` [PATCH v5 1/4] strbuf: pass correct alloc to strbuf_attach() in strbuf_reencode() Vaidas Pilkauskas via GitGitGadget
2026-02-23 14:20 ` [PATCH v5 2/4] strbuf_attach: fix call sites to pass correct alloc Vaidas Pilkauskas via GitGitGadget
2026-02-23 14:20 ` [PATCH v5 3/4] remote-curl: introduce show_http_message_fatal() helper Vaidas Pilkauskas via GitGitGadget
2026-03-10 17:44 ` Jeff King
2026-02-23 14:20 ` [PATCH v5 4/4] http: add support for HTTP 429 rate limit retries Vaidas Pilkauskas via GitGitGadget
2026-03-10 19:07 ` Jeff King
2026-02-24 0:07 ` [PATCH v5 0/4] " Junio C Hamano
2026-03-09 23:34 ` Junio C Hamano
2026-03-10 19:10 ` Jeff King
2026-03-10 19:19 ` Junio C Hamano
2026-03-17 13:00 ` [PATCH v6 0/3] " Vaidas Pilkauskas via GitGitGadget
2026-03-17 13:00 ` [PATCH v6 1/3] strbuf: pass correct alloc to strbuf_attach() in strbuf_reencode() Vaidas Pilkauskas via GitGitGadget
2026-03-17 13:00 ` [PATCH v6 2/3] strbuf_attach: fix call sites to pass correct alloc Vaidas Pilkauskas via GitGitGadget
2026-03-17 13:00 ` [PATCH v6 3/3] http: add support for HTTP 429 rate limit retries Vaidas Pilkauskas via GitGitGadget
2026-03-21 3:30 ` Taylor Blau
2026-03-21 3:31 ` [PATCH v6 0/3] " Taylor Blau
2026-03-21 4:57 ` Junio C Hamano
2026-03-23 6:58 ` Vaidas Pilkauskas
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=ae0087cd1c7fbb6b748d6767b476c1bd1a19996f.1764160227.git.gitgitgadget@gmail.com \
--to=gitgitgadget@gmail$(echo .)com \
--cc=git@vger$(echo .)kernel.org \
--cc=vaidas.pilkauskas@shopify$(echo .)com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox