From: Karthik Nayak <karthik.188@gmail•com>
To: git@vger•kernel.org
Cc: gitster@pobox•com, ps@pks•im,
"Karthik Nayak" <karthik.188@gmail•com>,
"Jean-Noël Avila" <jn.avila@free•fr>
Subject: [PATCH v4 0/4] refs: allow setting the reference directory
Date: Mon, 02 Feb 2026 13:26:29 +0100 [thread overview]
Message-ID: <20260202-kn-alternate-ref-dir-v4-0-3b30430411e3@gmail.com> (raw)
In-Reply-To: <20251119-kn-alternate-ref-dir-v1-0-4cf4a94c8bed@gmail.com>
While Git allows users to select different reference backends, unlike
with objects, there is no flexibility in selecting the reference
directory. Currently, the reference format is obtained from the config
of the repository and the reference directory is set to the $GIT_DIR.
This patch extends the config 'extensions.refStorage' to take in
multiple forms of inputs:
- A format name alone (e.g., `reftable` or `files`) which uses the
default location (the repository's common directory).
- A URI format `<format>://<location>` which explicitly specifies
both the format and location (e.g., `reftable:///foo/bar`).
We also add in a new ENV variable GIT_REFERENCE_BACKEND which can be
used to override the config.
One use case for this is migration between different backends. On the
server side, migrating from the files backend to the newly introduced
reftable backend can be achieved by running 'git refs migrate'. However,
for large repositories with millions of references, this migration can
take from seconds to minutes.
For some background, at GitLab, the criteria for our migration was to
reduce the downtime of the migrate ideally to zero. So running 'git refs
migrate --ref-format=reftable' by itself wouldn't work, since it scales
with the number of references and we have repos with millions of
references, so we need to migrate without loosing any information. We
came up with the following plan:
1. Run git-pack-refs(1) and note timestamp of the generated packed-refs
file.
2. Run git refs migrate –dry-run.
3. If there are no ongoing reference requests (read/write)
a. Lock the repository by blocking incoming requests (done on a
layer above git, in Gitaly [1]).
b. If the timestamp of the packed-refs file has changed, unlock
the repo and repeat from step 1.
c. Apply all the loose refs to the dry-run reftable folder (this
requires support in Git to write refs to arbitrary folder).
d. Move the reftable dry-run folder into the GIT_DIR.
e. Swap the repo config
f. Unlock repo access
Using such a route, scales much better since we only have to worry about
blocking the repository by O(ref written between #1 and #3a) and not
O(refs in repo). But for doing so, we need to be able to write to a
arbitrary reference backend + path. This is to add the missing
references to the dry-run reftable folder. This series, achieves that.
Since there was a long gap between v3 <> v4, the version 4 onward is
based on top of 2258446484 (RelNotes: correct "fast-import" option name,
2026-01-30).
[1]: https://gitlab.com/gitlab-org/gitaly
---
Changes in v4:
- Mostly re-wrote the code to also support worktree. Now, the existing
backends will store worktree references in 'ref_dir/worktrees/wt_id'
and add corresponding stubs in 'git_dir/worktrees/wt_id'.
- We also support relative paths in the reference directories. These
relative paths are resolved relative to the GIT_DIR.
- Link to v3: https://patch.msgid.link/20251201-kn-alternate-ref-dir-v3-0-c11b946bc2fa@gmail.com
Changes in v3:
- Cleanup some stale code which wasn't removed.
- Localize strings which will be output to the user.
- Remove additional defensive checks which are not needed.
- Link to v2: https://patch.msgid.link/20251126-kn-alternate-ref-dir-v2-0-8b9f6f18f635@gmail.com
Changes in v2:
- Added more clarification and proper intent in the cover message.
- Changed the format from '<ref_backend>://<path>' to
`<ref_backend>://<URI-for-resource>` as it much clearer.
- Added logic to check for the '//' in the provided URI and a test for
the same.
- In the tests:
- Use test_must_fail() instead of ! git
- Fix looped tests not using the variables correctly and ensure that
the test description is correct.
- Link to v1: https://patch.msgid.link/20251119-kn-alternate-ref-dir-v1-0-4cf4a94c8bed@gmail.com
---
Documentation/config/extensions.adoc | 16 ++-
Documentation/git.adoc | 5 +
builtin/clone.c | 10 +-
builtin/worktree.c | 35 +++++++
environment.h | 1 +
refs.c | 61 +++++++++++-
refs.h | 13 +++
refs/files-backend.c | 18 +++-
refs/packed-backend.c | 1 +
refs/packed-backend.h | 1 +
refs/refs-internal.h | 15 +++
refs/reftable-backend.c | 38 +++----
repository.c | 9 +-
repository.h | 5 +-
setup.c | 59 ++++++++++-
setup.h | 2 +
t/meson.build | 1 +
t/t1423-ref-backend.sh | 187 +++++++++++++++++++++++++++++++++++
18 files changed, 433 insertions(+), 44 deletions(-)
Karthik Nayak (4):
refs: allow reference location in refstorage config
refs: extract out `refs_create_refdir_stubs()`
refs: parse and use the reference storage payload
refs: add GIT_REFERENCE_BACKEND to specify reference backend
Range-diff versus v3:
1: a91a4d5ba3 < -: ---------- refs: support obtaining ref_store for given dir
2: a9f126d724 < -: ---------- refs: add GIT_REF_URI to specify reference backend and directory
-: ---------- > 1: c8ddb8ec8d refs: allow reference location in refstorage config
-: ---------- > 2: 02acdc16b4 refs: extract out `refs_create_refdir_stubs()`
-: ---------- > 3: 1b0e17e705 refs: parse and use the reference storage payload
-: ---------- > 4: 8ca379edd6 refs: add GIT_REFERENCE_BACKEND to specify reference backend
base-commit: 22584464849815268419fd9d2eba307362360db1
change-id: 20251105-kn-alternate-ref-dir-3e572e8cd0ef
Thanks
- Karthik
next prev parent reply other threads:[~2026-02-02 12:27 UTC|newest]
Thread overview: 131+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-11-19 21:48 [PATCH 0/2] refs: allow setting the reference directory Karthik Nayak
2025-11-19 21:48 ` [PATCH 1/2] refs: support obtaining ref_store for given dir Karthik Nayak
2025-11-20 19:05 ` Justin Tobler
2025-11-21 11:18 ` Karthik Nayak
2025-11-19 21:48 ` [PATCH 2/2] refs: add GIT_REF_URI to specify reference backend and directory Karthik Nayak
2025-11-19 22:13 ` Eric Sunshine
2025-11-19 23:01 ` Karthik Nayak
2025-11-20 10:00 ` Jean-Noël Avila
2025-11-21 11:21 ` Karthik Nayak
2025-11-20 19:38 ` Justin Tobler
2025-11-24 13:23 ` Karthik Nayak
2025-11-21 13:42 ` Toon Claes
2025-11-21 16:07 ` Junio C Hamano
2025-11-24 13:25 ` Karthik Nayak
2025-11-26 13:11 ` Toon Claes
2025-11-24 13:26 ` Karthik Nayak
2025-12-01 13:28 ` Patrick Steinhardt
2025-12-02 22:21 ` Karthik Nayak
2025-11-23 4:29 ` [PATCH 0/2] refs: allow setting the reference directory Junio C Hamano
2025-12-01 13:19 ` Patrick Steinhardt
2025-12-02 10:25 ` Junio C Hamano
2025-12-02 15:29 ` Karthik Nayak
2025-11-26 11:11 ` [PATCH v2 " Karthik Nayak
2025-11-26 11:12 ` [PATCH v2 1/2] refs: support obtaining ref_store for given dir Karthik Nayak
2025-11-26 15:16 ` Junio C Hamano
2025-11-26 11:12 ` [PATCH v2 2/2] refs: add GIT_REF_URI to specify reference backend and directory Karthik Nayak
2025-11-26 16:17 ` Junio C Hamano
2025-11-27 14:52 ` Karthik Nayak
2025-11-27 20:02 ` Junio C Hamano
2025-11-27 21:45 ` Karthik Nayak
2025-12-01 11:24 ` [PATCH v3 0/2] refs: allow setting the reference directory Karthik Nayak
2025-12-01 11:24 ` [PATCH v3 1/2] refs: support obtaining ref_store for given dir Karthik Nayak
2025-12-01 11:24 ` [PATCH v3 2/2] refs: add GIT_REF_URI to specify reference backend and directory Karthik Nayak
2026-01-05 15:13 ` [PATCH v3 0/2] refs: allow setting the reference directory Patrick Steinhardt
2026-01-05 20:13 ` Karthik Nayak
2026-01-20 21:03 ` Junio C Hamano
2026-01-22 12:36 ` Karthik Nayak
2026-02-02 12:26 ` Karthik Nayak [this message]
2026-02-02 12:26 ` [PATCH v4 1/4] refs: allow reference location in refstorage config Karthik Nayak
2026-02-06 14:33 ` Patrick Steinhardt
2026-02-09 12:25 ` Karthik Nayak
2026-02-02 12:26 ` [PATCH v4 2/4] refs: extract out `refs_create_refdir_stubs()` Karthik Nayak
2026-02-06 14:33 ` Patrick Steinhardt
2026-02-09 11:21 ` Karthik Nayak
2026-02-02 12:26 ` [PATCH v4 3/4] refs: parse and use the reference storage payload Karthik Nayak
2026-02-06 14:33 ` Patrick Steinhardt
2026-02-09 12:52 ` Karthik Nayak
2026-02-02 12:26 ` [PATCH v4 4/4] refs: add GIT_REFERENCE_BACKEND to specify reference backend Karthik Nayak
2026-02-06 14:33 ` Patrick Steinhardt
2026-02-09 12:53 ` Karthik Nayak
2026-02-06 14:33 ` [PATCH v4 0/4] refs: allow setting the reference directory Patrick Steinhardt
2026-02-06 17:50 ` Junio C Hamano
2026-02-09 12:53 ` Karthik Nayak
2026-02-09 15:58 ` [PATCH v5 " Karthik Nayak
2026-02-09 15:58 ` [PATCH v5 1/4] refs: extract out `refs_create_refdir_stubs()` Karthik Nayak
2026-02-09 15:58 ` [PATCH v5 2/4] refs: forward and use the reference storage payload Karthik Nayak
2026-02-09 16:34 ` Patrick Steinhardt
2026-02-10 10:09 ` Karthik Nayak
2026-02-10 22:46 ` Jeff King
2026-02-13 14:45 ` Karthik Nayak
2026-02-15 9:12 ` Jeff King
2026-02-09 15:58 ` [PATCH v5 3/4] refs: allow reference location in refstorage config Karthik Nayak
2026-02-09 16:34 ` Patrick Steinhardt
2026-02-10 13:02 ` Karthik Nayak
2026-02-10 22:44 ` Jeff King
2026-02-11 10:27 ` Karthik Nayak
2026-02-09 15:58 ` [PATCH v5 4/4] refs: add GIT_REFERENCE_BACKEND to specify reference backend Karthik Nayak
2026-02-09 16:34 ` [PATCH v5 0/4] refs: allow setting the reference directory Patrick Steinhardt
2026-02-09 18:02 ` Junio C Hamano
2026-02-10 13:02 ` Karthik Nayak
2026-02-10 15:35 ` Junio C Hamano
2026-02-14 22:34 ` [PATCH v6 0/6] " Karthik Nayak
2026-02-14 22:34 ` [PATCH v6 1/6] setup: don't modify repo in `create_reference_database()` Karthik Nayak
2026-02-17 7:24 ` Patrick Steinhardt
2026-02-17 9:15 ` Karthik Nayak
2026-02-14 22:34 ` [PATCH v6 2/6] refs: extract out `refs_create_refdir_stubs()` Karthik Nayak
2026-02-14 22:34 ` [PATCH v6 3/6] refs: receive and use the reference storage payload Karthik Nayak
2026-02-17 7:24 ` Patrick Steinhardt
2026-02-17 9:16 ` Karthik Nayak
2026-02-14 22:34 ` [PATCH v6 4/6] refs: move out stub modification to generic layer Karthik Nayak
2026-02-17 7:24 ` Patrick Steinhardt
2026-02-17 9:29 ` Karthik Nayak
2026-02-18 14:21 ` Toon Claes
2026-02-19 9:31 ` Karthik Nayak
2026-02-14 22:34 ` [PATCH v6 5/6] refs: allow reference location in refstorage config Karthik Nayak
2026-02-14 22:34 ` [PATCH v6 6/6] refs: add GIT_REFERENCE_BACKEND to specify reference backend Karthik Nayak
2026-02-17 7:24 ` Patrick Steinhardt
2026-02-17 9:32 ` Karthik Nayak
2026-02-17 10:15 ` Patrick Steinhardt
2026-02-18 15:27 ` Toon Claes
2026-02-19 9:35 ` Karthik Nayak
2026-02-19 9:38 ` [PATCH v7 0/6] refs: allow setting the reference directory Karthik Nayak
2026-02-19 9:38 ` [PATCH v7 1/6] setup: don't modify repo in `create_reference_database()` Karthik Nayak
2026-02-19 9:38 ` [PATCH v7 2/6] refs: extract out `refs_create_refdir_stubs()` Karthik Nayak
2026-02-19 9:38 ` [PATCH v7 3/6] refs: move out stub modification to generic layer Karthik Nayak
2026-02-20 15:21 ` Toon Claes
2026-02-19 9:38 ` [PATCH v7 4/6] refs: receive and use the reference storage payload Karthik Nayak
2026-02-20 15:32 ` Toon Claes
2026-02-22 20:12 ` Karthik Nayak
2026-02-19 9:38 ` [PATCH v7 5/6] refs: allow reference location in refstorage config Karthik Nayak
2026-02-20 15:36 ` Toon Claes
2026-02-20 16:53 ` Junio C Hamano
2026-02-22 20:15 ` Karthik Nayak
2026-02-19 9:38 ` [PATCH v7 6/6] refs: add GIT_REFERENCE_BACKEND to specify reference backend Karthik Nayak
2026-02-19 15:35 ` Patrick Steinhardt
2026-02-20 9:15 ` Karthik Nayak
2026-02-23 8:01 ` [PATCH v8 0/6] refs: allow setting the reference directory Karthik Nayak
2026-02-23 8:01 ` [PATCH v8 1/6] setup: don't modify repo in `create_reference_database()` Karthik Nayak
2026-02-23 8:01 ` [PATCH v8 2/6] refs: extract out `refs_create_refdir_stubs()` Karthik Nayak
2026-02-23 8:01 ` [PATCH v8 3/6] refs: move out stub modification to generic layer Karthik Nayak
2026-02-23 8:01 ` [PATCH v8 4/6] refs: receive and use the reference storage payload Karthik Nayak
2026-02-23 8:01 ` [PATCH v8 5/6] refs: allow reference location in refstorage config Karthik Nayak
2026-02-23 17:43 ` Kristoffer Haugsbakk
2026-02-24 13:09 ` Karthik Nayak
2026-02-24 13:20 ` Kristoffer Haugsbakk
2026-02-24 15:05 ` Karthik Nayak
2026-02-23 8:01 ` [PATCH v8 6/6] refs: add GIT_REFERENCE_BACKEND to specify reference backend Karthik Nayak
2026-02-25 8:50 ` Toon Claes
2026-02-25 9:41 ` Karthik Nayak
2026-02-23 10:54 ` [PATCH v8 0/6] refs: allow setting the reference directory Patrick Steinhardt
2026-02-23 13:37 ` Karthik Nayak
2026-02-23 20:05 ` Junio C Hamano
2026-02-25 9:42 ` Karthik Nayak
2026-02-25 9:40 ` [PATCH v9 " Karthik Nayak
2026-02-25 9:40 ` [PATCH v9 1/6] setup: don't modify repo in `create_reference_database()` Karthik Nayak
2026-02-25 9:40 ` [PATCH v9 2/6] refs: extract out `refs_create_refdir_stubs()` Karthik Nayak
2026-02-25 9:40 ` [PATCH v9 3/6] refs: move out stub modification to generic layer Karthik Nayak
2026-02-25 9:40 ` [PATCH v9 4/6] refs: receive and use the reference storage payload Karthik Nayak
2026-02-25 9:40 ` [PATCH v9 5/6] refs: allow reference location in refstorage config Karthik Nayak
2026-02-25 17:42 ` Junio C Hamano
2026-02-25 9:40 ` [PATCH v9 6/6] refs: add GIT_REFERENCE_BACKEND to specify reference backend Karthik Nayak
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20260202-kn-alternate-ref-dir-v4-0-3b30430411e3@gmail.com \
--to=karthik.188@gmail$(echo .)com \
--cc=git@vger$(echo .)kernel.org \
--cc=gitster@pobox$(echo .)com \
--cc=jn.avila@free$(echo .)fr \
--cc=ps@pks$(echo .)im \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox