From: "Kousik Sanagavarapu" <five231003@gmail•com>
To: "brian m. carlson" <sandals@crustytoothpaste•net>,
"Kousik Sanagavarapu" <five231003@gmail•com>,
<git@vger•kernel.org>
Subject: Re: Running out of inodes on an NFS which stores repos
Date: Mon, 08 Sep 2025 12:35:08 +0530 [thread overview]
Message-ID: <DCN87S14V9G8.3BAV5XX1BDHKM@gmail.com> (raw)
In-Reply-To: <aLxUkTzuVaZrWDs2@fruit.crustytoothpaste.net>
On Sat Sep 6, 2025 at 9:04 PM IST, brian m. carlson wrote:
> On 2025-09-06 at 14:16:12, Kousik Sanagavarapu wrote:
>> Hello everyone,
>
> Hi,
>
>> These git repos come from another service and there are typically
>> thousands of them each day. It is important to note that we only store
>> the .git dir and expose a url which is configured as the remote by
>> default to read and write into this repo.
>>
>> All of these are small repos; usually not many files and not many
>> commits too - I'd say ~5 commits on average.
>>
>> Historically, when we ran out of inodes, we had implemented a few
>> strategies where we used to repack the objects or archive the older
>> repos and move them into another store and bring them back into this
>> NFS and unarchive the repo.
>>
>> However, none of these totally mitigated the issue and we still run
>> into issue as the traffic increases. As a last resort, we increased
>> the disk size even though there was ton of free space left - just
>> for increasing the number of inodes.
>>
>> We can't delete any of these repos, no matter how old, because they are
>> valuable data.
>>
>> I was wondering if there was some other strategy that we could implement
>> here as this seems like a problem that people might often run into. It
>> would really help to here your thoughts or if you could point me to
>> anywhere else.
>
> There are a couple things that come to mind here. You can try to set
> `fetch.unpackLimit` to 1, which will cause of the objects pushed into
> the repository to end up in a pack. That means you'll usually have
> only two files, the pack and index, rather than the loose objects.
Thanks for this, I have tried this out and while going through the
surrounding documentation, found `transfer.unpackLimit`. This was exactly
what I was looking for.
> If you have a large number of references, you may wish to convert the
> repositories to use the reftable backend instead of the files backend
> (via `git refs migrate --ref-format=reftable`), which will also tend to
> use fewer files on disk. Note that this requires a relatively new Git,
> so if you need to access these repositories with an older Git version,
> don't do this.
>
> You can also periodically repack more frequently if you set
> `gc.autoPackLimit` to a smaller number (in conjunction with
> `fetch.unpackLimit` above). If you have repositories that are not
> packed at all, running `git gc` (or, if you don't want to remove any
> objects, `git repack -d --cruft`), which will likely reduce the number
> of loose objects and result in more objects being packed.
Yes, I have now set the following config surrounding gc
[receive]
autogc = true
[gc]
auto = 1
autopacklimit = 1
Curious to know if this will have any noticable performance impact
though. As I mentioned in my previous msg, these are small repos but the
number of repos being created and the operations performed on them are
large - mostly pushes,
> Finally, it may be useful to you to reformat the underlying file system
> in a way that has more inodes. I know ext4 supports a larger inode
> ratio for repositories with many small files. Alternatively, apparently
> btrfs does not have a fixed inode ratio, so that may be helpful to avoid
> running out of inodes. I can't speak to non-Linux file systems, though.
Unfourtunately, I can't reformat the NFS. It is currently on ext4 and
even though there are quite a few filesystems which don't impose a
threshold on inodes, I can't migrate to them.
next prev parent reply other threads:[~2025-09-08 7:05 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-09-06 14:16 Running out of inodes on an NFS which stores repos Kousik Sanagavarapu
2025-09-06 15:28 ` rsbecker
2025-09-06 15:34 ` brian m. carlson
2025-09-08 7:05 ` Kousik Sanagavarapu [this message]
2025-09-09 0:29 ` brian m. carlson
2025-09-09 7:39 ` Kousik Sanagavarapu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=DCN87S14V9G8.3BAV5XX1BDHKM@gmail.com \
--to=five231003@gmail$(echo .)com \
--cc=git@vger$(echo .)kernel.org \
--cc=sandals@crustytoothpaste$(echo .)net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox