It’s hard to imagine why cloning a git repo should be much slower than downloading an equivalent-sized file
Where are the experiments with custom git clients that clone faster
Why does it matter if sandboxes start up in 0.0001 nanoseconds if I then spend the next 20s downloading the git repo?
Many users praised the question about why Git cloning lags behind simple file downloads as insightful and supported faster alternatives like BunOS, while others dismissed Git as mediocre or suggested using Subversion instead.
No Digg Deeper questions have been answered for this story yet.
I would also bet there’s product changes that could be done to make it faster. Like, you could have a two step clone where it only downloads the files for the checkout initially then notifies the client it’s ready then any git commands block until it downloads the history lazily. Better to simply make it faster and not need to do that, but that would also be a worst case scenario option.
“You obviously don’t know how git is implemented”
Yes, I haven’t read much of git’s code. And that’s still true - downloading a big repo should not be much slower than downloading a big file.
“You obviously don’t know how git is implemented”
Yes, I haven’t read much of git’s code. And that’s still true - downloading a big repo should not be much slower than downloading a big file.
It’s hard to imagine why cloning a git repo should be much slower than downloading an equivalent-sized file
Where are the experiments with custom git clients that clone faster
https://github.com/oven-sh/bun/pull/32364
I have code locally demonstrating 2x - 3x is possible with zero server-side changes (github remote, no GitHub specific APIs). 5x sounds plausible but I’ll believe it when I see it.
I have code locally demonstrating 2x - 3x is possible with zero server-side changes (github remote, no GitHub specific APIs). 5x sounds plausible but I’ll believe it when I see it.
I will hide replies that suggest git cli flags or config - it’s not the ceiling of what’s possible.
I will hide replies that suggest git cli flags or config - it’s not the ceiling of what’s possible.
I would also bet there’s product changes that could be done to make it faster. Like, you could have a two step clone where it only downloads the files for the checkout initially then notifies the client it’s ready then any git commands block until it downloads the history lazily. Better to simply make it faster and not need to do that, but that would also be a worst case scenario option.
@jarredsumner You might be interested in https://blog.gitbutler.com/going-down-the-rabbit-hole-of-gits-new-bundle-uri
https://web.git.kernel.org/pub/scm/linux/kernel/git/mricon/korg-helpers.git/plain/linux-bundle-clone
Afaik the bottleneck isn’t usually clients.
@jarredsumner At Canva one way we speed it up in CI is by snapshotting the repo on a schedule into a tarball into eg s3.
Then the CI agents fetch and extract the tarball instead of cloning, then a git fetch only brings in changes since the snapshot and is much faster than a full clone.
@jarredsumner time to dig into the details here, fun side quest
@jarredsumner I'm building something to improve Git for agents (including "clone"): https://github.com/legendum/dojos
@kevincodex @jarredsumner @gitlawb_intern mirror this
most of it is file overhead right? server should just add a mode to serve up a blob of all files/objects in the repo as a single file (maybe with a zero compression zip/tar around it) and let client handle unpacking.
seems like political aspects of handling the change management would be 100x harder than the technical ones
@jarredsumner hi Jarred, I'm building @gitlawb a decentralized git primarily for AI agents. would love to hear your insights.
my agents can do a full git lifecycle in just one minute no email and passwords
@jarredsumner Anthropic were part of the Gitlab Transcend keynote last week with a design partnership on a new git backend. 15 min mark at https://about.gitlab.com/events/transcend/virtual/
@jarredsumner Does this mean you still have access to Mythos internally, or that there's another even better model available not subject to export controls?
@jarredsumner btw this is the repo https://github.com/Gitlawb/node
@jarredsumner Netlify use blobless clones by default, and the history is fetched lazily on demand
For some Docusaurus features like "last updated at" UIs we noticed this lazy dowbload makes it slower
For that reason we force fetch eagerly with "git backfill"
@jarredsumner Because downloading a zip is just one large blob. Git clone is not with one file. But multiple. It's same as trying to copy node_modules is slower than zipping it and copying it without high compression
@modpotatos @jjpcodes @jarredsumner Didn’t Microsoft do something like this. I thought they were working on a Virtual File System specifically for this issue like, a decade ago.
@jarredsumner Would this be something bun implements?
@jarredsumner http://gitcache.sh
