Does the Package Server always download the entire repository?

Questions & Answers
1

I have a package configured using the "Path within repository" option. I thought this would cause the package server to download only that part of the repository. The path is https://github.com/default-kramer/HermitsHeresy.git?path=hermits-heresy#release

But I just noticed it is running out of disk space "while updating the package checksum" and it appears to be downloading the entire repository. (Here it shows that it is downloading the /test subdirectory, which I hoped to exclude.)

copy-file: error writing destination file
  source path: /var/tmp/git17299539951729953995740/obj166
  destination path: /var/tmp/17299539951729953995740-default-kramer_HermitsHeresy_git_release/test/fresh-topias/4pvf1r91tm1.BIN
  system error: No space left on device; errno=28

Is it possible to configure the package so that only one subdirectory will be downloaded? Or do I need to move the desired package content into a separate repository?

2

I think it's only supposed to install where the info.rkt file is but I'm not sure. I reme this article has some relevant information: How to Organize Your Racket Library – Terminally Undead

1 Like
3

AFAIK When a package is backed by a git repo, raco pkg install or update needs to clone that git repo. AFAIK git doesn't let you clone only a subdirectory. So I don't see how this could work.

For the granularity of "what is downloaded", AFAIK you probably need to use distinct repos. At least, you won't have to wonder, it will be clear that's the granularity you're getting.

For the granularity of "what is installed/built", you can have multiple packages as subdirs within a repo.

If the issue is mainly that you have some huge example files/data used for test purposes, maybe you can move just those to be in a distinct repo, and the test itself does a git clone or wget or curl or whatever you prefer?

You could even have your test do that only when an env var like "CI" is defined, so this happens for say GitHub Actions. but not when users run tests as a result of building package locally.

1 Like
4

It is possible to limit what is downloaded without making a separate repository! When possible, the package manager will make a “shallow clone” (like git clone --depth=1, but implemented in pure Racket by the #:depth argument to git-checkout with only the contents of the desired commit, not the full history. So, you can create a release branch that leaves out the problematic directory. It looks like @default.kramer figured out that workaround in GitHub - default-kramer/HermitsHeresy at release-for-racket-pkg.

It seems like some fairly new features in Git might support automatically downloading only a desired subdirectory, particularly git clone --filter= with its <filter-spec> options and the experimental git sparse-checkout command (more docs). For Racket's purposes, what matters is not the git porcelain per se, but what we can do with the underlying wire protocol.

2 Likes
5

I can confirm that you can use Git to clone just a specific directory into a checkout: clone supports a sparse option that checks out only the top level, which you can then embiggen with sparse-checkout.

This is independent of partial clones using depth or filter.

3 Likes