Moving third-party packages to Codeberg

General
1

Hi. Does anyone knowledgeable about the Racket package system and Git want to comment on this plan?

I want to move 37+ Racket packages I wrote to Codeberg (which is like GitHub or GitLab, but warmer and fuzzier). The Racket package server is currently getting most of them from my Web site, not from Git. Some of these packages are in use by others. You can see most of them here: neilvandyke.org - Scheme & Racket

I'd like to have this migration work the first time, without breaking anyone's dependencies, and with a minimum of effort.

Example plan for package mcfly:

  1. I have releases: mcfly--2-0.zip mcfly--2-1.zip mcfly--2-2.zip

  2. Make repo: codeberg.org/neilvandyke/racket-mcfly

  3. Extract .zips to git tags: v2.0 v2.1 v2.2

  4. Add homepage/doc link in Codeberg.

  5. Change Racket pkg system entry to use repo.

  6. Some kind of measure to discourage the recent flood of LLM-generated pull requests that are drowning some other open source projects.

Then I would do the other few dozen packages like that.

If that exact plan should work smoothly, I can maybe do it in a couple hours. Putting on my software engineer hat, what I want to avoid is doing it but then later having to abruptly do urgent firefighting because something broke. Or having to go back and do something additional to each package.

Comments appreciated. Please feel free to reply here, or (I'll see it sooner) on https://mastodon.online/@neilvandyke/116756820844767929 or via email.

P.S., I also have a few unreleased packages, which I might polish up and release if this goes smoothly.

Thanks,
Neil V.

4 Likes
2

I don't see any particular issues with that plan.

Obviously this step is easier if the package sources are already tracked by Git (or another VCS that Git knows how to import from), but this also seems a reasonable way to map the existing archives into version control.

I don't know that the tags will matter too much to the Racket package server (though as above they'll be nice for humans) if you plan on pointing it at the "main" branch; if you plan to use another strategy[1], then you should be able to have the package server do something with the tags.

  1. Recall that Raco and Git and humans sometimes have differing notions of versions—and unless library consumers use a specific-enough URL to pin to a particular source revision, Raco will happily given them anything newer than the #:version constraint in their info.rkt (if any). ↩︎

1 Like
3

Good point about versions. One of the reasons for the Git tags (and for me increasing the priority to move to Codeberg) is that there was a recent headache involving versioning and the Racket package server.

It's also unclear to me whether the Racket package server is preserving old versions of packages.

4

Not to jinx myself but so far I have not experienced any automated (a.k.a. "agentic" ... pardon me I throw up in my mouth a little while typing that :smile:) pull requests.

I did get one human submitting a PR, the code for which was "AI" assisted. But just once. I updated the README to explain why I couldn't accept such PRs (TL;DR: no provenance).

Hopefully this factor won't turn out to be a big practical problem for you, either!

1 Like
5

This plan sounds great to me!

I've thought some about these sorts of things in the context of (very, very slowly) integrating Racket with Guix and archival preservation. I won't have time to write a whole lot of detail this week, but the bottom line is this sounds like an excellent direction, and many additional tools are able to help with archiving software in Git repositories than in ad hoc zip/tar/etc. archives on a website.

While it is possible for a library consumer to use a sufficiently-specific URL in-band to achieve pinning, it is usually better to do any needed pinning out-of-band through a catalog that maps the package name to the pinned URL. (And maybe that's what you had in mind, after all: the in-band way is fairly obscure, I think.)

The package server doesn't do anything about old versions of packages.

A good mental model of the package catalog protocol is a dictionary mapping “package names” to concrete, non-name “package sources.” That one layer of indirection turns out to be very useful for a variety of purposes. The Racket release process involves creating a catalog of all the packages in the release at the released version, which lets e.g. “minimal Racket” 9.2, configured to try the 9.2 release catalog first, install the drracket package and get the same libraries as the Racket 9.2 “main distribution.” The same catalog mechanism works for off-line builds, since the catalog can map names to local filesystem locations. In fact, that also gets used in the release process: a source distribution finds its bundled packages for initial installation through an off-line catalog.

I am enthusiastic about the fact that the catalog protocol does one simple job well. Nonetheless there are additional features one reasonably might want in a package system, like preserving old versions. My view is that the features are best approached by layering on top of the catalog protocol, rather “piling feature on top of feature” in the core protocol.

This is where I think it is very helpful, @neilvandyke, that you are planning to move your packages to Git. While a human can look at the URL for mcfly--2-1.zip in the catalog and infer the existence of mcfly--2-0.zip (and then hope that either you or e.g. the Internet Archive has preserved it), it would be less obvious to a tool, let alone if some less experienced and conscientious author had just published my-great-package.zip with no contextual clues. Using a Git repository ties the latest code to older and alternate versions, with the Racket package “checksum” recording the Git commit hash. It then becomes much more viable to leverage the work other communities have been doing to archive software in Git repositories.

(Other kinds of version-control repositories would have similar technical merits: Git is just the lingua franca at the moment.)

3 Likes
6

@neilvandyke With the status quo system, I think the idea/opinion is:

  • If you're adding new features/functionality, bump the version number. Dependents who need those new features can bump their deps to require your higher version number.

  • If you're deleting or changing (in a breaking way) existing features... well, please don't do that to an existing package. Backward compatibility is important. Breaking dependents isn't nice. Instead, please make a new package with a new name, and keep the old package available under the original name.

    • Note that dependents have no way to say "version <= X" in deps. That's OK in this view that "breaking changes mean a new package, not a new version of the same package".

Me? I think the system's opinion is wise and reflects the reality that dependency relationships are social not just technological.

But of course that's not everyone's opinion, or even the majority opinion.

Other ideas could be layered on top, for sure, as @LiberalArtist describes.

@neilvandyke One thing you could do is, go with the flow, status quo, and make a distinct Racket package for each older version, with something about the version number in the package name. IIUC each package could point back to some git version/branch/checksum in the same repo.

4 Likes
7

Are you using some CI/CD workflows in some of your packages? If so, do you have any plans to migrate these to Forgejo actions as supported by Codeberg?

We made a similar decision with one project (multiple packages) and I am also inclined to migrate my packages there. However some coordinated Racket support might help us all. I am thinking about specific reusable (cached) actions and/or container images that would be usable Codeberg-wide for running tests and/or compiling documentation for packages not on package server (or for compiling development snapshots documentation etc...).

Any thoughts?

8

You definitely could do that, but it would be more than is necessary to provide the equivalent of the status quo. IIUC, @neilvandyke currently has registered packages like mcfly once each, with the names pointing to sources like mcfly--2-2.zip. Sources for older versions remain available at the corresponding URLs like mcfly--2-1.zip, but anyone wanting to use them must do so manually, using the explicit URL or a catalog they have built (or https://racksnaps.defn.io, etc, etc), just like someone who wants to specify mcfly--2-2.zip even if future releases become available.

I think just providing the older releases as tags in the Git repository covers all the use-cases supported by the current setup, with an equivalent level of discoverability for humans. And I would suggest that other use-cases are niche enough that it would be better to address them only in response to a concrete need. The few packages I know of that try those sorts of things (which mostly respond to considerations like keeping code from a published paper working exactly) run into other downsides, and the tradeoffs seem harder to balance in the abstract.

On the catalog side, you could point the Git package source to a particular tag like v2.2, but you could equally well point to a main or stable branch holding the tagged releases, which would save having to update the catalog registration separately when releasing new versions.

I would suggest arranging to put all of the old releases on the same branch, so that Git understands the parent relationships between them. (Maybe that's what you'd do be default.) You could consider if you want to set the Git AuthorDate (vs. CommitDate) to the original release dates, if you want to be really fancy.

Tangentially, I realized the Package Build Service is an example of the kind of design I meant by “layering on top” of the core package catalog protocol. It is responsible for many useful features of the overall experience of Racket’s package system: generating the on-line documentation is probably the most obvious, but it also does CI testing and distributes packages in pre-built form. But it does all that without changing the core system. In some sense, you could think of it as a function from package catalog to package catalog.