Rhombus: global-install friendly way on already installed Racket?

Distro Packagers
1

Hi!

Rhombus 1.0 got released recently - Rhombus version 1.0 is now available! - HOORAY! I'm contemplating how it would be best to deliver Rhombus to OS/distribution users who already have Racket installed globally on their systems.

It is already possible to install Rhombus globally under Portage on Gentoo with the help of the racket-overlay for Gentoo (Racket Gentoo Overlay) but I was wondering if it could be better. I would envision that Rhombus supported compilation and install with already present Racket installation (via some mechanism managed by the Rhombus upstream). And since I believe the pkgs would have to be linked (referenced) from the main Racket links file maybe a small post-install script? (If I'm wrong here please correct me.) Currently in racket-overlay each pkg is installed individually, recorded in the Portage database (foll all packages) and then ensured that it is recorded in the Racket's installation DB (links.rktd). This is manageable but a but clunky - alternatively, with just one Rhombus package that installs the whole gang of Rhombus libs on top of existing Rakcet install updates and initial Rhombus pkg could be pushed to OSes/distros in a much easier way.

Thought? Ideas? Anything I missed about Rhombus?
Thanks in advance.

2 Likes
2

I'm not sure what you're looking for here, so I'll offer some pointers and you can clarify with "I already know all that, but..." to help me understand better:

  • There's a rhombus-main-distribution package that depends on all the packages in the Rhombus+Racket installer that are not in minimal Racket.
  • There's a catalog https://download.rhombus-lang.org/releases/1.0/catalog/ that provides specifically Rhombus v1.0 version of the packages, as opposed to the development versions referenced by https://pkgs.racket-lang.org.
  • As of Racket v9.2, the catalog https://download.rhombus-lang.org/releases/current/catalog/ is included as one of the catalogs to search by default. And that catalog is linked to the current Rhombus release, version 1.0. So, if someone running Racket v9.2 installs rhombus-main-distribution (or the shorthand rhombus-main), they currently get 1.0, even if the development version is later. I think this may be relevant to your question, because the catalog is configured even in the Racket source distribution, so you get Rhombus packages from the v1.0 catalog if you install using a build from source.
  • I think this is not relevant to your question, but the v1.0 Rhombus catalog serves built versions of the packages for each of Racket v9.2, v9.1, and v9.0. That makes installation of the Rhombus packages much faster (around 2 minutes instead of 10-20 minutes) on those versions of Racket, but only when using a Racket installer or a "source plus built pkgs" starting archive.
  • As an alternative to updating "links.rktd", the collection search path can be configured by updating "config.rktd", and that's connected to support for layered installations.
1 Like
3

So let's get the most important thing straight. When I'm thinking about packaging for a Linux distro (or other Unix, and MacPorts, whatever) I am thinking of a way that Racket (full version) is delivered, that is: configure, make, make install. When I am talking about better way to install Rhombus on Linux distros I'm thinking of a analogous way that Racket is installed EXCEPT that in this case Racket (full) is a required dependency of Rhombus. This will make it pleasant to many pkg distributors, and maybe even Windows package registry (winget). And why this would make it pleasant for them? - because they DO NOT have special programs to support with pkg setup, compilation and installation (like we have in racket-overlay). In fact I would imagine it would be currently impossible to have Rhombus as a Debian package.

I'm not sure what you're looking for here, so I'll offer some pointers and you can clarify with "I already know all that, but..." to help me understand better:
...

And here is a bit of misunderstanding, because Matthew, You are thinking in categories of raco user and Im thinking in categories of APT/DNF/Portage user.

There's a rhombus-main-distribution package that depends on all the packages in the Rhombus+Racket installer that are not in minimal Racket.

A big problem I found out with Rhombus packaging is that in the Github repo there are packages that depend on packages that are completely 3rd-party - what I mean is that for example actor is a pkg owned by Bogdan. If Rhombus had had deps on only full-version of Racket then a complex, yet manageable script to install and set up every pkg from Rhombus repo would suffice. In current case each 3rd-party dep (like actor-lib) would have to have a explicit package that is installed globally by a distro.

4

A catalog like https://download.rhombus-lang.org/releases/1.0/catalog/ refers to packages as .zip archives in https://download.rhombus-lang.org/releases/1.0/pkgs/. All dependencies that are not part of the Racket distribution are there—including actor, for example. It would be possible to gather up those archives, unzip them into a directory structure, and register the directories via link.rktd. The raco pkg catalog-archive command could help do that.

The .zo files that are in the .zip files are in machine-independent form, which means they are not yet compiled to machine code for the target platform. Finishing that up is normally the job of raco pkg install, but it could also be done with raco setup after setting up the right directories and links.

Documentation would be another detail to consider. The documentation is rendered within .zip archives so that it's already in place for a user-scope install of the package, but it is normally moved into place by raco setup (via raco pkg install) for installation scope.

I think you'd probably want to update more than just links.rktd so that the packages count as installed. That would matter if, for example, you want to let a user install additional packages via raco pkg that depend on Rhombus-distribution packages.

It's possible that a layered installation could help with the documentation and package-installation details, but I'm not sure about that.

5

After thinking a little more, let me synthesize into a more concrete suggestion/question:

We can gather all the relevant package sources into a directory with a corresponding catalog so that raco pkg install -i --catalog <that-dir>/catalog --auto rhombus-main would add Rhombus to a Racket installation. If that seems like the right idea, then we could even have a .zip of that directory created as part of the Rhombus release process. Would that be the right idea?

Part of what I'm wondering is whether the important thing is to have all the package sources in one archive, or whether the important thing is to avoid raco pkg install.

6

We can gather all the relevant package sources into a directory with a corresponding catalog so that raco pkg install -i --catalog <that-dir>/catalog --auto rhombus-main would add Rhombus to a Racket installation. If that seems like the right idea, then we could even have a .zip of that directory created as part of the Rhombus release process. Would that be the right idea?

This would be very helpful.

I wonder if some pkgs that are already in Racket-full would sneak into the catalog archive (if we would create a archive of pkgs from Rhombus catalog) but I guess Racket's dependency resolver would deal with it.

Part of what I'm wondering is whether the important thing is to have all the package sources in one archive, or whether the important thing is to avoid raco pkg install.

We can cope with it I imagine but the closer we are to what Racket-full install does the better.

Any by referencing Racket-full build & install process I mostly refer to ./configure && make && make DESTDIR="...." install.

7

All Rhombus package sources are now available together as
https://users.cs.utah.edu/plt/rhombus//1.0/installers/rhombus-1.0+9.2-pkg-src.tgz, which is listed in a new section of the "More Installers and Checksums" page.

There's a README.txt with more information, and a Makefile so that make can perform the slow platform-specific work before actually installing. There's also make install target that then actually installs, but no DESTDIR support, because actually installing uses raco pkg install to update an existing installation. The result of just make is standalone in the sense that it can be repackaged for a relatively quick installation step, roughly DESTDIR-like.

I wonder if some pkgs that are already in Racket-full would sneak into the catalog archive

No, they are specifically excluded from the Rhombus release catalog, including as gathered in the new archives. This is one way that the new archives are specific to a Racket version, since the set of packages can be version-dependent.

2 Likes
8

This is very helpful but the lack of some better install support makes it surprisingly harder to install for me. This is because in Gentoo we have to merge a DESTDIR image (the stuff that DESTDIR installs into a temporary throwaway directory) onto the system image (the live system).

In case of Racket this is very easy and idiomatic. In case of singular packages we do it this way:

  1. install it in user scope under a temporary build-only user (whose home is on a throwaway dir)
  2. merge the pkg sources + compiled files on to the system
  3. change-dir into the Racket pkgs dir and do: a) no-setup install (if the pkg is being installed 1s time) b) pkg setup
  4. if pkg is being removed 1st remove it via raco, then remove installed files.

As you can see, what I was looking for in case or Rhombus was whether that above thing could be easier and currently it is a bit more messy because we either have to a) gather all Rhombus pkg names and then do above for each pkg b) the same but hardcode the names into the build script.

Btw, is there a way to tell Racket this: "look into this dir, if you see any pkgs that is not linked to system DB link it"? This would help us in Rhombus because we can throw all the pkgs in build/9.2/pkgs into /usr/share/racket/pkgs/ and tell it to fix up the pkg links. currently we have to install each individually. (btw, the pkgs would already be compiled so literally all we need to do is to adjust global pkg DB - links.rktd, pkgs.rktd, etc...)

9

Step 1 is how the Makefile target starts, except that it uses racket -A instead of a separate user. After the second command in the Makefile, the build/<version>/pkgs directory contains all the package directories that you'd want to move using your current strategy. To get the list of relevant packages, you could record the list of directories that were moved (perhaps putting the recorded list somewhere in DESTDIR) and use that for a raco pkg install -i --link step.

I'm not sure I see the difference between that process and using the content of "build/rhombus-pkg-platform" as the content to initially merge as DESTDIR, installing from that content, and then removing the originally merged content as the last step of setting up. Either way, various files get moved and updated, and it will take about the same amount of time. Maybe it's a question of just how many files need to get updated, created, and moved.

It would be possible to add even more modes to raco pkg install so that the DESTDIR content could more directly reflect the final destination in the installed filesystem. I may look into that, but it would be for v9.3 and later.

10

I will experiment with --link.

11

I found a "bug" with current makefile - it will only set up the pkg deps if those are not installed (in global scope)

racket -A build -l- raco pkg install -u --auto --skip-installed --catalog catalog/ rhombus-main-distribution 
# For v9.3 and later, using \`--built\` with \`raco pkg catalog-archive\` would be better than \`find\`+\`rm\`: 
find build/9.2/pkgs -name 'synced.rktd' -exec rm {} \\; 
racket -l- pkg/dirs-catalog build/9.2/catalog build/9.2/pkgs 
Finding packages 
 Cataloging package rhombus-main-distribution 
Missing package authors for rhombus-main-distribution 
racket -l- raco pkg catalog-archive --relative build/rhombus-pkg-platform build/9.2/catalog 
== Archiving rhombus-main-distribution == 
checksum:  
packing /var/tmp/portage/dev-lang/rhombus-1.0/work/rhombus-1.0+9.2-pkg-src/build/9.2/pkgs/rhombus-main-distribution/
 into /var/tmp/portage/dev-lang/rhombus-1.0/work/rhombus-1.0+9.2-pkg-src/build/rhombus-pkg-platform/pkgs/rhombus-mai
n-distribution.zip 
writing package checksum to /var/tmp/portage/dev-lang/rhombus-1.0/work/rhombus-1.0+9.2-pkg-src/build/rhombus-pkg-pla
tform/pkgs/rhombus-main-distribution.zip.CHECKSUM 
Creating catalog build/rhombus-pkg-platform/catalog

I do not see a way to get the full dependency chain set up as of now regardless of what is installed in global scope (of course we have to have Racket-full).

Currently this is much harder for us to get working than installing each individual package and I wonder if it is worth the effort to create a one big Rhombus installer at this point. I think for Gentoo users racket-overlay (each-individual-pkg-installation way) will still be a recommended go-to way.

I can work on this but I believe it will take much more than current work, tho this is pretty far we got to!

For anybody wondering I post the Gentoo ebuild I was working on:

# Copyright 1999-2026 Gentoo Authors
# Distributed under the terms of the GNU General Public License v2

EAPI=9

MIN_RACKET_VERSION="9.2"

inherit multiprocessing

DESCRIPTION="Easy to use and uniquely customizable general-purpose programming language"
HOMEPAGE="https://rhombus-lang.org/
	https://github.com/racket/rhombus/"

SRC_URI="https://users.cs.utah.edu/plt/${PN}/${PV}/installers/${P}+${MIN_RACKET_VERSION}-pkg-src.tgz"
S="${WORKDIR}/${P}+${MIN_RACKET_VERSION}-pkg-src"

LICENSE="|| ( MIT Apache-2.0 )"
SLOT="0"
KEYWORDS="~amd64 ~x86"

RDEPEND="
	>=dev-scheme/racket-${MIN_RACKET_VERSION}:=[-minimal,doc,threads]
"
DEPEND="
	${RDEPEND}
"

src_configure() {
	export PLT_COMPILED_FILE_CHECK="exists"

	RACKET_VERSION="$(racket -e '(displayln (version))')"
	RACKET_BUILD_DIR="${S}/build/${RACKET_VERSION}"
	RHOMBUS_HOME="/usr/share/rhombus"
}

src_compile() {
	local -x TERM="dumb"

	edo racket --addon "${S}/build/" -l- \
		raco pkg install --scope user --auto --skip-installed \
		--jobs "$(get_makeopts_jobs)" --no-docs \
		--catalog "${S}/catalog/" \
		rhombus-main-distribution
}

src_install() {
	mkdir -p "${ED}/${RHOMBUS_HOME}" || die
	cp -r "${RACKET_BUILD_DIR}"/* "${ED}/${RHOMBUS_HOME}" || die

	cd "${ED}/${RHOMBUS_HOME}/pkgs" || die
	find . -maxdepth 1 -mindepth 1 -type d -printf "%f\n" \
		 > rhombus_pkgs.txt || die

	dobin "${RACKET_BUILD_DIR}/bin/${PN}"
}

pkg_prerm() {
	if [[ -z "${REPLACED_BY_VERSION}" ]] ; then
		if has_version "dev-scheme/racket" ; then
			while read -r pkg ; do
				if [[ -d "${EPREFIX}/${RHOMBUS_HOME}/pkgs/${pkg}" ]] ; then
					einfo "Uninstalling Racket package: ${pkg}"
					edo raco pkg uninstall --batch --force --no-trash \
						--scope installation --no-docs --no-setup --auto \
						"${pkg}"
				fi
			done < "${EPREFIX}/${RHOMBUS_HOME}/pkgs/rhombus_pkgs.txt"
		fi
	fi
}

pkg_postinst() {
	if has_version "dev-scheme/racket" ; then
		while read -r pkg ; do
			if [[ -d "${EPREFIX}/${RHOMBUS_HOME}/pkgs/${pkg}" ]] ; then
				einfo "Installing Racket package: ${pkg}"
				edo raco pkg install --batch --deps force --no-docs --no-setup \
					--scope installation --link \
					"${EPREFIX}/${RHOMBUS_HOME}/pkgs/${pkg}"
			fi
		done < "${EPREFIX}/${RHOMBUS_HOME}/pkgs/rhombus_pkgs.txt"
	fi

	edo raco setup --all-users --force --jobs "$(get_makeopts_jobs)" \
		--no-pkg-deps --pkgs rhombus-main-distribution
}
1 Like
12

I've made some small additions (for v9.3 and later) to raco pkg that I hope will help package management and cooperation with OS package managers. There's not one big idea here, but a bunch of little adjustments to potentially make things easier.

Improving installable-package creation (from installed to installable)

Installing into user scope to gather and compile packages is the way things are supposed to work, at least as the first step, and using racket -A is a good way to direct the output. But a user-scope installation directory is not always the right shape as the source for another installation, particularly when rendered documentation is included. Using raco pkg create --built --from-installed is the intended way to get from a user-scope installation to an installable package.

New pieces to help:

  • raco pkg create command can now create any number of packages at a time, instead of just one.

  • raco pkg create --format dir, which amounts to a shortcut for staying in directory mode, if that's useful. Along those lines, pkg/dirs-catalog recognizes checksum files that are created by --format dir; it's not clear that the checksums are important for the immediate goal, but having them closes a small consistency gap.

  • raco pkg create --adjacent-deps infers additional packages to create based on the dependencies of specified packages and directories that are siblings on the specified ones. This can reduce the need to manually list relevant packages.

It would be nice if raco pkg install --destdir <dir> (described in the next section) could do all of this itself. But the raco pkg install would have to restart Racket with -A, and while that doesn't seem so bad for command-line, it would be weird for the Racket-level API to do the same thing. So, I didn't try to merge the steps that way.

Improving package staging (from installable back to "installed" as staged)

After installable packages area available, they need to be put in a format to be added to a Racket installation.

New pieces to help:

  • raco pkg install --destdir <dir> stages a package installation (potentially with dependencies) into <dir>, instead of installing into the current Racket instance. If staging uses all the packages originally created by raco pkg create --format dir, then it turns out that the staged directory will be a copy of that one. In principle, though, this step converts from "installable packages" to "packages in installed form". Also, raco pkg install --destdir can potentially be used to follow dependencies (aided by a catalog or --adjacent-deps described below) to arrive at the right set of packages.

  • setup/doc-to-destdir supports moving rendered documentation within --destdir output to a separate directory, so that rendered documentation can be set up for copying into the final "doc" destination. Otherwise, the "packages in installed form" will still be modified further by raco setup to move documentation into the central "doc" directory. Of course, this step is not needed if documentation isn't being rendered.

Without docs, "installed" for user scope and "installed" for installation scope are currently the same shape, so it's tempting to skip the back-and-forth. But back and forth (with adjustments, sometimes) is how it's supposed to work.

Improving installation into an exsting Racket tree (from staged to really installed)

Assuming that we have the right packages (and perhaps documentation) staged, and there's no conflict with existing files in the Racket installation, we can to write the packages into the installation and link them. Although no files should be overwritten at first, linking will need to update files in the Racket installation: "share/info-cache.rktd", "share/links.rktd", "share/pkgs/pkgs.rktd", and similar documentation-related indices and regenerated entry points.

New pieces to help:

  • raco pkg install --attach is an alternative to --link when the relevant package directories (and, optionally, documentation) are already in place. The intent is that packages created via --destdir have been copied into the installed-package directory by the time --attach is used. This new --attach mode installs the package as if they came from a catalog, which means that each package can carry along a checksum and connect to an original source repo.

  • raco pkg install --adjacent-deps can discover dependencies without relying on a catalog. This isn't needed if all packages can be listed.

  • raco pkg install --no-promote is for the case when all packages are listed, but when there's one package that should count as explicitly installed and others are installed only as dependencies (so that raco pkg uninstall --auto on the main page can remove all of them). The --no-promote flag attaches all listed packages as auto-installed dependencies. After attaching all packages, the main one can be promoted to explicitly installed using raco pkg install --skip-installed, which is documented as promoting an auto-installed package, and has been changed to not require the original package source to be written again.

Improving installation into a separate tree (staged to really installed)

Suppose that we don't want to write package files into the Racket tree and instead keep them separate. We'll install the package in --link mode --- so, still updating some files in the Racket installation, but not adding new ones. Maybe we're doing that because there's a potential for conflicts, so we only want to link packages that are not already installed.

New pieces to help:

  • The --adjacent-deps flag can be used with --link more as well as --attach. Since dependencies will be searched only if they are not already installed, this can help avoid conflicts.

  • When listing all packages, combining --skip-installed with --no-promote solves a similar problem with --link as it solved with --attach: even though the packages to link are enumerated, they are not all made explicitly installed, and auto-installed existing packages are skipped the same as explicitly installed existing packages.

Putting the pieces together

This revised makefile for Rhombus (should work with the next snapshot, at least) shows how many of these pieces are meant to be put together:

3 Likes
13

Yes, I believe this solves it all now. I'm afraid I wont be able to test it before release but I will do my best to find available resources.
(I know that I can use 9.2/3 snapshot when its built but I'm not sure when I will have available slot to rebuild my Racket and then go back)

How would uninstall and update look like?
I think there might be cases where we might end up with broken pkg installs, like for example the pkg dir was removed but pkg not uninstalled with raco uninstall. Do we have any way to recover from this and tell Racket "please unregister all pkgs that you think are broken"?

14

Hi
When the release testing process starts you can get builds at https://pre-release.racket-lang.org/

If you stake a look I think it will still populated with 9.1.900 builds that were used in preparing the 9.2 release

And thank you for working one this. I’m sure our Linux users appreciate it.

Stephen :beetle: