(With apologies if the release-management tag is inappropriate.)
The recent thread about 8.17 package failures had me thinking about release candidates in other communities. For example, Git and Alacritty both publish -rc packages that early adopters are welcome to test and report issues with; bugs may still be fixed before the release is finalized.
Is there value to Racket—in the same spirit as being pro-active about testing packages—in publishing release candidates that mirror what will be in the final release (modulo bugfixes)?
I imagine, perhaps naïvely, that a combination of the 2 ideas might have uncovered the dc<%> breaking change and provided an opportunity to change the final release in some way that avoids the problem. (I won't pretend to guess at what that change might be; the important thing, in my mind, is that it would be a normal part of the process to fix "release bugs," even if they come from code that has mostly stabilized for folks living on the bleeding edge of Racket.)
@benknoble , I agree that it would be nice to discover problems with the release earlier; in this case, I think the fault is primarily mine. Specifically, it's the pkg-build regressions that uncovered this, and I think it's important to be checking this source of information substantially earlier. I've adjusted the release checklist to try to push this up, to make sure that these checks happen before the beginning of manual testing.
Ah, great! I thought I remembered something like that. Do those also get announced? Is there a period where we encourage folks to try them and report back? (I realize the subset of folks who do that might be limited to the folks that already build from HEAD each day, so there may not be much value here.)
I don't think I agree with this. Either you inherited an improvable release plan and you improved it, or someone needs to be monitoring the pkg builds outside the release process. Either way, blame lies elsewhere.
As years go by, I'm more firmly in the camp of checks being run as early, frequently, and automatically as possible.
So to engage in some "design by wishful thinking":
If pkg-build took just 10 seconds to run, then it could be added to the list of checks/tests for every PR on every Racket repo. That way, someone making a breaking change would get feedback on the PR (and, depending on repo settings, potentially even be unable to merge the problematic PR).
Heck, "third party" pkg authors could also add this to their own checks (to avoid breaking pkgs that depend on their pkg).
End wishful thinking:
The problem, of course, is pkg-build doesn't take just 10 seconds to run. Or even just 10 minutes. Adding this as a mandatory check is too "heavy", and of course no one would do this.
But could pkg-build run faster??
If it can't, in general, could there be a variation that uses dependency information to build/test a potentially much smaller subset??
IIRC, a long time ago I used your repo travis-racket to run once per week automated test on a few packages with the latest official version of Racket and also with the bleeding edge version (and perhaps with the release candidate(?)). It was enough to find errors early enough. Specially in repos that don't have too much activity and thrusting that most of the times Racket is backward compatible.
@samth Today I set this up for a few of my repos that don't change frequently (which would benefit most by a periodic test run).
Then I was rummaging through even more GitHub docs, and stumbled across this other documentation node -- which, alas, has this note:
In a public repository, scheduled workflows are automatically disabled when no repository activity has occurred in 60 days. For information on re-enabling a disabled workflow, see Disabling and enabling a workflow.
I'll live with this for awhile and see if the "cron" test runs disable themselves eventually on the repos with no commit or issue activity for 60 days....
Of course in principle I still think it would be ideal if upstream repos' CI could use a dependency graph to trigger re-running tests on downstream user repos -- that would re-test only the necessary subset, and do so sooner (when the feedback is most helpful/actionable to the person accidentally introducing some breaking change).
But I appreciate that setting up something like that is easier said than done.
