I am on my first not completely trivial Racket project, and it was going really fine. Nothing too demanding, maybe twenty files or so, < 100k source code, though it's meant to get bigger. I also use two libraries outside the code base, one is just my own goody bag, the other one is racket-cas.
Builds were really fast after the initial one, updating any file never caused more than two seconds wait for anything to rebuild. I was really pleased with the easy progress. Then suddenly everything ground down to snail's pace. Updating a file could cause the build to take as much as 40"-1' to give me the repl. It's a real damp squib, and no fun at all - and I do this for fun, I am a hobbyist not a pro.
I can' really see anything that's changed suddenly, I don't have any cycles in my requires or anything silly like that. I also found that if I went and deleted all the compiled files in all three locations (codebase and two aforementioned libraries) then the next build would be fine for a while, giving me the usual two-second build delay. This does not last very long, however, before everything reverts to dead-slow. Even just exiting DrRacket - while everything is going fine - makes the system revert to slow when restarting!!
So what I would like to know is what is the best and quickest way to learn about the build process, to see what I can optimize. I cannot really accept that recompiling one small file, forces a full system rebuild. Sure it works just dandy once it's built, but it's a real palaver to get there and doesn't make for happy developing.
All ideas hints suggestions are warmly welcomed. Thanks.
BTW, I did discover that DrRacket actually compiles files as soon as you save them, you don't have to do a run for that to happen! That's really neat, I like it, but unfortunately it's only any good when the system is working fast. When it's slow I think it only adds to the problem and can even snag DrRacket.
Are you experiencing this outside DrRacket too? DrRacket instruments programs so they run differently (read: slower) than they would at the command line.
You talked about the build "taking forever to get you a REPL." The slowness might not be from compiling code, but rather from loading it - the latter occurs when starting up a program to get you a REPL. Does raco setup on your code take forever? Once your code is compiled with raco setup, does loading it to get a REPL take forever? Slow compilation might be due to expensive macros, while slow loading might be due to expensive module loading graphs or expensive initialization code.
What version and flavor of racket are you using?
Are you using #lang racket or #lang racket/base in your code? The latter loads much faster than the former, at the expense of requiring you to declare your dependencies more explicitly.
I am using DrRacket exclusively. I have not yet ventured beyond its bounds, that is what I am looking to do.
The program involved will end up being a kind of #lang, so the REPL will always be involved, even at the command line. I have one "main" or "environment" file, which is available to the user to bridge to all the rest of the libraries, editing anything other than this is only done to edit the system itself.
Your explanation here re loading time is useful. I have not used raco setup on its own yet. This is what I want to learn to do. Essentially I'm looking for docs/tutorials/cookbooks that will get me up to speed on this ASAP. (I have no expensive macros either - as yet)
Fresh new DrRacket 8.15 under Ubuntu.
A-Ha! This is something I can try out pretty quickly - I had a few files under #lang racket/base with full dependency descriptions, but got lazy and said sod it, I'll make everything #lang racket for now, and sort it out once it's cobbled together a bit better. I've only just put the build under an RCS too, so I can't look back, but I think that may be when it started playing up.
I guess what I'm really looking for are the docs on Racket without DrRacket.
It sounds as if you have a single file that takes long to compile (maybe you accidently wrote a macro with O(n^2) behaviour). That file will be recompiled each time you make changes either to that file - or to files it depends on.
You can use the module browser in DrRacket (see the "View" menu) to get an overview of your modules. Try systematically to make a cosmetic change to modules at the top of the tree (away from your main file) and see if you can find the file that provokes the long compile time.
Other things to do (mostly if you can't find the cause):
disable background expansion (right click the circle in the lower, right corner of DrRacket).
Note: I accidently switched the meaning in the second sentence.
Every summer/fall I create a project that comprises some 8000 to 10,000 loc. At some point it does get slower to load and reach the repl (for just running tests). What helps is
menu: Language —> Choose Language
button: Details
radio button: Populate “compiled” directories (for faster loading).
If you have already turns this on, something else is slowing your code down and we may need more details.
(Background expansion is running in parallel. I doubt it has much of an impact.)
Well, I don't know what the heck I've done, but it seems that somehow I have brought the system back into order! Now I get the previous sub 3" builds on my rickety old laptop.
First I followed @notjack's advice and made sure all the files except the last one were #lang racket/base. Now the reproducible way to get it working is as follows:
Delete all three compiled directories, including "my-libs" and "racket-cas". Run my project's main.rkt file. It builds more or less as expected in 40"-50" or so. Now the compiled files for "my-libs" and my project are present. However, the compiled files for racket-cas are ztill not built. Dunno why. So I build these too, by loading running raket-cas's repl.rkt file. After this everything runs as well as before, even if I edit files that depend on racket-cas, the build time stays the same sub 3".
One day I'm sure I'll understand what's going on, too... But for now I'm happy.
I'll try to make a raco build, completely outside of DrRacket just to see how it goes.
I vaguely recall that there's a heuristic somewhere with DrRacket about libraries from other collections (where you write (require some-library) instead of (require "some-library.rkt"). I think some layer of the system assumes under certain circumstances (but I don't recall off hand exactly what layer or which circumstances) doesn't check that those kinds of dependencies have up-to-date compiled code, and you can end up repeatedly recompiling things on the fly, or other kinds of bad states.
My practice is to use raco setup some-library when I've changed a dependency. Hopefully someone else can be more specific and accurate about why.
My practice is to use raco setup some-library when I've changed a dependency. Hopefully someone else can be more specific and accurate about why.
This doesn't quite align with my own experience, but I mostly develop code in a (linked) package (even if I don't publish to the package server), so to compile my code I write
raco setup --pkgs my-package
and it does the rest. Even if I change a non-linked package dependency of my-package (such as gui-easy for Frosthaven Manager), all relevant modules get recompiled through this method.
PS I could also probably write raco setup my-collect in many cases, but for multi-collection packages that may not be what I want.
I think that DrRacket's automatic compilation will treat files differently at the package level of granularity. Specifically, if you have a file open in a particular package, it assumes it can write to those directories and thus tries to keep those .zo files up to date (assuming you have that option turned on) and it will use its own copies of the zo files (in compiled/drracket) in that case. If you don't have a file open in that package, then it will not do that.
I think that at some point we've come to the realization that this is not the best way to go but we aren't sure how to improve things more. Some of the relevant code is here and just generally in the following lines you can see how it configures compiler/cm if anyone has a good idea about how this should work and they want to try it out.
|