Scope and purpose of racket/gui

[LiberalArtist]

@ken

the "Feature requests/issues reported on Hacker News" issue from 2018

For reference: Feature requests/issues reported on Hacker News · Issue #115 · racket/gui · GitHub

From quickly skimming the list, I don't think all of the items are accurate: e.g. the editor data mechanism does #3, and #7 is supported by e.g. frame:standard-menus-mixin and application-about-handler.

---

@ken

Not only do I think that racket/gui/easy is the most reasonable way to write GUIs in Racket, but I think the "easy" label is wrong. It might be easier, but it'd be more accurate to say it's just a better abstraction.

I don't actually know, but when I first saw it I assumed the easy suffix was following @bogdan's net/http-easy, which similarly provides abstractions over net/http.

---

@ken

for now, I've settled on writing my GUIs using the platform toolkit (since I mostly care about macOS and iOS) and embedding Racket to handle the business logic.

Yep, this can be a feasible approach. I think the main reason I'm not doing that myself right now is simply because both of my main projects are still in the "research/experimental" phase, and it's hard to split up a program when you don't even know what it's going to do yet.

In case you aren't aware, Racket can be configured to use GTK instead of the "native" toolkit on Mac OS and Windows (though I think the GTK/Windows support is not regularly tested: I was surprised when I stumbled across it in the source). You can structure your code as a racket/gui program but, when needed, use the get-client-handle method of window<%> to grab a GtkWidget pointer and drop down into GTK.

---

Native Widgets & Custom Drawing

@ken

Is performance a feature? I ran into trouble with putting many controls on one panel (filed: #311), due to the deep invisible hierarchies that racket/gui builds behind the scenes. There are some very pretty racket/gui applications but they all appear to be mostly custom drawing, with not many native widgets. Using just native controls, I'm getting around 1fps on my new Ryzen system, which is unusable for the types of programs I want to write.

@robby

I would say that improving the performance would be awesome. For deep invisible hierarchies, we have some support (c.f., panel vs pane) but I'm sure there is more improvement that can be done.

(The following is a somewhat vague impression I've had: hopefully people who actually know things can chime in, either with supporting evidence or with explanations of what I'm getting wrong.)

(P.S.: This ended up being very long! I've been ruminating on some of these topics for a while, though I still think it's a bit amorphous.)

A number of platform GUI toolkits seem to have "recently"(-ish) undergone some sort of transition:

A	B	C
Qt Widgets	⮕	Qt Quick
AppKit	⮕	SwiftUI
Win32	⮕	WinUI [1]
GTK 3 rendering	⮕	GTK Scene Graph Kit (GSK) in GTK 4

One pattern I notice is that, in the "traditional" toolkits, widget objects are resource-intensive and do not scale well. Here's an extended fragment of an even longer old discussion:

Dmitry Pavlov wrote at 2014-07-09 04:50 AM:

I have to do a simple spreadsheet editor and I wonder
whether Racket suits my needs. The main challenge
is that the spreadsheet editor should be able to edit
tables as big as 1000x1000 or 10000x100 cells.

@mflatt wrote at 2014-07-09 05:15 AM:

… [I]nstances of button% (or generally control<%>) in a scrolling
panel will not scale well. The racket/gui library is not designed for
it.

I'm not sure how much the problem is in racket/gui versus the
underlying toolkits. Your example program scrolls nicely for me on
Windows and Mac OS X, but not Unix/Gtk, but I would not conclude from that
experiment that the problem is in Gtk. The problem might be the
Gtk-specific part of the implementation of racket/gui. Also, I
wouldn't expect any of the platforms to scale to 1000x1000 buttons.
When I tried 100x100 on Mac OS X, it took a couple of seconds to create
all of the buttons.

I think you would have to use canvas% and draw/manage the grid and
controls manually. It's possible that the classes of embedded-gui
will be useful, if you can set up a suitable harness for snips. (I've
always wanted to make a table% class to go along with text% and
pasteboard%, but I never got around to it.)

@greghendershott wrote at 2014-07-09 9:27 PM:

I don't think that a big-grid GUI application like Excel will use
controls for very much in its main window. Generally it is managing
all that itself.

It might use a few plain windows, such as one for the column names
on top, another for the row names on the left, and then a big one for
the main grid client area. Just to make it easier to clip output.

If the user clicks in a button-like area, it will handle that itself.
e.g. If you click somewhere in the column header, it will calculate
which column, and draw that column as selected.

If the user clicks in a region it calculates to be a cell, it might
create a text-edit control there for in-place editing -- but just
temporarily, and destroy it when editing finishes.

All the logic for scrolling... managed itself.

At least, that's how I did Windows GUI stuff like this, 15+ years ago.
Usings hundreds or thousands of windows/controls was just too much
overhead to get desirable speed and space. Although the overhead might
be less, now, I imagine if you want a really crisp UI it's probably
much the same story.

Neil Van Dyke wrote at 2014-07-10 6:39 PM:

For a million cells like that, when using any language and toolkit that
I know of, I would probably implement it with a mix of manual drawing
and using the occasional toolkit widgets in only small numbers at a time
(only for actively editing of a single cell).

…

I think that this simple approach of a manually-drawn grid and minimal
use of toolkit controls will be fast with even a million cells, without
much programming difficulty.

The "new" toolkits (or new features/emphasis in existing toolkits) seem to move toward approaches reminiscent of the "Virtual DOM" used by some JavaScript UI frameworks. They seem to be trying to make toolkit widgets more scalable and, at least in some cases, rendering APIs are getting closer to functional programming: for example, GTK Scene Graph Kit's GskRenderNodes are immutable. Some are doing more sophisticated compositing, doing more work on the GPU, or otherwise pursuing performance gains.

AIUI, gui/easy also takes steps in this direction through its functional shell of lightweight view<%> objects. But I think this would be a fruitful direction for future work, both in higher-level abstraction over racket/gui and in identifying platform functionality that should be supported.

That sort of leads to the second, related pattern I've noticed in the toolkit transitions, which I'd introduce with the assessment in chapter 7 of Leif Andersen's thesis, "Adding Visual and Interactive Syntax to Textual Programs" (p. 41):

Unfortunately, the racket/visr prototype falls short of the design requirement[ that "an interactive-syntax extension mechanism must use the existing GUI libraries of the chosen language as much as possible" (p. 12).] …

[The racket/gui] framework is not powerful enough, requiring developers to use separate implementations for their application view and interactive-syntax …

Most notably, visual syntax extensions in racket/visr cannot make use of the existing Racket GUI library. Instead visual syntax extensions must use a custom GUI library. This major limitation means that while racket/visr is useful for communicating ideas and thoughts via hybrid code, it is ultimately not usable.

This technical problem is due to a division of Racket’s GUI library. This GUI library comes in two parts: an operating-system widgets layer and a code-editor layer. Code editors can be placed into both widgets and other code editors. Widgets, in contrast, can only be placed in other widgets. The design of racket/visr, however, necessitates embedding widgets in code-editors.

To work around this, I created a widget library that attempts to parody the Racket widget library inside of code editors. In principle, programs written for one library can be easily ported to the other. In practice though, porting these programs from one library to the other is a significant effort. As a result, to make use of interactive-syntax extensions, programmers are forced to develop two implementations of their GUI.

An unfortunate side effect of using a custom interactive-syntax GUI framework is that this custom framework is inevitably less performant than its user-facing counterpart. This means that while developers are able to read code with that uses interactive-syntax extensions, their ability to write new code is severely limited.

In racket/gui—and, I think, in "traditional" toolkits (or ways of using them) more generally—a canvas<%>, be it a canvas% or an editor-canvas%, is basically a "leaf" from the toolkit's perspective. Outside, you have one set of relatively high-level abstractions like message%, radio-box%, and group-box-panel%. Inside, you have a surface for Cairo drawing (racket/draw): any abstractions you create over it, even ones as powerful as the editor<%> and and snip% systems provide, still look to the platform just like drawing.

The "new" toolkits seem to be trying in various ways to remove some of these weaknesses and restrictions.

One motivation is performance: as the blog post I linked to above about the motivations for GSK put it, "we can incrementally transition from the current immediate more rendering model to a more structured tree of rendering operations that can be reordered and optimized for the target graphics layer."

More interesting to me are the possibilities for expressiveness. I think many, perhaps all, of the "new" toolkits have functionality that we could use to allow toolkit widgets to be placed inside of canvas<%>es (or perhaps some new canvas<%>-like abstraction would be needed), solving problems like those encountered by racket/visr. Those cross-platform GUI toolkits that mostly eschew "native" widgets face similar challenges, and some seem to have found solutions. For example, Flutter can embed an android.view.View or an iOS UIView, and on most platforms Qt^[2] can wrap a "native" pointer with QWindow::fromWinID() and embed it in Qt Widgets via QWidget::createWindowContainer() or in (the latest version of) Qt Quick via WindowContainer.

In a different way, communicating more structure to the to the platform is needed to fix DrRacket Is Inaccessible with Screenreaders · Issue #219 · racket/drracket · GitHub, an especially important area for improvement (though also a challenging task to take on).

I don't have an especially concrete conclusion. I, for one, find it very helpful to hear from people like @bogdan and @ken about their experiences working with the platform toolkits directly as a way to think about what might be useful to have in racket/gui.

For some kinds of functionality, an alternative to finding common API for all platform backends would be to adopt some cross-platform library, analogous to the way we use Cairo for racket/draw. (There are also obvious downsides to this approach!) For "scene graph"-like functionality, GSK, QRhi (the Qt Rendering Hardware Interface), and Webrender (a Rust rendering engine used by Firefox—not actually web-related itself) are all cross-platform.

Well, one concrete step would be adding support for GTK 4: I have given thought to this, but there are several other things I'd need to finish before I could work on it myself.

---

1. I am least clear on the details for WinUI: I haven't routinely used Windows, much less programmed on it, since the days of Visual Basic 6. One of the reasons I'm enthusiastic about racket/gui (and Racket's cross-platform portability more generally, e.g. Windows path support) is that I can write portable code that almost always "Just Works" for the Windows users I need to support. ↩︎

2. As a KDE Plasma user, Qt is the "native" toolkit for my platform, but Qt also brings its own widgets and rendering when used on other platforms. ↩︎