# Imports as completion candidates for hash-langs **URL:** \ **Category:** General\ **Created:** [November 17, 2023, 2:19pm UTC](https://racket.discourse.group/t/imports-as-completion-candidates-for-hash-langs/2511 "2023-11-17T14:19:04Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![greghendershott](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/greghendershott/32/98_2.png) [@greghendershott](https://racket.discourse.group/u/greghendershott)\ **Post date:** [November 17, 2023, 2:19pm UTC](https://racket.discourse.group/t/imports-as-completion-candidates-for-hash-langs/2511/1 "2023-11-17T14:19:04Z")
Given ```scheme #lang rhombus imp ``` with the cursor after "imp", if I either - In DrRacket: Press C-/. - In Emacs, with `racket-xp-mode` enabled: Press TAB (or trigger `complete-symbol` in some other manual or automatic way). then the choices include - `import`: Good. - `impersonate-box`: Oops. Using imported identifiers (from `module` and `#%require` in fully-expanded code) as a source of completion candidates has been, in my experience, reliable and useful for `#lang racket`. That's what `racket-xp-mode` is doing here. I assume similar in DrRacket? But based on this example, it looks like this can't be the whole story for arbitrary hash-langs, including `#lang rhombus`. Does a hash-lang need to be able to supply some kind of function to filter and/or transform the imported identifiers, to make them suitable as completion candidates? (Maybe omit some. Maybe transform some to be suitable, e.g. `#{impersonate-box}` IIUC for rhombus?) Thoughts? * * * p.s. This program ```scheme #lang rhombus #{impersonate-box} ``` gives the error `impersonate-box: unbound identifier in: #{impersonate-box}`. So maybe (as usual) I'm confused what the problem even is here. ---
**Author:** ![samth](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/samth/32/3_2.png) [@samth](https://racket.discourse.group/u/samth)\ **Post date:** [November 17, 2023, 3:07pm UTC](https://racket.discourse.group/t/imports-as-completion-candidates-for-hash-langs/2511/2 "2023-11-17T15:07:12Z")
The fully expanded version of: ```scheme #lang rhombus 1 ``` is ```scheme (module anonymous-module rhombus (#%module-begin (#%declare #:realm rhombus #:require=define) (#%app call-with-values (lambda () '1) print-values) (module configure-runtime racket/base (#%module-begin (module configure-runtime '#%kernel (#%module-begin (#%require racket/runtime-config) (#%app configure '#f))) (#%require rhombus/runtime-config))))) ``` That doesn't have any imports that involve `racket/base`, except in the submodules. So I don't know where either DrRacket or racket-mode is getting the relevant bindings. ---
**Author:** ![greghendershott](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/greghendershott/32/98_2.png) [@greghendershott](https://racket.discourse.group/u/greghendershott)\ **Post date:** [November 17, 2023, 4:05pm UTC](https://racket.discourse.group/t/imports-as-completion-candidates-for-hash-langs/2511/3 "2023-11-17T16:05:10Z")
OK I see now. The `configure-runtime` module uses `racket/base` and contributes these candidates. The status quo approach in `racket-xp-mode` is to err on the side of more candidates. (The philosophy is to save typing. Not a perfect mad lib copilot.) So it throws candidates from all submodule langs and imports into one big pool. Ignoring `configure-runtime` modules might be a reasonable tactical hack (for this example "it works"). * * * The approach in `pdb` (still WIP) is a little smarter. It scopes candidates by submodule (and is module+ aware). Requests for completion candidates can supply a position. That avoids this. Even so, `pdb` isn't currently attempting to limit candidates perfectly down to intra-module lexical scope. And, when the file is in an error state, even `pdb` falls back on the "one big pool" approach -- you get all candidates from all modules, from the previous successful expansion. After all, if the file has an expansion error, strictly speaking we can't know in what submodule the user wants completion candidates. Completion is weird this way -- the program is being edited and is almost always in a temporarily invalid state. ---
**Author:** ![samth](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/samth/32/3_2.png) [@samth](https://racket.discourse.group/u/samth)\ **Post date:** [November 17, 2023, 4:48pm UTC](https://racket.discourse.group/t/imports-as-completion-candidates-for-hash-langs/2511/4 "2023-11-17T16:48:35Z")
I think it might be better to ignore `module` subforms (and consider only `module*` submodules) for the purposes of providing completions outside the submodule. Another probably good idea is to ignore submodules that have no source locations from the actual module. ---
**Author:** ![greghendershott](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/greghendershott/32/98_2.png) [@greghendershott](https://racket.discourse.group/u/greghendershott)\ **Post date:** [November 17, 2023, 5:07pm UTC](https://racket.discourse.group/t/imports-as-completion-candidates-for-hash-langs/2511/5 "2023-11-17T17:07:19Z")
I agree. `pdb` already does that -- tracks imports by module, understands `module*` nesting visibility, knows the srcloc extents of module forms (even `module+` forms spliced from disjoint source spans). The "tactical hack" idea is for `racket-xp-mode`'s simplistic back end. I'm trying to get to the point with `pdb` where `racket-xp-mode` can become a legacy thing used only by people who need to work with older versions of Racket. (I'd prefer `racket-xp-mode` just change to use `pdb` on the back end. Unfortunately `pdb` needs some things added only in very recent versions of Racket. So for older Rackets, I need to keep the old `racket-xp-mode` back end for awhile. I just don't want to invest _too_ much time in enhancing it.) ---
**Author:** ![greghendershott](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/greghendershott/32/98_2.png) [@greghendershott](https://racket.discourse.group/u/greghendershott)\ **Post date:** [November 17, 2023, 7:16pm UTC](https://racket.discourse.group/t/imports-as-completion-candidates-for-hash-langs/2511/6 "2023-11-17T19:16:31Z")
p.s. I guess more broadly there's some entire other story when it comes to completion candidates and dotted names. Looking at `gui_demo.rhm` we have things like `math.max`. OK. Let's make a simpler example: ```rhombus #lang rhombus math.max(1, 2) ``` If a user wanted to type `math.` and get completions including `max`... I don't see how this is supposed to work. First, the `rhombus` exports include what seems like a couple dozen `phase+space` variations. From looking at the result of ```racket (parameterize ([current-namespace (make-base-namespace)]) (namespace-require '(lib "rhombus")) (module->exports '(lib "rhombus"))) ``` I see: - `math` is exported under phase+space `(0 . rhombus/namespace)`. And FWIW check-syntax draws an arrow from it to the `rhombus` module language. - But `max` is not exported anywhere. And FWIW check-syntax doesn't draw an arrow for it. Is there some "rhombus-namespace-\>exports" function? If so, I could imagine discovering things like `max` that way. But even if so, is there some lang-independent mechanism? * * * p.s. The following is N/A for discovering completion candidates -- which needn't already be present in the user program, of course -- but just of note, in fully expanded code `max` is apparent but `math` isn't: ```racket (module rhombus-math rhombus (#%module-begin (#%declare #:realm rhombus #:require=define) (#%app call-with-values (lambda () (#%app max (quote 1) (quote 2))) print-values) (module configure-runtime racket/base (#%module-begin (module configure-runtime '#%kernel (#%module-begin (#%require racket/runtime-config) (#%app configure (quote #f)))) (#%require rhombus/runtime-config))))) ``` However `max` does have a syntax property `'rhombus-dotted-name` with value `'math.max`. ---
**Author:** ![samth](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/samth/32/3_2.png) [@samth](https://racket.discourse.group/u/samth)\ **Post date:** [November 17, 2023, 8:30pm UTC](https://racket.discourse.group/t/imports-as-completion-candidates-for-hash-langs/2511/7 "2023-11-17T20:30:25Z")
I think ~0 thought has gone into APIs that could be used for completions in the rhombus context. One thing to note is that the expansion of ```scheme import rhombus/gui as G ``` expands to (roughly) ```scheme (for-meta 0 (portal G ((import (lib "rhombus/gui.rhm") (lib "rhombus/gui.rhm") mod-ctx) G G))) ``` which may be useful for finding identifiers to complete after `G.` ---
**Author:** ![greghendershott](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/greghendershott/32/98_2.png) [@greghendershott](https://racket.discourse.group/u/greghendershott)\ **Post date:** [November 17, 2023, 9:20pm UTC](https://racket.discourse.group/t/imports-as-completion-candidates-for-hash-langs/2511/8 "2023-11-17T21:20:20Z")
So IIUC there is some prefix mechanism for things like `math.min`. And there is another prefix mechanism using portal syntax, with `import as`. One confusion I have is that [`#%require`](https://docs.racket-lang.org/reference/require.html#%28form._%28%28quote._~23~25kernel%29._~23~25require%29%29) documents `portal` as: > (portal _portal-id_ _content_) but there is no grammar for _content_, just this prose: > The portal form provides a way to define [portal syntax](https://docs.racket-lang.org/reference/stxtrans.html#%28tech._portal._syntax%29) at any phase level. A (portal _portal-id_ _content_) defines _portal-id_ to portal syntax with _content_ effectively quoted to serve as its content. On the one hand "quoted anything" seems super flexible for a lang to use, which is great. On the other hand this is opaque to a tool -- how can it know what `content` _means_? If something like `([import (lib "rhombus/gui.rhm") (lib "rhombus/gui.rhm") mod-ctx] G G)` isn't following a documented grammar, is the idea that some new lang-info function can cash this in for more information? Or some other idea? (Beyond completion candidates: This seems discouraging because I'm now realizing this impacts `pdb`'s goal to track names across files/modules to support multi-file find-references and renaming. I thought I had nearly all the bases covered but it seems like the onion has more layers. Or... maybe it's good news, once I understand it, because `(#%require [prefix _])` relies on renaming, which forces using syntax props to retain the original pieces -- but maybe this new approach avoids that lossy-ness?) ---
**Author:** ![robby](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/robby/32/7_2.png) [@robby](https://racket.discourse.group/u/robby)\ **Post date:** [November 17, 2023, 9:29pm UTC](https://racket.discourse.group/t/imports-as-completion-candidates-for-hash-langs/2511/9 "2023-11-17T21:29:56Z")
I'm forgetting the pragmatics of portal syntax, but I think it is a way to put a definition directly into a provide which you can then access without actually running the corresponding module body. So it is a way to export something and let outsiders access it without having to run a bunch of code. I'm totally blanking on why it is there but surely it is something to do with rhombus. So, while I don't have a helpful technical nugget to add here but I just wanted to say that I agree with @samth that there is some wide open design space here to figure out the Right Way to add information into the macro system for use with completion and figuring something out seems like a fun design challenge. (Of course, academics are rewarded for doing that kind of thing and then publishing papers about it so that colors my view of these things ...) ---
**Author:** ![greghendershott](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/greghendershott/32/98_2.png) [@greghendershott](https://racket.discourse.group/u/greghendershott)\ **Post date:** [November 17, 2023, 9:47pm UTC](https://racket.discourse.group/t/imports-as-completion-candidates-for-hash-langs/2511/10 "2023-11-17T21:47:35Z")
> I'm forgetting the pragmatics of portal syntax, but I think it is a way to put a definition directly into a provide which you can then access without actually running the corresponding module body. So it is a way to export something and let outsiders access it without having to run a bunch of code. I'm totally blanking on why it is there but surely it is something to do with rhombus. And that aspect sounds wonderful for tools! > So, while I don't have a helpful technical nugget to add here but I just wanted to say that I agree with @samth that there is some wide open design space here to figure out the Right Way to add information into the macro system for use with completion and figuring something out seems like a fun design challenge. (Of course, academics are rewarded for doing that kind of thing and then publishing papers about it so that colors my view of these things ...) Understood about completion candidates, and for some users that's more of a nice-to-have than a must-have. I do want to underline the thing buried at the end of my last message: For `pdb` I'm trying to track definition and use sites across files ("inter-file arrows"). So if a user program has `math.min`, I need to know where the `math` and `min` pieces come from. Likewise for imported or exported `prefix.original`. I'd eventually managed to work that out for the `#%require` and `#%provide` forms' documented grammar, AFAIK, including making some PRs to preserve components' srclocs. But now... argh. To me this feels more fundamental. (And given that information reliably, I think it's free or at least pretty short strokes to completion candidates.) ---
**Author:** ![robby](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/robby/32/7_2.png) [@robby](https://racket.discourse.group/u/robby)\ **Post date:** [November 21, 2023, 2:01pm UTC](https://racket.discourse.group/t/imports-as-completion-candidates-for-hash-langs/2511/11 "2023-11-21T14:01:43Z")
re the expansion of `import` in `rhombus`: is the information in the portal portion of the import part of some public information that follows a format tools are expected to use? Or is it internal information that rhombus is using? The documentation for `portal` would seem to suggest the former, which suggests that changes in rhombus are needed before tools are going to be able to autocomplete, I suppose? Is this also a correct expectation for Greg's inter-file arrows that `pdb` wants? ---
**Author:** ![mflatt](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/mflatt/32/6_2.png) [@mflatt](https://racket.discourse.group/u/mflatt)\ **Post date:** [November 21, 2023, 2:24pm UTC](https://racket.discourse.group/t/imports-as-completion-candidates-for-hash-langs/2511/12 "2023-11-21T14:24:31Z")
Currently, that format is intended as internal to Rhombus. Although some part of the format could be pinned down, that would still be Rhombus-specific. I'm not sure of the right direction here to support tools. Maybe there should be something in a language that points to a tool for extracting this kind of information from a module expansion, similar to the way that a language says how to handle coloring or indentation on source. Or maybe there should be a designated submodule name, where a module expansion defines the submodule to expose this kind of information. ---
**Author:** ![samth](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/samth/32/3_2.png) [@samth](https://racket.discourse.group/u/samth)\ **Post date:** [November 21, 2023, 3:10pm UTC](https://racket.discourse.group/t/imports-as-completion-candidates-for-hash-langs/2511/13 "2023-11-21T15:10:25Z")
I feel like it's worth distinguishing two different issues here: 1. How do we draw a check-syntax arrow for `math` in @greghendershott's example program? Maybe the answer here is as simple as using a `'disappeared-use` in the resulting `max` identifier, but at no point in the macro expansion steps visible in the macro stepper is `math` even shown as an identifier that has a useful binding (although that may be a space issue?). 2. Since Rhombus makes heavy use of `a.b` references, how do we do completion of such forms? Racket's completion support has never been as good as in other languages, for a variety of reasons, and we have a chance to do better here with Rhombus, but right now we're in a worse position instead. (2) seems like a problem that will take a bunch of thought and iteration, but (1) seems like a more immediate regression wrt things that we expect Racket-based languages to support, and also seems like it will probably be easier to fix. ---
**Author:** ![samth](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/samth/32/3_2.png) [@samth](https://racket.discourse.group/u/samth)\ **Post date:** [November 21, 2023, 3:47pm UTC](https://racket.discourse.group/t/imports-as-completion-candidates-for-hash-langs/2511/14 "2023-11-21T15:47:18Z")
I just realized I got `math` and `max` backwards in this post -- `math` gets an arrow currently, but `max` does not because it is not `syntax-original?` even though it appears in the fully-expanded code. ---
**Author:** ![robby](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/robby/32/7_2.png) [@robby](https://racket.discourse.group/u/robby)\ **Post date:** [November 21, 2023, 3:48pm UTC](https://racket.discourse.group/t/imports-as-completion-candidates-for-hash-langs/2511/15 "2023-11-21T15:48:06Z")
@samth 's thinking makes sense to me but I'd be more ambitious on point 2. Since Rhombus has a different expander, maybe we should be asking "can a change to the way the expander works mean that we have completion information more readily available with less work on the part of the macro author?". Note that I have no idea how to get a positive answer to that! ---
**Author:** ![greghendershott](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/greghendershott/32/98_2.png) [@greghendershott](https://racket.discourse.group/u/greghendershott)\ **Post date:** [November 21, 2023, 3:49pm UTC](https://racket.discourse.group/t/imports-as-completion-candidates-for-hash-langs/2511/16 "2023-11-21T15:49:55Z")
Yes, two distinct issues. But I think maybe two sides of same coin? IMHO any lang ought to expand exports and imports into the grammar for fully-expanded programs, which includes the grammars for `#%require` and `#%provide`. (But let's exclude the `portal` clause which is a kind of quote of undocumented content.) If that grammar isn't optimal, then let's extend the grammar in a documented way? It's currently not optimal for exporting/importing identifiers composed of multiple other identifiers (e.g. prefixes). - For example `#%require` has a `prefix` clause, which is great because it allows tracing the provenance of the prefix and prefixed identifiers. But it doesn't support nested `prefix`. (I added syntax properties for this, but only `racket` `require` and `provide` add those.) - `#%provide` lacks even non-nested `prefix`. (Ditto.) What if we extend the grammar for `#%require` and `#%provide` with new clauses that expose the composite identifiers in a documented, principled way? p.s. If a tool knows that `Math.min`, `Math.max`, etc. are the identifiers, then it can provide completions for all of those after typing `Math.`. Maybe it can even restrict choices based on originating from the same clause? Anyway, I understand there may be fancier completion _validation_ and choice reduction tactics, and that's cool, and maybe research-y. But my simpler concern atm is (a) provenance for features including but not limited to completion and (b) basic completion. p.p.s. I realize any program can smuggle things in using `dynamic-require` and things built on that like `lazy-require` or `rackunit`'s `require/expose` (which also uses `module->namespace`). I don't see any great answer for that, except user's will probably understand tools can't see through such tricks. I just think that "normal" langs should not work invisibly, if possible. ---
**Author:** ![greghendershott](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/greghendershott/32/98_2.png) [@greghendershott](https://racket.discourse.group/u/greghendershott)\ **Post date:** [November 21, 2023, 3:55pm UTC](https://racket.discourse.group/t/imports-as-completion-candidates-for-hash-langs/2511/17 "2023-11-21T15:55:03Z")
p.p.p.s. If `#%require` and `#%provide` are growing too baroque, or assume too much: There could also be a clean sheet of paper and new `#%import` and `#%export` forms? ---
**Author:** ![robby](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/robby/32/7_2.png) [@robby](https://racket.discourse.group/u/robby)\ **Post date:** [November 21, 2023, 4:03pm UTC](https://racket.discourse.group/t/imports-as-completion-candidates-for-hash-langs/2511/18 "2023-11-21T16:03:05Z")
All of this makes sense to me, and we can probably even make the change in a backwards compatible way if we use properties (although maybe that's icky). On this point: > [@greghendershott](#): > > p.p.s. I realize any program can smuggle things in using `dynamic-require` and things built on that like `lazy-require` or `rackunit`'s `require/expose` (which also uses `module->namespace`). I don't see any great answer for that, except user's will probably understand tools can't see through such tricks. I just think that "normal" langs should not work invisibly, if possible. Sometimes those macros that generate those dynamic-requires are going to know the names (or know where to get them?), and they are there only to avoid pulling in the depenencies / running code at inopportune times. Maybe such things could communicate the information that a tool would want by leaving behind such information in properties too. (At this stage, maybe just keeping in mind that we might want to get that done is enough.) ---
**Author:** ![greghendershott](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/greghendershott/32/98_2.png) [@greghendershott](https://racket.discourse.group/u/greghendershott)\ **Post date:** [November 21, 2023, 4:30pm UTC](https://racket.discourse.group/t/imports-as-completion-candidates-for-hash-langs/2511/19 "2023-11-21T16:30:34Z")
> [@robby](#): > > Sometimes those macros that generate those dynamic-requires are going to know the names (or know where to get them?), and they are there only to avoid pulling in the depenencies / running code at inopportune times. Maybe such things could communicate the information that a tool would want by leaving behind such information in properties too. (At this stage, maybe just keeping in mind that we might want to get that done is enough.) I mentioned that just to show I understand that "expand to a documented grammar for composite identifiers" doesn't magically cover all possible cases. 🙂 I agree situations with `dynamic-require` aren't necessarily hopeless. I just feel it's more nice-to-have or "version 2". Like, as a user, I might _want_ a multi-file find-references or rename command to know about all instances of "foo" in `(provide foo)`, `(define foo)`, `(dynamic-require 'mod 'foo)`. But I wouldn't be shocked if it missed the last one. Whereas in situations like `(provide (prefix-out pre foo))` and in some other file `(begin (require (prefix-in pre mod)) pre-pre-foo)` -- the last "foo" IMHO is a must-have. And so are each of the distinct "pre"s. That ought to work for all langs that do prefixed imports/exports with whatever surface syntax. ---
**Author:** ![samth](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/samth/32/3_2.png) [@samth](https://racket.discourse.group/u/samth)\ **Post date:** [November 21, 2023, 4:32pm UTC](https://racket.discourse.group/t/imports-as-completion-candidates-for-hash-langs/2511/20 "2023-11-21T16:32:30Z")
Right now, Rhombus is expanding into a program the certainly follows the grammar for fully-expanded programs. But it effectively works like this: ```scheme ;; module M #lang racket (define-syntax (scope stx) (syntax-case stx (math string) [(_ math id) (with-syntax ([i (syntax-e #'id)]) #'(let () (local-require (only-in racket/math i)) i))] [(_ string id) (with-syntax ([i (syntax-e #'id)]) #'#'(let () (local-require (only-in racket/string i)) i))])) (provide scope) ``` ```scheme ;; module N #lang racket (require M) (scope math max) ``` Except that `math` and `string` in Rhombus are lifted out to be top-level definitions so they could be potentially renamed. I don't think it could be changed to use an extension to `require` for hierarchical names, because those are too flexible in Rhombus -- the name before the dot has full control over the meaning of the dotted name, and that's used to implement everything from namespaces to import prefixes to struct field access. But I also don't think we need to do that, at least to handle renaming. Instead, we can use the tools we already have, or extend them a little (eg, by providing syntax properties that talk about dotted names specifically and could also be used for `syntax-parse`), and continue to build things from the fully-expanded program. As a proof-of-concept, this commit: [Add disappeared-use for fields. · samth/rhombus-prototype@4c700bd · GitHub](https://github.com/samth/rhombus-prototype/commit/4c700bd47c7fce387fffff792fa85cb93086608f) improves things a bit (there's now a link to the docs for `max`) but it doesn't yet let you jump to the definition (and it doesn't do anything useful for `math.pi`). [Next page](https://racket.discourse.group/t/imports-as-completion-candidates-for-hash-langs/2511.md?page=2)