Shriram Krishnamurthi shared a pointer to this interesting series of pieces on "Transparent Telemetry" for open-source projects. Should products like Racket collect information about their use? Are there statistics that are sufficiently anonymous but might also be helpful in detecting problems? Russ Cox argues that there are. Would something like this make sense for Racket?
2 Likes
Recording what people search at docs.racket-lang.org could be interesting.
Especially it would reveal search terms that have 0 results.
1 Like
Ouch... that's not even telemetry, that's just "actually reading your web logs", more or less. Well, sigh, maybe we should assign all of our Racket Inc. summer interns to this.
I'm interested in this. Every failed onboarding is another person who misses out on either using the great ideas implemented in Racket. Every success is a chance for these ideas to be improved and/or spread to other languages.
What causes new users to stumble before they even ask a question is worth knowing;
- How do they get Racket?
- how do they first interact with Racket?
I got some useful feedback in a comment from u/vplatt responding to What are the most frequently asked questions about Racket? on https://www.reddit.com/r/Racket:
Having just gone through some of this, and trying various things out in the IDE without having read any of the documentation yet at that point, I can tell you the things that brought me up short at first:
- Where are the samples and how do I run them?
- Why don't the snippets in the help files (which are supposed to be complete) work? (Requires 'requires'!)
- How do I debug a program and step through code, and along with that, how do I view local variables, the stack trace, etc.? Can I do that in DrRacket or on the REPL? Why can't I set breakpoints in DrRacket?
- How do I write a program that takes command line arguments? What's the proper way to do this in Racket or Scheme? (Example program could cover this.)
- How do I compile the current program in DrRacket to an executable file? (Answer is 'raco' right, but how is that different from the menu option in DrRacket? )
- What are these different "languages" in Racket? Aren't they all just Lisp, or Scheme and how is Racket different or special compared to just normal Scheme? Is Racket the same as R5RS Scheme? Could probably just link the Wikipedia Racket article: Racket (programming language) - Wikipedia
- Why am I now unable to open my .rkt file in my other text editors? What did DrRacket do to my file and how do I fix it?
I'm as bad as anyone else - I know I should 'Read The Fine Manual' but I often just jump in and start pressing buttons and turning knobs - and I don't think I'm terribly unusual.
I believe a significant number of new users jump straight to the Racket command - perhaps based on experience with other programming languages.
With this in mind I believe opt-in to telemetry should be part of installation (including installarion via package managers), and to have effective coverage telemetry should cover
- The
racketcommand; commands that trigger an error - DrRacket; language selected, first errors (and what triggered)
- The Racket LSP Server; errors (and what triggered)
If possible Racket-Mode should be included as a significant number choose Emacs.
I believe there is a case for extending the onboarding in DrRacket beyond language selection, but it would be nice to have the data to show this is the case. Similarly I believe adding onboarding to the racket command may be valuable for users who never start DrRacket.
Stephen 
Related:
1 Like
Lots of the searching is handled client-side so it's not quite that easy.
2 Likes
Stephen De Gabrielle via Racket Discussions
notifications@racket.discoursemail.com writes:
With this in mind I believe opt-in to telemetry should be part of installation (including installarion via package managers), and to have effective coverage telemetry should cover
- The
racketcommand; commands that trigger an error- DrRacket; language selected, first errors (and what triggered)
- The Racket LSP Server; errors (and what triggered)
Maybe this is obvious to everyone, but it wasn't for the golang people
before a lot of pushback.
As someone who works on distro packaging for racket, I can't emphasize
enough how important the opt-in part is. If it's not opt-in, it will be
patched-out.
d
6 Likes
If this kind of stuff is opt-in, it becomes essentially useless. The people who would know how to opt-in to this usually aren't inexperienced newcomers, and those are the users whose toolchain problems we want to understand the most.
I'm not saying we should do it, just that from my perspective this is a go-big-or-go-home scenario.
What I want most is the number of download counts for packages registered at the package index. The package index server already has an ability to record this information, since it will get requested on raco pkg install/update.
Download count seems uncontroversial. Most package registries have it:
- PyPI has https://packaging.python.org/en/latest/guides/analyzing-pypi-package-downloads/
- npm has weekly downloads as a part of the interface: https://www.npmjs.com/package/leftpad
- Rust's crates are similar to npm: https://crates.io/crates/leftpad
Why is it useful? It allows people to discover packages based on popularity, which can often be a good proxy for maturity and quality. This can be especially helpful to people who are new to Racket, who don't know what is available in the package index.
5 Likes
The Transparent Telemetry link made a case for starting recording data immediately, but only sending once permission was granted.
Stephen De Gabrielle via Racket Discussions
notifications@racket.discoursemail.com writes:
The Transparent Telemetry link made a case for starting recording data
immediately, but only sending once permission was granted.
I would not support recording without the user giving explicit permission.
1 Like
This makes me think that logging might be a simpler and better choice. Unlike Go we don’t have the resource to monitor or even run a telemetry server.
Stephen
People have very strongly held views on metrics collection in open-source tools. I have observed the existence of telemetry features fracture other communities and cause long-term reputational harm to their projects.
I hope Racket will avoid any kind of opt-out metrics process (recording, sending, anything), as the risk of long-term harm feels too great to me.
I understand that the data may seem useful to answer various questions about user behaviour (I've wanted similar answers in the past for other projects), but I would recommend looking for other ways to answer those questions that ensures user consent along the way. For example, other projects seem to gain insight from annual user surveys. Opt-in metrics could work, but there's still a risk of burning time and reputation on something when it's unclear whether it will pay off in the end.
4 Likes