Markdown to Scribble converter

Questions & Answers
, , ,
1

In my opinion, it's often reasonable to start with some documentation in Markdown to get things going, but convert the documentation to Scribble later so it can be included in the Racket or package documentation to look nicer and to be included in the Racket documentation search function.

I'd be interested in whether a converter from Markdown to Scribble exists. My web searches didn't find one. Do you know such a converter?

By the way, as far as conversion between text formats goes, checking out Pandoc is kind of obvious, but unfortunately Pandoc doesn't support Scribble output. That said, there's an API to plug in custom writers in Lua. It would be great to implement a Scribble writer like this, but personally I don't expect to find the time for this anytime soon. :frowning:

3 Likes
2

You could create a pollen project and then write some code that outputs scribble documentation instead :wink:

Personally I am not a big fan of markdown, I have trouble understanding its popularity.
6.2 Optional reading: the case against Markdown

Pollen markup however has felt quite satisfying after an initial learning curve and getting used to how to structure and implement things using it. I am far from a pollen expert, but I really like how it allows me to mold it to what I want to do with it.

With markdown I am just forced to take it as it is, probably that is why it is used, maybe it is more a data exchange format / more about compatibility.

Anyways pollen also has a markdown mode, so you still could use pollen to generate scribble from markdown and switch to pollen markup if it turns out useful later. Although that probably doesn't make the process of writing documentation easier if that is the intent.

I am not sure building a direct/dedicated markdown -> scribble converter is actually that helpful, I feel like you would quickly run into cases where you want to do things that scribble can do, but markdown can't. Then you would have to convert to scribble and start using it as the source of truth / source code and at that point you have to use scribble anyway. So personally I would just start with scribble, instead of later on having to switch to it.

The other case I could imagine is that eventually more and more conventions are bolted on to markdown, to allow more extra cases of "this one simple thing" gets converted to "this thing" in scribble, at that point you no longer have a standardized markdown format, which makes it lose its only benefit?
In that situation a pollen project using markdown (or pollen markup) also would make more sense, because you basically have a personalized conventions and transformations project, which seems like a good fit for pollen, to me.

If you feel like markdown has other benefits, I am interested to hear about those.

Having said all that I am unsure in which cases it would actually be enjoyable / advisable to generate scribble using pollen, over just writing scribble directly. But if somebody does this I would definitely like to take a look at the code.

Maybe a more reasonable approach would be to implement some scribble dialect #lang scribble/easy that maybe isn't as powerful but you can get started with it very quickly and maybe even has some kind of "upgrade" command that converts the file from #lang scribble/easy to another #hash scribble/...?
(this would basically be what bsl is in relation to racket)

I think that sounds good, but it isn't a priority for me, so I can't actually commit to implementing something like this.

1 Like
3

From Greg Hendershott's markdown, there's xexprs->scribble-pres, which I've used for real purposes.

I think Alexis King also had something based on her more recent CommonMark: Standard Markdown package, and certainly using its structured representation would be a good basis from which to build something. Maybe I was thinking of this, for her blog? lexi-lambda.github.io/markdown.rkt at source · lexi-lambda/lexi-lambda.github.io · GitHub.

Alexis has a comparison of the two packages here.

3 Likes
4

I'll try to answer some of your points and will probably miss a few others. :wink:

Why Markdown?

While you can compare Scribble (or Pollen) with Markdown, in my opinion it's not that useful to conclude that one of the two is "better." There are just different tradeoffs for both formats.

These are features I like about Markdown:

  • Relatively easy to read even in non-rendered mode

  • Easier to write. For me, this means that writing Markdown breaks my writing flow much less than writing Scribble. That's why I like to write (at least a draft version) in Markdown, even if I can add more sophisticated markup only later with Scribble.

  • More likely to get contributions because of the previous point and because much more people are familiar with Markdown than with Scribble. This may be not or less true for people in the Racket ecosystem. I really can't tell; I'm quite sure it'll depend on the person you ask.

Use cases

I'll try to describe a bit more why I asked for such a conversion tool from Markdown to Scribble. To integrate well with other Racket documentation, using Scribble (at least eventually) (or Pollen?) is a must. Now the question is: when/why would I write something in Markdown instead of directly in Scribble?

  • One use case is that I wrote a README.md for one of my projects. It makes sense to have this rendered from Markdown on Sourcehut (but would similarly apply to Github or Gitlab). It would be nice if I could create a Scribble version of the README.md file for the documentation on the Racket package server without converting the Markdown to Scribble purely manually. This rationale would also apply to other README files and projects.

  • My other use case: For the glossary project that I mentioned in another thread, I hope to get contributions more easily if I make the threshold for contributions as low as possible. I can hopefully write many of the glossary entries myself, but for others I'm bound to need some help. So having a low "contribution barrier" has the highest priority for me, even if that means I have to convert Markdown contributions to Scribble myself. And of course it would be nice if I didn't have to make these conversions manually. :slight_smile:

For the first use case, the current situation is that I wrote the API documentation for the project directly in Scribble. I may create Scribble documentation for the command line tool later.

For the second use case, I think an either/or approach isn't necessary. Since I rely on contributions, I'll change the README of the glossary project to encourage contributors to send me Markdown or Scribble content, whatever they prefer. :slight_smile:

Markdown extensions

I don't think it should be a goal to make Markdown as expressive as Scribble, Pollen or LaTeX. Markdown isn't designed for that. Moreover, bigger changes/enhancements to Markdown would defeat the purpose of having a format that's easy to write and read.

Relatedly, a middle ground between Markdown and Scribble is reStructuredText, which is quite common in the Python ecosystem. (For example, it's used for the Python documentation.) ReStructuredText allows some things that Markdown doesn't, but I usually dislike writing in reStructuredText compared to Markdown. (That said, the two are much closer to each other than to LaTeX or Scribble.)

2 Likes
5

I consider the possibilities of Markdown a (small) subset of those of Scribble. Therefore, I asked only for the conversion from Markdown to Scribble, not the other way around. :slight_smile:

1 Like
6

That was a mistake on my part I only meant markdown -> scribble

7

Just you weren't aware, another possibility for this scenareo is generating Markdown from Scribble. The Racket build guide is a notable example: it is rendered to racket/build.md at master · racket/racket · GitHub in the main Racket Git repository so you can read it on GitHub, but it is generated from Scribble (racket/pkgs/racket-build-guide/build.scrbl at master · racket/racket · GitHub), so you can also read it on the docs site (Building, Distributing, and Contributing to Racket) with all the rich cross-references.

2 Likes
8

The CommonMark parser by Alexis produces structures representing the parsed markdown.
So a recursive descent emitting Scribble formatted output to a file seems to be relatively straight forward.

https://docs.racket-lang.org/commonmark/index.html#(part._structure)

2 Likes
9

Thanks for all your replies! :slight_smile:

@simonls @soegaard So Alexis King's CommonMark parser in combination with a to-be-written Scribble serializer seems to be the way to go. That said, I might even prefer the Pandoc approach. I'm not sure yet which one I'd take.

@LiberalArtist Nice! Indeed I wasn't aware of the Scribble-to-Markdown conversion. Although using this approach (for my first use case) would lose the simplicity of writing Markdown, it would avoid maintaining two formats for the same document.

I marked @soegaard 's reply as the "Solution" because it comes closest to one (apart from the Pandoc approach).

Probably I'll implement a Markdown-to-Scribble conversion sooner or later, but I don't know when. If someone reading this post in the future wants/needs such a converter or if they want to work on one, please contact me in a message on this forum (private or in this thread), so we can avoid double work. :slight_smile:

3 Likes
10

Another example of this (generating markdown from scribble) is the quickscript-extra package:

2 Likes
11

The source code for Alexis's blog seems to have a Markdown-to-Scribble converter, which might be a useful starting point: lexi-lambda.github.io/blog/markdown.rkt at source · lexi-lambda/lexi-lambda.github.io · GitHub

3 Likes
12

I don't fully understand the code, but I think it is for converting from Markdown to Scribble datastructures. What I'd like instead is a conversion to Scribble code, so that it can be edited further. (But maybe you meant exactly what I wrote here. :slight_smile: )

1 Like
13

To throw another approach into the mix: I’ve been working for a few days on a #lang that combines Alexis’s CommonMark parser with a Scribble-like reader. The intent is to have something that acts something like Pollen, but uses Markdown as a starting point, with the ability to add custom markup via Racket functions where desired. What you end up with is an AST that is like CommonMark’s but in the form of a regular quoted list, not structs, which can be converted to HTML (or Scribble or LaTeX or whatever) in a fairly straightforward way. More details here.

1 Like