Announcing uni-table - a Unicode text table renderer with ECMA-48 SGR support

[dominik.pantucek]

IT SLICES! IT DICES!

It causes a 100,000 light year-diameter quantum explosion!

Based on one innocuous remark[1] and a Sunday evening package
popularity poll results[2], I finally decided to put together various
bits and pieces of code that have been scattered across my
private projects for a few years now and I assembled them into a
package anyone can use.

Rendering tables is fun - especially if you want each and every cell
to behave like an ECMA-48 terminal with proper support for (most of)
Set Graphics Rendition control codes. Colors, italics, underlines - if
you can name it, it is there. See ECMA-48 Fifth Edition - June
1991[3], section 8.3.117 for reference - or the ECMA SGR module of
this package. Combining solid, dashed, thick and thin borders in all
combinations makes the tabular data even more pleasing to the eye.

The public API is fairly straightforward and contains just two
procedures - one for printing the result to (current-output-port) and
one for obtaining the result as a string value. An example showing the
most important features is right on the front page of package's
documentation[4]. The package also comes with an extensive
documentation of its internals - all private modules are completely
scribbled using scribble/srcdoc.

In addition to the documentation, the whole package internals are also
heavily contracted using both inter-module and intra-module
contracts. I hope that others will have a look at the code[5] and it
will fuel the ongoing discussion about contract usage in various
scenarios. My two cents would be that it was extremely useful to have
the intra-module contracts in place during development - especially in
connection with the extensive tests. This combination allowed me to
quickly test some of the various approaches available and decide
on-the-fly what works the best (at least for me).

Although the internal package version is 0.5.1 - meaning it is far
from complete - there are no plans to change the public API and/or its
behavior. That is the reason why I am releasing it right now. The
internals may (and probably will), however, change a lot in the
future. There are still some features missing which I need to
implement before it can fully replace all my other code so expect more
to come in the future.

In the meantime, I'd love to hear any feedback, suggestions or any
comments that anyone may have regarding both the functionality and the
code itself. At the upcoming Racket meetup next Saturday I plan to
show some of the features live in REPL and I will be available for
answering questions there.

Special thanks go out to:

• Jay McCarthy - for reminding me that I am not the only old-timer
  spending most of his time in the text-only terminal (and for
  organizing RacketCon),
• Stephen De Gabrielle - for that Sunday evening package popularity
  poll (and his relentless popularization of Racket itself), and
• Laurent Orseau - for the text-table package, of course.

Cheers!
Dominik

[1] RacketCon 2021 - Day 1 - Session 3 - Bogdan Popa & Ryan Culpepper - YouTube
[2] What are the most used packages? - #6 by spdegabrielle
[3] https://www.ecma-international.org/wp-content/uploads/ECMA-48_5th_edition_june_1991.pdf
[4] Unicode Tables
[5] Dominik Joe Pantůček / uni-table · GitLab

[jbclements]

I'm embarrassed to admit this, but I really don't know what ECMA-48 SGR support allows; I clicked through to your package documentation to see examples, but it wasn't immediately obvious what was the new thing, or when I would use this rather than text-table. Would it make sense to add a few more examples to the first page of the documentation that show off the new features?

[dominik.pantucek]

I wouldn't worry - people usually refer to it as "ANSI terminal" or similar. Improving the user documentation is a high priority on my TODO list, I expect to add more examples and better description there. In the meantime, try running:

racket -l uni-table

It shows some basics without having to write any ECMA CSI SGR (ANSI) codes manually.

Trouble is that I have currently no clue how to render colored examples in documentation. You can try a simple test program in your terminal:

#lang racket/base
(require uni-table)
(define cells '(("\e[90m.\e[33mo\e[93mO"
                   "   \e[1;3;4muni-table\e[0m   "
                   "\e[93mO\e[33mo\e[90m..."
                   )
                  ("\e[44mX"
                   " * easy-to-use \n * UTF-8 \n * ECMA-48 SGR "
                   "\e[44mZ")
                  (1 2 3)
                  ("A" "B" "C")))
  (print-uni-table cells
                   #:row-borders '((heavy solid) () () (light))
                   #:cell-borders '(dashed)
                   #:column-borders '(() () (heavy)))

You should get something like:

Basically if you have an "ANSI terminal" compatible output somewhere and you want to put it into a table - it works.

And of course, thank you for the feedback! It's greatly appreciated.

[Laurent.O]

Looks great. Trying to render unicode and text colours in docs has been a annoying experience for me. I would suggest using screenshots, even though that sounds like missing the whole point

[soegaard]

In case you are using macOS, you can use the neat screencapture utility using system.

#lang racket
(require racket/gui)


(define (grab-screen)
  ; -c   use clipboard
  (system "screencapture -c")
  (send the-clipboard get-clipboard-bitmap 0))

(grab-screen)

There is an option to capture a single window too, but I don't know a simple way of getting the window id. The workaround must be to put the terminal window in full screen and then crop the output after the fact.

[dominik.pantucek]

I would like to include the output in textual form. With the internal representation (snapshots of the state of the terminal) it is relatively easy to output HTML or similar markup. Getting that in scribble output might be much trickier though. And for the screenshots - pict with fixed-size font% goes well with scribble, so I would opt for that if pure text transformation won't be feasible - (meta)pict is available on all systems.

[Laurent.O]

Then here's the CSS I'm using with scribble to display unicode frames correctly:

Metaxal/text-block/blob/main/example.css

And usage examples here:
https://github.com/Metaxal/text-block/blob/main/scribblings/math.scrbl

It took me some time to get it right...

[dominik.pantucek]

Do I understand correctly that you fixed the line stretch for text-block documentation but left it as-is in text-table documentation? I'll try that and see how it renders. It doesn't, however, solve the problem with colored, italic, underline..
But no matter what, your table-block/math is a great stress-test for the SGR machinery:

Colors, italics and bold kinda works - although the difference is only for normal characters, superscripts and subscripts.

#lang racket

(require text-block uni-table)

(print-uni-table
 (for/list ((y 5))
   (for/list ((x 5))
     (cond ((and (eq? x 0) (eq? y 0)) "")
           ((eq? y 0) (integer->char (+ (sub1 x) (char->integer #\A))))
           ((eq? x 0) y)
           ((and (eq? y 1) (eq? x 1))
            ($formula '(+ (sqrt
                           (/ (log (/ (+ x 3)
                                      (- x 2)))
                              (- (expt x y) z)))
                          (f a b (/ c (+ a b))))))
           ((and (eq? y 1) (eq? x 2))
            (happend @sigma
                     " = "
                     ($sqrt (happend
                             ($/ 1 'N)
                             " "
                             ($sum "i=1" 'N)
                             ($sqr ($- ($_ 'x 'i)
                                       @mu))))))
           ((and (eq? y 1) (eq? x 3))
            ($_^ ($square-bracket (happend "3" ($sqrt 'x)))
                 "x=1"
                 ($sqr "n")))
           (else ""))))
 #:cell-borders '(light)
 #:row-borders '((heavy)())
 #:row-style '((bold white #:bg black)(#:bg black))
 #:row-align '((center)(middle))
 #:column-style '((bold)(red)(green italic)(#:bg DarkGrey yellow bold)())
 #:column-align '((middle)())
 #:border-style '(#:bg black)
 #:column-borders '((heavy)()))

[Laurent.O]

The docs for text-table don't look so bad, apart maybe from a single pixel, so I didn't feel much pressure to fix it. Does it look different on other browsers? (I'm using chrome)

ecma-csi-remove does work for displaying in DrRacket, but I wonder if it would be better to have a parameter that controls what happens at the individual procedure level (although of course having an ECMA-48-compliant text% would be ideal, hint hint )

FYI, in case that's any helpful (though I bet not), here's what your example looks like in gnome-terminal:

It looks fantastic!

Xterm doesn't work so well though, unfortunately:

No background color, and it doesn't revert to non-italics on failure .

[dominik.pantucek]

And here we go again - version 0.6 is out. But after this one, I really need to take a break

Based on the feedback, I revamped the documentation here and there, provided more rationale and examples to various parts of the package. I also implemented another feature that was basically ready for release - explicit and implicit repetitions in style templates.

Now if you want to specify something for head columns and something different for tails columns, it is pretty straightforward. The same applies for rows. Also groups of rows and columns with the same style specifications are now easy to write:

  (define tbl
    (for/list ((y 10))
      (for/list ((x 10))
        (case (list x y)
          (((4 4)) "big\nbigger\nthe biggest")
          (((0 0)) "")
          (else (format "r~ac~a" y x))))))
  (print-uni-table tbl
                   #:col-borders '((solid heavy) (left light dashed))
                   #:row-borders '((solid heavy) (top light dashed))
                   #:row-style '((brred) () ... (White))
                   #:col-style '((brmagenta) ())
                   #:row-align '((center) 3 (right)())
                   #:col-align '(4 (bottom) ())
                   #:border-style '(cyan)
                   #:table-border '(heavy solid))

Of course, the ellipsis pattern can be used only once in given template. And yes, this example show any actual difference only in the lower left corner for the head/tail row style specification

Also big thanks @Laurent.O for pointing me in the right direction with documentation styling, although I went with minimalist approach and basically just "patched" the default style using nested-flow wrapper:

.example table tr *{ 
    line-height: normal;
 }

And the rest can be seen in:

racketeer/uni-table/-/blob/master/scribblings/uni-table.scrbl

It will take some time before package build server kicks in and the compiled documentation is shown though.

Any testing will be highly appreciated too - I always try it under gnome-terminal, uxterm and konsole. If anyone can confirm that it works as expected on MacOS and Windows (really no idea about the latter, to be honest), that would be great too!

Btw, the MacOS approach suggested by @soegaard works in Ubuntu under GNOME with "gnome-screenshot -c" - and with "-w" it screenshots only the active window! But no, I'd like to do it properly by rendering it somehow...

ECMA SGR for text%? Yeah... I am thinking about that - but really not that soon. But generally speaking, I'd like to play with that a bit. Although that pict rendering will really be easier for a start...

[dominik.pantucek]

Couldn't resist and hacked a proof-of-concept:

Although I wonder, why the ((get-current-code-font-size)) does not return something closer to DrRacket code font size when run under DrRacket. I will finish the ECMA SGR pict renderer and release a minor update with colored examples in the documentation.

Creating a thin table->pict wrapper around this "virtual terminal" will do the trick, I am sure

[samth]

You might want to have a look at the simple-tree-text-markup package as well.

[jtp]

I wonder if it would make sense to have a print-to-HTML function. I don't know if anyone would actually use it for web sites but it would be an option for one issue I realized wile testing this package with some real world data I had in CSV files. A text based tool can't hide part of the contents of a cell the way a GUI spreadsheet can. That means that each column will be as wide as the widest piece of content in that column. One way around this would be to collapse down longer content and make it a link to another place with the full content.

[jtp]

I've been following various different table tools in Racket for a while now. One of the things missing is a tool which can work with large data sets with hundreds of thousands of rows or more. Unfortunately, uni-table takes several minutes to render a table, with about 23K rows, which LibreOffice Calc can open and display in less than five seconds.

I was wondering if this tool might work well for a preview feature for importing CSV files. This would be like the dialog box you see in LibreOffice Calc or Excel, where you can choose the delimiters and quote characters and see the results as you go. With the csv-reading package, you can create a reader which reads a CSV file one line at a time so that might be a good combination with uni-table.

[dominik.pantucek]

Actually text wrapping and hidden overflow for fixed-width columns is planned quite soon - that should put it on-par with web browsers rendering but in text terminal. The HTML output is fairly easy to implement as well, but I am not sure how to support certain features. But yes, in the future, some kind of HTML output is desirable.

My current goal is not speed but correctness. However with fixed-column widths (either explicitly specified or calculated using some heuristics) in place the next logical step is lazy rendering. However that needs some refactoring of the internals and therefore I've been postponing this until it is otherwise feature-complete.

Apparently your requirements @jtp are the same as mine - future versions will hopefully fulfill those requirements.

Btw, for low-level speedup Typed Racket is to be used once everything is in place.

[dominik.pantucek]

Although the pict renderer turned out to be a very useful and versatile thing, it yields visually quite different results on my laptop (Ubuntu 21.10, Racket 8.3 CS from PPA) and on the package build server. The example colored output as seen on[1] looks like this:

And the one rendered on my laptop is:

I guess it is due to the fact, that 'modern maps to different fonts on different platforms (that makes sense), but I am not sure how to get unified output - except for providing a "custom" font (DejaVu Sans Mono or similar) in the package. Which is - of course - an absolutely insane idea. Any other ideas?

And also the dark grey bars between appended picts are ugly - but I suspect some border drawing/not-drawing issues there, which means that I at least know what to investigate.

Apart from these limitations, 0.7 is out with hidden borders, pict renderer and the possibility of styling individual cells manually.

[1] Unicode Tables

[sschwarzer]

@dominik.pantucek

I guess it is due to the fact, that 'modern maps to different fonts on different platforms (that makes sense), but I am not sure how to get unified output - except for providing a "custom" font (DejaVu Sans Mono or similar) in the package.

I also think the different output is due to different fonts. At least I didn't see differences that couldn't be explained by different fonts.

Is it really important to get the same output (font) in all environments? I would rather see it the other way around: If a user has installed/configured a certain font for their terminal, they'll most likely expect that it'll be used. I certainly would.

I think a compromise could be that you just use the font that's currently used by the user, but in case a user sees odd font rendering, you can recommend in the documentation fonts that should work well. But you'll probably need some research for that, so I wouldn't it consider mandatory to have this information in the documentation.

You could even turn this around: You can mention in the documentation that some fonts lead to low quality output and ask users for font recommendations. In this case, I think it would make sense to provide an example program for that so that users don't recommend a font that doesn't work well with more complex layouts.

Stefan

[sschwarzer]

I now see that you wrote specifically about the pict renderer, so some of what I wrote may not apply. Still, I wouldn't worry too much about providing the same output on all platforms. As a compromise, you could add an optional keyword argument to choose a default font of your choice and/or a keyword argument to choose a font of their choice. Or maybe you could combine this in one keyword argument.

That said, if I were you I'd only implement this if someone asked for it. No need to make software unnecessarily complex.

Stefan

[dominik.pantucek]

After some experimenting, this is the best I can get with pict without going to dc%-based pixel-precision rendering:

Drawing background "border" and separating the background compositing from foreground and compositing them only when the line of text is to be completed seems to be the key here.
It still leaves me wondering why the small artifacts in the horizontal direction (see magnified top border for an example) and - even before any attempts to clean it up - no vertical artifacts. Any thoughts?

[sschwarzer]

My assumption is that for the horizontal line characters they just don't touch each other, maybe because of rounding errors.

On the other hand, for the vertical direction, it's maybe possible to specify (intentionally or unintentionally) the vertical distance of the text lines and if the vertical lines in the font are longer than the vertical distance of the text lines, the vertical lines reliably blend into each other. I don't think there's a corresponding concept for horizontal character distance, but maybe there is.