Sqlite3-connect crashes with "invalid memory reference", but only when racket/gui is required

Questions & Answers
1

I have an app of medium complexity that uses sqlite, with recent racket versions (I think since racket 8.2 maybe 8.1 dot something) while trying to connect to the database I get an error that doesn't give a lot of clues:

invalid memory reference.  Some debugging context lost
  context...:
   .../private/minimatch.rkt:33:17: fail
   /usr/share/racket/collects/racket/contract/private/arrow-val-first.rkt:555:3
  <<< more of my module stuff here... calling sqlite3-connect >>>

(I started the app with a racket git checkout I had laying around for a while, because I didn't have time to look into the error and it worked with that racket installation (and I thought that maybe I had messed up my other installation somehow), that checkout is version "Welcome to Racket v8.1.0.6 [cs]." commit e032be434e5918b92ee93c571b5531078642524c if it is a bug in racket it might have been introduced after that)

I use libsqlite, so I tried without it and I also get the error.
I tried a bunch of different parameter combinations for sqlite3-connect, even using a newly created database, nothing changed anything.

Then I tried to create a minimal example as a separate module / project.
And it simply worked, I was puzzled about what is the difference between my example program and my real one?

I debugged my program with the debugger couldn't see anything new,
except then I clicked on stuff in drracket and got this:

Bildschirmfoto_2021-10-23_14-02-48
Bildschirmfoto_2021-10-23_14-02-481055×226 32.3 KB

(while simply clicking the text editor in drracket)

So that got me thinking, maybe it is some kind of strange interaction between racket/gui and sqlite?
So I found that this program works for me, but fails when I require racket/gui:

#lang racket/base

(require db)
(require racket/gui) ;; comment out this line and it works

(define (test)
  (define c (sqlite3-connect #:database 'memory #:mode 'create))
  (displayln (query-value c "SELECT sqlite_version();"))
  (displayln "done"))

(module+ main
  (test))

fail output:

invalid memory reference.  Some debugging context lost
  context...:
   body of (submod "/home/sze/development/workspace/racket_projects/tools/sqlitetest.rkt" main)

working output:

3.36.0
done

If you open the non working program in drracket you get the internal error popup when you click on text, or at least I do with Racket v8.3 [cs]. Actually it seems the popups only start appearing when you have typed something within the editor.

Can anyone reproduce this error on their machine? I am running manjaro linux, if that matters.
Are there any other needed / helpful things I could post?

2

It works for me on:

macOS 10.15.5 (19F101)
GNU Emacs 25.3.1
Racket v8.0 [cs]

(Sidebar: I need to get around to fixing Emacs's idea of which Racket it should use!)

1 Like
3

I think this problem only started to appear for me, with racket 8.1 or later. (not sure anymore if I had the problem with the 8.1 release already or only starting with 8.2)

I think it might be related to new features in the db package, but that is currently only a guess, because there are some new features there (os-thread) maybe that introduced something in the internal "plumbing", that is somehow affected? @ryanc could that be the case?

4

I haven't been able to reproduce the problem yet. I've tried running the program you provided with Racket 8.2 CS and Racket 8.3 CS, both racket and drracket, both sqlite version 3.31.1 (default on Ubuntu 20.04.3) and 3.36.0 (built myself from the sqlite-autoconf tarball).

Yes, the changes to support os-threads also touched the code for "normal" mode. There was also a finalization fix around that time.

A wild guess: does the problem go away if you change the three occurrences of #:blocking? #t in racket/collects/db/private/sqlite3/ffi.rkt to #:blocking? #f and run raco setup?

5

Same error as before.
I might try to update my git checkout to a recent version and see whether I also get the error there on my machine, otherwise I might poke around in the db code and add a bunch of print statements to binary search my way to where exactly the error is being triggered.
I think there are already some logging things in the code I could activate to get more context, how do I do that?

6

I though maybe I can use lldb to run racket and get more info, I got a little bit.
If there are other things I could try with that, for example setting breakpoints in certain places, etc., let me know, I have only a little bit experience with using it. (also I think I might not have debug symbols, so it might not be that useful?)

lldb --arch x86_64 -- racket sqlitetest.rkt
(lldb) target create --arch=x86_64 "racket"
Current executable set to 'racket' (x86_64).
(lldb) settings set -- target.run-args  "sqlitetest.rkt"
(lldb) 
(lldb) run
Process 6127 launched: '/usr/bin/racket' (x86_64)
Process 6127 stopped
* thread #1, name = 'racket', stop reason = signal SIGSEGV: invalid address (fault address: 0x0)
    frame #0: 0x0000000000000000
error: memory read failed for 0x0
(lldb) bt
* thread #1, name = 'racket', stop reason = signal SIGSEGV: invalid address (fault address: 0x0)
  * frame #0: 0x0000000000000000
    frame #1: 0x00007ffff068d2ec libsqlite3.so`sqlite3Malloc + 60
    frame #2: 0x00007ffff068d6de libsqlite3.so`sqlite3MallocZero + 14
    frame #3: 0x00007ffff074dc56 libsqlite3.so`openDatabase + 150

This makes me think that the error is that the sqlite-lib's allocation fails for some reason on my system?
But I will try the other things too and maybe try replacing the sqlitelib with another version.

7

That's interesting. Could you try the following program? In particular, what does it print out, and does it crash in the same way? If it does crash, does it also crash with the other definition of flags?

#lang racket/base
(require racket/gui)

(module+ main
  (require ffi/unsafe db/private/sqlite3/ffi)

  (define-sqlite sqlite3_config (_fun _int -> _int))
  (define SQLITE_CONFIG_SERIALIZED 3)
  (eprintf "config returned ~s\n" (sqlite3_config SQLITE_CONFIG_SERIALIZED))

  (define-sqlite sqlite3_compileoption_get (_fun _int -> _string/utf-8))
  (eprintf "~s\n"
           (let loop ([i 0])
             (define s (sqlite3_compileoption_get i))
             (if s (cons s (loop (add1 i))) null)))

  (define flags (+ SQLITE_OPEN_READWRITE))
  ;; (define flags (+ SQLITE_OPEN_READWRITE SQLITE_OPEN_FULLMUTEX))
  (define-values (db status) (sqlite3_open_v2 #":memory:" flags))
  (eprintf "flags = ~s, db = ~s, status = ~s\n" flags db status)
  (sqlite3_close db)
  (eprintf "done\n"))
1 Like
8

It seems I get the same output for both:

config returned 0
("COMPILER=gcc-5.5.0 20171010" "ENABLE_FTS3" "ENABLE_FTS3_PARENTHESIS" "ENABLE_FTS4" "ENABLE_FTS5" "ENABLE_JSON1" "ENABLE_LOAD_EXTENSION" "ENABLE_MATH_FUNCTIONS" "ENABLE_RTREE" "ENABLE_STAT4" "ENABLE_UPDATE_DELETE_LIMIT" "HAVE_ISNAN" "SOUNDEX" "SYSTEM_MALLOC" "TEMP_STORE=3" "THREADSAFE=1" "USE_URI")
invalid memory reference.  Some debugging context lost

and

config returned 0
("COMPILER=gcc-5.5.0 20171010" "ENABLE_FTS3" "ENABLE_FTS3_PARENTHESIS" "ENABLE_FTS4" "ENABLE_FTS5" "ENABLE_JSON1" "ENABLE_LOAD_EXTENSION" "ENABLE_MATH_FUNCTIONS" "ENABLE_RTREE" "ENABLE_STAT4" "ENABLE_UPDATE_DELETE_LIMIT" "HAVE_ISNAN" "SOUNDEX" "SYSTEM_MALLOC" "TEMP_STORE=3" "THREADSAFE=1" "USE_URI")
invalid memory reference.  Some debugging context lost

so both times it crashes on sqlite3_open_v2, if I comment (require racket/gui) it again doesn't crash and outputs:
(define flags (+ SQLITE_OPEN_READWRITE)):

config returned 0
("COMPILER=gcc-5.5.0 20171010" "ENABLE_FTS3" "ENABLE_FTS3_PARENTHESIS" "ENABLE_FTS4" "ENABLE_FTS5" "ENABLE_JSON1" "ENABLE_LOAD_EXTENSION" "ENABLE_MATH_FUNCTIONS" "ENABLE_RTREE" "ENABLE_STAT4" "ENABLE_UPDATE_DELETE_LIMIT" "HAVE_ISNAN" "SOUNDEX" "SYSTEM_MALLOC" "TEMP_STORE=3" "THREADSAFE=1" "USE_URI")
flags = 2, db = #<cpointer>, status = 0
0
done

(define flags (+ SQLITE_OPEN_READWRITE SQLITE_OPEN_FULLMUTEX)):

config returned 0
("COMPILER=gcc-5.5.0 20171010" "ENABLE_FTS3" "ENABLE_FTS3_PARENTHESIS" "ENABLE_FTS4" "ENABLE_FTS5" "ENABLE_JSON1" "ENABLE_LOAD_EXTENSION" "ENABLE_MATH_FUNCTIONS" "ENABLE_RTREE" "ENABLE_STAT4" "ENABLE_UPDATE_DELETE_LIMIT" "HAVE_ISNAN" "SOUNDEX" "SYSTEM_MALLOC" "TEMP_STORE=3" "THREADSAFE=1" "USE_URI")
flags = 65538, db = #<cpointer>, status = 0
0
done
9

So I updated my racket git checkout (sepparate local racket) to version Racket v8.3.0.8 [cs] commit 795c2bd1b17e337a6f6a99e6229304fdd6c6123a and recompiled it.

config returned 0
("COMPILER=gcc-11.1.0" "ENABLE_COLUMN_METADATA" "ENABLE_DBSTAT_VTAB" "ENABLE_FTS3" "ENABLE_FTS3_TOKENIZER" "ENABLE_FTS4" "ENABLE_FTS5" "ENABLE_JSON1" "ENABLE_MATH_FUNCTIONS" "ENABLE_RTREE" "ENABLE_STMTVTAB" "ENABLE_UNLOCK_NOTIFY" "HAVE_ISNAN" "MAX_EXPR_DEPTH=10000" "MAX_VARIABLE_NUMBER=250000" "SECURE_DELETE" "SYSTEM_MALLOC" "TEMP_STORE=1" "THREADSAFE=1")
flags = 65538, db = #<cpointer>, status = 0
0
done

All programs working without a crash, so I am confused because I expected it to not work.
I do raco pkg install libsqlite3 and I get the error again raco pkg remove libsqlite3 and I still get the error. So now I am thinking maybe the error only happens if you had libsqlite3 installed at some point.
Honestly at this point I am just confused :crazy_face:

Took a bit of a brake and reran the compile option one.

config returned 0
("COMPILER=gcc-5.5.0 20171010" "ENABLE_FTS3" "ENABLE_FTS3_PARENTHESIS" "ENABLE_FTS4" "ENABLE_FTS5" "ENABLE_JSON1" "ENABLE_LOAD_EXTENSION" "ENABLE_MATH_FUNCTIONS" "ENABLE_RTREE" "ENABLE_STAT4" "ENABLE_UPDATE_DELETE_LIMIT" "HAVE_ISNAN" "SOUNDEX" "SYSTEM_MALLOC" "TEMP_STORE=3" "THREADSAFE=1" "USE_URI")
invalid memory reference.  Some debugging context lost

What I find troubling is that even after I removed libsqlite3 it seems its library stays installed.
And further that gcc-5.5.0 20171010 seems very old and could explain the different and strange behavior.
What I don't know yet is how I can get back to the state as if I had never installed libsqlite3.

Cause found?

So it really seems to be the libsqlite3 package.
I looked at its package definition racket-libsqlite3/info.rkt at 9e5c5696c65ddd3f8359e234fd4ebd93ad1bfc3c · Bogdanp/racket-libsqlite3 · GitHub
and then executed raco pkg remove libsqlite3-x86_64-linux and it actually removed something.

After that it now works again.

Bug:

To me it seems there is a bug with the package manager here: platform specific package dependencies aren't marked as automatically installed / dependent on the meta package and thus aren't removed when the meta package is removed?

Thanks

Thank you @ryanc for your help and example program, without it I wouldn't have found out that there was still the platform specific sub package of libsqlite3 hanging around after removing the package.

Where to go from here?

I am still not sure what is going on with the libsqlite3 package if that is a problem others could have? So far nobody but me could reproduce this error...
Maybe it would disappear if the system that builds the libraries used a less old gcc?
(@bogdan it seems the crash in this post is somehow caused by libsqlite3, but unless someone can reproduce it, this is simply to let you know in case someone has the same problem)

For now it seems uninstalling the not auto-removed sub-package fixes the problem for me.

10

@simonls can you try https://racket.defn.io/libsqlite3-x86_64-linux-3.36.0.1.tar.gz instead? This is a different version from the one that’s currently published, built on a more recent version of Ubuntu. You can install it by running raco pkg install --name libsqlite3-x86_64-linux https://racket.defn.io/libsqlite3-x86_64-linux-3.36.0.1.tar.gz .

2 Likes
11

With that I also get the error (using Ryan's script):

config returned 0
("COMPILER=gcc-7.5.0" "ENABLE_FTS3" "ENABLE_FTS3_PARENTHESIS" "ENABLE_FTS4" "ENABLE_FTS5" "ENABLE_JSON1" "ENABLE_LOAD_EXTENSION" "ENABLE_MATH_FUNCTIONS" "ENABLE_RTREE" "ENABLE_STAT4" "ENABLE_UPDATE_DELETE_LIMIT" "HAVE_ISNAN" "SOUNDEX" "SYSTEM_MALLOC" "TEMP_STORE=3" "THREADSAFE=1" "USE_URI")
invalid memory reference.  Some debugging context lost

seems the compile options are a bit different between the two:

racket sqlite:
("COMPILER=gcc-11.1.0" "ENABLE_COLUMN_METADATA" "ENABLE_DBSTAT_VTAB" "ENABLE_FTS3" "ENABLE_FTS3_TOKENIZER" "ENABLE_FTS4" "ENABLE_FTS5" "ENABLE_JSON1" "ENABLE_MATH_FUNCTIONS" "ENABLE_RTREE" "ENABLE_STMTVTAB" "ENABLE_UNLOCK_NOTIFY" "HAVE_ISNAN" "MAX_EXPR_DEPTH=10000" "MAX_VARIABLE_NUMBER=250000" "SECURE_DELETE" "SYSTEM_MALLOC" "TEMP_STORE=1" "THREADSAFE=1")

libsqlite3:
("COMPILER=gcc-7.5.0" "ENABLE_FTS3" "ENABLE_FTS3_PARENTHESIS" "ENABLE_FTS4" "ENABLE_FTS5" "ENABLE_JSON1" "ENABLE_LOAD_EXTENSION" "ENABLE_MATH_FUNCTIONS" "ENABLE_RTREE" "ENABLE_STAT4" "ENABLE_UPDATE_DELETE_LIMIT" "HAVE_ISNAN" "SOUNDEX" "SYSTEM_MALLOC" "TEMP_STORE=3" "THREADSAFE=1" "USE_URI")

@bogdan maybe that has something to do with it? But at this point I am only guessing.
To be more strategic about debugging this I guess I would have to compile libsqlite with debug symbols and try to step through what happens with a debugger.

1 Like
12

I've tried reproducing the issue this morning inside a Docker container, and I haven't been successful, so it does seem like the problem might be specific to your system. Maybe you're running out of memory, so the allocation fails, and that leads to the segfault. That would explain why requiring racket/gui makes a difference since it increases memory usage by quite a bit. However, it doesn't explain why the problem only happens with the newer versions of the SQLite library (one reason could be that they're bigger (more code, not stripped, etc.) and that they try to allocate more space).

To be more strategic about debugging this I guess I would have to compile libsqlite with debug symbols and try to step through what happens with a debugger.

Yup, that's what I'd recommend doing. This script builds the lib so you can refer to it for the build flags and such.

3 Likes