I suspect the thing that's tripping you up is that scheme is a fundamentally different paradigm than your usual c/java style language. So I'm guessing for scala (I haven't used it so I could be totally wrong) you could write code like you would in most languages and then sprinkle in a new concept here and there like functional programming or type stuff etc. That doesn't really work for scheme based languages in my experience. I think its really important to have concepts like lambdas, list recursion, functional patterns, and other scheme basics down before messing with the higher level stuff. Reading the guide and reference won't be enough to use racket if you don't have those basics imo.
And its not just something you can read and understand, you have to practice writing in that style for a bit, and its not gonna be pleasant because you'll feel like everything is super rudimentary (as you suggested). However, once you get past that, you will start to see how both the scheme concepts map to your current knowledge and how you can use what you already know in scheme. And then you should have a pretty good time. There will still be rabbit holes, but you'll be gaining useful techniques instead of arbitrary concepts when you go down them.
As others have suggested, the material which teaches you the basics is 2htdp (slow and thorough) and SICP (fast but takes more effort).
Now if you did that and I'm being presumptuous than I apologize, but from what you wrote I'm guessing that this might be why you're having a hard time. Also I could be wrong and maybe its something else, ymmv.
@hasn0life I think this is a very important point. I got into functional programming via (some) Haskell several years ago, and despite decades of experience - with imperative languages - I sometimes felt like a total beginner in programing in general. I think without this prior experience learning Racket would have been much more difficult.
That said, even beyond the hurdle of functional programming, I still find (or found) Racket difficult to get into. All I wrote about myself and my Racket experience in this thread is with this background.
But again, I wonder if what you wrote means we should recommend beginners that they learn Scheme first. I think this would be less difficult than starting with Racket directly. Or, if we want to help people starting with Racket directly, it's important to explain the functional programming concepts quite thoroughly. Also, I think exercises are very important to get the hang of it.
It's OK to assume (I do it all the time) but in this case Scala does tend to be different. In fact, I had no trouble assimilating Haskell either as I saw is as a "ground up" simple(r) kind of a language that actually looked pretty appealing to me (until I started dealing with Cabal). Way back when I was younger I studied Eiffel (mandatory in our 2nd year CS degree data structures and algorithms class) and was exposed to contracts. The concept stuck with me so they do not look foreign in Scheme. You can write "bad Scala" with mutable variables and it may even try and look like Java but for the most part everything you read discourages you from doing so - it is rare (well at least for me) to run into someone with Scala experience that still writes bastardized Java in Scala. It can happen but rare. Way back when I was in college I actually wrote a Prolog interpreter in C++ and I wrote a small LISP in x86 asm. Concepts are not foreign to me, in case of Racket it is the sea of information coming at me at once. I wonder if I suffer from "fear of missing out" "syndrome" and I get overwhelmed with all the "cool"/"different" concepts people mention so I am always chasing the new shiny (Racket) thing....Yikes, I am traumatized! (just kidding)
I should clarify: many of my first racket programs were simple.
For example, hacking on a HW grader mostly involved writing unit tests, though eventually I added a macro or two and hacked on some things related to JSON and files. Everything but the macro system was familiar to me conceptually: tests, JSON, files.
Building GitHub - benknoble/slack-archive-viewer was a longer process but the fundamental concepts are processes, data transformation, and files—again, things I'm familiar with conceptually. There are some macros in there but by then I had a better grasp on the system and knew where to get help I did spend some time with the Pollen docs as well, since I rely on it for many pieces.
I think my point here is that if I take this list of complex things:
I encountered very few of them at the conceptual level of my early programs (and still haven't really encountered some). I felt immediately quite productive with the basics of data manipulation and "system stuff" like files.
Perhaps these are the kinds of programs you should start with? Something conceptually familiar. These are things that are practicable and feel quite productive ("it works!") without needing large new concepts; coincidentally, they are also the types of programs I would write when starting with Scala, or Rust, or Go.
That's the funny thing. Actually I needed only the simpler stuff (though you could argue to which extent contracts are simple), but I wasn't sure whether the simple things would be enough.
I remember at some point I was looking for a way to free a resource at the end of the program run, and a search brought me to executors and wills. In the end, I found that I couldn't achieve with them what I wanted, but I still read about them to find out that they wouldn't work for me.
I guess I have the same "problem" as @maketo. I come across some concept in the documentation and to understand what it is, I read about it. Only after reading I know how useful (or not) these concepts will be for me. And before I read more about them I can't tell whether they're "advanced" or rarely used concepts, or idiomatic things I should know.
Ironically, I didn't have that problem with macros and continuations. When I started with Racket, I knew enough about the concepts that I was relatively sure I wouldn't need them at that time, so I delayed learning more about these.
I got into functional programming via (some) Haskell several years ago, and despite decades of experience - with imperative languages 
- I sometimes felt like a total beginner in programing in general. I think without this prior experience learning Racket would have been much more difficult.
(just kidding)