# Reading a web directory (slow) vs. reading a file (quick)

**URL:** <https://racket.discourse.group/t/reading-a-web-directory-slow-vs-reading-a-file-quick/2182>\
**Category:** Questions & Answers\
**Created:** [July 31, 2023, 8:23pm UTC](https://racket.discourse.group/t/reading-a-web-directory-slow-vs-reading-a-file-quick/2182 "2023-07-31T20:23:06Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![gknauth](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/gknauth/32/142_2.png) [@gknauth](https://racket.discourse.group/u/gknauth)\
**Post date:** [July 31, 2023, 8:23pm UTC](https://racket.discourse.group/t/reading-a-web-directory-slow-vs-reading-a-file-quick/2182/1 "2023-07-31T20:23:06Z")

</div>

I'm reading a couple of NOAA directories on the web, just to find out what the latest data is. But that is taking forever, like 500 seconds (!). Whereas once I find the file I need, I can read its 28756136 bytes in 6 seconds. The directory listings are really short. I'm trying to figure out why reading them it taking so long.

```scheme
#lang racket

(require net/url html-parsing)

(define nbm-base "https://nomads.ncep.noaa.gov/pub/data/nccf/com/blend/prod/blend.")

; (nbm-webdir "20230731")
; => "https://nomads.ncep.noaa.gov/pub/data/nccf/com/blend/prod/blend.20230731/"

(define (nbm-webdir ymd8)
  (string-append nbm-base ymd8 "/"))

; (nbm-textdir "20230731" "18")
; => "https://nomads.ncep.noaa.gov/pub/data/nccf/com/blend/prod/blend.20230731/18/text/"

(define (nbm-textdir ymd8 hour)
  (string-append (nbm-webdir ymd8) hour "/text/"))

; (nbm-product "20230731" "18" 'hourly)
; => "https://nomads.ncep.noaa.gov/pub/data/nccf/com/blend/prod/blend.20230731/18/text/blend_nbhtx.t18z"

(define (nbm-product ymd8 hour product-code)
  (string-append (nbm-textdir ymd8 hour) "blend_nb" (product-letter product-code) "tx.t" hour "z"))

(define (product-letter product-code)
  (match product-code
    ['hourly "h"]
    ['short "s"]
    ['extended "e"]
    ['super-extended "x"]
    ['probabilistic-extended "p"]
    [_ (error (format "unknown product-code: ~a" product-code))]))

(define (nmb-product ymd8 hour product-code)
  (string-append (nbm-textdir ymd8 hour) "/"))

(define (get-webdir-xexp ymd8)
  (html->xexp (get-pure-port (string->url (nbm-webdir ymd8)))))

(define (get-textdir-xexp ymd8 hour)
  (html->xexp (get-pure-port (string->url (nbm-textdir ymd8 hour)))))

(define (get-product-bytes ymd8 hour product-code)
  (port->bytes (get-pure-port (string->url (nbm-product ymd8 hour product-code)))))

(define x-webdir-xexp (time (get-webdir-xexp "20230731")))
(define x-textdir-xexp (time (get-textdir-xexp "20230731" "18")))
(define x-product-bytes (time (get-product-bytes "20230731" "18" 'hourly)))

```

```scheme
cpu time: 67555 real time: 500331 gc time: 93
cpu time: 64171 real time: 500488 gc time: 81
cpu time: 2867 real time: 6364 gc time: 77

```

---

<div class="post-metadata">

**Author:** ![sschwarzer](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/sschwarzer/32/1940_2.png) [@sschwarzer](https://racket.discourse.group/u/sschwarzer)\
**Post date:** [August 1, 2023, 6:54am UTC](https://racket.discourse.group/t/reading-a-web-directory-slow-vs-reading-a-file-quick/2182/2 "2023-08-01T06:54:28Z")

</div>

Since the cpu time is much lower than the real time, your problem seems to be limited by the download. But the cpu time also seems quite large.

Can you maybe for comparison try to download the file in a webbrowser and measure how long it takes?

---

<div class="post-metadata">

**Author:** ![gknauth](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/gknauth/32/142_2.png) [@gknauth](https://racket.discourse.group/u/gknauth)\
**Post date:** [August 1, 2023, 12:09pm UTC](https://racket.discourse.group/t/reading-a-web-directory-slow-vs-reading-a-file-quick/2182/3 "2023-08-01T12:09:27Z")

</div>

In a web browser (Firefox), the 27MB file loads pretty quickly, and the directory listings are instaneous.

In DrRacket, the 27MB file loads quickly enough for me (6 secs is fine), but I'm puzzled why the directory listings take 8+ minutes each. The directory listings are very short.

---

<div class="post-metadata">

**Author:** ![greghendershott](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/greghendershott/32/98_2.png) [@greghendershott](https://racket.discourse.group/u/greghendershott)\
**Post date:** [August 1, 2023, 2:49pm UTC](https://racket.discourse.group/t/reading-a-web-directory-slow-vs-reading-a-file-quick/2182/4 "2023-08-01T14:49:20Z")

</div>

I'd suggest using `net/http-client` instead of `net/url` to interact with today's web servers. It will make HTTP 1.1 requests, it handles compression like gzip and deflate, it follows redirects, and it supplies other request headers that might impact this somehow. In general, it's closer to what Firefox would be doing.

For example, rewriting your program to this:

```scheme
#lang racket

(require net/http-client
         ;html-parsing
         )

(define nbm-host "nomads.ncep.noaa.gov")
(define nbm-base "/pub/data/nccf/com/blend/prod/blend.")

(define (get-from-noaa uri-path)
  (define-values (_status _headers input-port)
    (http-sendrecv nbm-host
                   uri-path
                   #:ssl? #t))
  input-port)

;; Dummy because I don't have html-parsing installed.
(define (html->xexp in)
  (port->bytes in))

; (nbm-webdir "20230731")
; => "/pub/data/nccf/com/blend/prod/blend.20230731/"

(define (nbm-webdir ymd8)
  (string-append nbm-base ymd8 "/"))

; (nbm-textdir "20230731" "18")
; => "/pub/data/nccf/com/blend/prod/blend.20230731/18/text/"

(define (nbm-textdir ymd8 hour)
  (string-append (nbm-webdir ymd8) hour "/text/"))

; (nbm-product "20230731" "18" 'hourly)
; => "/pub/data/nccf/com/blend/prod/blend.20230731/18/text/blend_nbhtx.t18z"

(define (nbm-product ymd8 hour product-code)
  (string-append (nbm-textdir ymd8 hour) "blend_nb" (product-letter product-code) "tx.t" hour "z"))

(define (product-letter product-code)
  (match product-code
    ['hourly "h"]
    ['short "s"]
    ['extended "e"]
    ['super-extended "x"]
    ['probabilistic-extended "p"]
    [_ (error (format "unknown product-code: ~a" product-code))]))

(define (nmb-product ymd8 hour product-code)
  (string-append (nbm-textdir ymd8 hour) "/"))

(define (get-webdir-xexp ymd8)
  (html->xexp (get-from-noaa (nbm-webdir ymd8))))

(define (get-textdir-xexp ymd8 hour)
  (html->xexp (get-from-noaa (nbm-textdir ymd8 hour))))

(define (get-product-bytes ymd8 hour product-code)
  (port->bytes (get-from-noaa (nbm-product ymd8 hour product-code))))

(module+ example
  (define x-webdir-xexp (time (get-webdir-xexp "20230731")))
  (define x-textdir-xexp (time (get-textdir-xexp "20230731" "18")))
  (define x-product-bytes (time (get-product-bytes "20230731" "18" 'hourly))))

```

I get times like these:

```scheme
cpu time: 13 real time: 553 gc time: 1
cpu time: 15 real time: 126 gc time: 0
cpu time: 555 real time: 838 gc time: 65

```

Note this is cheating because I'm not actually parsing the HTML using `html-parsing` (I don't have it installed on this computer). However, if you add that part back in, and the times explode, which seems unlikely, at least we'll have narrowed it down to that.

* * *

Note: I'm using the simplest, least efficient `http-sendrecv` which uses a fresh connection each time, as does `get-pure-port`. You probably don't care, but if you do, there's an API where you make a connection and reuse it for multiple requests.

* * *

Note: Because `http-client` is closer to reality, there's no URL abstraction over host, path, scheme, port, etc. In most cases I find that's fine for web-service-y purposes. e.g. You can just split your `nbm-base` URL into `nbm-host` and `nbm-base-path`, assume SSL, etc.

---

<div class="post-metadata">

**Author:** ![soegaard](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/soegaard/32/19_2.png) [@soegaard](https://racket.discourse.group/u/soegaard)\
**Post date:** [August 1, 2023, 2:49pm UTC](https://racket.discourse.group/t/reading-a-web-directory-slow-vs-reading-a-file-quick/2182/5 "2023-08-01T14:49:55Z")

</div>

Below I inserted a `port->string` call in order to fetch the entire file at once and then start parsing.

```scheme
(define (get-webdir-xexp2 ymd8)
  (html->xexp (port->string (get-pure-port (string->url (nbm-webdir ymd8))))))

(define (get-webdir-xexp ymd8)
  (html->xexp (get-pure-port (string->url (nbm-webdir ymd8)))))

(define x-webdir-xexp2 (time (get-webdir-xexp2 "20230731")))
(define x-webdir-xexp (time (get-webdir-xexp "20230731")))

```

The timings were:

```scheme
cpu time: 27 real time: 500713 gc time: 0
cpu time: 34 real time: 500584 gc time: 0

```

I therefore come to the same conclusion as [sschwarzer](https://racket.discourse.group/u/sschwarzer).  
The bottleneck must the be the download.

Are there redirects involved in resolving the address?

Hmm. The command line tool `curl` is not too happy about the url  
[Index of /pub/data/nccf/com/blend/prod/blend.20230731](https://nomads.ncep.noaa.gov/pub/data/nccf/com/blend/prod/blend.20230731/)

```scheme
% curl --verbose "https://nomads.ncep.noaa.gov/pub/data/nccf/com/blend/prod/blend.20230731/"                             

* Trying 95.166.120.178:443...
* Connected to nomads.ncep.noaa.gov (95.166.120.178) port 443 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
* CAfile: /etc/ssl/cert.pem
* CApath: none
* [CONN-0-0][CF-SSL] (304) (OUT), TLS handshake, Client hello (1):
* [CONN-0-0][CF-SSL] (304) (IN), TLS handshake, Server hello (2):
* [CONN-0-0][CF-SSL] (304) (IN), TLS handshake, Unknown (8):
* [CONN-0-0][CF-SSL] (304) (IN), TLS handshake, Certificate (11):
* [CONN-0-0][CF-SSL] (304) (IN), TLS handshake, CERT verify (15):
* [CONN-0-0][CF-SSL] (304) (IN), TLS handshake, Finished (20):
* [CONN-0-0][CF-SSL] (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN: server accepted h2
* Server certificate:
* subject: CN=nomads.ncep.noaa.gov
* start date: Jul 17 12:10:12 2023 GMT
* expire date: Oct 15 12:10:11 2023 GMT
* subjectAltName: host "nomads.ncep.noaa.gov" matched cert's "nomads.ncep.noaa.gov"
* issuer: C=US; O=Let's Encrypt; CN=R3
* SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* h2h3 [:method: GET]
* h2h3 [:path: /pub/data/nccf/com/blend/prod/blend.20230731/]
* h2h3 [:scheme: https]
* h2h3 [:authority: nomads.ncep.noaa.gov]
* h2h3 [user-agent: curl/7.87.0]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x7f921c010a00)
> GET /pub/data/nccf/com/blend/prod/blend.20230731/ HTTP/2
> Host: nomads.ncep.noaa.gov
> user-agent: curl/7.87.0
> accept: */*
> 
* HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)
* Connection #0 to host nomads.ncep.noaa.gov left intact
curl: (92) HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)

```

Maybe it's a bad interaction between the `net/url` library and the web-server?

---

<div class="post-metadata">

**Author:** ![gknauth](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/gknauth/32/142_2.png) [@gknauth](https://racket.discourse.group/u/gknauth)\
**Post date:** [August 1, 2023, 3:19pm UTC](https://racket.discourse.group/t/reading-a-web-directory-slow-vs-reading-a-file-quick/2182/6 "2023-08-01T15:19:03Z")

</div>

This is really great, thank you so much, I'll use this for now and maybe tweak it as you suggest later. Your numbers are way better than mine, I'm not doing anything fancy parsing-wise, your solution is perfect for now.

What I'm doing this for: I have CAP [Civil Air Patrol] gliders and tow planes to move back and forth between two locations, one in Pennsylvania and one in Vermont. For aerotow I need good visibility for 250+ miles over a 4-hour period, and generally favorable weather. I need to look ahead up to a week but certainly several days ahead. There are multiple stations along the route, and the NBM (National Blended Model} is very good for all the surface weather stuff plus ceilings, cloud bases and cloud cover. I'm using this to put together a visualization of when the best windows of opportunity are (if they exist). Then I also have to check other things, like winds aloft at 10,000' (a good height for aerotow in case emergency glides to an airfield are needed by either tow plane or glider). It's a data synthesis / visualization effort to help decision making.

---

<div class="post-metadata">

**Author:** ![gknauth](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/gknauth/32/142_2.png) [@gknauth](https://racket.discourse.group/u/gknauth)\
**Post date:** [August 1, 2023, 3:20pm UTC](https://racket.discourse.group/t/reading-a-web-directory-slow-vs-reading-a-file-quick/2182/7 "2023-08-01T15:20:44Z")

</div>

Thank you, I should have used `curl` as you did to diagnose things, you found what I suspected, that the client side of things was waiting for something that wasn't happening right on the server side. I'm glad `html-client` is an easy way out of that mess.

---

<div class="post-metadata">

**Author:** ![benknoble](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/benknoble/32/16_2.png) [@benknoble](https://racket.discourse.group/u/benknoble)\
**Post date:** [August 1, 2023, 3:46pm UTC](https://racket.discourse.group/t/reading-a-web-directory-slow-vs-reading-a-file-quick/2182/8 "2023-08-01T15:46:35Z")

</div>

IIRC, the http123 package and library offers a good abstraction over http-client that includes some URL abstraction and connection pooling.

---

<div class="post-metadata">

**Author:** ![gknauth](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/gknauth/32/142_2.png) [@gknauth](https://racket.discourse.group/u/gknauth)\
**Post date:** [August 1, 2023, 3:47pm UTC](https://racket.discourse.group/t/reading-a-web-directory-slow-vs-reading-a-file-quick/2182/9 "2023-08-01T15:47:25Z")

</div>

Thanks, I'll check that out too!

---

<div class="post-metadata">

**Author:** ![sschwarzer](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/sschwarzer/32/1940_2.png) [@sschwarzer](https://racket.discourse.group/u/sschwarzer)\
**Post date:** [August 1, 2023, 7:04pm UTC](https://racket.discourse.group/t/reading-a-web-directory-slow-vs-reading-a-file-quick/2182/10 "2023-08-01T19:04:25Z")

</div>

> [@gknauth](#):
>
> Thank you, I should have used `curl` as you did to diagnose things, you found what I suspected, that the client side of things was waiting for something that wasn't happening right on the server side.

From the curl messages alone it's not clear to me whether the client was waiting for the server or the other way around (i.e. the server waiting for more data in the request).