# Extracting lines from big file

**URL:** <https://racket.discourse.group/t/extracting-lines-from-big-file/888>\
**Category:** Questions & Answers\
**Tags:** question\
**Created:** [April 17, 2022, 7:59am UTC](https://racket.discourse.group/t/extracting-lines-from-big-file/888 "2022-04-17T07:59:41Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![jkleiser](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/jkleiser/32/498_2.png) [@jkleiser](https://racket.discourse.group/u/jkleiser)\
**Post date:** [April 17, 2022, 7:59am UTC](https://racket.discourse.group/t/extracting-lines-from-big-file/888/1 "2022-04-17T07:59:41Z")

</div>

Hi. I want to extract and/or process certain lines from a rather big file (330 MB, 6.5 million lines). The condition for which lines to handle is based on whether a certain part of the line read is found in a hash set of 9300 words. I may be old-fashioned, but I think I should read the big file line by line, not to use too much memory.  
I have not used Racket too much yet, but I found the recursive style "read file line by line" example here: [https://rosettacode.org/wiki/Read\_a\_file\_line\_by\_line#Racket](https://rosettacode.org/wiki/Read_a_file_line_by_line#Racket)  
The thing that troubles me with that solution, is how do I fit the needed condition and processing in there. I need to pass references to my existing hash set and out-file.  
I feel a more "old school" loop would be easier to do, but will it be as efficient as a recursive solution?

---

<div class="post-metadata">

**Author:** ![swsch](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/swsch/32/165_2.png) [@swsch](https://racket.discourse.group/u/swsch)\
**Post date:** [April 17, 2022, 8:23am UTC](https://racket.discourse.group/t/extracting-lines-from-big-file/888/2 "2022-04-17T08:23:44Z")

</div>

One way: Use [parameterize](https://docs.racket-lang.org/guide/parameterize.html) to provide a binding to a function used inside the iterator.

Another way: Define a function taking your "external references" as arguments, which produces a function suitable for `call-with-input-file`.

---

<div class="post-metadata">

**Author:** ![jjsimpso](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/jjsimpso/32/602_2.png) [@jjsimpso](https://racket.discourse.group/u/jjsimpso)\
**Post date:** [April 17, 2022, 3:58pm UTC](https://racket.discourse.group/t/extracting-lines-from-big-file/888/3 "2022-04-17T15:58:49Z")

</div>

Here's one way to do it:

```scheme
(define (process-file infile outfile print-line-number?)
  (define (process-lines [line-number 1])
    (define line (read-line (current-input-port) 'any))
    (unless (eof-object? line)
      (if print-line-number?
          (printf "~a: ~a~n" line-number line)
          (printf "~a~n" line))
      (process-lines (add1 line-number))))
  
  (with-output-to-file outfile
    (lambda () (with-input-from-file infile process-lines))))

```

Since the helper function `process-lines` is defined within the scope of `process-file`, it has access to `print-line-number?`. You can use the same technique for your hash table. And you don't need to pass the out-file either since `with-output-to-file` takes care of that.

I also used an optional argument to `process-lines` to track the line numbers.

---

<div class="post-metadata">

**Author:** ![swsch](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/swsch/32/165_2.png) [@swsch](https://racket.discourse.group/u/swsch)\
**Post date:** [April 17, 2022, 9:03pm UTC](https://racket.discourse.group/t/extracting-lines-from-big-file/888/4 "2022-04-17T21:03:31Z")

</div>

```scheme
(define my-function (make-parameter println))

(define (read-next-line-iter file)
	   (let ((line (read-line file 'any)))
	     (unless (eof-object? line)
               ((my-function) line)
	       (read-next-line-iter file))))

(call-with-input-file "d:/foobar.txt" read-next-line-iter)
(newline)

(parameterize ((my-function (lambda (line)
                              (println (if (odd? (string-length line)) "odd" "")))))
  (call-with-input-file "d:/foobar.txt" read-next-line-iter))

(newline)

(define (make-my-function dict)
  (lambda (file)
    (let loop ((line (read-line file 'any)))
      (unless (eof-object? line)
        (println (hash-ref dict line 'nope))
        (loop (read-line file 'any))))))

(call-with-input-file "d:/foobar.txt"
  (make-my-function (hash "22" 'yep)))

```

gives the following output for a suitable `foobar.txt`:

```scheme
"1"
"22"
"333"
"4444"
"55555"
"4444"
"333"
"22"
"1"

"odd"
""
"odd"
""
"odd"
""
"odd"
""
"odd"

'nope
'yep
'nope
'nope
'nope
'nope
'nope
'yep
'nope

```

---

<div class="post-metadata">

**Author:** ![jkleiser](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/jkleiser/32/498_2.png) [@jkleiser](https://racket.discourse.group/u/jkleiser)\
**Post date:** [April 18, 2022, 9:53am UTC](https://racket.discourse.group/t/extracting-lines-from-big-file/888/5 "2022-04-18T09:53:09Z")

</div>

Thanks a lot! I found this solution easy to understand and adapt, so I'll stick with that one.