Home/ Questions/Q 8208475
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-07T09:22:52+00:00

I have a data file which I’d like to read into R which is

  • 0

I have a data file which I’d like to read into R which is something like the following:

STARTOFDATA 2011-06-23 35
143 6456 23 646 123.53A 864.95 23B
343 634 24 545 65.3 235.2 94C
...
524 542 45 245.4 24 245A 45B
STARTOFDATA 2011-06-24 84
245 6532 24.4 624.2 542 23B 35A
241 4532 13.5 235.12 534.23 54 32B
etc...

As you can see, it’s basically a 2D dataset (each of the columns between the header lines is a different variable) which is stored for a number of dates, specified by the STARTOFDATA lines, which split up the different days. The number at the end of the header line is the number of lines of data before the next header line. The A’s, B’s and C’s etc are quality control information which can basically just be discarded – probably just as a gsub on the text I get from the file.

My question is: how should I go about reading this into R? Ideally I’d like to be able to read either the whole file, or a specified date (or date range). I should probably point out that the file is over 200,000 lines long!

I’ve done some thinking and researching about this, but can’t seem to work out a sensible way to do it.

As far as I can see it, there are two questions:

  1. How to read the file: Is there a way to move a pointer around within a file in R? Some other languages I’ve worked with have had that ability, in which case I could read the first line, read the date, see if I want that date or not, then if not skip the number of lines listed at the end of the header (preferably without reading them!) and read the next header line. I can’t see anything in the documentation about a function that would let me do that without actually reading in the lines. It seems that if I create a connection object manually then that will keep track of where I am in the file, and I can use repeated calls to readLines (in a loop) to read in chunks of the file, discarding them once read if they’re not needed.

  2. How to store the data: Ideally I want to store the 2D dataset for each date in a dataframe, then I can continue to do any analysis on them fairly easily. However, how should I store loads of these 2D datasets? I’m thinking of a list of data-frames, but is that the best way to do it (in terms of being able to index the list sensibly)?

Any ideas or comments would be much appreciated.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Use readLines to read your data as a character vector and then manipulate this vector. Here is some code that splits your sample data into a list of blocks:

    Use readLines to read the data:

    x <- readLines(textConnection(
"STARTOFDATA 2011-06-23 35
143 6456 23 646 123.53A 864.95 23B
343 634 24 545 42 65.3 235.2 94C
...
524 542 45 245.4 24 542.54 245A 45B
STARTOFDATA 2011-06-24 84
245 6532 24.4 624.2 542 23B 35A
241 4532 13.5 235.12 534.23 54
etc..."))

    Determine the positions of STARTOFDATA, then split into a list of blocks:

    positions <- c(grep("STARTOFDATA", x), length(x)+1)
lapply(head(seq_along(positions), -1), 
       function(i)x[positions[i]:(positions[i+1]-1)])

[[1]]
[1] "STARTOFDATA 2011-06-23 35"          
[2] "143 6456 23 646 123.53A 864.95 23B" 
[3] "343 634 24 545 42 65.3 235.2 94C"   
[4] "..."                                
[5] "524 542 45 245.4 24 542.54 245A 45B"

[[2]]
[1] "STARTOFDATA 2011-06-24 84"      
[2] "245 6532 24.4 624.2 542 23B 35A"
[3] "241 4532 13.5 235.12 534.23 54" 
[4] "etc..."

    Now each block of data is an element in a list and you can process that as required, using a second lapply()

    • 0