Question

I use F# once every few months or so and in between it seems I forget everything, so I hope you’ll excuse my ignorance. My code below is pulling data from Yahoo. It’s a good example that represents what I need to do. The first row returned has the column headings. I need to take the data (the tail of the list) and insert it into the database. What’s the best way to generate an insert statement based on the column headings returned (the column headings match the database column names)?

In the example below dataWithHeaders.[0] will contain “Date,Open,High,Low,Close,Volume,Adj Close.” Should I just take that string and put brackets around the headings to create the insert? Then in insertData add the value as a parameter? Is there a more elegant solution?

let url = System.String.Format("http://ichart.finance.yahoo.com/table.csv?s={0}&g=d&ignore=.csv", "FB")

let splitLineIntoArray (line : string) = 
    line.Split(",".ToCharArray())

let insertData (data : string[]) =
    // insert data
    ()

let client = new WebClient()
let dataWithHeaders = 
    client.DownloadString(url).Split(Environment.NewLine.ToCharArray())

let data =
    dataWithHeaders
    |> Array.toList
    |> List.tail
    |> List.map(splitLineIntoArray)
    |> List.iter insertData

Answer 1

If you’re loading the data into SQL Server you can use this excellent CSV reader (free) and the SqlBulkCopy class. It’s simple and efficient.

let loadStockPrices ticker =
  use client = new WebClient()
  let url = sprintf "http://ichart.finance.yahoo.com/table.csv?s=%s&g=d&ignore=.csv" ticker
  use stringReader = new StringReader(client.DownloadString(url))
  use csvReader = new CsvReader(stringReader, hasHeaders=true)
  use con = new SqlConnection("<connection_string>")
  con.Open()
  use bulkCopy = new SqlBulkCopy(con, DestinationTableName="<destination_table>")
  bulkCopy.WriteToServer(csvReader)

The destination table should have the same columns as the incoming data (OHLC, etc).