Home/ Questions/Q 6844323
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-27T00:21:46+00:00

I am operating on chunks of panel data with about 3*10^6 rows and 11

  • 0

I am operating on chunks of panel data with about 3*10^6 rows and 11 columns. This data is stored in a SQL database (the data I use is a subset of an even larger dataset). I want to process chunks of the data (each date seperately) without importing the entire thing all at once, but the operations I have to run on each chunk are computationally intensive, so I would like to use the foreach package.

However, I am running into some problems with running the sql call in parallel. If I do a simple call such as (sqlQuery(channel, "select 1")) I can run that in parallel. If I do a more complex statement I get SIGPIPE errors. Has anyone else run into similar problems?

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    You did not tell us what database you were using. On a SQLServer connected with RODBC, if have successfully done this by using transactions.

    channel = odbcConnect(database)
odbcSetAutoCommit(channel,FALSE)
# Make requests here
odbcEndTran(channel,TRUE)

    I am not sure if this will also help when you have only read request as in your case, but it might be worth giving a try because it is only twp additional lines. I asked about the server, because transactions work nicely with MSSqlServer under ODBC, and I had some mixed results with others (but did not try hard).

    • 0