Question

Is there an option to comunicate a TVF/UDF in a DBMS with an external IDE or language like C? Doing it without writing to a table?

I know there is a way of ‘memory mapping’ or a way to share block of memory

POSIX mmap() function
Windows OpenFileMapping() function

I am using Windows, so Is there a way to communicate a DBMS using memory mapping or sharing with something like C?
But how would you avoid writing to a table, or a file, using just memory?

Answer 1

Shared Memory is available as a data transport to and from SQL providers. You don’t have to write any additional code for this if you are using built-in drivers to access your provider. Instead, you would just configure the driver and the server to use this, and your application would have to reside on the same server as your SQL provider.

The ODBC drivers available for windows support shared memory for SQL activities. To write code for these from C, you would use the ODBC API to communicate with your provider. Here’s a link with a function reference.

ODBC Function Summary @ MSDN

Also note that there is support for BLOBs for all SQL providers that can handle arbitrary binary data. A list of the types known to ODBC API is available here. There’s no strict requirement that your statement results must be expressible in tabular form.

SQL Data Types @ MSDN

On the other hand, if are concerned about communicating with internal SQL entities on your own terms, you might be able to patch something together via extensions to the SQL service you are using. For example, MS SQL Server allows extensions via Ole Automation Procedures or CLR (.net) Integration (available in MS SQL Server). You could potentially use these make something to communicate out-of-band. However, neither of these is easily created with a pure C solution.

Ole Automation Procedures in SQL Server @ MSDN

CLR Integration in SQL Server @ MSDN

However, I recommend that you avoid doing this, as you will find that you’re at the mercy of the environment of the host service and you may not be able to participate in transactions.

If your dataset size requirements are so large that you consider RAM and direct access your best option, your needs would probably be better fulfilled by communicating only the parts that change in the dataset held outside of SQL. In addition, as a shared-memory solution is restricted to one machine, you would probably want to consider splitting the work on your dataset across multiple machines. It is more likely that you would see a performance/productivity improvement by such means than by changing how you reference data in SQL.

Last, it is tough to dictate to a SQL provider that it should avoid using filesystem storage. For MS SQL Server, one possible option is to force tempdb to reside in RAM. Here’s a KB article with more details. Other DBMSs may have similar configuration options.

INF: When to use tempdb in RAM

However, the use of disk storage isn’t necessarily a cause for concern. I’m unable to find a good example of how SQL providers manage a RAM / Filesystem balance, but one good analogue for SQL server is how windows is affected by page-file use. Here’s a great link that details how windows behaves at high limits of operation, and how memory use doesn’t necessarily correspond into overflow to disk use. Also note that applications written to run on windows are also adversely affected when the host’s operation approaches these limits.

Pushing the Limits of Windows: Virtual Memory @ TechNet