Question

I’m connecting to a large postgresql database(300 million records) with libdbi and perform a SELECT * query, then show the result line by line. I’m getting full swap and memory so it seems autocommit is enabled and it loads the whole result set in the memory. Is there any option to disable autocommit or at least hold cursors after commit like ResultSet.HOLD_CURSORS_OVER_COMMIT in java? I didn’t find any option for dbi_conn_set_option to do it. here is my code:

dbi_conn conn;
dbi_result result;
int64_t id;

dbi_initialize(NULL);
conn = dbi_conn_new("pgsql");

if (conn == NULL)
{
    printf("connection error.\n");
return EXIT_FAILURE;
}

dbi_conn_set_option(conn, "host", "127.0.0.1");
dbi_conn_set_option(conn, "username", "postgres");
dbi_conn_set_option(conn, "password", "123456");
dbi_conn_set_option(conn, "dbname", "backup");

if (dbi_conn_connect(conn) < 0)
{
    printf("could not connect to database.\n");
    return EXIT_FAILURE;
}

result = dbi_conn_query(conn, "SELECT * FROM tbl");
if (result)
{
    while (dbi_result_next_row(result))
    {
        id = dbi_result_get_longlong(result, "_id");
        printf("This is _id: %ld\n", id);
    }

    dbi_result_free(result);
}

dbi_conn_close(conn);
dbi_shutdown();

Answer 1

It’s not related to autocommit.

The solution to the out-of-memory problem is indeed to use a cursor to fetch N results at a time as opposed to all results in one step.

libdbi doesn’t provide an abstraction for SQL cursors, so it needs to be done with SQL queries.

The documentation’s page on FETCH has a complete sequence of queries in its example that shows how it’s done. You need to call these queries with libdbi in C with two loops: an outer loop calling FETCH N from cursor_name until there’s nothing left to fetch, and an inner loop processing the results of that FETCH, just like your current code processing the results of the select * itself.