Question

Just trying to learn C++. I’m starting off with a scraper. The idea is I want to scrape a bunch of pages, apply a regex, and write my findings to a file.

However, I’m currently stuck trying to assign the curl handle to write into a string variable. Any pointers (tips I mean…)?

#include <stdio.h>
#include <curl/curl.h>
#include <string>
#include <iomanip>
#include <boost/lexical_cast.hpp>

// Typedefs
using std::string;      using boost::lexical_cast;
using std::cout;
using std::endl;
using std::ostringstream;

// CURL directives
static const string BASE_URL = "http://www.XXXXXX.com/action?page=";
static char* USER_AGENT_STRING = "C++ Scraper"; 

// CURL variables
static CURL* curl;
static CURLcode res;

       void setPageOpts(CURL* c, int count, const ostringstream stream) {
            cout << "Got to the set opts method." << endl;

            // Set URL
            string new_url = BASE_URL + lexical_cast<string>(count);
            curl_easy_setopt(curl, CURLOPT_URL, new_url.c_str());
            curl_easy_setopt(curl, CURLOPT_WRITEDATA, &stream);
        }

        void setConstOpts(CURL *c) {
            // User agent string
            curl_easy_setopt(curl, CURLOPT_USERAGENT, USER_AGENT_STRING);
            //curl_easy_setopt(curl, CURLOPT_VERBOSE, 1);
        }

        void loadCurl() {
            int status = curl_global_init(CURL_GLOBAL_ALL);

            if (status != 0) {  
                cout << "An error has occurred with exit status " << status << endl;
                exit(status);
            }   
        }

        int main() {
            loadCurl();
            curl = curl_easy_init();

            if(curl) {
                setConstOpts(curl);

                for (int i = 1; i < 2; ++i) {           
                    const ostringstream stream;

                    setPageOpts(curl, i, &stream);

                    res = curl_easy_perform(curl);  

                    string output = stream.get();
                    cout << "And the data was: " << endl << endl << output << endl;
                }

                curl_easy_cleanup(curl);
            }

            // Everything went as planned
            return 0;
        }

Current errors I’m getting:

learn.cpp:15: warning: deprecated conversion from string constant to 'char*'
learn.cpp: In function 'int main()':
learn.cpp:62: error: conversion from 'const std::ostringstream*' to non-scalar type 'std::ostringstream' requested
learn.cpp:67: error: request for member 'get' in 'stream', which is of non-class type 'const std::ostringstream*'

The way I see it, I want to get CURL the location of the stream in memory so that it can write there. So I assumed I needed to use a & character and hand that in. But that doesn’t seem to work.

Answer 1

Your definition:

 void setPageOpts(CURL* c, int count, const ostringstream stream)

Is likely to cause you problems. It looks like you mean to pass the stream in by reference so that it can be updated:

void setPageOpts(CURL* c, int count, ostringstream& stream) 

(and don’t forget to use c when you pass it in).

And then instead of this:

                const ostringstream stream;
                setPageOpts(curl, i, &stream);

Call your function:

        ostringstream stream;
        setPageOpts(curl, i, stream);

This will allow stream to be updated.

std::ostringstream has no ::get(). I think you meant:

        string output = stream.str();

You may also find this answer useful, without setting CURLOPT_WRITEFUNCTION to specify a callback function to use your std::ostringstream pointer, you’ll get an access violation, when libcurl tries to write to it.