Question

I have to do a HTTP post in java every second after building a json object.
The json object is built from reading a CSV file which is huge (200Mbs+), so my problem is
how do I read x number of lines build x objects and post that every second(as it is not possible to parse the whole 200mb file in less than a second) and continue reading the next x lines.

Please let me know your thoughts..

Can I use Java timer class, and keep reading the CSV file and at the same time post the json object to the server every second with the formed json?

Answer 1

I would make it depending on the file size and not depending on time.

BufferedReader fin = null; //create it
    Gson gson=new Gson();  //Google code open source library for JSON in Java  
    ArrayList<JSONObject> jsonList=new ArrayList<JSONObject>();    
    while (((line = fin.readLine()) != null)) {
                            if ( line.length()==0 ){
                                //"Blank line;
                            }else{
                                        currJSON=loadJSON(line);//You have to load it in a Java Object

                               if ( jsonList.size()<MAX_JSON){
                               jsonList.add(currJSON);
                            }

        if (JsonList.size()==MAX_JSON){ //Define the maximum size of the list you want to post
        gson.toJson(jsonList); //Convert to JSON
        //You should post your Json with some Http Connection to your server
    jsonList.clear();