Question

I have a self-made framework/API and it does some database searches that can take a long time. A long time means several seconds up to minutes depending on input. The search puts found results into a java.util.concurrent.BlockingQueue. Hence the first search result is immediatley available.

I would now like to create a search page and after submitting the search the results page instantly loads with these first hits and then periodically adds subsequent hits until the search completed. How could I do that?

Answer 1

OK. I came up with a solution.

BIG WARNING:
untested for production use!

Used javascript dependencies: JQuery, jquery datatables plug-in (http://datatables.net) and fnStandingRedraw plugin for datatables.

Java Dependencies: googles Gson library

I have a search page that submits to a servlet.

The servlet starts the search and puts the search thread and the BlockingQueue into session and then redirects to the results page:

final LinkedBlockingQueue<Integer> hits =
    new LinkedBlockingQueue<Integer>();
Thread searcher = new Thread() {

@Override
public void run() {

    query.search(hits);
}
};
searcher.start();
session.setAttribute("searchQueue", hits);            
session.setAttribute("searchThread", searcher);
response.sendRedirect("SearchResult.jsp");

Note that the Queue is filled with the primary key from the database, hence integer.

The results page needs following references:

<link href="css/datatables/demo_table.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="js/jquery-1.6.2.js"></script>
<script type="text/javascript" src="js/jquery.dataTables.min.js"></script>               
<script type="text/javascript" src="js/datatables.fnStandingRedraw.js"></script>  

and following javascript code to initialize the table and add rows:

<script type="text/javascript">    
    var resultsTable;
    var myTimer;

    $(document).ready(function() {
        resultsTable = $('#result').dataTable( {
            "bDeferRender": true,
            "bProcessing": true,
            "sPaginationType": "full_numbers",
            "iDisplayLength": 4,            
            "aLengthMenu": [[4, -1], [4, "All"]],
            "sScrollY": "580", 
            "aaSorting": [],
            "aoColumns": [
            /* column1 */   {"bSearchable": true, 
                    "bSortable": true,
                    "sWidth": "50px"},
                /* column2*/{"bSearchable": false, 
                    "bSortable": false,
                    "sWidth": "510px"}
            ]
        } );            
        getAndAddRows();
        startTimer();
    } ); 

    function startTimer() {
        myTimer = window.setInterval( function() {
            getAndAddRows();
        }, 500);
    };

    function stopTimer(){
        window.clearInterval(myTimer);
    }


    function getAndAddRows(){
    $.get(
        'getSearchHits',
        function(data){
            if(data[0] == 'searchCompleted'){
                    stopTimer();
                    $('#hitsFound').text('Hits Found: ' 
                        + resultsTable.fnSettings().fnRecordsTotal()
                        + ' Search Completed');  
                    return;
            }
            $('#result').dataTable().fnAddData(data, false);            
            resultsTable.fnStandingRedraw();
            $('#hitsFound').text('Hits Found: ' 
                + resultsTable.fnSettings().fnRecordsTotal()
                + ' Searching...');  
        }, 
        "json"
        );    
    } 
</script>

<h1 id="hitsFound">Hits Found:</h1>

<table id="result">
    <thead>
        <tr>
            <th>column1</th>
            <th>column2</th>
        </tr>
    </thead>
    <tbody>
    </tbody>
</table>

Please also refer to the datatables homepage on how to setup such a table.
The JQuery get-function is called periodically based on a timer. It calls another servlet that reads from the Queue and returns new hits formated as JSON:

int limit = 50; //amount of records ro return
int counter = 0;
BlockingQueue<Integer> hits = (BlockingQueue<Integer>) session.getAttribute("searchQueue");
Thread searcher = (Thread) session.getAttribute("searchThread");
Gson gson = new Gson();
ArrayList<ArrayList> rows = new ArrayList<ArrayList>();

while (hits.size() > 0 && limit > counter) {
    ArrayList row = new ArrayList();
    Integer key = hits.take();
    // get desired data and add it to current row example:
    String myData = dataccessObject.getMyData(key); 

    row.add(key);
    row.add(myData);

    rows.add(row);      
    counter++;
}

if (rows.size() < 1 && !searcher.isAlive()) {
    out.print("[\"searchCompleted\"]");
} else {
    String json = gson.toJson(rows); //==> json is [[1,"myData1"],[2,"myData2"]]        
    out.print(json);
}

The limit and timers might need to be adjusted depending on your purpose. The limit is there so that new results are actually added periodically. (If getting the data is slower than the rate at which new hits are found, then nothing would be returned and displayed until search finishes).

Note that the string myData can be HTML and it is rendered as HTML. In my case I return image tags. And those image tags have another servlet as source that creates the images dynamically. Note that the datatables option “bDeferRender”: true only renders the according rows (and in my case images) once they get visible for the first time. Eg. if a user only looks at the first 3 pages, pages 4-100 never actually get rendered. And if a user clicks on the last page only the last one gets rendered not all in between.

Please comment in case of questions or improvements / possible bugs.