Question

I’m creating a a web page scraper. I’m attempting to fill a JTable from data on the page, retrieved using XPath. I want the data for a single row put into an array.

{"Name","Phone","Address","City","State","Postal Code","Link"}

I retrieve the webpage into a string to be parsed. There are no \n’s(Sorry, I don’t know the proper word for this. New lines?)

The array represents the data in a column. The way I want to do it is to make it so it retrieves the individual data row that row, then inserts it into the array, then it’s placed into the JTable. Then the array is set to null, the process is repeated.

Another option which I REALLY do not want to have to deal with would be placing all of the individual cells into array lists for each column, then do a while loop. The issue with this is that a entry might return empty, which would mean the indexes might not be perfectly synced.

I’m a rather “new” programmer, with about 6 months of experience of actually doing most things on my own. I’m still working on building my methodology, however here I’m stuck.

I’m wondering what would be the best solution in this situation, there’s probably a better solution that I have not thought of.

This is not a raw code problem, rather a flow problem.

It’s messy, yes. However it was very easy for me to crawl using Google Docs’s xPath & ImportXML functions.

Here is what I would put in the cells:

Name:

=importxml("http://www.yellowpages.com/[location]/[type]","//a[@class='no-tracks url ']")

Address:

=importxml("http://www.yellowpages.com/[location]/[type]","//span[@class='street-address']")

Phone:

=importxml("http://www.yellowpages.com/[location]/[type]","//span[@class='business-phone phone']")

I didn’t do the others, however I know it’s possible.

Removed the HTML example for privacy reasons, sorry.

Answer 1

Generate an Object that best represents the data (array). Use the data from the array to construct the object and add the object to the table model…

public class Data {
    private String name;
    private String phone;
    private String address;
    private String city;
    private String state;
    private String postalCode;
    private String link;
    public Data(String[] data) { 
        name = data[0];
        phone = data[1];
        address = data[2];
        city = data[3];
        state = data[4];
        postalCode = data[5];
        link= data[6];
    }

    public String getName() {
        return name;
    }

    public String getPhone() {
        return phone;
    }

    public String getAddress() {
        return address;
    }

    public String getCity() {
        return city;
    }

    public String getState() {
        return state;
    }

    public String getPostalCode() {
        return postalCode;
    }

    public String getLink() {
        return link;
    }
}

Create a table model that is capable of taking advantage of this data object…

public class MyDataTableModel extends AbstractTableModel {

    private List<Data> dataList;

    public MyDataTableModel() {
        dataList = new ArrayList<Data>(25);
    }

    @Override
    public int getRowCount() {
        return dataList.size();
    }

    @Override
    public int getColumnCount() {
        return 7;
    }

    @Override
    public String getColumnName(int column) {
        String name = "??";
        switch (column) {
            case 0:
                name = "Name";
                break;
            case 1:
                name = "Phone";
                break;
            case 2:
                name = "Address";
                break;
            case 3:
                name = "City";
                break;
            case 4:
                name = "State";
                break;
            case 5:
                name = "Postal Code";
                break;
            case 6:
                name = "Link";
                break;
        }
        return super.getColumnName(column);
    }

    @Override
    public Object getValueAt(int rowIndex, int columnIndex) {
        Object value = null;
        Data data = dataList.get(rowIndex);
        switch (columnIndex) {
            case 0:
                value = data.getName();
                break;
            // Other case statements per column...
        }
        return value;
    }

    public void add(Data data) {
        dataList.add(data);
        int index = dataList.size() - 1;
        fireTableRowsInserted(index, index);
    }
}

Once you have a new Data object, use this models add method to add it to the model.