Home/ Questions/Q 7036527
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-28T01:26:06+00:00

Using the following code I am trying to scrape a call log from our

  • 0

Using the following code I am trying to scrape a call log from our phone provider’s web application to enter the info into my Ruby on Rails application.

desc "Import incoming calls"
task :fetch_incomingcalls => :environment do

    # Logs into manage.phoneprovider.co.uk and retrieved list of incoming calls.
    require 'rubygems'
    require 'mechanize'
    require 'logger'

    # Create a new mechanize object
    agent = Mechanize.new { |a| a.log = Logger.new(STDERR) }

    # Load the Phone Provider website
    page = agent.get("https://manage.phoneprovider.co.uk/login")

    # Select the first form
    form = agent.page.forms.first
    form.username = 'username
    form.password = 'password

    # Submit the form
    page = form.submit form.buttons.first

    # Click on link called Call Logs
    page = agent.page.link_with(:text => "Call Logs").click

    # Click on link called Incoming Calls
    page = agent.page.link_with(:text => "Incoming Calls").click

    # Prints out table rows
    # puts doc.css('table > tr')

    # Print out the body as a test
    # puts page.body

end

As you can see from the last five lines, I have tested that the ‘puts page.body’ works successfully and the above code works. It successfully logs in and then navigates to Call Logs followed by Incoming Calls.The incoming call table looks like this:

| Timestamp    |    Source    |    Destination    |    Duration    |
| 03 Jan 13:40 |    12345678  |    12345679       |    00:01:01    |    
| 03 Jan 13:40 |    12345678  |    12345679       |    00:01:01    |    
| 03 Jan 13:40 |    12345678  |    12345679       |    00:01:01    |    
| 03 Jan 13:40 |    12345678  |    12345679       |    00:01:01    |

Which is generated from the following code:

<thead>
<tr>
<td>Timestamp</td>
<td>Source</td>
<td>Destination</td>
<td>Duration</td>
<td>Cost</td>
<td class='centre'>Recording</td>
</tr>
</thead>
<tbody>
<tr class='o'>
<tr>
<td>03 Jan 13:40</td>
<td>12345678</td>
<td>12345679</td>
<td>00:01:14</td>
<td></td>
<td class='opt recording'>
</td>
</tr>
</tr>
<tr class='e'>
<tr>
<td>30 Dec 20:31</td>
<td>12345678</td>
<td>12345679</td>
<td>00:02:52</td>
<td></td>
<td class='opt recording'>
</td>
</tr>
</tr>
<tr class='o'>
<tr>
<td>24 Dec 00:03</td>
<td>12345678</td>
<td>12345679</td>
<td>00:00:09</td>
<td></td>
<td class='opt recording'>
</td>
</tr>
</tr>
<tr class='e'>
<tr>
<td>23 Dec 14:56</td>
<td>12345678</td>
<td>12345679</td>
<td>00:00:07</td>
<td></td>
<td class='opt recording'>
</td>
</tr>
</tr>
<tr class='o'>
<tr>
<td>21 Dec 13:26</td>
<td>07793770851</td>
<td>12345679</td>
<td>00:00:26</td>
<td></td>
<td class='opt recording'>
</td>
</tr>
</tr>

I’m trying to work out how to selects just the cells I want (Timestamp, Source, Destination and Duration) and output those. I can then worry about outputting them to the database rather than in Terminal.

I have tried using Selector Gadget but it just show either ‘td’ or ‘tr:nth-child(6) td , tr:nth-child(2) td’ if I select multiple.

Any help or pointers would be appreciated!

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    There is a pattern in the table that is easy to leverage using XPath. The <tr> tag of rows with the required information lack the class attribute. Fortunately, XPath provides some simple logical operations including not(). This provides just the functionality we need.

    Once we’ve reduced the number of rows we’re dealing with, we can iterate over the rows and extract the text of the necessary columns by using XPath’s element[n] selector. One important note here is that XPath counts elements starting from 1, so the first column of a table row would be td[1].

    Example code using Nokogiri (and specs):

    require "rspec"
require "nokogiri"

HTML = <<HTML
<table>
  <thead>
    <tr>
      <td>
        Timestamp
      </td>
      <td>
        Source
      </td>
      <td>
        Destination
      </td>
      <td>
        Duration
      </td>
      <td>
        Cost
      </td>
      <td class='centre'>
        Recording
      </td>
    </tr>
  </thead>
  <tbody>
    <tr class='o'>
      <td></td>
    </tr>
    <tr>
      <td>
        03 Jan 13:40
      </td>
      <td>
        12345678
      </td>
      <td>
        12345679
      </td>
      <td>
        00:01:14
      </td>
      <td></td>
      <td class='opt recording'></td>
    </tr>
    <tr class='e'>
      <td></td>
    </tr>
    <tr>
      <td>
        30 Dec 20:31
      </td>
      <td>
        12345678
      </td>
      <td>
        12345679
      </td>
      <td>
        00:02:52
      </td>
      <td></td>
      <td class='opt recording'></td>
    </tr>
    <tr class='o'>
      <td></td>
    </tr>
    <tr>
      <td>
        24 Dec 00:03
      </td>
      <td>
        12345678
      </td>
      <td>
        12345679
      </td>
      <td>
        00:00:09
      </td>
      <td></td>
      <td class='opt recording'></td>
    </tr>
    <tr class='e'>
      <td></td>
    </tr>
    <tr>
      <td>
        23 Dec 14:56
      </td>
      <td>
        12345678
      </td>
      <td>
        12345679
      </td>
      <td>
        00:00:07
      </td>
      <td></td>
      <td class='opt recording'></td>
    </tr>
    <tr class='o'>
      <td></td>
    </tr>
    <tr>
      <td>
        21 Dec 13:26
      </td>
      <td>
        07793770851
      </td>
      <td>
        12345679
      </td>
      <td>
        00:00:26
      </td>
      <td></td>
      <td class='opt recording'></td>
    </tr>
  </tbody>
</table>
HTML

class TableExtractor  
  def extract_data html
    Nokogiri::HTML(html).xpath("//table/tbody/tr[not(@class)]").collect do |row|
      timestamp   = row.at("td[1]").text.strip
      source      = row.at("td[2]").text.strip
      destination = row.at("td[3]").text.strip
      duration    = row.at("td[4]").text.strip
      {:timestamp => timestamp, :source => source, :destination => destination, :duration => duration}
    end
  end
end

describe TableExtractor do
  before(:all) do
    @html = HTML
  end

  it "should extract the timestamp properly" do
    subject.extract_data(@html)[0][:timestamp].should eq "03 Jan 13:40"
  end

  it "should extract the source properly" do
    subject.extract_data(@html)[0][:source].should eq "12345678"
  end

  it "should extract the destination properly" do
    subject.extract_data(@html)[0][:destination].should eq "12345679"
  end

  it "should extract the duration properly" do
    subject.extract_data(@html)[0][:duration].should eq "00:01:14"
  end

  it "should extract all informational rows" do
    subject.extract_data(@html).count.should eq 5
  end
end
    • 0