Home/ Questions/Q 6344891
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-24T20:43:36+00:00

How do I parse a document in a table and send it across as

  • 0

How do I parse a document in a table and send it across as a JSON file to another db.

Detailed Desc:
I have crawled and taken data into a table from websites using anemone. I need to now parse it and transfer it as a JSON file to another server. I think, I will have to first convert the document in the table into nokogiri document which can be parsed and converted to json file. Any idea how can I convert the doc into nokogiri document or if anyone has any other idea to parse it and send it as a json file ?

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Okay I found the answer long time back, I basically made use of REST to send message form one application to another, i sent it across as a hash. And the obvious one, I used nokogiri for parsing the table. 

    def post_me
     @page_hash = page_to_hash

    res = Net::HTTP.post_form(URI.parse('http://127.0.0.1:3007/element_data/save.json'),@page_hash)
  end

    For sending the hash from one application to another using net/http.

    def page_to_hash
    require 'rubygems'
    require 'nokogiri'
    require 'open-uri'
    require 'domainatrix'

    #page = self.page.sub(/^<!DOCTYPE html(.*)$/, '<!DOCTYPE html>')
    hash={}
    doc = Nokogiri::HTML(self.page)
    doc.search('*').each do |n|
      puts n.name
    end

    Using Nokogiri for parsing the page table in my model. page table had the whole body of a webpage.

    file_type = []

    file_type_data=doc.xpath('//a/@href[contains(. , ".pdf") or contains(. , ".doc")
                          or contains(. , ".xls") or contains(. , ".cvs") or contains(. , ".txt")]')
    file_type_data.each do |href|
      if href[1] == "/"
        href = "http://" + website_url + href
      end
      file_type << href
    end
file_type_str = file_type.join(",")
    hash ={:head => head,:title => title, :body => self.body,
      :image => images_str, :file_type => file_type_str, :paragraph => para_str, :description => descr_str,:keyword => key_str,
      :page_url=> self.url, :website_id=>self.parent_request_id, :website_url => website_url,
      :depth => self.depth, :int_links => @int_links_arr, :ext_links => @ext_links_arr
    }

    A simple parsing example and how i formed my hash.

    • 0