Home/ Questions/Q 9151281
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-17T11:45:56+00:00

I’m trying to extract all the links from a website. I used Jsoup in

  • 0

I’m trying to extract all the links from a website. I used Jsoup in previous programs to do this, the problem here is that “more content” is generated by pressing the more button and it doesn’t change pages, it simply loads more content so I’m not sure how to see all of the available links using Java and Jsoup.

The website is http://seekingalpha.com/symbol/msft and I’m simply trying to extract all the links to articles for a specific company, such as Microsoft.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    You need to get yourself something you can spy on the requests you’re making over the wire. You could watch the http traffic using the Network tab in Chrome, but personally I like Charles. Anyway, if you check out what happens when you click the more button you would see that a POST request is being made (using AJAX of course) the looks like this:

    http://seekingalpha.com/account/ajax_headlines_content 200 POST seekingalpha.com /account/ajax_headlines_content 432 ms 5.94 KB Complete

    In the headers the params are:

    type all
    page 2
    slugs msft
    is_symbol_page true

    So if I were you I’d just emulate that by making the POST request with the page param counting up until you’ve got all the content you want. By the way the content returned was an html fragment so easy to parse, eg:

    > <div class="symbol_articles_list mini_category">
>     <div>
>       <ul>
>           <li>
>               <div class="fl">
>                 <a class="small_picture" href="/author/the-part-time-investor">
>                   <img height="20" width="20" src="http://static1.cdn-seekingalpha.com/images/users_profile/000/541/113/small_pic.png?1352646193"/>
>                 </a>
>               </div>
>             <div class="content">
>               <div class="symbol_article">
>                 <a href="/article/1109901-microsoft-and-4-other-undervalued-stocks"
> sasource="portfolio_rel">Microsoft And 4 Other Undervalued Stocks</a>
>                 <div class="date_on_by">
>                     <a sasource="portfolio_rel" href="/author/the-part-time-investor">The Part-time Investor</a>
>                   <span class="bullet">&bull;</span>
>                   Mon, Jan 14<span class='bullet'>&bull;</span> <span class='comments'><a
> href='/article/1109901-microsoft-and-4-other-undervalued-stocks#comments_header'
> sasource='headlines_tabs'>44&nbsp;Comments</a></span>               
> </div>
>               </div>
>             </div>
>             <div class="cleaner"></div>
>           </li>
    • 0