Home/ Questions/Q 7054957
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-28T03:41:24+00:00

I want to use java.net.url to crawl some websites and retrieve some data. I

  • 0

I want to use java.net.url to crawl some websites and retrieve some data.

I am confused about the following issues–

(1) Suppose I configure the crawler to visit a video sharing webpage, for eg You Tube. Now, the crawler is set to visit a specific You Tube video page– does this mean that when the crawler actually visits that page, it will by default download all elements on that page, including the FLV Video? Or can I control which files to retrieve. The aim being, minimisation of bandwidth utilisation on Google App Engine. Specifically, initially I want only the HTML web page itself to be retrieved, without retrieving images/videos/other attachments on that web page… is this possible, either on Google App Engine, or as part of a regular java web app?

(2) What is the quick and easy way to know the exact bandwidth being utilised for visiting a single specific site? So that I can keep track of bandwidth utilisation?

Also keeping the above 2 issues in mind, do you recommend usage of java.net.url or low level API? Or do you think I should not stick with App Engine (and use for eg. AWS)?

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    (1) Your crawler will only load what the web-server responds for a specific URL, which normally is pure HTML. In case of YouTube, just right-click with your browser on a page and select View Source. That is what you’ll download if you load the page automatically. No video, just text.

    (2) when you read the content of the webpage, just count the bytes you received. That is your bandwidth.

    • 0