Home/ Questions/Q 6071613
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-23T10:04:02+00:00

I checked for this specific question and could not find any. I am writing

  • 0

I checked for this specific question and could not find any. I am writing a program in Java which analyses content from web pages, so I need a regular expression which can weed out all the links and tags (href, img, etc…), so that I could display only the pure content written and visible in the webpages. Thanks a lot.

Hi I wanted to make it more specific: 

URLConnection connection = wordURL.openConnection("http://en.wikipedia.org/wiki/Bloom_filter");
BufferedReader br = new BufferedReader(new InputStreamReader(connection.getInputStream()));
String line;
String word = "bloom filter";
String regexp2 = word; 
Pattern pattern2 = Pattern.compile(regexp2);
String HTML_REGEX = "(<.+?>)+"; // as per your answer(Martijn Courteaux)
while ((line = br.readLine()) != null)
{
       String content;
       if ( (content = line.replaceAll(HTML_REGEX, "\n") )!= null)
       {
              Matcher matcher2 = pattern2.matcher(line);
              if(matcher2.find())
              {
                   System.out.println(line);
              }
        }
 }

But unfortunately it still prints out paragraph (<p>) tag and also <li> tag with some rubbish inside </li>. I would like to restrict it to display only those words where “bloom filter” is present.Thanks again.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    I really know it isn’t good to use a regex with html. But if he really wants to this might help:

    String HTML_REGEX = "<.+?>";
String yourHTML = "<html><body><h1>Lorem Ipsum</h1>" + 
                  "<p>Lorem <i>Ipsum</i> dolorem sedet. Set nihil amat. " + 
                  "<sub>I don't know the text</sub></p></body></html>"

String content = yourHTML.replaceAll(HTML_REGEX, "\n");
System.out.println(content);

    prints:

    


Lorem Impsum

Lorem 
Ipsum
dolorem sedet. Set nihil amat. 
I don't know the text

    As you can see, it will work, but it is definitely not what you want.

    You can reduce the number of newlines by using this regex:

    String HTML_REGEX = "(<.+?>)+";
String yourHTML = "<html><body><h1>Lorem Ipsum</h1>" + 
                  "<p>Lorem <i>Ipsum</i> dolorem sedet. Set nihil amat. " + 
                  "<sub>I don't know the text</sub></p></body></html>"

String content = yourHTML.replaceAll(HTML_REGEX, "\n");
System.out.println(content);

    prints:

    
Lorem Impsum
Lorem 
Ipsum
dolorem sedet. Set nihil amat. 
I don't know the text

    I tried your code, and it didn’t work indeed. After some editing this worked:

    URLConnection connection = new URL("http://en.wikipedia.org/wiki/Bloom_filter").openConnection();
BufferedReader br = new BufferedReader(new InputStreamReader(connection.getInputStream()));
String line;
String word = "bloom filter".toLowerCase();
String HTML_REGEX = "(<.+?>)+"; // as per your answer(Martijn Courteaux)
while ((line = br.readLine()) != null) {
    String content;
    if ((content = line.replaceAll(HTML_REGEX, "\n")) != null) {
        if (content.toLowerCase().contains(word)) /* Changed: regex match -> contains() */
        {
            System.out.println(content); /* CHANGED: line -> content */
        }
    }
}

    What you did wrong was:

    1. You didn’t print content, but line which of course contains the tags…
    2. You tried to find the word “bloom filter” using a regex, which is case sensitive. So, just lowercase the strings and use String.contains(CharSequence target), which tells you if the target string is a part of the whole string.
    • 0