🔒 Locked

[java] Grabbing HTML source off the web

Started by Kevinator Sep 14, 2005 at 7:01 PM 2 replies 1.1k views

Original Post

Sep 14, 2005 at 7:01 PM

Hi guys. I want to implement an application in Java that lets the user type in a URL, then it will grab the source code at that URL, process it, and spit out some results. I have a clear vision of how I will proceed once I have the html file, but I don't know how to snag the source code off the internet in the first place. Can anyone point me in even vaguely the right direction?

SiCrane

Mod

473

SiCrane

Sep 14, 2005 at 7:03 PM

Provided you already know how to open a TCP socket, then take a look at RFC 2616 (HTTP 1.1)

bleb

Sep 15, 2005 at 9:15 AM

... or you could avoid reinventing the wheel and try this:

 private String fetchHTML( String url ) {  StringBuffer buff = new StringBuffer();   try  {   URL query = new URL( url );   URLConnection connection = query.openConnection();   connection.connect();   BufferedReader br = new BufferedReader( new InputStreamReader( connection.getInputStream() ) );   String line = null;   do   {    line = br.readLine();    buff.append( line );   }   while( line != null );      br.close();  }  catch( Exception e )  {   e.printStackTrace();   return "";  }  return buff.toString(); }

There's probably clean-up problems that might have to be handled in a finally clause, but it should get you started.

Kevinator

Author

Kevinator

Sep 15, 2005 at 5:52 PM

WOW! Thanks, bleb! I figured there must be something somewhere in Java's gigantic library to handle something like this, I just didn't know where to look. [grin]

Topic Locked

This topic has been locked by a moderator. New replies are not allowed.