🔒 Locked 🎮 Unity

[java] Speeding up threads

Started by Sfpiano Dec 6, 2005 at 5:05 PM 5 replies 650+ views

Original Post

Dec 6, 2005 at 5:05 PM

I'm trying to make a simple webcrawler that gets links and images off of websites. I maintain a list of links to visit, and in my run() method I take one of the links, and send it to a processing thread I have and call start on the thread:


run() {
while(true) {
Thread t = new Thread ( new Processor(url) );
t.start();
}
}

However, the program runs really slowly, and I was wondering if there was somethign I could do t ospeed up the threads.

demonkoryu

131

demonkoryu

Dec 6, 2005 at 5:11 PM

1. Don't create too many threads. 20-50 threads at once are maybe enough.
2. If you write a crawler, have it obey to robots.txt AND don't hammer the site, ie. have a short delay (1 sec) between requests to the same site. If you crawl many sites at once, your crawler will still run at full speed.

aidan_walsh

114

aidan_walsh

Dec 6, 2005 at 5:22 PM

Use a Thread Pool instead of constantly creating new threads.

I would also agree with the robots.txt and timing comments above.

Whackjack

Dec 7, 2005 at 1:28 PM

I would use a queue and just a single thread. Each item you pull off the page, add it to the bottom of a queue (or Vector in Java) and have the thread just pull off each item from the top of the queue. If the queue empties, you can have your thread sleep for a while or something. Vectors are also nice to use because they're synchronized.

Edit: Actually, I mean use 2 threads and a single queue. One thread pushes items onto the queue, the other pops them off and processes them.

Raghar

Dec 7, 2005 at 2:26 PM

java.nio ?

CaptainJester

101

CaptainJester

Dec 12, 2005 at 12:20 PM

public void run() {  while(true) {    Thread t = new Thread ( new Processor(url) );    t.start();    try {      Thread.sleep(500);    catch(InteruptedException e){}  }}

Anytime you have a thread that has a short run loop, you should introduce a sleep or it will hog CPU and is very hard for the O/S to break out of to allow other threads CPU time.

I think Thread.yield() will work too.

Topic Locked

This topic has been locked by a moderator. New replies are not allowed.