FREE Delivery in the UK.
Only 1 left in stock (more on the way).
Dispatched from and sold by Amazon. Gift-wrap available.
Spidering Hacks: 100 Indu... has been added to your Basket

Dispatch to:
To see addresses, please
Please enter a valid UK postcode.
+ £2.80 UK delivery
Used: Good | Details
Sold by Brit-Books
Condition: Used: Good
Comment: Simply Brit: We have dispatched from our UK warehouse books of good condition to over 1 million satisfied customers worldwide. We are committed to providing you with a reliable and efficient service at all times.
Have one to sell?
Flip to back Flip to front
Listen Playing... Paused   You're listening to a sample of the Audible audio edition.
Learn more
See this image

Spidering Hacks: 100 Industrial-Strength Tips & Tools Paperback – 7 Nov 2003

2.0 out of 5 stars 1 customer review

See all 4 formats and editions Hide other formats and editions
Amazon Price
New from Used from
Kindle Edition
£10.98 £1.75
Note: This item is eligible for click and collect. Details
Pick up your parcel at a time and place that suits you.
  • Choose from over 13,000 locations across the UK
  • Prime members get unlimited deliveries at no additional cost
How to order to an Amazon Pickup Location?
  1. Find your preferred location and add it to your address book
  2. Dispatch to this address when you check out
Learn more
£23.99 FREE Delivery in the UK. Only 1 left in stock (more on the way). Dispatched from and sold by Amazon. Gift-wrap available.
click to open popover

Enter your mobile number or email address below and we'll send you a link to download the free Kindle App. Then you can start reading Kindle books on your smartphone, tablet, or computer - no Kindle device required.

  • Apple
  • Android
  • Windows Phone

To get the free app, enter your mobile phone number.

60 Kindle Books for £1
Browse our selection of Kindle Books discounted to £1 each. Learn more

Product details

  • Paperback: 428 pages
  • Publisher: O'Reilly Media; 1 edition (7 Nov. 2003)
  • Language: English
  • ISBN-10: 0596005776
  • ISBN-13: 978-0596005771
  • Product Dimensions: 15.2 x 2.5 x 22.9 cm
  • Average Customer Review: 2.0 out of 5 stars 1 customer review
  • Amazon Bestsellers Rank: 209,034 in Books (See Top 100 in Books)
  • Would you like to tell us about a lower price?
    If you are a seller for this product, would you like to suggest updates through seller support?

  • See Complete Table of Contents

Product description

About the Author

Kevin Hemenway, coauthor of Mac OS X Hacks, is better known as Morbus Iff, the creator of, which bills itself as "content for the discontented." Publisher and developer of more home cooking than you could ever imagine, he'd love to give you a Fry Pan of Intellect upside the head. Politely, of course. And with love.

Tara Calishain is the creator of the site, ResearchBuzz. She is an expert on Internet search engines and how they can be used effectively in business situations.

Excerpt. © Reprinted by permission. All rights reserved.

Hack #18 Adding Progress Bars to Your Scripts
Give a visual indication that a download is progressing smoothly.

With all this downloading, it’s often helpful to have some visual representation of its progress. In most of the scripts in this book, there’s always a bit of visual information being displayed to the screen: that we’re starting this URL here, processing this data there, and so on. These helpful bits usually come before or after the actual data has been downloaded. But what if we want visual feedback while we’re in the middle of a large MP3, movie, or database leech?

If you’re using a fairly recent vintage of the LWP library, you’ll be able to interject your own subroutine to run at regular intervals during download. In this hack, we’ll show you four different ways of adding various types of progress bars to your current applications. To get the most from this hack, you should have ready a URL that’s roughly 500 KB or larger; it’ll give you a good chance to see the progress bar in action.

The Code
The first progress bar is the simplest, providing only a visual heartbeat so that you can be sure things are progressing and not just hanging. Save the following code to a file called and run it from the command line as perl scriptname URL, where URL is the online location of your appropriately large piece of sample data:

#!/usr/bin/perl -w
# Progress Bar: Dots - Simple example of an LWP progress bar.#
# This code is free software; you can redistribute it and/or
# modify it under the same terms as Perl itself.

use strict; $|++;
my $VERSION = "1.0";

# make sure we have the modules we need, else die peacefully.
eval("use LWP 5.6.9;"); die "[err] LWP 5.6.9 or greater required.\n" if $@;

# now, check for passed URLs for downloading.
die "[err] No URLs were passed for processing.\n" unless @ARGV;

# our downloaded data.
my $final_data = undef;

# loop through each URL.foreach my $url (@ARGV) {
print "Downloading URL at ", substr($url, 0, 40), "... ";

# create a new useragent and download the actual URL.
# all the data gets thrown into $final_data, which
# the callback subroutine appends to.
my $ua = LWP::UserAgent->new( );
my $response = $ua->get($url, ':content_cb' => \&callback, );
print "\n"; # after the final dot from downloading.

# per chunk.
sub callback {
my ($data, $response, $protocol) = @_;
$final_data .= $data;
print ".";

None of this code is particularly new, save the addition of our primitive progress bar. We use LWP’s standard get method, but add the :content_cb header with a value that is a reference to a subroutine that will be called at regular intervals as our content is downloaded. These intervals can be suggested with an optional :read_size_hint, which is the number of bytes you’d like received before they’re passed to the callback.

In this example, we’ve defined that the data should be sent to a subroutine named callback. You’ll notice that the routine receives the actual content, $data, that has been downloaded. Since we’re overriding LWP’s normal $response->content or :content_file features, we now have to take full control of the data. In this hack, we store all our results in $final_data, but we don’t actually do anything with them.

Customer reviews

2.0 out of 5 stars
Share your thoughts with other customers
See all 1 customer reviews

Top customer reviews

on 14 January 2006
Format: Paperback|Verified Purchase
0Comment| 5 people found this helpful. Was this review helpful to you?YesNoReport abuse

Most helpful customer reviews on 4.0 out of 5 stars 19 reviews
2 people found this helpful.
4.0 out of 5 starsA Classic (is it outdated?)
on 12 October 2010 - Published on
Format: Paperback|Verified Purchase
3.0 out of 5 starsA Little Knowledge Goes a Long Way
on 26 March 2010 - Published on
Format: Paperback|Verified Purchase
5.0 out of 5 starsFive Stars
on 19 June 2015 - Published on
Format: Paperback|Verified Purchase
2.0 out of 5 starsNot for me...
on 6 February 2008 - Published on
Format: Paperback|Verified Purchase
5.0 out of 5 starsEnter the Spider
on 9 February 2010 - Published on
Format: Paperback|Verified Purchase
Pages with related products. See and discover other items: api programming

Where's My Stuff?

Delivery and Returns

Need Help?