XML, the Perl Way

Perl & LWP

Perl & LWP
author Sean M. Burke
pages 264
publisher O'Reilly and Associates
rating 9
reviewer mir
ISBN 0-596-00178-9
summary Excellent introduction to extracting and processing information from web-sites


Perl & LWP is a solid, no-nonsense book, that will teach you how to do screen-scraping using Perl.. It describes how to automatically retrieve and use information from the web. An introduction to LWP and related modules from simple to advanced uses and various ways to extract information from the returned HTML.

The good: nice style, good coverage of the subject, includes introduction to all the modules used, reference material and good, well-developed examples. It really liked the way it describes the basic methodology to develop screen-scraping code, from analyzing an HTML page to extracting and displaying only what you are interested in.

The bad: not much, some chapters are a little dry, some times the reference material could be better separated from the rest of the text. The book covers only simple access to web sites, I would have liked to see an example where the application really dialogs with the server. The appendixes are not really useful.

More Info

If it had not been published by O'Reilly, "Perl and LWP" could have been titled "Leveraging the Web: Object-Oriented techniques for information re-purposing", or "Web Services, Generation 0". An even better title would have been "Screen-scraping for fun and profit": one day we might all use Web Services and easily get the information we need from various providers using SOAP or REST, but in the meantime the common way to achive this goal is just to write code to connect to a web server, retrieve a page and extract the information from the HTML. In short "screen-scraping". This will teach you all about using Perl to get Web pages and extract their "substantifique moëlle" (the pith essence, the essentials) for your own usage. It showcases the power of Perl for that kind of job, from regular expressions to powerful CPAN modules.

At 200 pages, plus 40 pages of appendixes and index, it is part of that line of compact O'Reilly books that cover only a narrow topic but that covers it well. Just like Perl & XML its target audience is Perl programers that need to tackle a new domain. It will give them a toolbox and basic techniques that will give them a jump start and avoid many mistakes.

"Perl & LWP" starts from the basics: installing LWP, using LWP::Simple to retrieve a file from a URL, then goes on to a more complete description of the advanced LWP methods for dealing with forms and munging URL's. It continues with 5 chapters on how to process the HTML you get, using regular expressions, an HTML tokenizer and HTML::TreeBuilder, a powerful module that... builds a tree from the HTML. It goes on with how to allow your programs to access sites that require cookies, authentication or the use of a specific browser. The final chapter wraps it all up in a bigger example: a web-spider.

The book is well-written and to-the-point. It is structured in a way that mimics what a programer new to the field would do: start from the docs for a module, play with it, write snippets of code that use the various functions of the module, then go on to coding real-life examples. I particularly liked the fact that the author often explains the whys and not only the hows of the various pieces of code he shows us.

It is interesting to note that going from regular expressions to ever more powerful modules is a path followed also by most Perl programers, and even by the language itself: when Perl start being applied to a new domain first there are no modules, then low-level ones start appearing, then, as the understanding of the problem grows, easier-to-use modules are written.

Finally I would like to thank the author for following his own advice on including interesting examples and above all for not including anything about retrieving stock-quotes.

An other recommended book on the subject is Network Programming with Perl by Lincoln D. Stein, which covers a wider subject but devotes 50 pages to this topic and is also very good.

Breakdown by chapter