It's hard to
argue with the proposition that the World Wide Web is the largest repository
of information that has ever existed. In just over a decade, the Web has
moved from a university curiosity to a fundamental research, marketing and
communications vehicle that impinges upon the everyday life of most people in
the developed world. But there's a catch, of course. As the amount of
information on the Web grows, that information becomes ever harder to keep
track of and use.
This vast
amount of freely available information is spread over billions of Web pages,
each with its own independent structure and format. So how do you find the
information you're looking for in a useful format -- and do it quickly and
easily without breaking the bank?
Search Isn't Enough
Search
engines are a big help, but they can do only part of the work, and they are
hard-pressed to keep up with daily changes. For all the power of Google and
its kin, all that search engines can do is locate information and point to
it. They go only two or three levels deep into a Web site to find information
and then return URLs. They also find and return meta descriptions and meta
keywords embedded in Web pages, but these may well be inaccurate.
Consider that
even when you use a search engine to locate data, you still have to do the
following tasks to capture the information you need:
- Scan the content until you find the information.
- Mark the information (usually by highlighting with a mouse).
- Copy the information.
- Switch to another application (such as a spreadsheet, database or word
processor).
- Paste the information into that application.
A better
solution, especially for companies that are aiming to exploit a broad swath
of data about markets or competitors, lies with Web harvesting tools.
Web
harvesting software automatically extracts information from the Web and picks
up where search engines leave off, doing the work the search engine can't.
Extraction tools automate the reading, copying and pasting necessary to
collect information for analysis, and they have proved useful for pulling
together information on competitors, prices and financial data of all types.
Harvesting Techniques
There are
three ways we can extract more useful information from the Web.
The first
technique, Web content harvesting, is concerned directly with the specific
content of documents or their descriptions, such as HTML files, images or
e-mail messages. Since most text documents are relatively unstructured (at
least as far as machine interpretation is concerned), one common approach is
to exploit what's already known about the general structure of documents and
map this to some data model.
Another
approach to Web content harvesting involves trying to improve on the content
searches that tools like search engines perform. This type of content
harvesting goes beyond keyword extraction and the production of simple
statistics relating to words and phrases in documents.
Another
technique, Web structure harvesting, takes advantage of the fact that Web
pages can reveal more information than just their obvious content. Links from
other sources that point to a particular Web page indicate the popularity of
that page, while links within a Web page that point to other resources may
indicate the richness or variety of topics covered in that page. This is like
analyzing bibliographical citations -- a paper that's often cited in
bibliographies and other papers is usually considered to be important.
The third
technique, Web usage harvesting, uses data recorded by Web servers about user
interactions to help understand user behavior and evaluate the effectiveness
of the Web structure.
General
access-pattern tracking analyzes Web logs to understand access patterns and
trends in order to identify structural issues and resource groupings.
Customized
usage tracking analyzes individual trends so that Web sites can be
personalized to specific users. Over time, based on access patterns, a site
can be dynamically customized for a user in terms of the information
displayed, the depth of the site structure and the format of the resources
presented.
Also Known As . . .
Over the past
decade, the terminology used to describe Web harvesting has undergone several
changes. In 1996, researcher Oren Etzioni wrote a paper called "The
World Wide Web: Quagmire or Gold Mine?" which was published in the
journal Communications of the ACM. Etzioni defined Web mining as the use of
data mining techniques to automatically discover and extract information from
Web documents and services.
In the late
1990s, Richard Hackathorn coined the term Web farming to describe a
discipline combining aspects of data warehousing, Web data mining and
knowledge-base creation.
Around the
turn of the millennium, Web harvesting began to replace Web mining as the
fashionable buzzphrase, although it can mean different things to different
people. Web harvesting can be synonymous with Web mining, Web farming and Web
scraping, but it can have other meanings as well. One widespread usage of the
term refers specifically to the searching of Web pages for e-mail addresses
for resale and use in commercial solicitations (i.e. spam).
The Web site
of the Medical University of South Carolina defines Web harvesting as "the
process of downloading RSS feeds and consolidating them for display."
Another
related term is Web scraping, an obvious derivation from the 1980s
catchphrase "screen scraping," where PC- or mini-based applications
accessing mainframe systems emulated 3270 or VT100 terminals. Such
applications were quick and cheap but not always reliable. Similarly, Web
scraping applications process a Web page's HTML to extract meaningful data,
often from live data feeds or by manipulating specific applications. Web scrapers
are also cheap and useful but of questionable reliability.
Kay is a Computerworld contributing writer in Worcester, Mass.
Contact him at russkay@charter.net.
| 
