CSCI E-1 | Section | Treasure Hunting

The Internet provides a means of communications between computers in the world. But what exactly do we communicate to each other? In general, the answer is information, be it in the form of a webpage you read, an email you receive, or a conversation you have with a friend via, say, GoogleTalk.
The World Wide Web, in particular, is a huge collection of information, making billions of webpages available to you and to anyone in the world. However, the information does not go to you; instead, you have to fetch it by either specifying the URL for the webpage in the address bar of your browser, or by clicking on a link that will take you to this page. Information retrieval, in this sense, is client-based: you request a certain webpage to be displayed in your browser's windows. The question that arises is, How do you know which page to load when you have to choose among billions?
Explosive growth of the World Wide Web has led to the problem of information overloading
What are some ways to select webpages?
- Known websites (e.g., cnn.com)
- Recommendations (from your family, friends, coworkers, etc.)
- Links from other websites you're browsing (but this is sporadic)
- Bookmarks
- ...but most importantly, searching!
Today, to retrieve new information from the WWW, it is often necessary to search for it first.

Search engines are programs that store a large number of tagged URLs.
The tagging is done match particular webpages to certain content.
Words (or, rather, sequences of characters) that you search for are used to determine what results will be returned.
Sophisticated algorithms are used to determine in what order this particular set of results is returned.
- Older search engines, such as AltaVista, used algorithms based on the number of times a certain word was found in a webpage. This made it possible for webpage authors to "bump up" their pages in the result tables by appending lots of extraneous content to the bottom of a webpage.
- Modern search engines use algorithms based on a webpage's popularity, where "popularity" may be defined in a number of ways. Google was the first search engine to assign popularity based on the number of pages that link to the page in question.
The result of any search engine query is simply a list of URLs, perhaps augmented with snippets of content.
Most Web search engines today are free, and get their revenue from advertising.

Different types of information may be found using different tools.
Searching for images (e.g., 'image' link/tab, thumbnail searches)
Searching for news (e.g., Reuters.com)
Searching for products to buy (e.g., Froogle, eBay)
Searching for information from other users (e.g., Google Groups, electronic bulletin boards)
Craigslist: Searching for private postings
Searching for jobs (Monster.com, Careerbuilder.com)
Searching for directions (Google Maps, Yahoo Maps, MapQuest)
Searching white & yellow pages
Searching Wikis and nodes (Wikipedia, Everything2)
Searching for anything else!

Students are given an opportunity to exercise their newly improved search skills by finding several challenging pieces of information on the Internet (similar, in spirit, to questions 11-14 of Problem Set 3).

Section 4: Treasure Hunting