Data Mining, Search, and the World Wide Web
http://infolab.stanford.edu/~sergey/349/
CS 349: Data Mining, Search, and the World Wide Web
http://www-db.stanford.edu/~sergey/cs349.htmlTuesdays and Thursdays 4:15 - 5:30 in Bldg 370, Room 370 on the Main Quad
Instructors: Sergey Brin and Lawrence Page
Tues and Thurs 5:30 - 7:00 or by appointment.
sergey@cs.stanford.edu and page@cs.stanford.edu
Course Assistant: Diane Tang
Gates 416: Mon - Wed 11:15 - 12:15 or by appointment.
dtang@cs.stanford.edu
Description
Over the past two years there has been a close collaboration between the Data Mining Group (MIDAS) and the Digital Libraries Group at Stanford in the area of Web research. It has culminated in the WebBase project whose aims are to maintain a local copy of the World Wide Web (or at least a substantial portion thereof) and to use it as a research tool for information retrieval, data mining, and other applications. This has led to the development of the PageRank algorithm, the Google search engine, the DIPRE algorithm, and a number of other works which represent the cutting edge of research on the Web today (see WebBase Publications).The topics of this class are data mining and information retrieval in the context of the World Wide Web. First, we will cover background material in data mining and information retrieval that is relevant to the class. Second, we will cover recent advances made at Stanford (PageRank, DIPRE,...) and elsewhere (Kleinberg, Mitchell,...). Third and most important students will get the opportunity to work hands on with the WebBase as this will be a project class. We have already modularized a large part of the code to give people the opportunity to work with it and will continue to do so throughout the summer. Several people have already taken advantage of the code. The current WebBase repository consists of roughly 25 million web pages amounting to 150 GB of HTML.
Prerequisites
- A strong knowledge of C.
- Working knowledge of C++.
- Very basic statistics, graph theory and linear algebra.
Very Tentative Syllabus
- Introduction: 1
- 9/24 Introduction:
- 9/29 WebBase 1 (slides)
The Anatomy of a Large-Scale Hypertextual Web Search Engine
- Data Mining: 5
Publications of IBM's QUEST project- 10/1 Market Basket (slides)
R. Agrawal, T. Imielinski, A. Swami: ``Mining Associations between Sets of Items in Massive Databases'', Proc. of the ACM SIGMOD Int'l Conference on Management of Data, Washington D.C., May 1993, 207-216. PDF format. Abstract.
Dynamic Itemset Counting and Implication Rules for Market Basket Data
by Sergey Brin. Rajeev Motwani, Jeffrey D. Ullman and Shalom Tsur.
We present and algorithm for counting large itemsets faster than previous algorithms. We rely on partial results to guide the mining process.
Proceedings of the ACM SIGMOD International Conference on Management of Data, pp. 255-264, Tuscon, Arizona, May 13-15 1997. (html , postscript,gzipped ps, bibtex) - 10/6 Causality
Scalable Techniques for Mining Causal Structures by C. Silverstein, S. Brin, R. Motwani, and J. Ullman. VLDB '98.
Abstract ~ Postscript - 10/8 WebBase 2
- 10/13 Classification and Singular Value Decomposition (slides - html postscript)
SGI's MLC++ Library - 10/15 Clustering Techniques (slides - html postscript)
Berkeley Clustering Demo
- *** Project Proposals Due ***
- 10/20 Data Mining in the Real World
- Search: 3
- 10/22 Standard IR
- 10/27 New Technologies
- 10/29 Latent Semantic Indexing
Bellcore's LSI site - 11/3 WebBase 3
- *** Milestone Due ***
- 10/1 Market Basket (slides)
- Web: 6
- 11/5 Search Engines 1 - basics, size, evaluation
- 11/10 Search Engines 2 - crawling, robots.txt, ...
- 11/12 PageRank, Kleinberg
- 11/17 DIPRE
- 11/19 DEC Research
- 11/24 Classification of Web Pages
- *** Final Project Due ***
Mailing List
Sergey Brin Last modified: Sat Oct 24 23:18:37 PDT 1998