Data Mining on the Web

When visitors interact with your site, they provide information about themselves and how they respond to your content: which links visitors click, where they spend most of their time, which search terms they use, and when they browse. Some visitors may even fill out a lifestyle survey or provide names and addresses. Complex content also contains important information, such as words in articles, job descriptions and resumes, and features of competitive or complementary products. All this information is often stored in a database.

As a result, you have a lot of information on your Web visitors and content, but you probably aren't making the best use of it. Data warehouse reporting systems, such as those provided by traffic analyzers, aggregate and report facts over different dimensions. (See my article titled "Tracking Users," Web Techniques, July 1999.)

These warehouse reporting systems are commonly called online analytic processing (OLAP) systems. OLAP systems can report only on directly observed and easily correlated information. They rely on you to discover patterns and decide what to do with them. OLAP systems won't tell you that people frequently buy potato chips, onion soup mix, and sour cream at the same time, and they won't discover that some people love any movie that contains an explosion. The information is even too complex for humans to discover these patterns using an OLAP system.

To solve this problem, marketers and business analysts use data-mining techniques. These are machine learning algorithms that find buried patterns in databases, and report or act on those findings. There are many data-mining techniques, and it's difficult for one person to understand the entire field. The best we can do in one article is provide an introduction to the problems that data-mining techniques can solve, mention the techniques usually applied to those problems, and give some insight into vendors offering solutions.

Know Your Visitor

To use data mining on your Web site, you have to establish and record visitor and item characteristics, and visitor interactions.

Visitor characteristics include demographics, psychographics, and technographics. Demographics are tangible attributes such as home address, income, purchasing responsibility, or recreational equipment ownership. Psychographics are personality types that might be revealed in a psychological survey, such as highly protective feelings toward children (commonly called "gatekeeper moms"), impulse-buying tendencies, early technology interest, and so on. Technographics are attributes of the visitor's system, such as operating system, browser, domain, and modem speed. If you have a phone number or address, you can sometimes obtain household demographic or psychographic information through direct marketing service providers, such as Webcraft or Acxiom. Business demographics are available through Dun & Bradstreet.

Item characteristics include Web content information media type, content category, URL as well as product information -- SKU (stock-keeping unit, basically a product number), product category, color, size, price, margin, available quantities, promotion level, and so on.

Visitor statistics accumulate when visitors interact with items, the Web site, or the company. Visitor-item interactions include purchase history, advertising history, and preference information. Purchase history is a list of products and purchase dates. Advertising history indicates which items were shown to a visitor. Preference information refers to item ratings provided by a visitor. Click-stream information is a history of hyperlinks that a visitor has clicked on. Link opportunities are hyperlinks that have been presented to a visitor.

Visitor-site statistics are typically per-session characteristics, such as total time, pages viewed, revenue, and profit per session with a visitor. Visitor-company information might include total number of customer referrals from a visitor, total profit, total page views, number of visits per month, last visit, and so on. Visitor-company information can include brand measurements. Brand associations, for example, are lists of positive or negative concepts a visitor associates with the brand, which can be measured by surveying visitors periodically. Permissions are attributes that a visitor provides indicating how marketing information contributed by the visitor can be used, such as permission to send email, to share information with marketing partners, and so on.

If you do nothing else in response to this article, I urge you to do two things: First, decide how you might use information recorded about your site's visitors, write a privacy statement, and make that statement available on your Web site. See http://www.truste.org/ for assistance. Think about privacy from the visitor's point of view. Visitors prefer to view products and pages that interest them, so they usually share information for that purpose. However, they typically want you to ask for permission before sending them marketing email, or sharing their contact information with partner companies. If you provide a privacy statement documenting your intended uses, and give visitors an email address for comments, your visitors will let you know whether the policy is acceptable.

Second, record the data now, even if you do not have a data-mining process in place. You will find most data-mining tool vendors allow for an initialization step in which they incorporate historical data into your data-mining system.

List Your Goals

The great advantage of Web marketing is that you can measure visitor interactions more effectively than in brick-and-mortar stores or direct mail. Data mining works best when you have clear, measurable goals. The following are some goals you might consider:

Increase average page views per session;

Increase average profit per checkout;

Decrease products returned;

Increase number of referred customers;

Increase brand awareness;

Increase retention rate (such as number of visitors that have returned within 30 days);

Reduce clicks-to-close (average page views to accomplish a purchase or obtain desired information);

Increase conversion rate (checkouts per visit).

If you've instrumented your site to record the visitor, content, and interaction characteristics, and you've determined a set of measurable marketing goals, congratulations! You are farther along than most marketers. Now you can gain value from data mining.

Understand Your Problem

The first step to solving a problem is articulating the problem clearly. Common problems Web marketers want to solve are how to target advertisements, personalize Web pages, create Web pages that show products often bought together, classify articles automatically, characterize groups of similar visitors, estimate missing data, and predict future behavior. All involve discovering and leveraging different kinds of hidden patterns.

Targeting. Marketers use targeting to select the people receiving a fixed advertisement, to increase profit, brand recognition, or other measurable outcome. Targeting on the Web must account for different advertising ad space costs. Web sites with valuable visitors typically charge more for ad space.

On sites where visitors register, advertisers can target on the basis of demographics. For example, people living in different parts of the country or visiting different Web sites may have differing propensities to purchase sports-team-branded apparel, gay travel tours, or discount car parts. Therefore, if you target the people most likely to purchase your product, you can reduce your cost for an ad campaign and increase the total profit.

Some sites let you target ads on the basis of IP address, under the theory that DNS registration information or surveys provide the physical location of the IP address. However, because national dial-up ISPs often share a pool of IP addresses, this is not a reliable method. As we say in the business, "Half the U.S. population lives in Vienna, Virginia" (AOL's corporate address).

Data mining can help you select the targeting criteria for an ad campaign. Web publications have a set of variables by which they can target advertisements. By performing a test ad using "run-of-site" (that is, untargeted) ad space you can associate demographic variables with conversion. People "convert" when they accomplish the marketing goal, such as performing a click-through, purchase, registration, and so on. Data mining can identify the combination of criteria that maximizes the profit. For example, data mining might discover that targeting based on the logical expression (java-consultant) or (software-engineer and purchasing-authority < 10,000) will increase the click-through on a JavaBean banner ad.

There is a huge variety of data-mining tools that support targeting, because targeting is extensively used in direct mail marketing.

Personalization. Marketers use personalization to select the advertisements to send to a person, to maximize some measurable outcome. Here we use "advertisement" loosely to refer to any recommendation or item offered by a site. Even a simple hyperlink in a menu or an article could be considered an advertisement.

Personalization is the converse of targeting. Targeting optimizes the types of people that will see an advertisement, reducing cost by showing the advertisement to more people in a broader campaign. It is most useful for prospecting -- finding people who haven't visited your site yet -- because there's a cost to advertising on outside Web sites. But targeting is pointless on your own site, where advertisements are free. Why would you not show your products to a person visiting your own site?

In contrast, personalization optimizes the advertisements that a person sees, raising revenue because the person sees more interesting stuff. Personalization can be used for external advertising, but you're more likely to use it on your own site. External sites don't usually give you enough information about individual visitors to do good personalization.

Some personalization systems, such as Broadvision One-to-One, rely on the marketer to write rules for tailoring advertisements to visitors. These are "rules-based personalization systems." If you have historical information, you can buy data-mining tools from a third party to generate the rules. Rules-based personalization systems are usually deployed in situations where there are limited products or services offered, such as insurance and financial institutions, where human marketers can write a small number of rules and walk away.

Other personalization systems, such as Andromedia LikeMinds, emphasize automatic realtime selection of items to be offered or suggested. Systems that use the idea that "people like you make good predictors for what you will do" are called "collaborative filters." These systems are usually deployed in situations where there are many items offered, such as clothing, entertainment, office supplies, and consumer goods. Human marketers go insane trying to determine what to offer to whom, when there are thousands of items to offer. As a result automatic systems are usually more effective in these environments. Personalizing from large inventories is complex, unintuitive, and requires processing huge amounts of data.

Association. Also called market-basket analysis, association identifies items that are likely to be purchased or viewed in the same session. If you place references to these items together on the same page in a Web catalog, you may remind your visitor to purchase or view something otherwise forgotten. If you hold a promotion on one item in an association group, you're likely to increase purchases of other items in that group.

Association can be deployed in situations even where you have static catalog pages. In this case, you rely on the visitor to select the first catalog page to view, and then serve up related items as cross-sells. Association is the data-mining solution Amazon uses when it says, "Customers who bought The Grapes of Wrath also bought The Great Gatsby."

Clustering. Sometimes called segmentation, clustering identifies people who share common characteristics, and averages those characteristics to form a "characteristic vector" or "centroid." Clustering systems usually let you specify how many clusters to identify within a group of profiles, and then try to find the set of clusters that best represents the most profiles.

Clustering is used directly by some vendors to provide reports on general characteristics of different visitor groups. These techniques require training, and suffer from drift on Web sites with dynamic Web pages. (Again, see the article "Tracking Users," Web Techniques, July 1999.)

Estimation and Prediction. Estimation guesses an unknown value, such as income, when you know other things about a person. Prediction guesses a future value, such as the probability of buying a car next year, when a person hasn't done it yet, or the expected number of stocks that a person will trade in the coming year. The same algorithms can perform estimation and prediction.

Estimation is often used in demographics to fill in the blanks. If you don't know what income a person has, an estimator can identify other variables that correlate well with income -- such as location, car preference, job title -- then find other people with similar traits and use them to estimate income and confidence value.

Prediction can compute important future attributes of a person -- such as lifetime monetary value, next visit interval, learning speed, promotion susceptibility, and so on -- based on the same approach. These values can be used in personalization applications.

Marketers often aggregate information to understand groups of customers. Even adding up or averaging past events over different dimensions -- such as visitor category, content category, referrer, and time -- can provide useful information. This simple aggregation is called OLAP, online analytic processing: online because the marketer uses an online reporting engine to interactively move through the data; analytic because the marketer is passively looking through past data, not trying to change it.

Prediction can be applied in combination with OLAP techniques to generalize properties of groups of people visiting a Web site. This can help a marketer to slice and dice the data to find which item attributes or site characteristics appeal to the most valuable customers.