Field Of Internet Search Exploding With Innovation

Mercury News

It's a global battle whose foot soldiers will be engineering teams working inside a few square miles of Sunnyvale and Mountain View, with billions of dollars in advertising at stake.

Almost a decade after Google became a household name, Microsoft's launch of its Bing search engine, followed by Microsoft and Yahoo's deal to collaborate on search, could give the world's dominant Internet search engine its first serious challenge in years, as search becomes a key front in the looming competition between Google and Microsoft.

But regardless of who wins this competition, the beneficiaries are everyone who uses search engines, as quickening innovation improves the quality of information and delivers it in more useful packets. This year for the first time, a majority of the roughly 180 million U.S. adult Internet users typed a query into a search engine on a typical day, and search is gaining on e-mail as the most common online task.

Thanks to new technology, users will get their answers faster, from more than just text, and if the companies are successful, may find search engines are better at understanding what they are looking for.

"Search is going to change more in the next year than it has in the past five years," said Ben Schachter, an analyst with Broadpoint AmTech, who believes the pace of search innovation is the greatest in at least a decade.

Deluge of innovation

The pace of new features rolled out by Google, Yahoo and Microsoft has been furious in recent weeks.

At the Oct. 20-22 Web 2.0 industry conference in San Francisco, Microsoft announced a deal that allows Bing to search up-to-the-minute postings on Twitter, with much of the software engineering done at Microsoft's Mountain View campus. Google scrambled to announce its own real-time search deal with Twitter several hours later.

Not to be outdone, Google last week unveiled a new service that allows people to search for a specific song title and see a link to that song on MySpace or Lala in their search results — a service Google described as yet another way to speed users to results.

Within minutes of Google's music launch, Yahoo posted a company blog reminding that its search engine has had the ability to show links to free audio files in a partnership with Rhapsody since 2008.

Google has been unveiling so many search changes — even tweaking the size of the search box on its sacrosanct home page and pinching advertising on the results page slightly toward its center— that it has begun a "This Week in Search" item on its company blog to track new features.

Yahoo, which has been working aggressively to make sure the look and feel of its search engine remains distinct, even though its underlying results eventually will be generated by Bing, announced a new "Search Experience" in September. Among the changes: Yahoo allows users to bundle their search results from an array of topical providers they can select. A search for a restaurant would allow a user to click on a link to Yelp results; a sports search would offer bundles of results from ESPN, or a local newspaper.

"Now the real competitors have emerged, and it's mainly Google and Microsoft, with Yahoo in there because of its brand identity," said Greg Sterling, principal of Sterling Market Intelligence.

Beyond the blue links

On each of the three biggest search sites, the basic 10 blue hyperlinks that have been the essential product of an Internet search are rapidly being augmented or replaced by deeper, richer and more detailed nuggets of data — for example, a map, photos, restaurant reviews embedded in Yahoo search results for "San Jose sushi," not just the basic links to restaurant Web sites.

Since Microsoft launched Bing in June — calling it a "decision engine" for its ability to filter out unimportant information — the new search engine has gained more than 156 million monthly searches, while Google has seen a slight decline, according to comScore.

While Google and Yahoo say Bing is not driving the innovation surge, some analysts are not convinced. "I do think Bing has put some pressure on Google," Sterling said.

Microsoft, Yahoo and Google say they are innovating because people's expectations for a search engine are far higher than they were even five years ago. People no longer search for a Web site; now they expect to find a specific piece of information, like the cheapest airfare to Chicago. This represents a whole new set of challenges and opportunities in organic website optimization.

"We increasingly find that people think about search the way they think about a public utility," said Susannah Fox of the Pew Internet & American Life Project, which compiled the search data. "When you turn on a tap you expect clean water to come out, and when you do a search you expect good information to come out."

The big three search engines also search more than words. Bing offers visual searches, allowing users to browse and filter images of politicians, celebrities, album covers, or even yoga poses, as they search for information.

Clicks that count

Increasingly, a successful search is about an engine's ability to reveal a "Web of objects" — images, videos, audio files, or blog posts — rather than just a web of pages, said Larry Cornett, Yahoo's vice president for consumer products and search.

"We kicked off this huge innovation in search engines well over a year ago," Cornett said, "before anyone was doing anything else but the 10 blue links."

The new Yahoo page offers a "Search Pad" where users can note their searches. In an effort to make the results more relevant to an individual by tracking their search history, Yahoo is reading it, too.

"Every search engine looks at clicks," Cornett said. "We tried to be very open about that and say, not only is that going on, but, hey, do you want to use this for your benefit?"

At Google, speed remains king, said Johanna Wright, Google's director of product management for search.

In a recent experiment, it slowed its Web site by 0.4 seconds. The result, Wright said: People searched less.

Among the changes Google rolled out in the past three weeks are a "Jump to" link in the search results that allow a user to go directly to a keyword buried deep within a document, saving the user time, like the music search Google rolled out this week.

"Speed is something we take almost manically seriously," said Jack Menzel, group product manager for search. "We obsess about tens of milliseconds."

Microsoft, which has its Search Technology Center in Mountain View, says its philosophy boils down to helping people make a choice, sometimes limiting results when Bing decides a person knows what he is looking for.

A search for "UPS", for example, produces little on the results page but a box to enter your package tracking number, and the customer service number for UPS.

"What it amounts to is trying to build a mind-reader, to understand people," said Qi Lu, head of online services for Microsoft.

Microsoft also realized that its old search identity, called "Live Search,'' wasn't exactly hip, said Stefan Weitz, director of Bing SEO Search.

"We wanted to make sure you could use it as a verb," Weitz said. "You want people to be able to say, 'I Binged that.' "

Bing Losing Search Share

Story from Information Week

Searchers used Bing less in September than in previous months, ending three consecutive months of growth.

Microsoft (NSDQ: MSFT)'s Bing search engine lost market share for the first time since its launch in May and its Internet Explorer browser continued its long, slow slide.

A turnaround could be in the offing: Windows 7 launches on October 22, and Microsoft's considerable marketing muscle may generate some lift beyond the operating system market.

But in the calm before the ad blitz, Microsoft's search and browser power is suffering.

Bing lost 0.2 percentage points, dropping to a 3.39% share of the global search market, according to NetApplictions. Google lost a similar number of percentage points, dropping to 83.13% global search share. Yahoo (NSDQ: YHOO) lost 0.44 percentage points to end up at 6.84%.

A different metrics firm, StatCounter, said that Bing declined slightly on a global basis, from 3.58% to 3.25%, a drop mirrored by Yahoo, which fell from 4.84% to 4.37%.

Globally, StatCount saw Google (NSDQ: GOOG) gaining rather than slipping, as NetApplications did: Google's global search share increased to 90.54% in September, from 89.57% in August, according to StatCounter's yardstick.

StatCounter's figures also show Google US search share rising to 80.08% in September from 77.83% in August.

Bing's share of the US search market in September dropped to 8.51% from 9.64% in August, according to StatCounter. Yahoo's US search share during this period dropped from 10.5% to 9.4%.

With regard to Web browser usage, Internet Explorer went from 58.69% in August to 58.37% in September, as StatCounter reports. According to NetApplications, Internet Explorer's decline went from 66.97% to 65.71%.

Firefox reached a global market share of 23.75%, according to NetApplications, a level just below its 23.84% high-water mark in April. Google's Chrome browser and Apple's Safari browser also posted gains to end up at 3.17% and 4.24% respectively.

As StatCounter sees it, Chrome actually passed Safari in August and increased its lead in September. Chrome finished the month with global market share of 3.69% while Safari came in at 3.28%.

Hunch: And Now For Something Completely (Okay, Somewhat) Different
After the failure of several community Q&A sites, Caterina Fake - co-founder of Flickr - has launched Hunch. Will it catch on, or go the way of Jeeves?
Story from Business Week

When Caterina Fake was picking out a new bag for her laptop computer, she didn't turn to friends, browse store aisles, or page through magazines for advice. Instead, she consulted her new Web site, Hunch. In response, the tool asked her a series of questions, including "How much are you planning to spend?" and "Would you prefer leather?", before dishing out three recommendations.

Opened to the public on June 15 after a three-month preview for select users, Hunch is a tool for finding answers to a wide variety of questions—from mundane shopping decisions to dilemmas as serious as "Should I get a divorce?" The questions and answers are created by users, and the site uses feedback from the community to refine the relevance of results.

Hunch is the latest of several decision-making sites that have cropped up in recent years that are designed to shake up a Web-search landscape that has long been dominated by keyword queries. The sites, including Answers.com (ANSW), Mahalo, Aardvark, ChaCha, and Yahoo Answers (YHOO), aim to do what Google and other keyword-based search engines have trouble doing: delivering results tailored to specific human situations and problems without sending users to many different sites on the Internet. "Google can't provide all the answers," says Danny Sullivan, editor-in-chief of Web site Search Engine Land.

Hunch co-founder Fake, who also co-founded image-sharing site Flickr and sold it to Yahoo! in 2005, isn't out to topple Google (GOOG). But she does contend that in an online world where people frequently contribute edits to Wikipedia and thumbs-up stories on Digg, the power of online crowds has yet to be fully harnessed to help people make better everyday decisions. "One little action is actually fairly insignificant in the grand scheme of things, but in the aggregate, these things together create something that's incredibly valuable," she says.
Fake: Flickr, Yahoo Answers, and now Hunch

Fake burnished her search skills while still at Yahoo after the Flickr deal. She helped to create one of the most successful question-and-answer sites, Yahoo Answers. Launched in 2005, each month the site attracts tens of millions of visitors who pose questions and get rewarded with recognition for providing the most popular answers to others' questions. In fact, the site gets many of its visitors from traditional search engines like Google, since Yahoo Answers turns up in results when people search for questions like: "Does chocolate spoil?" Fake left Yahoo last year.

Hunch arrives at a response after asking users about five to 10 questions. Going a step beyond Yahoo's community-voting model, Hunch determines the best response for each individual user based on their past clicks. The site has voluntary survey questions on its front page, in the section "teach Hunch about you," where it asks simple, fun questions and then factors in those preferences when it's advising you on a decision.

Answer engines have a poor track record of late. In May, Microsoft (MSFT) closed its MSN QnA, a service similar to Yahoo Answers. In March, Wikipedia founder Jimmy Wales announced he was closing Wikia Search, a short-lived project intended to create user-influenced results to keyword queries. Google once offered a collaborative search site called Google Answers, where people could offer money rewards for thoroughly researched responses to queries. The company axed the site in 2006, telling users "the Answers community's limited size and other product considerations made it more effective for us to focus our efforts on other ways to help our users find information."

A large base of active users is essential for making these types of sites useful, says Brady Forrest, a technical evangelist at O'Reilly Media who previously worked on the search team at Microsoft and helped develop the prototype of MSN QnA. "You have to constantly be able to pull in a certain crop of answerers who will invest the time to go in and talk about whether you should buy this laptop or that laptop and keep it updated," he says.
"they might get more valuable advertisers"

Profits also hinge on these sites reaching a wide audience. "Many of these services are ad-based, which is problematic, says Greg Sterling, founding principal at technology market researcher Sterling Market Intelligence. '"In order to make that work, you have to have a massive audience.""

Could answer services one day be as lucrative as the search business? Eschewing banner or text ads, Hunch plans to get all of its revenue from fees it receives by sending users to shopping sites like Amazon.com (AMZN). So while it may profit from only a small portion of the questions on its site, search experts say these links could be very profitable since people are already coming to the site with the intention to buy something. "Potentially, they might get more valuable advertisers," says Search Engine Land's Sullivan.

Sullivan says Google and other keyword search engines may lose a small portion of users to decision sites like Hunch, but generally he thinks the category is creating new searches that weren't done before at all. "They enlarge the search pie overall," he says, adding that when video started appearing on the Web, it prompted a similar boom in new types of searches that didn't exist before.

That isn't stopping traditional search engines from trying to compete with these upstarts by trying to offer smarter searches. In May, Microsoft unveiled Bing, a new incarnation of its Web search that's billed as the "decision engine." The site includes information such as average plane ticket prices for a search of "Chicago to London" right on the first page of search results.

Microsoft Banking On Former Yahoo! Wizard To Compete With Google
Story from Business Week

Qi Lu has had more than his fill of losing to Google. For a decade, the technologist led development of Yahoo!'s Internet search technology and watched Google eclipse his company to become the Internet's brightest star. He left Yahoo in August with vague plans to start a company or return to his native China. Then Microsoft CEO Steve Ballmer came calling. He wanted Lu to consider taking over Microsoft's online operations and lead the charge against Google one more time. Ballmer promised the company was willing to invest vast resources to compete, even if it took years to pay off. "The more I thought about it, the more it seemed like a duty," says Lu, during an interview at the software giant's Redmond (Wash.) headquarters on a balmy May day. "There's a chance—a genuine chance—that we can make the search landscape a whole lot more competitive and healthy."

He may be one of the few people who believe that. But Microsoft is giving Lu more backup than he's ever had before. On May 28, the company was set to unveil an ambitious new search offering called Bing. Instead of just finding promising Web links, the site is designed to help consumers more easily make complex decisions—like what car to buy or where to go on vacation. The goal is to create a loyal base of fans who routinely use Bing for certain types of queries, rather than default to Google. To support the launch, Microsoft is gearing up its biggest search marketing blitz ever, one that could run as much as $100 million. "We're going [to show] consumers that the two guys that really care about helping you navigate the Internet are us and Google," says Ballmer "This will be the first time we will step out and say, 'We're not just sort of a general online player. We're really a player in search.' "

The stakes are high. Microsoft was slow to recognize the importance of search, only starting to build its own technology in 2003. Since then, the effort has contributed to barrels of red ink in the company's Internet business, including more than $3.5 billion in losses in the past three years. Yet Microsoft has only lost ground with its Live Search, dropping to just 8% of U.S. searches, while Google has grown to 64%. Microsoft has missed out on billions in potential revenues that might have goosed its stock, which is stuck at the same level it hit in 1998. Worse, Google is using its dominance in search to attack Microsoft's most lucrative businesses—including its Windows operating system and Office suite of business software.

Most search experts believe Lu (whose name is pronounced CHEE-loo) will struggle to do much better against Google this time. The search kingpin is refining its own technology all the time, and the data it gathers from handling almost two-thirds of people's queries give it deep insight into how they react to various search alternatives. "Microsoft is making some nice changes, but [there are] no game changers," says Danny Sullivan, editor in chief of the blog Search Engine Land who has been briefed on Microsoft's new site. "I still don't think Microsoft fully realizes how far behind Google they are."

Lu and other Microsoft executives argue they have an opening few experts see. The company's extensive research has turned up a surprising vulnerability at Google: While Web surfers may say they're happy with search technology, the data show they don't find what they're after almost half the time. Microsoft has designed the new search offering to remove the roadblocks. One example: Microsoft researchers found that 25% of clicks on search pages involve going back to the previous page, suggesting a frustrated search. So Microsoft developed a feature to avoid the wasted effort: When users hover over a Web link without clicking, Microsoft's computers generate a pop-up summary of the link.

Lu and his team have also designed a pane on the left third of the search page that generates a "table of contents" for each search. Entering "U2" brings up categories such as "songs," "tickets," and "biography," while a search for "Honda Accord" offers to lead you to "used," "reviews," and "specs." "For anything beyond finding a Web site—say, finding a person, buying a product, finding a relationship—today's search experience is not compelling," says Lu.

The marketing blitz will hit a similar note. In one TV spot, Microsoft will poke fun at Google by comparing its search to a bad relationship where your significant other takes too long to respond to questions and then gives the wrong answers three out of four times. Microsoft has also spent hundreds of millions on distribution deals that will make Bing the default search engine on Hewlett-Packard (HPQ) and Dell (DELL) PCs and Verizon phones.

LOVING THE 19-HOUR DAY

Google is certainly paying attention. The company has been adding a number of new features to its own search engine in recent months. And at a press event on May 12, Google unveiled an option to open a new left-pane feature that resembles Microsoft's technology. Marissa Mayer, vice-president for search products and user experience, declined to comment on Microsoft specifically, but says, "Search is really in its infancy. We're just really getting started."

Lu, 47, has been as involved as anyone in the technology's history. After a brief stint at IBM, he helped meld three Yahoo acquisitions to launch its first search offering in the late 1990s and later oversaw development of the technology that lets Yahoo make money by placing ads alongside search results. Along the way, he earned a reputation for having both technical chops and relentless work habits. He wakes at 3:00 a.m. most days, takes a five-mile run, and often works until 10:00 p.m. "It doesn't feel long because I love every day," says Lu, who is married with two children.

These qualities made him something of an institution at Yahoo, where he constantly pressed for management to pour more dollars into building the technology infrastructure necessary to keep pace in search. In the end, sources say he lost faith in the company's ability to do so, and left. Former Yahoo engineer Amit Kumar says Lu was "universally well liked" and at his going-away party T-shirts were handed out that read: "I worked with Qi. Did you?"

A FATEFUL ENCOUNTER

Lu has faced tough challenges since he was a boy. Facing persecution during China's Cultural Revolution, his parents sent him from their Shanghai home to live with his grandfather in a tiny town in Jiangsu province, five hours away. Lu lived without electricity or plumbing, and was so poor that meat was a once-a-year luxury. His first two choices to escape poverty were closed off: His slight build left him short of government weight mandates for coveted ship-building jobs, and his eyesight was too poor to pass requirements for becoming a physicist.

That left computer science, which he hoped might help him land a job in a radio factory. Instead, after earning his master's degree, he was assigned to a $10 a month teaching job at Fudan University in Shanghai. One weekend a rainstorm prevented his weekly bike ride home to see his parents, so he was in his dorm room when a student knocked and pleaded with him to attend a talk by Carnegie Mellon professor Edmund M. Clark since only a few students had showed up. Impressed with Lu's questions, Clark asked to see his research papers and then offered him a scholarship to earn his PhD—even waiving the $45 application fee that Lu says he could never have come up with.

Even fans question whether he has the business acumen to be Microsoft's savior. One former Yahoo executive thinks Lu's main allure to Microsoft is that he'd be the perfect person to integrate Yahoo's search operation if Ballmer ever manages to gain control of the business, which Microsoft bid for last year. Ballmer disputes this. "Qi is here because he's absolutely the best guy on the planet to run a search business."

When Lu arrived in December as president of Microsoft's Online Services Division, he inherited a division that began planning its search offensive in mid-2007. That's when Ballmer asked Susan Athey, a young Harvard economist who studies auctions, to help rethink the search effort. In an initial session in Redmond, Athey quelled fears that there may not be room for a second player in the business, but only if Microsoft got much larger and learned to innovate much faster. Just buying Yahoo, as Ballmer was trying to do, wouldn't be enough. Later hired as Microsoft's chief economist, Athey led the research effort that uncovered consumers' frequent troubles with search. "There's no reason Microsoft couldn't catch Google," she says.

Since acccepting, Lu's priorities are to set long-term strategy while tightening up operations. One of his mantras: "Have your head above the clouds but your feet on the ground." He arrives at meetings with stacks of documents, many with notes jotted down in the margins, and requires that a summary be written up afterwards. He is also pushing through changes so that the search group can forecast revenues on a daily rather than monthly basis, to react more quickly to what's working.

Still, he thinks Microsoft's real strength is its willingness to lay out a multi-year plan to gain on Google. Lu's group is working more closely with Microsoft Research so new technologies can be integrated into search. Soon, people with mobile phones will be able to speak search terms, rather than type them. Eventually, Lu says those looking for answers will be able to push beyond the limits of typing a few words in a rectangular box. "This is just the first step in a long journey," he says.

Low Value Personal Information Sites Clogging Search Engines

As posted by: Wall Street Journal

In the first week of January, New York graphic designer Nicholas Felton will boil down everything he did in 2008 into charts, graphs, maps and lists.

The 2007 edition of his yearly retrospective notes that he received 13 postcards, lost six games of pool and read 4,736 book pages. He tracked every New York street he walked and sorted the 632 beers he consumed by country of origin.

Part experimentation, part self-help, such "personal informatics" projects, as they are known, are gathering steam thanks to people like Mr. Felton who find meaning in the mundane. At their disposal are a host of virtual tools to help them become their own forensic accountants, including Web sites such as Dopplr, which allows people to manage and share travel itineraries, and Mon.thly.Info, for tracking menstrual cycles. Parents can document infant feeding schedules with Trixie Tracker. And couples can go from between the sheets to spreadsheets with Bedpost, which helps users keep track of their amorous activities.

The objective for Mr. Felton and others is to seize data back from the statisticians and the scientists and incorporate it into our daily lives. Everyone creates data -- every smile, conversation and car ride is a potential datapoint. These quotidan aggregators believe that the compilation of our daily activities can reveal the secret patterns that govern the way we live. For students of personal informatics, the practice is liberating because it shows that our lives aren't random, and are more orderly than some might expect.

Mr. Felton calls his compilation the Feltron Annual Report; the slight alteration of his name connotes the mechanical nature of his autobiographical cataloging effort, now entering its fourth year. He plans to continue his project over the next decade in what he hopes will result in a modern-day spin on James Boswell's famously detailed biography of Samuel Johnson. "I want to create connections where I didn't know that they existed," Mr. Felton says. "I'm a natural annotator."

The elegantly graphical reports, as much design projects as they are data compilations, are posted online by Mr. Felton. He also creates hard-copy limited editions, available free of charge. They have become so popular that he recently launched a Web site with his friend Ryan Case called Daytum, which helps fellow chroniclers track the details of their own experiences.

The culture of sharing information online has shifted in recent years, from a focus on blog ramblings to the ubiquitous micro-movements of posters' daily lives. Microblogging sites like Twitter have become commonplace. President-elect Barack Obama, for example, had his own Twitter account and used it to keep his supporters up to date on his campaign's daily comings and goings. (It's been silent since the election.) Facebook's News Feed feature initially drew criticism from members because it offered a running log of users' minute postings and updates, but has since became a core part of the Web site's community. Some sites collect data automatically for their users. Last.fm keeps a record of all of the songs users have listened to, and Netflix keeps track of members' movie-watching habits.

"It's a natural progression from people sharing things like movies, photos and videos," says Dennis Crowley, founder of Dodgeball, an early social-networking service for mobile phones which was sold to Google in 2005. "What's left to share? Basic data."

Yannick Assogba, a graduate student at MIT's Media Lab, created a site called Mycrocosm to help users compile and share the "minutiae of daily life" in the form of multicolored bar charts and pie charts. Mr. Assogba, for example, tracks his ping-pong winning streaks and what days he spends the most money. Created in August, Mycrocosm now has 1,300 registered users. "We're living in an era of data," Mr. Assogba says.

Today's info-chroniclers are just the latest in a long history of diarists and scientists who kept notes by hand. Nineteenth-century English inventor and statistician Francis Galton, who introduced statistical concepts such as regression to the mean, was an obsessive counter who created the first weather map and carried a homemade object called a "registrator" to, among other things, measure people's yawns and fidgets during his talks. (Mr. Galton's preoccupation with data, specifically with human hereditary traits, also yielded an unsavory by-product -- eugenics.)

In 1937, a social research organization called Mass Observation in London used about 2,000 volunteers to develop an "anthropology of ourselves." For more than a decade, participants recorded such things as their neighbor's bathroom habits and what end of their cigarettes they tapped before lighting up. Personal tracking also showed up in "Cheaper by the Dozen," a 1948 book about efficiency experts Frank Bunker Gilbreth and Lillian Moller Gilbreth and their attempts to track and optimize the daily routines of their 12 children (including when they brushed their teeth and made their beds).

Several technological shifts in the last decade have helped turn personal informatics into a mainstream pursuit. The iPhone, for example, has several applications such as Loopt that use the product's internal global positioning system to record a user's location and then share it with others. Low-cost products such as Wattson, an energy monitor that tracks real-time power consumption, make it easy to record otherwise nebulous data.

To help women prepare for their period or try to get pregnant, Chicago Web designer Heather Rivers created Mon.thly.Info, a site that sends alerts and tracks users' menstrual cycles. Ms. Rivers says her interest was purely practical; it's the only data about herself that the University of Chicago student records. "I'm not interested in biorhythms for the sake of being interested. It's just helpful in terms of throwing tampons in your backpack. This is one of the details I'd rather not worry about," Ms. Rivers says. "It's not so I can go back and fondly reminisce about my past periods."

Some of the new data collectors hope to make better decisions about their activities and improve their quality of life. For the last four months, Alexandra Carmichael, the founder of a health research Web site called CureTogether in San Francisco, has been tracking more than 40 different categories of information about her health and personal habits. In addition to her daily caloric intake, her morning weight and the type and duration of exercise she performs, she also tracks her daily mood, noting descriptions such as "happiness" and "feeling fat."

From her initial readings, she concluded that her mood went up when she exercised and went down when she ate too much. "I realized my relationship with food is a distorted, unhealthy one," Ms Carmichael says. She has concluded that she may have an eating disorder and has decided to seek counseling.

Andy Stanford-Clark, an inventor for IBM, began tracking the power usage of his 16th-century thatched cottage on the Isle of Wight in an unusual way. Everything in his house, from his phone to his doorbell, is hooked up to automated sensors. Each time water is used, or a light goes on or off, it's catalogued publicly on Twitter for all to see, along with the total household water and electricity consumption. Mr. Stanford-Clark says he now tries harder to conserve power. "I just couldn't believe how much money that was wasting," he says.

Keeping track of personal data online can yield unexpected consequences. "Initially, it sounds like a great idea, such as the social aspects," says Christopher Soghoian, a fellow at Harvard University's Berkman Center for Internet and Society. But "for most users, the costs outweigh the benefits," he says. Specifically, Mr. Soghoian points to the legal concept called the "third-party doctrine" which eliminates the right to privacy for users who voluntarily place their information on Web sites. "If you're cataloging every movement, that might come up if you get divorced," he say.

Private investigators and the federal government could also use such information in some circumstances. In the application for jobs with Mr. Obama's administration, applicants are asked to list all of the social networks that they are involved in and to supply any potentially problematic blog posts from their online past. "All this stuff is creating a huge digital paper trail that could come back and haunt you," says Mr. Soghoian.

Personal data collection can get in the way of living, some people admit. "It becomes an obsession," says Toli Galanis, an aspiring filmmaker in New York who tracks everything from his mercury levels to his vitamin D consumption. He says that he's had to forgo outings with friends when he's trying a new diet that requires scheduled mealtimes, and elicits strange looks from his parents when he measures his dinner food to the ounce.

Still, he adds, "Life and its goals are like a lab. Why not use it like a scientist? Then you'll really know what you want to. There's so much info that it'd be a shame not to track it."