How accurate are Zestimates? Zillow awards $1M to trio for improving them

Zillow's estimate of a home's value, called the Zestimate, can be powerful: Some homeowners track them like a stock, and when it gets to a certain point, they may decide to sell. Home shoppers gauge the estimate against the list price of a home. Others use it just to gawk at their neighbor's home values.

But it's far from perfect: In Seattle, the Zestimate is off by a median of 4.7 percent compared to the actual sale price, according to the company — a $35,000 difference on the typical house. Real estate brokers have long complained that the numbers give sellers, in particular, a distorted view of their home's true worth.

Now the Zestimate, that little number that appears at the top of every home's Zillow page and updates daily, is in line to get more accurate.

Last week, the Seattle-based company awarded a $1 million prize to the winners of a public contest to improve its algorithm. The winning team, three guys from Raleigh, Toronto and Morocco who teamed up despite never having met in person, came up with a way to beat Zillow's own data scientists to a better estimate.

The contest started a year and a half ago with 3,800 teams from 91 countries and was narrowed down to 100 finalists last year. The teams were given seven years' worth of data on a sample of millions of homes across the country and were tested to see how closely their estimated values for each home matched up with the actual sale prices of homes that sold in the ensuing months.

Jordan Meyer, the American on the winning team, reduced his workload at his day job as CTO of an analytics company and poured about six hours a day into the contest, communicating with his teammates, Moroccan computer science professor Chahhou Mohamed and Canadian artificial intelligence startup founder Nima Shahbazi, on the messaging application Slack.

Meyer started by finding every data source he could — the exact longitude and latitude of houses could be used to determine the proximity to streets and therefore determine noise near the house. Slight differences in distance from a body of water could influence a home price by thousands of dollars. In the end each home had hundreds of different data points.

"It was extremely hard," Meyer said in an interview. He called the process "relentless experimentation" and echoed Shahbazi, who said in a statement: "For every idea that worked, there were a hundred that didn't work. But we kept going."

Zillow has slowly improved its Zestimate from a median error rate of 14 percent when it started in 2006 to 5.7 percent when the contest began in mid-2017. It's now down to 4.5 percent nationally (it's higher in some cities and lower in others), and once the winners' tweaks to the algorithm are incorporated, the company expects the error rate to dip to about 4 percent.

Humans are still better than machines. Homes nationally sell on average for about 2 percent less than the list price set by brokers, according to data from Redfin. Brokers have access to information that an algorithm often doesn't — the Zestimate relies on publicly available data and voluntary input from homeowners, which can give an incomplete picture of a house.

"There are way too many factors for a certain algorithm to work," said Sam Mansour, a managing broker with John L. Scott in Lynnwood, Wash. He said he constantly has to battle with clients who cling to their Zestimate. "I've been to homes and people say, 'my Zestimate is worth X amount,' and I'm like, 'no, no.' "

Mansour said the biggest factor a computer can't track is the emotional appeal of a home, which can vary wildly from buyer to buyer and is a primary driver in how much people offer.