A Brief History of Prediction Markets: From Gambling to Prediction Machines

Original author: Benny Attar

Compiled by: LlamaC

"Recommendation: Mainly introduces the development history, working principles of prediction markets, and their advantages compared to traditional polls, and demonstrates the successful application of prediction markets in actual predictions through the example of the 2024 U.S. presidential election."

November 4

Today is November 4, 2024. There is only one day left until the U.S. presidential election. Polls, historical models, and experts unanimously believe that this election will be exceptionally close. Renowned political forecaster Nate Silver conducted 80,000 simulations this morning and announced that Kamala Harris narrowly defeated Donald Trump by just 12 votes, with a mere 0.0015% advantage. However, the actual results the next day showed that the eventual winner won by a landslide. But experts calculated that such a scenario was absolutely impossible! Who could have predicted all of this?

The fact is that on the same day, the two major prediction markets, Polymarket and Kalshi, both gave Donald Trump a roughly 60% chance of winning the election. The final result matched this probability, with Trump winning all seven swing states and the popular vote.

Before that, prediction markets were often regarded as gambling, dirty, and unregulated hedging products. However, their success in accurately predicting presidential elections has shown the world their true value: prediction machines. How did we get to this point? Why are they really effective?

Prediction markets are markets for real-world events. Traders can choose to bet on binary outcomes such as "Donald Trump" or "Kamala Harris." The price of each option (called a contract) is proportional to the probability of them winning at that time. For example, a $0.62 contract for Donald Trump corresponds to a 62% chance of him winning. The ultimately winning contract will receive a full payment of one dollar, while the total cost of all other contracts becomes profit. Like any market, contract prices are determined by supply and demand factors, similar to the stock market. When traders find an informational advantage in a contract, they will buy that contract, and the price will rise accordingly.

Polymarket Presidential Election Winner on November 1, 2024

Compared to traditional polls and expert predictions, the difference with prediction markets is that they can aggregate dispersed private information in real time, continuously update, and participants have economic incentives. To understand this perspective, it is necessary to know the history of prediction markets.

A Brief History

Markets and betting have been popular for centuries. Betting on papal elections is a recently re-emerged practice and is one of the earliest recorded election markets, originating in the early 16th century. Insider information leaked by informed cardinals would quickly influence and permeate betting odds, allowing merchants to hedge their economic interests. Although Pope Gregory XIV issued a decree in 1591 strongly cracking down on such gambling, the practice reappeared in the early 20th century. Since then, major bookmakers have started offering services for betting on election outcomes. Today, this practice has become entrenched—betting on real-world events has become commonplace.

Although the predictions and betting markets that existed before the advent of the internet had developed to a certain extent in terms of accuracy and influence, the rise of online trading and betting provided the necessary impetus for the growth of these markets. The emergence of digital prediction markets began in the late 1980s, initiated by the University of Iowa. After achieving early success in the academic political arena, there were several attempts at commercialization. In the early 2000s, Intrade and Cantor Exchange, two subsidiaries backed by large financial sponsors and institutions, launched prediction markets as tools for speculating on and hedging against economic and cultural events. PredictIt, another exchange that is still active today, also entered the market during this period. However, regulatory challenges ultimately forced most platforms to either shut down or lose a significant number of users. It wasn't until the 2020 election that prediction markets became more mainstream, with their popularity rising due to political interest and advancements in the cryptocurrency space. In 2022, Kalshi became the first retail prediction market platform to gain CFTC approval, marking an important step in regulatory terms.

Although prediction markets are now often seen as retail-oriented "gambling-like" products, an extension of gambling, their original purpose was entirely different. In the early 21st century, many companies utilized prediction markets to forecast internal milestones. Google launched an internal prediction market project called Prophit to achieve its mission of "organizing the world's information and making it universally accessible and useful." Executives would create markets around the objectives and key results they were concerned about, send them to their teams, and then observe the aggregated results of the employees' collective wisdom.

For example, a senior manager needs to forecast the future computing demand for the newly launched Gmail product. They pose a simple question: "How many users will Gmail have by the end of Q2?" and then send it to the engineering team, customer success team, product managers, and anyone who might have insights into the results. Those associated with the product will vote during lunch breaks or while waiting for code to compile. In this way, management can gather direct data sources from those who know the situation best and uncover hidden information that is often obscured by bureaucratic channels. A few days later, managers can see who voted for what, for what reasons, and accurately quantify the overall predictions of the company’s viewpoint. Bo Cowgill, the founder of Google’s prediction market, later explained that at the time, 10% of Google employees (about 1,500 people) participated in the market voting, and the results "outperformed other forecasts available to management." In fact, Google’s original goal was to launch the prediction market as a public product, but this vision was shelved due to concerns about financial innovation after 2008.

The accuracy of the market predictions within Google (Dan Schwarz) )

Gambling

Besides the corporate environment, prediction markets have also achieved significant success in several other areas. Notably, DARPA and the CIA have chosen to utilize prediction markets to enhance U.S. intelligence capabilities. Operators, policy experts, and agents place bets on the likelihood of future events in internal markets to prepare and quantify strategic intelligence. This approach demonstrated a highly successful outlook and strong intelligence gathering capabilities before Congress halted it, citing "terrorism gambling venues." CIA operative Puong Fei Yeh emphasized in his after-action report that Congress's decision was "premature" and that "a large amount of evidence about prediction markets indicates that they are reliable aggregators of decentralized information." Since then, with the revival of prediction markets, several government agencies have begun collaborating with the RAND think tank to use prediction markets internally.

Just like at Google, the internal market at the CIA shows potential due to its unique capabilities, allowing traders to bet on what they believe will happen, thus aggregating information from a large crowd of unrelated individuals. Compared to opinion polls, where respondents express a static viewpoint only once, prediction markets allow for Bayesian updates of probabilities based on newly acquired information or information discovered by traders. Supporters of the efficient market hypothesis believe that the more information and traders there are in the market, the closer the outcome will be to the true probability. At Google and the CIA, each respondent has unique sources of information, which, when aggregated, provide deeper insights into the predictions of the true experts in the room. As the saying goes: the wisdom of the crowd leads to the best judgment.

After achieving success with Google and the CIA, and with countless academic publications emphasizing the benefits of these markets, an increasing number of companies have begun to experiment with prediction markets. For example, Microsoft uses prediction markets to establish product development timelines and claims that it is "extremely accurate, sometimes even surprisingly so" in uncovering hidden bottlenecks in engineering schedules. Ford employs prediction markets to forecast weekly car sales, achieving a "25% reduction in mean squared error," significantly better than their expert forecasts. Eli Lilly uses prediction markets to predict new infectious disease outbreaks and track which drugs can gain FDA approval. In the case of infectious diseases, the data shows that the market can "accurately predict statewide seasonal flu activity 2-4 weeks in advance." In all these cases, executives benefiting from improved markets have mentioned that prediction markets can integrate data from a large number of members across the organization in a fast, accurate, and low-cost manner.

Predictive Machine

After achieving these successes, Polymarket and Kalshi were established in the early 2020s, aiming to bring these prediction tools to the public. As a new type of financial instrument aimed at real-world events—some kind of event derivative—these markets quickly gained attention on topics relevant to retail traders, such as geopolitics, sports event outcomes, and macroeconomic indicators. However, their influence was truly solidified on the eve of the 2024 presidential election. With Biden's sudden withdrawal and the drastic fluctuations in the news cycle following Trump's assassination attempt, investors sought some comfort from static and infrequently released polls. Prediction markets naturally became a channel for obtaining immediate, probabilistic, and generally accurate market views, helping people understand the direction of the election. In contrast, polls have a time lag, with sometimes reliable reports spaced about a week apart, while the continuous online presence of prediction markets allows anyone to instantly check the latest developments.

Moreover, in the fiercely contested and controversial realm of federal elections, polls have been found to be prone to biases and susceptible to sampling errors. Historically, Republican candidates tend to perform better than poll results suggest, as respondents often lie when answering. On the other hand, the fact that traders "bet real money" during elections helps to predict a more accurate representation of the true weighted probabilities reflecting the outcomes. When negative videos of any candidate start to go viral, the market is able to reflect this immediately.

David Rothschild from Microsoft Research made a clear distinction between prediction markets and opinion polls: "I can create an opinion poll that mimics a prediction market in every way," he said, "but prediction markets have an incentive mechanism that makes you come back at 2 AM to update your answer." In fact, traders with information advantages—often referred to as "alpha" in finance—are compelled to adjust their trades to reflect the latest expectations, thereby enhancing the market's ability to generate accurate and dynamically updated predictions.

Data shows that the election market indeed played a role. The Economist wrote on Polymarket: "Despite Kamala Harris's entry into the race generating excitement and leading her in many swing state polls, Trump's chances of winning never significantly dropped below 50%." A post on X by Polymarket showed that in all seven swing states, the market probability of the eventual winner had already exceeded 95% hours before the Associated Press announced the results. On election night at Mar-a-Lago, "everyone in the room was using Polymarket to gauge the election situation, including the President himself," explained a presidential advisor. Kalshi founder Tarek Mansour revealed that on election day, Kalshi had 500 million unique visitors—equivalent to 7% of the global population. There is no doubt that prediction markets left a profound mark on this election.

*Source: The Economist

So, why are these markets able to operate? As evidenced by data from Google, the CIA, and many other companies that have used these tools, prediction markets excel at aggregating large amounts of information and unifying that information in an equitable manner. When the thoughts of analysts, product designers, executives, and engineers are directly channeled into a market, an accurate weighted average unifies them. This leads to the next point: when someone has a stronger source of information than others, the market encourages them to bet more. In polls, all votes and opinions carry equal weight. In prediction markets, weighted bets push the market proportionately based on the information advantages held by the traders. Thus, the market tends to favor those with unique information by tilting towards them. Furthermore, as social psychologist and prediction model researcher Russ Clay explains, monetary incentives are a powerful reward for information sharing.

“It’s one thing to let ‘experts’ express their views on the economy, politics, or manned moon missions, but apart from having a certain impact on their reputation, when experts’ predictions go wrong, it usually doesn’t lead to significant consequences, and they can often provide a seemingly reasonable explanation afterward for why their predictions didn’t materialize. However, if many people could lose money when predictions go wrong, this helps to eliminate the sources of bias in everyone’s predictions, and when the numerous individual predictions made in this way are averaged, we actually obtain a fairly useful predictive tool.”

In this case, I would quote Charlie Munger's aptly stated words: "Show me the incentive and I will show you the outcome."

Paris trader Théo

In late October 2024, just weeks before the election, a Paris trader named Théo demonstrated the advantages of prediction markets over polls. At that time, all polls unanimously indicated that Kamala Harris was clearly ahead, yet Théo bet $30 million on Trump on Polymarket. He later regarded his "bet as essentially a counter-bet on the accuracy of the polling data." Explaining his winning bet to reporters, he pointed out that no polling organization could provide him with such precise judgments, nor could they react in real-time like prediction markets: survey results are often weeks apart, and the results cannot be directly input into trading algorithms. In contrast, prediction markets can turn every rumor, every data release, and every piece of gossip into continuously updating prices, which reflect and reward accurate information. This is why Théo had enough confidence to place his bet. Théo's experience—and the $50 million prize he ultimately received—demonstrates that markets are not merely substitutes for polls; for anyone needing to convert uncertainty into actionable intelligence, markets are fundamentally a superior tool. This ability to quantify uncertainty into real-time inputs rather than lagging snapshots makes markets superior to static polling data and naturally prompts us to rethink why we rarely consider them as default forecasting tools.

Polymarket election ad. Source: The Information

When reviewing the history of prediction markets, one theme has always been present: markets can always utilize private information and real-time incentives more effectively than static surveys. Initially, it was merely speculation about papal elections, but it has now evolved into a prediction engine capable of aggregating dispersed insights, continuously updating probabilities, and linking each prediction to actual monetary interests. Despite the evident superior performance of prediction markets—outpacing expert models, corporate forecasts, and all major electoral polls—they still live in the shadow of traditional polling and are limited by people's misunderstandings of gambling. If our goal is to make wiser judgments under uncertainty, then relying on the same polling cycle every four years and complaining about its failures afterward must give way to the market. It is time to change the default choice and embrace the market. By doing so, we will have the clearest perspective on insights into the future: the wisdom of the crowd.