Why We Built Palantir

Celebrating the public listing of my first company: PLTR

In 2010, I gave a speech on the computer science concept of “man-machine symbiosis”. Computers are great at integrating and organizing data from thousands of sources to build a common operating picture in ways humans cannot. Humans have a comparative advantage at higher levels of abstraction: creativity, intuition, and holistic judgements. Each is necessary. The best technologies do not automate complex problems, as many assume; they equip people to solve them faster and more effectively.

In this talk, I used the example of confronting a pandemic. You can’t “automate” the fight against a viral disease. But great information technology can help health providers, epidemiologists, and governments test and trace, map PPE supplies, and manage hospital traffic while keeping health records private in the process. In a crisis, leaders demand the best solution available to inform their decisions. Palantir’s platform was the common operating system for this year’s pandemic response efforts in 35 countries.

When I started recruiting most of the first 100 people to join Palantir in 2003, the theme of using man-machine symbiosis to solve the 21st century’s hardest problems was at the forefront of our discussions. By the early 2000s, Silicon Valley’s leading technologists were lightyears ahead of IT departments in government agencies. We were obsessed with our vision of using technology to transform the information processes that power government and enterprise.

At that time, rooting out terrorists was an existential imperative. The US was spending tens of billions gathering data but many of its IT projects were years late or hundreds of millions over budget. DC had fallen behind. We resolved to start by equipping the American Intelligence Community (IC) and DoD with a new type of collaborative information platform that would protect civil liberties at home. Our software eventually allowed counter-terrorism experts and special forces to neutralize thousands of adversaries (including infamous ones) and prevent dozens of attacks on the United States.

Palantir’s intuitive visual environment for analysts to explore and monitor terrorist networks relied on extremely complicated “back-end” architecture: dynamic ontologies for organizations with different data models, databases enabling new types of audit trails, and other systems supporting new ways of monitoring structured and unstructured information at scale based on concepts deemed relevant by analysts. We built AI to integrate hundreds of types of legacy databases, each with its own language and security paradigm. We organized data to allow computers and analysts to communicate in real time according to strict protocols and made sure our platform could flexibly accommodate changing collaboration policies between organizations and countries.

Our ambition was for the American and allied IC to securely share intelligence across a distributed network to prevent the next 9/11, without sacrificing American civil liberties. We achieved our vision within the first decade.

Palantir is often lambasted as an evil panopticon or “surveillance system”, which is the height of irony because Palantir is, to this day, the most sophisticated privacy engine in the world. I blame this faulty perception partly on a company culture that rewards building technology to solve hard problems rather than PR victories. Well before conversations about data privacy became popular, we designed our platform to protect civil liberties by defining and strictly enforcing new policies, masking private data from government agents and “watching the watchers” with a variety of safeguards and techniques. In fact, we named our company “Palantir” after the cautionary tale in the Lord of the Rings about the power of “seeing stones” that later fall into the wrong hands. Subsequent revelations of government misuse of data have only strengthened our original convictions.

Shortly after winning our initial government contracts, we made our technology available to the private sector, which now accounts for about half of Palantir’s business. We continued to develop new ways of solving the most important problems for large organizations with teams of forward-deployed engineers (FDEs) who then incorporated their solutions into the core platform. Today, a couple of FDE’s can perform in days what would otherwise require months of labor for hundreds of IT service professionals.

Palantir currently helps global energy companies integrate thousands of sources of data to make key investment and trading decisions. It helps aerospace giants coordinate with hundreds of airlines and manufacturers to improve supply, maintenance, and aircraft design. Leading banks, healthcare organizations, and others solve problems with Palantir in days or weeks that would have otherwise taken hundreds of millions of IT dollars of one-off work.

Products beat services, and our goal was always to develop a flexible platform to accommodate complex decision making based on “big data”, customizing solutions with top talent as needed and integrating workflows back into the platform offerings. After releasing “Foundry”, a more comprehensive ‘productized’ platform, Palantir’s profit margins in the private and public sectors grew, and they continue to grow today.

Palantir is now one of the most powerful tools on the planet solving vital problems for governments and corporations. Our original thesis that technology should augment, not replace human beings has proven itself out hundreds of times as the world’s most important institutions have turned to Palantir to solve critical problems.

From the start, our goal was to put a top tech culture to work in the service of the US and its allies, while preserving the civil liberties that define our civilization. 17 years on, Palantir has kept our country secure and saved governments and corporations billions of dollars in the process. I’m excited to see what it does next.