Palantir

How I learned to design for data visualisation at a petabyte scale

Published 5 min read

What you probably know about Palantir is that they’re a data integration and analysis company – they build software that connects data, technologies, humans and environments. They have two main products that they offer to customers — Gotham and Foundry — for enriching, managing and analysing enterprise and quantitative data.

What you’re less likely to know is that Palantir work on a slew of custom solutions like Slate and Contour that are tailored to specific clients or problems. Many of these new projects were introduced to slowly phase out the legacy systems, replicating their features on a case-by-case basis.

For a few months in late 2015, I worked as a Product Design intern at Palantir's Palo Alto HQ. I was on an amazing team tasked with designing an anti-fraud focused pilot project. Palantir was my first purely design-focused role at a company. I can’t talk much about what I worked on but it did kickstart my design career in an interesting way — designing a platform to visualise and connect petabytes of highly secure information.

What I can share with you is how to think like a Palantir designer.

Data visualisation

Anyone can hit up a database and show a bunch of text and graphs on a screen. The real challenge is in figuring out what this data means and how humans naturally interpret it, then transforming those insights into actionable information. Data visualisation is at the core of what Palantir does.

The project I was working on built on the core layout and visualisation techniques of Gotham. The fundamental visualisation of the product was object-node graphing (called Graph View), similar to Cytoscape or Sigma but at a quantity that neither of these libraries likely support. Datasets would typically be drilled-down and isolated in the Graph View and the subset brought into a relative, time-based visualisation called Stacks.

Stacks helped show object frequency and positioning over time which could be used to find recurring patterns. The level of granularity required for most cases meant we needed a library that didn’t use a bezier curve to simplify things and allowed for dynamic scaling of time range (years, months, weeks, minutes). We had our own open-source library for this called Plottable.

turned on monitoring screen
Photo by Stephen Dawson / Unsplash

The last part was an Object Viewer — an inspector for individual or small groups objects. It showed the object’s properties with varying visualisation based on property type — string, pointer, geolocation, etc. It also listed that object’s relationships to other important items. By doing this, you were able to inspect a particular object in detail then traverse it’s relationships.

Without context, this seems like a pretty easy task. Just smash some libraries together into a React app and hook it up to a database. But what you have to remember is that we’re talking about rendering terabytes or petabytes of data in a Google Chrome window. At that point, guiding the user to isolate a subset of data becomes more important than rendering everything.

The point of this is that we built tools for analysts, and an analyst is only as effective as their tools. Rather than focusing on aesthetics from the start, just use Bootstrap or something. Get the user experience right first by giving the user what they need to do their work effectively.

Problem-focused Interface Design

At the time, Palantir had so many clients and projects that there was no time to reinvent the button stylesheet. They developed an internal (now open-source) styleguide called Blueprint to handle the common aesthetics.

This was an important step for the new web-based projects. When you remove the need for designers to craft a user interface and experience, they’re able to focus on solving problems and designing completely new components which could later become part of the ecosystem.

Of course, working on the web means we were limited as much as we were empowered. As the design team mentioned in their article about Blueprint, the platform’s interfaces used to run on Java. “When browsers evolved and new web technologies were developed (e.g., Node, Sass, ES6), suddenly we could deliver the full depth of our data analysis software right in the browser.

person working on blue and white paper on board
Photo by Alvaro Reyes / Unsplash

But browsers bring their own chaotic background with them. Our apps had to be containerised to support legacy browsers which is a concept that a lot of developers and designers tend to ignore. There’s a trend on the internet now called “Fuck IE” which involves unashamedly and ruthlessly dropping support for older versions of Internet Explorer.

Here’s the thing — if you want to design a platform that’s accessible then you need to support legacy IE. One of the unfortunate perks of the job is that you don’t get to determine who your customers are or what browser they use. I hate to use a cliché “startup vitamin” quote but think about what Steve Jobs said — “if a user is having a problem, it’s our problem”.

This isn’t limited to browser support — consider the deaf, the blind and the colourblind. There’s plenty of ways the web caters for the disabled, through ARIA tags or colour contrast checkers. There’s too many designers that are focused on reinventing the wheel and taking shortcuts where it counts.

Pushing the boundaries

The most inspiring part of working at Palantir is learning that just because something is a certain way, doesn’t mean it has to be. One of the biggest things I learned was to get out of your comfort zone when designing, consider all the possibilities and explore all of them.

It’s not just in terms of design. How many other companies can you work at where you wake up in the morning, have breakfast then spend the rest of your day helping expose financial fraud, eliminate slavery, coordinate disaster relief or discover new medicines? It’s unreal.

It was hard adopting to Palantir’s design culture initially. Australia doesn’t have many opportunities like it, so my design career started boxed-in and limited to what I was told was correct. Silicon Valley is one of the few places you can design something because you and your team think it’s right, rather than arguing over how big the call-to-action should be.

I learnt so much from the design team at Palantir, particularly in understanding the importance of design to people’s lives and the value of knowing your craft. I learnt so much there that it’s impossible to summarise in a single Medium article, so maybe I’ll post some more later. For now though, I’ve got many more products to build.

Palantir
Share this

Want my help on a cool project?

Get in touch on the Erudito contact page — my team and I would love to help you out!

View more →

Stay up to date! Get all the latest & greatest posts delivered straight to your inbox