Kristina Chodorow's Blog

Altair is a beautiful graphing library for Python. I’ve been using it a lot recently, but it was a real struggle to get started with. Here’s the guide I wish I’d had.

I’m going to be using https://colab.research.google.com/, but this should work fine in any other interactive notebook you want to use.

Getting started

First, you’re going to want to import numpy and pandas as well as altair. They’ll make working with data easier.

import altair as altimport numpy as np
import pandas as pd

To start with, we’ll generate a random dataframe and graph it using pandas. It’ll use matplotlib and look pretty ugly:

Instead, if you use altair:

Not much prettier, but it’s a start. There are several important things to note:

• There are three separate parts to creating this graph:
  1. Passing in the data you’re using (the alt.Chart call).
  2. What kind of marks you want. There are dozens of options: dots, stacks, pies, maps, etc. Line is a nice simple one to start with.
  3. What x and y should be. These should be the names of columns in your dataframe.
• From point #3 above: Altair does not understand your indexes. You have to reset_index() on your dataframe before you pass it to Altair, otherwise you can’t access the index values. (The index becomes a column named “index” above.)
• The API is designed to chain calls, each building up more graph configuration and returning a Chart object. The default behavior for showing a returned chart is displaying it.

Using this slightly more complicated configuration, you get a more attractive graph that you can do more with. However, as you try to do more with Altair, it just feels… not quite right. And it took me a while to figure out why.

Why Altair’s API feels weird

Why doesn’t Altair let you pass in a column (instead of a column name)? Why is typing and aggregation done in strings? Why is the API so weird in general?

The reason (I think) is that Altair is a thin wrapper around Vega, which is a JavaScript graphing library. Thus, if you take the code above and call to_json(), you can get the Vega config (a JSON object) for the graph:

chart = alt.Chart(df.reset_index()).mark_line().encode(
    x='index',
    y='val'
)
print(chart.to_json())

{
  "$schema": "https://vega.github.io/schema/vega-lite/v4.8.1.json",
  "config": {
    "view": {
      "continuousHeight": 300,
      "continuousWidth": 400
    }
  },
  "data": {
    "name": "data-54155f6e9cef9af445e6523406ab9d2b"
  },
  "datasets": {
    "data-54155f6e9cef9af445e6523406ab9d2b": [
      {
        "index": 0,
        "val": 0.772999594224295
      },
      {
        "index": 1,
        "val": 0.6175666167357753
      },
      {
        "index": 2,
        "val": 0.824746009472559
      },
      {
        "index": 3,
        "val": 0.23636915023034855
      },
      {
        "index": 4,
        "val": 0.730579649676023
      },
      {
        "index": 5,
        "val": 0.507522783979701
      },
      {
        "index": 6,
        "val": 0.6662601853327993
      },
      {
        "index": 7,
        "val": 0.39232102729533436
      },
      {
        "index": 8,
        "val": 0.9814526591403565
      },
      {
        "index": 9,
        "val": 0.6932117440802663
      }
    ]
  },
  "encoding": {
    "x": {
      "field": "index",
      "type": "quantitative"
    },
    "y": {
      "field": "val",
      "type": "quantitative"
    }
  },
  "mark": "line"
}

The cool thing about Vega charts is that they are self-contained, so you can copy-paste that info into the online Vega chart editor and see it.

In general, I’ve found there are slightly confusing Python equivalents to everything you can do in Vega. But sometimes I’ve run into a feature that isn’t yet supported in Python and had to drop into JS.

Lipstick on the pig

We can give everything on this chart a nice, human-readable name by passing a title to the constructor, x, and y fields:

alt.Chart(df.reset_index(), title='Spring Rainfall').mark_line().encode(
    x=alt.X('index', title='Day'),
    y=alt.Y('val', title='Inches of rainfall'),
)

You can also use custom colors and such, but the last graph I made someone asked why it was puke-colored, so that’s left as an exercise to the reader.

Poking things

The real strength of Altair, I think, is how easy it is to make interactive graphs. Ready? Add .interactive().

alt.Chart(df.reset_index(), title='Spring Rainfall').mark_line().encode(
    x=alt.X('index', title='Day'),
    y=alt.Y('val', title='Inches of rainfall'),
).interactive()

Now your graph is zoomable and scrollable.

However, you might want to give more information. In this totally made up example, suppose we wanted to show who had collected each rainwater measurement. Let’s add that info to the dataframe, first:

rangers = (
    pd.Series(['Rick', 'Scarlett', 'Boomer'])
    .sample(10, replace=True)
    .reset_index(drop=True))
df = df.assign(ranger=rangers)

Now we’ll add tooltips to our chart:

alt.Chart(df.reset_index(), title='Spring Rainfall').mark_line().encode(
    x=alt.X('index', title='Day'),
    y=alt.Y('val', title='Inches of rainfall'),
    tooltip='ranger',
).interactive()

Which results in:

Pretty nifty! Give it a try yourself in a colab or the Vega editor, and let me know what you think!

At this point, I’ve fallen so far behind of where JS developers are that I don’t think I’ll ever be able to figure out what’s going on. However, Vercel is a portfolio company of GV’s, so I decided to give it a valiant effort.

Thus, I started at vercel.com. I went through their deploy flow for a Gatsby template, linked my GitHub account, and ended up with a static webpage. This created a new Gatsby repository on my GitHub account. Unfortunately, I also have no idea how to use Gatsby. However, I’ve also been meaning to learn Gatsby, so let’s dive in.

I cloned the repository and opened up in Visual Studio Code. Unfortunately, I’m not super familiar with VS Code, either, so then I had to look up how to add the damn folder to my workspace. (The weird thing about working at Google is that I have the best tools in the world at my disposal… just not the ones anyone else in the world uses.)

One quick StackOverflow search later, I’m suspiciously inspecting index.js in VS Code. This seems to be the business end of the app, but unfortunately I’m not familiar with React nor Helmet, both of which seem to be doing some lifting here.

Usually I’ve found the best way to learn a new thing is to mess around with it, so let’s start by changing the front end. I change the h1, commit, and push.

I head to the Vercel equivalent of the github page (e.g., my repo is github.com/kchodorow/gatsby, so my Vercel dashboard for it is https://vercel.com/kchodorow/gatsby. Nice. After a second, it updates and shows my new commit as the deployed version. Very nice. It also has been emailing me about its actions each step, which is a bit much for a personal project but would be nice in general.

Okay, time to get serious. How do I actually connect Vercel to a backend? Googling around for this, it looks like I’m going to be writing serverless functions. Guess what else I’m not familiar with? However, this looks interesting. Basically I can put node.js functions in files like api/foo.ts and this becomes a server request my app can make (/api/foo). I rename date.ts to hello.ts and push it out.

Vercel displays “Build failed.” Clicking on it, It gives me the build logs:

I take a look at index.js and realize that there’s some code that calls the backend function and loads it into a variable, which I completely neglected to change. Well, that’s good, just having {hello} work would be a bit too magic for my blood (and how would nested directories in /api be specified?). I update index.js and this time, cleverly, run yarn run build before pushing.

Sigh. Fine. I install yarn. Then I run yarn. It immediately fails because I needed to run npm install first. So I install dependencies, then I run yarn. Success! A push later, a successful build, and:

Verdict: Vercel is very cool. And I feel a little less behind the curve.

See the actual code behind this paragon of frontend programming at https://github.com/kchodorow/gatsby.

There is something delightful about LaTeX. However, the last time I bothered with it was in college, since I don’t have much call for PDFs in day-to-day life. I recently came across Overleaf, which is an online LaTeX editor. The nice part is that it live-renders your work and you can right-click->Save as an PNG. Thus, you can suddenly embed gorgeously formatted math anywhere. For example, here’s one of my favorite proofs, that the square root of two is not a rational number:

Source code:

\documentclass[varwidth=true, border=10pt]{standalone}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}

\begin{document}

Suppose $\sqrt{2}$ was rational. Then we could write:

\[ \sqrt{2} = \frac{a}{b} \]

...where $a/b$ is in lowest terms. Squaring both sides yields:

\[ 2 = \frac{a^{2}}{b^{2}} \]

Now multiply both sides by $b^{2}$:

\[ 2b^{2} = a^{2} \]

$a^{2}$ must be even, since $b^{2}$ is multiplied by 2. For $a^{2}$ to be even, $a$ must be even, so we can say that $a = 2c$ for some $c$. 

Thus, we can write this equation as:

\[ 2b^{2} = (2c)^{2} \]

or:

\[ 2b^{2} = 4c^{2} \]

Now we can divide both sides by 2... but we end up with $b^{2} = 2c^{2}$, which is shaped the same as $2b^{2} = a^{2}$ above!

We can continue expanding this equation out forever, so there are no whole numbers that $a$ and $b$ can resolve to.

Thus, $\sqrt{2}$ is irrational.

\end{document}

Gorgeous.

Sorry to keep posting financial stuff, but whatever, it’s my blog.

It’s interesting how the amount of investment risk that a human can put up with is very relevant to how much they have invested, and it isn’t linear. Let’s take the case of three investors, all of whom currently can invest $1k/month and need $1M in assets to live comfortably off of assets alone. While more is more, suppose these people aren’t particularly driven to keep accumulating wealth beyond their needs ($1M).

They start with:

• Investor A: $1,000 in investments
• Investor B: $1,000,000 in investments
• Investor C: $1,000,000,000 in investments

To simplify things, we’ll say they keep 100% of their assets in stocks. Now, let’s say the market plunges by 90%: $100 invested in the market is now worth $10. What happens to each investor?

• Investor A now has $100 in investments
• Investor B now has $100,000 in investments
• Investor C now has $100,000,000 in investments

I would argue that investors A & C are in a similar boat here, ironically. Investor A started out .1% of the way towards their goal and next month, they will be back to that. Not much has changed for them: the market set them back by one month.

Conversely, it doesn’t really matter what happens to Investor C’s portfolio. They’re doing fine regardless: greater than 100% of the money they need is still greater than 100%, even if it’s less than before.

Thus, Investor B is the only one in the danger zone. They were exactly at their investment goal, and now they’re only 1/10th of the way there! Theoretically, they’re now 7 years (900 months) away from $1M!

I was reading about “bond tents” as a way to defend against stock market crashes at retirement: you don’t want a market crash right when you retire, because then you’ll sell your stocks and have no way to replenish them to take advantage of the market recovery. (This is called sequence of returns risk, which ERN does a great job explaining.) Thus, it’s a good idea to increase your bond allocation going into your retirement so you don’t have to sell any stocks if there is a crash. Bond tents might be a good mechanism for investors like Investor B, too: if you’re near your goal you have more to lose than any other time.

Not the stocks, mind you, but the market itself. There are less than 4,000 companies listed, and new companies have less and less appetite for going public. Conversely, there are over 8,000 private-equity-backed companies in the US, and growing.

There are a couple of problems with the stock market mouldering. One is that this is where most of America keeps their retirement. Common wisdom is that if you invest in a nice index fund, then your money should grow apace to the US economy. This works until the economy grows without the market.

Suppose the public market continues to dwindle. New companies stay private, so the companies on the market get older and older. Old companies have a tendency to grow more slowly and, eventually, go out-of-business. Without new blood, I’d guess the market would ossify around a few large surviving stocks.

Without the possibility of explosive growth in the public markets, any money that could go into private equity would (and is). Unfortunately, at the moment, private equity is only open to accredited investors (i.e., the already well-off) so most of America won’t be able to participate at all.

It’s important that everyone in the US:

1. Has the opportunity to benefit from economic growth and
2. Able to participate in a way where they won’t get fleeced.

As I understand it, the SEC doesn’t allow the middle nor lower class to invest in private companies because they don’t want people who are barely making ends meet to be hoodwinked out of all of their money.

Why is investing private equity riskier than public companies? Small companies tend to be riskier, but there are plenty of private companies that are larger than public ones. If I understand correctly, most of it is related public companies being better-regulated than private ones.

My knee-jerk reaction is, then: how can we get more companies to go public? And maybe that’s a good goal. However, why are we in the pocket of big-market here?

The goal is to allow more people to benefit from private companies, not bail out banks/NYSE/NASDAQ. So one possible solution is making the private market more liquid while adding protections for investors.

Making the private markets more liquid is easy: just let anyone buy and sell shares without being accredited and without needing the company’s approval.

However, given that something like 50% of VC firms don’t even return the capital they invest and (almost) everyone there is doing this professionally and on the up-and-up, making sure people aren’t fleeced seems more difficult. My strawman proposal is to tighten up regulations on private companies based on top-of-line revenue. Making $1M/year? Great, you need some sort of S-1-like prospectus for investors. Making $10M/year? Great, you can’t just spout off about “funding secured” on Twitter. Making $50M/year? Great, you basically have to follow all of the rules public companies do. I dunno, it’s a first draft. But I think it’s important the SEC gets on making private equity more accessible before the middle and lower classes are left in the dust.

There’s a lot of research on loss aversion: how bad people feel when they lose $50 vs. how good they feel good about gaining $50. This research is kind of taking an absolute value of emotion, positive or negative. We can represent this with emoji equations:

• 😀 == 😫 (large-magnitude happiness or sadness)
• 🙂 == 🙁 (small-magnitude happiness or sadness)
• 😫 > 🙂 > 😐 (ordering magnitude of emotion)

What research has shown is that, when someone loses $X and feels 😫 (large-magnitude), if they gain $X they feel 🙂 (small-magnitude). Gaining money doesn’t make them as happy as losing money makes them sad!

My hypothesis is that people have an intuitive feeling about the math behind these experiments and the results make a lot of sense if you look at percentages instead of values.

For example, suppose you invest $100 and your investment goes up to $150. Holding that money in your hand, $100 is a distant memory, when you had 33% less than you currently have. ($150 – (33%*$150) = $100)

Now let’s say your investment doesn’t do well and you’re standing there with $50, feeling sad. Now you have 50% of what you started with.

My hypothesis is that loss aversion is really an intuition about the difference between 33% and 50%. My guess is that the emoji magnitude would be (roughly) equal if you gained and lost the same percentage. E.g., these would have equal magnitudes of satisfaction:

If we extend this out to an infinite number of plans, we can see it breaks down at the ends. I don’t think losing 80% would feel the same magnitude as making 4x, but it also feels hard to imagine. Losing 80% of a meaningful amount of money would feel terrible, but would it feel as terrible as 4x-ing my money feels good? 10x-ing? 10x feels more significant than 4x, but 4x-5x? I’m not sure I’d be materially cockier.

Anyway, I assume the ratios are a bit different for different people. But it always struck me with the loss aversion studies that most people understand that going from $1 to $2 doubles your money, but going from $2 to $3 does less.