Algorithmic progress: In the coming decade, AI labs will invest tens of billions in algorithmic R&D, and all the smartest people in the world will be working on this; from tiny efficiencies to new paradigms, we'll be picking lots of the low-hanging fruit. We probably won't reach any sort of hard limit (though “unhobblings” are likely finite), but at the very least the pace of improvements should slow down, as the rapid growth (in $ and human capital investments) necessarily slows down (e.g., most of the smart STEM talent will already be working on AI). (That said, this is the most uncertain to predict, and the source of most of the uncertainty on the OOMs in the 2030s on the plot above.)

I recently read Ashenbrenner's several hundred page missive about the coming inevitability of AGI filled with charts that go up and to the right, and my only thought was: “go f**** yourself”. But then I realized that if he spent his time carefully interviewing people at OpenAI and writing hundreds of pages to lay out his argument, I should at least throw together one poorly researched page to rebut it.

My biggest issue is with his statement “most of the smart STEM talent will already be working on AI”. Wrong. Most people just do what the market has support for, and the market has limited support for people getting paid >300k a year to hunt for algorithmic inefficiencies on which they WILL NOT MAKE MONEY. These are not high frequency algorithmic trading systems. The major returns from foundation model companies to this point have been from sale to large FAANG companies looking to bolster their own internal capabilities, NONE OF WHICH MAKE MEANINGFUL MONEY ON GENAI. “GENAI” does not currently make any money for anyone other than the Cloud Providers (OpenAI, etc.), the top 2-3 chat providers (ChatGPT, Claude, Characters, etc.), and the top 2-3 image generators (ChatGPT / Imagen, Adobe, StableDiffusion, etc.) - “AI” as a business enabler is far more than GENAI methods including large scale recommendation algorithms (Amazon, TikTok, Netflix, Youtube), RL systems (market makers, ad pricing systems ), and traditional “ML” / robotics systems (warehouses, drone systems, construction, logistics) to improve core business processes. All of Meta and Google's revenue comes from Ads. The core use of “AI” in those businesses is to improve their profit centers. Google's / Microsoft / Amazon's revenue from GENAI is a drop in the bucket on their P&L, and should really be viewed as another line in the cloud services business.

I smell a grift. Let's look at base incentives. Forget the rhetoric for a second. Pretend you're an LLM / “AI” researcher - one of the real ones from the top CS programs (Stanford, Berkeley, MIT, CMU, Princeton, Oxford, etc). You likely started your academic career after AlexNet in 2012, and finished your doctorate sometime 2014-2024. If you finished your Ph.D. before that, you could easily be an esteemed professor who did the hard work of defining the field, and can still probably make money lending your name as an advisor to an AI company or a dual appointment at a tech company, Let's say this pool of extremely elite people is ~5000 people (10 schools, 50 Phds awarded in relevant research areas, 10 years), but we all know that's generous. There's actually like ~1000 people who really define the field. We don't have to pretend someone with a Ph.D. in language theory, or geometric computer vision, distributed system, or hell, even cryptography, has made any real contribution to the LLM specific arms race we're in now. No. It's guys like Sutskever, Bengio, Shamir, Hassabis and a few other people who either wrote the core research or joined OpenAI / Google Research / DeepMind early enough to catch the wave. Cool, great. These people, by the way are world-class academics, elite organization builders, and highly proficient engineers. Now hand them billions of dollars of investor capital and an army of research engineers that are the best at setting up large compute clusters, and getting models to concurrently run on thousands to tens of thousands (and more) GPU's concurrently using really really good distributed code.

AI progress won't stop at human-level. Hundreds of millions of AGIs could automate AI research, compressing a decade of algorithmic progress (5+ OOMs) into 1 year. We would rapidly go from human-level to vastly superhuman AI systems. The power—and the peril—of superintelligence would be dramatic.

Continuing the incentives talk, you're one of these people with all the knowledge, the keys to the kingdom in the form of equity in a highly valued LLM venture, and we're at the industry wide benchmarks of GPT-4o and Llama V2. These systems are amazing, but if Ashenbrenner is to believed, we're still on the exponentially rising curve that will end with AGI in 2030, so your equity in OpenAI / Alphabet / Nvidia is still exponentially undervalued with the AGI to come. Why would you sell any shares, or for that reason leave at all? You've got an equity stake in literally bringing our AGI overlords to fruition - surely that stake will be honored with piles of money by the 2030's “at the latest”.

My view is much more sober and, frankly, far less fun.

The brilliant people who built these systems have seen the writing on the wall - that we can continue to improve these systems via exponential investment in compute, but there are a limited number of OOMs up for grabs and funding is finite because VC funds have ~5-7 years to deliver returns. Furthermore, the differentiation between the best models for the mean of knowledge work is declining - so pricing power and margins will decline as well. There will be a handful of winners that will win because of network effects and enterprise adoption, squeezing out upstarts that have to burn all their funding on compute, expensive researchers, and outmatched sales teams. Now is the time cash out equity and raise money to form new ventures while the AI investment thesis is still de jour and there is a still a premium placed on AI expertise.

It's a perfectly hedged decision given the downside is rejoining a FAANG research group in 2-3 years.

Here's what I believe is happening at a blocking and tackling level. Collect a few brand name Ph.Ds who worked at any "world-class" LLM research organization that have some risk tolerance, split off to form a new company. Use the academic star power and “ex” mafia to raise >100mm dollars, build a chat or GenAI offering aimed at enterprise in some knowledge work centered vertical (law, health, finance, advertising, marketing, etc), go to war with the other >100mm funded enterprise AI companies for a finite pool of companies with the scale and means to use AI to improve their knowledge work processes but CRITICALLY not technologically sophisticated enough to either (1) deploy free open source models into their organization (e.g. Llama) or (2) be one of the ultra-profitable FAANG enterprises that won't use your AI tools anyways. Now play startup for awhile. If your company finds PMF, great! If it doesn't or you can't fundraise more because the economics don't support it (read - you aren't profitable), then get acquired / merge back into FAANG at a reasonable multiple.

We've already seen this quite a few times. Shamir left, went to Captions, now back at Google. Ilya left OpenAI, raised something like >100mm to start SSI…. but we'll see. I'd be willing to bet he'll be back at one of Alphabet, Microsoft, Apple, or Nvidia within 5 years. A bunch of the best robotics and RL people left to go do robotics foundation models at Physical Intelligence, but unless they actually solve that and build a model that completely revolutionizes all future robotics problems (which is unlikely), they'll be rehired as the DeepMind robotics research group within 5 years. Eric Jang is doing this at X1, and Fei-Fei Li (et. Al) are doing this at [I FORGET]. At some point Mistral will fold into a European conglomerate. Anthropic is essentially funded by Amazon and will eventually fold back in.

This time is not different, we're just seeing the circle of life.

Venture capital is de-facto funding the best academics to keep doing their core research, but at wildly inflated multiples. And for the the thousands of lesser known companies building “AI for X”, the question is whether their tools will so drastically improve the productivity / revenue of their target market that they can get pricing power. If you're selling me AI for CRM management, and I only had 1 person beforehand doing that job, I can now fire that person and improve my profitability via the productivity improvement, but will it completely revolutionize my business to where I'll pay your SaaS fees? Probably not. I don't buy that this productivity improvement will lead to a new economy, or even wholesale change many industries, it will just shuffle the productivity formula of knowledge work, causing some low-skill knowledge jobs to be replaced by software, and new knowledge jobs emerge. Meanwhile industries that actually employ large numbers of people (healthcare, manufacturing, trade labor) will be unaffected in the near term.

All of this to get back to why Ashenbreener can go f**** himself; he can show me a hundred curves that are up and to the right, but those curves are descriptive - none of them tell me how an AI system, which is just a piece of software running on machines, will start to run itself. Because at the end of the day, even the most sophisticated LLM “AI” models we have now are just wildly effective seq2seq models. We've managed to compress all previously written (and soon visual) knowledge implicitly into a function parameterized by trillions of numbers and begun injecting that system into our business operations, but it's still just a system without a self. Anyone who speaks this confidently on something deserves to be checked - because think about it, if you were actually this CERTAIN that AGI is coming by 2030, shouldn't you quit your job and move to to await the end times with the other doomsday cults? What is the point of life and building these systems if you truly believe the end is INEVITABLE. I personally don't think Ashenbrenner believes it either, because he has the gall to lay out all the heavy statements about inevitability but the cowardice to use phrasing like “probably”, "would", and "could" before every statement. Well sir, I won't be bullied by “charts” and “math” into accepting our AGI overlords, and neither should you. Show me your trades Ashenbrenner - you coward, and go f**** yourself.

-- Goodman

Don’t be a consumer, it’s doesn’t lead to a life worth living.

We hear all the time about the strength of the American Consumer, or the consumer economy, but what actually is a consumer? Not in the obvious “a consumer consumes” sense - a consumer is someone who buys things of course, but how we label our economy as driven by consumers. When you start looking, you can find lots of wannabe-VC-god-king dichotomies of people in the world, but the Consumer vs. Creator dichotomy is real - there are people who make things: products, companies, content, and then there are people who buy them. The buyers vastly outnumber the sellers, which is how you get an economic pyramid that concentrates wealth in the hands of a few individuals. Take a second to think of all the things you buy and consume: services, products, food, media. Do you make any of these?

From what I’ve seen, most people “make” only one thing - they do their job for their employer, “making” economic value. They then consume a salary for that work which they funnel into more consumption of all the stuff I described above. For every influencer that has some unbelievably rich 5-9 after they’re 9-5, most people just go to work, maybe exercise, then watch TikTok / Netflix / media, then go to bed. The whole day they consume food and media and services, and they only put 1 thing into the world.

The Consumer is what capitalism pushes us to be, consumers of everything around us up to and beyond how much money we have, consuming experiences and new restaurants and taking the same trips that we’ve seen online to the same places with the same photos. I speak from personal experience, there is a hollowness to being a Consumer - it’s akin to gluttony or hedonism, you just have to fill your role, consume, and then die.

So my advice is to go N > 1. What does that mean? It means rewiring your brain and daily routines. Our brains make it incredibly easy to fall into a pattern of joy through consumption - pulling up TikTok for 30 minutes leads to a nice dopamine wave, but you get nothing from it in the end. Writing an essay (like this one) that will probably be bad and no one will ever read is hard and doesn’t make you happy most of the time, but you made a thing, and you can point at it and say “I made a thing”. Making a thing whether it’s art or music or software or a meal worth sharing is the whole point. That probably means you’ll be a less traditionally "fun" person because you’ll start slipping on contacts and social media references, but you have to make choices, and after a median lifetime of consuming, you want to drastically over-correct.

I still don't get crypto, but I kind of get one of the needs for crypto. If you run in my "elite" ¹ post-grad circle you'll notice that a lot of people talk about finance. Every quantitatively oriented software engineer I meet will introduce themselves and be like "I'm John. In my free time I like hiking, hanging out with friends, and I do some quantitative trading just for fun." Just for fun?! Is quantitative trading fun? I mean, it's lucrative if you have good strategies and the right infrastructure, but is it actually so "fun" that we would do it in our unpaid free time? What I'm getting at is lots of people in the academic elite that know quant trading is a way to (1) be smart (2) make gobs of money where (3) the amount of money you make is correlated with how smart you are. And if you consider the kinds of people who make it to elite institutions, this is the dream job. Most of us learn young that how much you earn isn't tied to how smart you are, but this is tough pill for people who won at "being smart". You spend a lifetime jumping through academic hoops, winning the race, and you get to the end where statistically, it hardly matters ². But then, there's this one magical niche field, quantitative trading, where the two strongly correlate, so naturally the demand to enter the field is high. I studied electrical engineering, but after looking at the EE job market as a rational economic actor, I realized joining the field would be silly. If you love EE and would die doing it in your free time, sure go work in EE. But if you're me and only really liked EE, you do what it takes to switch towards a field with higher compensation. Why the low EE salaries? I'm not an economist, but rising income inequality, international labor markets, and skilled worker visas (H-1B) successfully commoditized the EE profession. So in the end, EE and other hard science / engineering careers pay well, but not nearly the money software engineering and quant finance pay.

The Supply: exceptional organizations that don't need many people

But from the quantitative finance side ³, the demand for new talent is far lower than supply. What is this statement based on? Anecdotal evidence, so call foul if I'm off. But just eyeballing my peer group, it looks like Jane Street / etc. take a handful of new graduates a year maximum. Assume a small multiple above new grads for experienced hires. Any of these places employ at maximum ~2000 people, and it's pretty clear these small exceptional organizations minimize their headcount so everyone gets paid more. Nothing surprising here. And so, we kind of see that quant finance shops always need new talent, because people flame out or retire, but not nearly as much talent as elite institutions produce. From personal conversations, it feels like >50% of computer science, statistics, math, and physics Ph.Ds. have considered quant finance.

The Mismatch

There's a mismatch here. Lot's of highly intelligent, credentialed people that could probably do these jobs, not many jobs because the firms are massively scalable. Citadel Securities reports they account for 26% of all US equities volume with ~1400 employees ⁴. The larger quant hedge funds have less than 1k employees, and Renaissance famously employs like what, 300 people? Quant finance is just software and finance IP - these firms don't produce tangible goods and stay lean to maximize profitability. And worse, more market participants means more efficient markets - i.e. less alpha. Again, un-cited, but this why IP is so closely guarded at these firms. Strategies disappear as soon as other market participants can execute the same strategy. It's reductive, but I just think of all these guys as alpha miners. Finite amount of gold on Earth, finite amount of alpha in the markets. Unless we could just create more markets....

Cryptocurrencies = Infinite "Amateur Golden Windows"

So here you go: I think below the surface, some of the demand for crypto is just masked demand for new markets. We need crypto because we need more markets. We need more markets because there are a lot of smart people who want to correlate their earnings with their intelligence. Alpha decreases monotonically in existing markets, so the solution is new markets. Cryptocurrencies present an elegant way to create new FX instruments.

Q: What's a challenge with FX trading from the growth perspective?
A: Currencies are 1:1 with nation-states, and the world isn't getting any new nation-states.

Q: But why can't we make money trading traditional FX?
A: You can make money, but there's a difference between betting on directions/movements (gambling) and systematic trading. FX is the largest and most liquid market in the world. With so many market participants and massive liquidity, you can bet it's pretty goddamn difficult to find alpha, let alone arbitrages.

Q: So what makes crypto different?
A: From one perspective, nothing. Every new cryptocurrency just creates N new currency pairs with N existing currencies. But because the cryptos have different flavors / features / auto-generated copy, you get weird cross currency dynamics. Figuring out weird market dynamics with algorithms is mining alpha.

Q: Couldn't we create markets with other asset classes?
A: Of course, but isn't it easier to go with something where you just click a button and a new instrument springs into existence globally?

Q: So how is this a solution for people who don't already work professionally in traditional quantitative finance?
A: There's a golden window of time before trading volume draws in professionals where amateurs can rediscover existing strategies and apply them new instruments. You can bet that as $t \rightarrow \infty$, the big HFT shops will deploy the same cross-exchange latency arbitrage strategies they use now, but those strategies either can't be guaranteed to work or aren't worth the cost to deploy until:

1. Instrument trading volumes are large
2. Market data infrastructure is stable
3. There is enough market data to backtest new strategies

To hammer it home, cryptocurrencies are in one way valuable because they provide an infinite number of golden windows for amateurs to exploit before volumes draw in bigger players. By the time bigger players enter the market, some kid can hit a button and create a new market on an existing blockchain, or a group of engineers can write a new blockchain. Rinse and repeat. Your smart friends can trade crypto and conceivably make money that isn't just random gambling. Larger quant finance institutions support this because they get to deploy large-scale / high-volume strategies and take the alpha if any cryptocurrencies become widely successful (e.g. Bitcoin, Ether). Are cryptocurrencies "good"? Unlikely. Are cryptocurrencies valuable? They are if people value them.

This was for the Wikipedia competition hosted by Kaggle competition to predict future page view counts on Wikipedia. Like so many of the solutions now the winning approach used RNNs to learn a set of hyperparameters on the entire dataset that are then fed in to a predict locally on each time series. This begs comparison to multivel hierarchical models from a Bayesian perspective, but I'll save that for a later post. I treat these competitions as a way to explore new classes of techniques more than as competition to win. And it's from that perspective that I settled on Dynamic Time Warping as the technique I wanted to explore. The big question I had was:

Instead of learning hyperparameters on our entire corpus of time series, wouldn't it be great if we could find a way to sub divide our corpus into highly similar groups, and then learn hyperparameters on those groups?

In the context of the [winning solution]¹ this clustering layer would sit a layer above the RNN. In the language of hierarchical multilevel models, this would adding another level to the model that where we then compute posterior distributions on the hyperparameters for each group, with additional hyperparameters for the entire population. This yielded the question of how to cluster time series data. I looked to the research and settled on Dynamic Time Warping (DTW) from Keogh et. al from UC Riverside. For any pair of time series signals, we can find the energy of the signal that minimizes the distance between two input signals. DTW can easily show that time warped signals are the same. This technique has been highly effective in classifying EEG and other time-varying signals. As

Literature Review: DTW

This review is informal but should provide sufficient grounding. Most of the important work has been done by Keogh

1. Rakthanmanon, T., Campana, B., Mueen, A., Batista, G., Westover, B., Zhu, Q., … Keogh, E. (n.d.). Searching and Mining Trillions of Time Series Subsequences under Dynamic Time Warping.

2. Mueen, A., & Keogh, E. (2016). Extracting Optimal Performance from Dynamic Time Warping, 2129–2130.

3. Lemire, D. (2009). Faster retrieval with a two-pass dynamic-time-warping lower bound, 42, 2169–2180. http://doi.org/10.1016/j.patcog.2008.11.030

4. Keogh, E. (n.d.). Clustering of Time Series Subsequences is Meaningless : Implications for Previous and Future Research.

5. H. Sakoe and S. Chiba, “Dynamic programming algorithm optimization for spoken word recognition,” IEEE Trans. Acoust. Speech, Lang. Process., vol. 26, no. 1, pp. 43–50, 1978.

6. Müller, Meinard. Information Retrieval for Music and Motion. Springer, 2010.

Theory: DTW

For an excellent review of the computational aspects of DTW I recommend "Abdullah Mueen, Eamonn J. Keogh: Extracting Optimal Performance from Dynamic Time Warping. KDD 2016: 2129-2130". Refer to Meinard for a solid review of DTW theory.

Dynamic Time Warping is a path-searching algorithm. DTW finds the minimum cost path between the complete matrix of pairwise distances between two time-series we will label $X$ and $Y$. Define $X := (x_1, x_2, ..., x_n)$ and $Y:=(y_1, y_2,...,y_n)$. This matrix of pairwise distances is referred to as the cost matrix $C$. Define the function $c(x,y)$ as the cost or local distance function between two points $x$ and $y$. If $c(x,y)=0$, the two points are identical. Therefore, low cost implies similarity, high cost implies dissimilarity. DTW finds a path through the cost matrix of minimum total cost. Each valid path through the cost matrix is called a "warping" path. The set of all warping paths is labeled $W$.

This recursive function gives the minimum cost path:

$\gamma(i,j)~=~d(q_i, c_j)+min\{\gamma(i-1,j-1), \gamma(i-1, j), \gamma(i, j-1)\}$

Figure 1. DTW computed optimal path between sinusoid & sinusoid + sinc over $[0,6\pi]$

DTW Nearest-Neigbhor Clustering

The Wikipedia challenge provide 145,000 time-series to predict. Ideally, I would have computed pairwise DTW distances for all 145,000 pairs, and then run agglomerative clustering with Ward linkage to generate clusters. The computation wasn't feasible on my desktop setup, and I wasn't willing to go the AWS route for this exercise. Instead, I decided to randomly sample 10,000 rows, compute pairwise DTWs for the row sample, run hierarchical clustering on the pairwise distance matrix, and finally assign all non-sampled rows to the cluster matching of their nearest neighbor. Admittedly this was not the ideal solution. By assigning all non-sampled rows to the cluster containing their nearest neighbor (i.e. minimizing DTW distance), I likely unbalanced the clusters. Furthermore, nearest-neighbor clustering is meaningless with high-dimensionality inputs. I considered nearest-neighbor defensible because my dimensionality was unitary. That is, I only clustered on the DTW distance and so avoided have high-dimensionality inputs. I didn't compute any rough heuristics to quantify the distribution of non-sampled rows to clusters due to time constraints.

"As noted by (Agrawal et al., 1993, Bradley and Fayyad, 1998), in high dimensions the very concept of nearest neighbor has little meaning, because the ratio of the distance to the nearest neighbor over the distance to the average neighbor rapidly approaches one as the dimensionality increases." [2]

I ending up doing a minor rewrite of Prof. Eamonn Keogh's UCR-DTW C++ tool to do nearest-neighbor search instead of sub-sequence search. The tool now searches a list of time-series to find the one that minimizes the DTW distance instead of sub-sequence search to find the sub-sequence that minimizes distance. Because of optimizations in the UCR-DTW tool (via research by Lemire, Kim, et. al) the tool only computes the full DTW in 10% of cases.

After clustering, about 80,000 rows belong to one cluster - so we could conceptualize that cluster as the base data set and run our standard model. I would be curious to train the winning solution on each of my clusters to see if that improved its performance. The smaller cluster had predictable features like single large spikes, uniformity, two spikes, clear seasonality, etc. Using different linkage whole and partial linkage patterns instead of ward only increased the size of the largest cluster. This indicates that there is no signicant distinction for the majority of the time series signals in the corpus.

Code Appendix

    %% Define original signal
    x = 0:pi/64:6*pi;
    y = 0.25*sin(x);

    %% Modify a section of the signal
    N = length(y);
    xind = (floor(0.40*N):floor(0.60*N));
    ymod = y;
    ymod(xind) = sinc(x(xind));

    %% Compute DTW
    zg = repmat(y, N, 1);
    zgmod = repmat(ymod', 1, N);
    d2 = sqrt((zg - zgmod) .^ 2);
    [~,ix,iy] = dtw(y, ymod);

    %% Plottingm
    [xg, yg] = meshgrid(1:N);
    d2(sub2ind(size(d2), iy, ix)) = 1.5*max(d2(:));
    mesh(xg, yg, d2)

I’ve been extensively testing Fable 5 (Mythos class) after the recent launch. My tests pushed its capabilities from spatial reasoning, critical reasoning, long running research and feedback, multi-step tasks, cinematography / visual design, and other more minor tasks.

Examples:

• help research, design, and procure CNC milled parts
• assist with optics procurement
• do cinematography for my company landing page hero (vigilautonomy.com)

My TLDR feedback is that without a doubt Anthropic cooked with this one. Using Opus 4.8 or its competitor GPT 5.5 xhigh always felt like working with a brilliant intern in their first month - it would do EXACTLY what you said because it was afraid of screwing up and wanted to please you. So if you put a typo in your prompt by gosh it would do its damndest to execute that typo.

On the one hand that’s great? Like I want this AI to be aligned with my intended use, so if I tell it something crazy and I meant it, then I don’t want pushback; just do it. But like most people I make typos all the time, and when I try to use voice to text transcription for technical problems those typos pop up even more often. The ideal alignment here is judgement of my prompt with reasoning to align with the intentions of my words, not the exact words themselves.

Usage Patterns

I do all of my tasks with permissions completely removed - mainly because I think that pattern is the future and the rewards absolutely outweigh the risks. Consider the idea of placing a human (me) in the middle of a loop of operations that may seem inscrutable, so when Fable is asking if I’m okay with an operation like ripgrep or curl or some chained together shell operation my only real reason to interject is not because I disagree with the METHOD, because I do not even attempt to understand the method, it’s because I’m scared of the blast radius and privilege escalation.

I fully believe that as we get more computer use data from Fable 5 users and the flywheel for the frontier labs spins up that post-training will be able to build in better guardrails on the system that make it harder to blow things up. But honestly, the consequences fall on me and I find them worth it to get the full agentic “go off and do this” experience so I can focus on other work.

The only best practice I always engage in is specifying allowed tools in my CLAUDE.md / AGENTS.md that I consider “safe” (chrome browser, ffmpeg, FreeCAD, python virtualenv, isolated worktrees), with explicit instructions to halt immediately if any execution ever requires changing GPU drivers or establishing persistent long running background tasks that will remain after prompt execution. I know the “safety” this brings me is untrue in so many ways, but in my scoped usage it has been effective. I think as long as you don’t say unbounded tasks like “go build me a 1 billion dollar business”, you’re going to be fine. METR is a decent mental heuristic here - as long as the task I’m asking it to do is scopable within 1 day of human work, it’s likely the tool usage is contained enough.¹

For example I’ve already seen Fable 5 taking pains to limit its own blast radius even when I give it my personal thinkmaxxing cheat code of “exhaustively”:

• clone my code into an isolated worktree
• spin up an ephemeral and completely separate python virtual environment for the debugging
• use playwright to control an isolated chrome browser
• all of this gets wiped as soon as the task is done

Within the standard bounds of “use my computer to do a thing that is not crazy in a moderately well bounded filesystem with good enough available tools” Fable nails it every time.

Case Study 1: Research, CAD, Procure

I gave Fable the task of producing CAD artifacts of M6 adapter plates for my work at Vigil Autonomy I could send to a machine on demand vendor that would be accepted without any issues, based on hardware casing specifications buried in PDFs and across websites. Consider a few of the steps involved if you were doing this manually:

• for each part go find the relevant spec document. If you’re lucky this is a STEP file, if unlucky it’s a technical drawing buried in a PDF
• Translate that spec to a CAD file of an adapter plate that fits to an M6 optical breadboard
• The CAD artifacts must be acceptance ready for a print on demand vendor (Xometry, SendCutSend) and must use language a human reviewer would understand including acceptable technical drawing PDFs
• Actual uploading of the CAD files to those vendor sites using menu usage to specify taps at each hole for the CNC mill
• Procurement of any mounting equipment from a separate vendor like McMaster-Carr

Put another way this is a complex chain of:

• OCR / Diagram Extraction
• Tool use (CAD)
• Judgement / Reasoning: Do the CAD files we designed make sense? Will they mount to the optical breadboard? Not only does the math check out, but do we have underflush? When we’re operating in field in Kuwait will the heat cause warping?
• Search (for best practices working with vendors)
• Tool Use + Judgement (Browser): now that we have a browser and a vendor site, how do we upload our “correct” CAD files in a way the vendor accepts? How do we examine other vendor sites to identify SKUs that will allow one-shot mounting?

Analysis

We’ll only know after mounting and field use, but Fable’s planning and execution matched the exact process I would have taken, accomplishing in 3 hours of execution with intermittent feedback to push for continued use in producing more CAD files, and the file synthesis into the browser what would have taken me 3-4 days of work?!

Obviously adapter plates are the ideal solid you could ask a model to generate - they are rigid, bounded, no deformations, single standardized material, and would require a human’s limited mental context to maintain (taps, flush, proud, underflush). But still…. holy shit.

When I asked it to produce validation diagrams showing its understanding of layout on the breadboard, how the screws would work on the board with threading distance, underflush available, it nailed it. When I asked it to confirm that our designs would succeed assuming the tolerances of a 3rd party machinist, Fable explained that obviously it had already considered this, surfaced its thinking tokens to prove it, and made it clear that any error here would be because the machinist did not match their quoted tolerance. Time will tell.

What I love the most is that Fable finally feels like that intern that’s been there for awhile - it’s still slavishly devoted to your goals, but when you’re wrong in your prompts it will reason about your actual goal, do that, and explain that it knew what you meant. I finally had zero times where it executed based on a typo (if I type “produce a STEO cad file” it knows I mean STEP).

I’m guessing as part of their post-training Anthropic is pushing more reasoning tokens during planning before any execution takes place. This makes a ton of sense - for safety aligned AI you need to know the INTENTION of someone’s prompt and filter red team prompts that may read as innocuous but are part of a malicious prompt chain. If someone asks for a list of chemical reagents or electronics related to explosives, reasoning would make a normal person ask “wait, why do you need this? I’m suspicious”, and as future prompts arrive I have that suspicion tied to reasoning, and can apply judgement to stay aligned with my safety policy.

By actively blocking usage for biotech / cybersecurity applications Anthropic is aggressively enforcing their safety mandate² - which I tend to agree with, and the downstream effect of all the upfront reasoning is a model that nails intention alignment.

The multi-step execution for long running flows is solved. The 1M token contexts are more than enough memory for my isolated workflows, Anthropic has clearly made improvements in context efficiency, and when in doubt memory files are the right solution for picking up from an earlier checkpoint. There was no task I gave it, including the cinematography tasks for vigilautonomy.com, where it failed, we’re talking long running ffmpeg processes, wiring videos into the landing page, and visual analysis of other best practices landing sites.

Case Study 2: Hero Video Production

Until I hire professional designers I’ve been stress testing LLM visual reasoning by repeatedly redesigning the landing page for my company Vigil Autonomy (vigilautonomy.com - hit me up if you like aerial autonomy). LLMs are crushing all of the technical benchmarks but we all know they still don’t nail aesthetic taste.

I think this will continue to be true for a few reasons: (1) design inspiration may be from the distribution (i.e. “inspiration” or “reference”), but new designs are inherently out of distribution (2) design language is ambiguous (what is “clean” design, describe “blue” to a blind person) and (3) LLMs are not inherently human - the designs they produce cannot link to a gut feeling (the Golden Arches suggest nothing in the transistors of an LLM).

I gave it the task of generating a split-screen piece of cinematography I could use in the hero. On the left I had some UAV footage taken during a test flight from above, on the right I wanted paired footage from a ground station detecting the UAV.

You get into interesting problems - on the left it’s more about visually inspiring and cinematic shots with “clean” clip transitions (see even I can’t help use ambiguous language) - on the right I wanted my ground station UAV capture footage, but I didn’t just want hover shots of the drone, I wanted to capture the actual importance of why this data matters - the drone flying from out of frame or a distance towards the camera. I had a loose idea of the clips that looked nice, but didn’t want to do any labeling of frames or help with trajectory segment selection.

Here’s what Fable 5 did:

• first planned how it would accomplish a split-screen design with ffmpeg
• then did clip selection from my ground footage by collating the known UAV trajectories, where the UAV would land in frame in post projection, filtering segments down to make sure even after cropping we’d see the UAV fly into the split screen frame
• executed long running FFmpeg background subprocesses
• updated my landing page to use the video

Analysis

I don’t know much about cinematography (or if that’s even the right word) but as a computer vision person I’ve spent enough time looking at video to know this looks better than average. Like is it cinema grade? No. Is it better than what I had before? Yes. I liked that Fable 5 was able to handle the FFmpeg usage to do fairly obvious things like clipping / splicing that have always been painful. This was a series of annoying manual steps that are finally in reach because of a tool like this.

Final Thoughts

I’m going to need a better way to work around or with the robots.txt and agents.txt policies more websites are using and that Anthropic respects.³ My intended usage is not malicious - if I want to go buy a list of things from McMaster-Carr, I need to make sure I have the full up-to-date list of SKUs and specs so Fable 5 can reason, but the policies block this (fairly, IMO).

The tradeoff here will be how much I personally care, the future plan is to just have a less scrupulous agent (local LLM) go execute the scrape, build the cache, RAG it, then feed that local file to Fable 5 as it executes. Less token usage and we have the data we need so I can speed run my purchases. I’m dead certain Anthropic / OpenAI / Google / Stripe are fighting to sell this integration for agentic commerce into vendors like this, but in the meantime I still need to accelerate.

Finally, using a model like Fable 5 with high, xhigh, or max effort makes me as the operator wonder if I’m being ambitious enough. It feels like if I can come up with a very long complicated task with a healthy amount of detail the LLM will nail it - so I’m now bounded by big problems I can come up with. This is a joy - drudgery is now outsourced and the hard physical problems that need solving can get my entire focus without sacrificing anything.

Per my earlier article the conclusion is this feels like a step change even if that change is just because the model is 25% better - it has surpassed some threshold. I’m not excited for this to move behind API pricing as it’s currently worth it, but at usage based pricing I’ll have to re-evaluate. I may find it worth it to use it as a coordinator leaving execution to dumber models, but we’re talking about more scaffolding, more mental overhead, more complexity. As they say, the first hit is free.

Footnotes

1. METR, “Measuring AI Ability to Complete Long Tasks” (arXiv:2503.14499) — the 50%-task-completion time horizon metric. The heuristic: if a skilled human could finish the task in about a day, the agent’s tool use tends to stay well bounded.
2. Anthropic, “Introducing Claude Fable 5 and Claude Mythos 5” — Fable 5 and Mythos 5 share the same underlying model; Fable 5 ships with additional safety measures for dual-use capabilities, while Mythos 5 is available only to approved organizations.
3. Anthropic, “Does Anthropic crawl data from the web, and how can site owners block the crawler?” — ClaudeBot, Claude-User, and Claude-SearchBot honor robots.txt directives and anti-circumvention technologies.

Theory

TODO

Requirements

Language: Python

The lingua franca of modern DL is Python. All the major research codebases are in Python, the major DL frameworks all have Python bindings, and the vast majority of tooling is written for Python users. I don't know of any big league research organizations that use anything other than Python.

CPU Training on Machine

This is the "Hello World" of DL, but we'll explore this scenario in some detail so we can reference it in later sections. Let's use the torchvision library from the PyTorch team to train a simple classifier on MNIST.

import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import torchvision.transforms as transforms

from torch.utils.data import DataLoader
from torchvision.datasets import MNIST

transform = transforms.Compose([
                               transforms.ToTensor(),
                               transforms.Normalize(
                                 (0.1307,), (0.3081,))
                             ])

# NOTE: Dataset downloaded to local machine
trainset = MNIST(root='./data', train=True, download=True, transform=transform)
trainloader = DataLoader(trainset, batch_size=4, shuffle=True, num_workers=2)

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
        self.conv2_drop = nn.Dropout2d()
        self.fc1 = nn.Linear(320, 50)
        self.fc2 = nn.Linear(50, 10)

    def forward(self, x):
        x = F.relu(F.max_pool2d(self.conv1(x), 2))
        x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
        x = x.view(-1, 320)
        x = F.relu(self.fc1(x))
        x = F.dropout(x, training=self.training)
        x = self.fc2(x)
        return F.log_softmax(x)

net = Net()

criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)

for epoch in range(2):
    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):
        # NOTE: Data loaded from disk into RAM
        inputs, labels = data

        optimizer.zero_grad()

        # NOTE: CPU computes forward pass, backward pass, update weights
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        running_loss += loss.item()
        if i % 2000 == 1999:
            print('[%d, %5d] loss: %.3f' % (epoch + 1, i + 1, running_loss / 2000))
            running_loss = 0.0

Several things verify that our model works correctly:

1. There were no interpreter errors when we run the code (i.e. no crashes)
2. The loss decreases
3. The accuracy on the validation increases

This example is representative of how tutorials and most universities teach deep learning, but it is not useful in practice. Let's dig in on several of the assumptions that make this example so elegant. First, the MNIST dataset is tiny, only 9.9 MB, and static. Why do we focus on this? Because in practice, the datasets we want to use to power insights or new features are massive and dynamic (constantly changing). It is easy in this example to download MNIST to our local device to train the model - it only takes a few seconds to make an HTTP request to the server with the tarball of the MNIST dataset and download the whole dataset. And once the dataset is in memory, the time cost of loading small batches of the dataset into our model on the CPU is effectively optimal. MNIST is just 28x28 pixel images - it takes almost no time to load a small batch of 28x28 images, and those images take up almost no RAM. You can't beat the speed of just loading data from disk onto CPUs without fancy I/O optimizations which we'll cover in a later section. If we were lazy and inefficient, we could always re-download the MNIST dataset from the server every time we trained the model and it would still work. We would only add a few seconds to each training run, and the amount of memory we consume is only 9.9 MB.

Also observe that all of the operations in the function above.. TODO

Distributed Deep Learning

Now we that we've shown the example above, we can throw away almost every assumption we just made. In real life, DL / AI / ML is not magic. All we are doing is using backpropagation to compute gradient updates for an optimization function. The magic is cleverly selecting a model that maps our input data to our objective, sometimes being clever with how we train the model(s), and training on as much data we have available. For all of the literature on "efficient training" and network architectures that have stronger capacity to generalize, in practice we want to train the model using as much data as humanly possible. Practically any clever DL trick we can think of can be ignored if we have more training data. So if you're a DL practitioner in an organization, and the goal is to get a model you can actually use (in business or research), your main goal is to acquire as much data as possible. If you're successful and can acquire a massive dataset (>10TB), you will be able to train something useful, but your dataset does not fit in memory. Furthermore, your dataset is probably generated via an internal ETL pipeline, or by interns tasked with getting you more data, which means it changes over time.

It will be useful to understand the role of distributed communications in DL before digging into distributed deep training. To speed up model training on a large dataset using multiple GPUs, we turn to "data parallel training". The plain english explanation is we can speed up model training by sending different chunks (i.e. "batches") of the very large dataset to each GPU, have each GPU compute weight updates separately, and then add up those weight updates (i.e. "gradients") to produce one global weight update at each time step. The step where we add up the weight updates is known as all_reduce(), and we'll cover it in more detail in the MPI section. The copy of the model on each GPU will have the same weights after each iteration of backpropagation, but we've now consumed # gpu's times the data in a single iteration. Throughout this process the GPU's need to be in constant communication, and so a distributed communications protocol is required.

Hardware

The bottom layer of our distributed training stack is hardware. Our network has CPUs, GPUs, and in the future perhaps TPUs and FPGAs. For modern deep learning, the only GPUs anyone uses are sold by NVIDIA - they hold a monopoly on the market. Why? How? The short story: NVIDIA owns CUDA. CUDA makes it easy to write code that can take advantage of GPU parallelization. Deep learning, and especially computer vision based deep learning, is a lot of convolutions. Convolutions are "shift, add, sum" - very easy math to parallelize, extremely fast operation when parallelized. All the major numerical computation libraries built in CUDA support to use GPUs since it was first and easiest. Only NVIDIA GPUs support CUDA, so by default you can now only use NVIDIA GPUs for deep learning.

CUDA is a parallel computing platform and programming model that makes using a GPU for general purpose computing simple and elegant. The developer still programs in the familiar C, C++, Fortran, or an ever expanding list of supported languages, and incorporates extensions of these languages in the form of a few basic keywords.... These keywords let the developer express massive amounts of parallelism and direct the compiler to the portion of the application that maps to the GPU.

In 2016 Google announced the Tensor Processing Unit (TPU), an application specific integrated circuit (ASIC) specifically built for deep learning. Without going into hardware detail, TPUs are significantly faster than GPUs for deep learning. GPUs are multipurpose - they are graphics processing units, literally designed to handle graphics workloads. It just so happens that GPUs are effective for deep learning, but they are not power or memory efficient. Unlike GPUs, TPUs leverage systolic arrays to handle large multiplications and additions with memory efficiency ¹. Google initially used its TPUs internally to handle its massive DL workloads, but now offers TPU enabled training and inference as a cloud offering and for sale.

Network Switching

Message Passing Interface (MPI)

Message Passing Interface (MPI) surfaced out of a supercomputing computing community working group in the 1990's. From the MPI 4.0 specification²:

MPI (Message-Passing Interface) is a message-passing library interface specification... MPI addresses primarily the message-passing parallel programming model, in which data is moved from the address space of one process to that of another process through cooperative operations on each process. Extensions to the “classical” message-passing model are provided in collective operations, remote-memory access operations, dynamic process creation, and parallel I/O

MPI specifies primitives (i.e. building blocks) that can be used to compose complex distributed communications patterns. MPI specifies two main types of primitives:

1. point-to-point : one process communicates with another process (1:1)
2. collective : group of processes communicates within the group (many:many)

These primitives are obviously useful for distributed deep learning. Each GPU process has weight gradients we need add up? Use the all_reduce() primitive. Each GPU process P_i comes up with a tensor x_i that needs to be shared to all other GPU processes? Use the all_gather() primitive, etc. Below is a helpful graphic directly from the MPI 4.0 standard to cement the concept.

Distributed Training Frameworks

Horovod: https://youtu.be/SphfeTl70MI

Comparison of NVLink vs alternatives: https://arxiv.org/pdf/1903.04611.pdf

Single GPU Training

This is the same scenario as "CPU Training on Machine", but we now have 1 GPU in addition to our CPUs.

https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader

Multi GPU, Single Node Training

This is the same scenario as "Single GPU Training", but we now have >1 GPUs on our node. Most deep learning practitioners in academia, small industry research groups, and hobbyists operate under this scenario. Simply put, you take as many GPU's as you can afford and put them on a single physical machine. This avoids the need to consider inter-machine networking, and in reality, this still works for most SOTA research.

As seen in the figure below, the deep learning framework creates a process to manage each GPU's training. The GPU's each receive a different mini-batch of the data for the forward pass, compute gradients, and then proceed to all_reduce. Now we have two main ways to perform the all_reduce.

The first way is to designate one of the processes the master process, usually the process managing GPU 0, and have each GPU process send gradients to the master process. The master process accumulates all gradients and sends the reduced gradients back to each GPU process. Now each GPU has the same gradients to update weights. This is particularly effective for this scenario because the GPUs are all on the same machine; they are connected by a dedicated IO bus that supports extremely fast inter-GPU communication. In PyTorch this the DataParallel[^3] model

TODO: Does the second way exist??? What are the tradeoffs if so.

https://pytorch.org/tutorials/intermediate/dist_tuto.html https://pytorch.org/docs/stable/distributed.html

Multi GPU, Multi Node Training

All processes in the process group³ use ring communication

Multi GPU, Multi-Node Training Cloud

Truly large-scale models are trained in the multi-gpu, multi-node setting. Examples include GPT-3, DeepMind's DOTA model, and Google's DL driven recommendation algorithms. Recall that a 'node' is typically a physical machine with 8 GPU's. When practitioners say a model was trained on 1024 GPU's, that precisely means the model was trained on 1024 / 8 = 128 physical nodes, each with 8 GPU's.

Let's take a moment to think of the broad challenges here before getting into specifics. We're now training a model across 128 separate physical machines. This introduces problems related to communication, i.e. that the physical nodes will have to communicate with each other over some network, and problems of speed. It takes time to load each batch of data from the data store to each physical machine. What if the data size is large (videos, raw audio), and what if DB reads are slow?

6a. Distributed Training Frameworks (Training Backends) a. MPI b. DDP (NCCL) c. Horovod https://github.com/horovod/horovod d. BytePS e. Parameter Servers

https://pytorch.org/tutorials/intermediate/rpc_param_server_tutorial.html

Cloud Workflows for Deep Learning

Deep Learning Frameworks

Higher Level Frameworks

Keras, Ignite, Lightning

Data Ingestion Patterns

Frameworks*

Let's talk about frameworks for a second. This section has an asterisk because I want to make it clear these are opinions, not facts. Frameworks provide higher level patterns / abstractions that structure your usage of a tool and in theory enable you to do more complex tasks with less effort. I say in theory because at worst, the frameworks themselves have a steep learning curve and don't enable you to do anything the underlying tool couldn't do.

PyTorch Ignite with PyTorch

• Elegant engine construct to manage logging, hooks, training execution
• Clean interface over PyTorch distributed module
• Narrow set of excellent utilities for logging, checkpoint management, etc.
• Poor support for custom code - rewriting open source code to operate with an ignite engine is difficult
• Low-level, sits one level above PyTorch so not too abstract
• Small but active developer community

PyTorch Lightning with PyTorch

• Strong abstractions for new developers
• Excellent support for multi GPU training
• Integrations with upcoming deep learning backends
• Large developer community
• The guy who started PyTorch Lightning is shameless, he goes for personal attention at every opportunity, and they misrepresent other's work as the Lighning's work. -10.

Keras with TensorFlow

• Excellent abstractions over TensorFlow for new developers
• Such widespread usage that Google moved to provide first-class support
• TensorFlow adopted many core ideas
• No longer needed to advanced TensorFlow
• Doesn't play nice with the entire Google Cloud DL ecosystem

Reading List (Links)

• Andrej Karpathy's Recipe for Training Neural Networks (2019)
• https://lambdalabs.com/blog/introduction-multi-gpu-multi-node-distributed-training-nccl-2-0/

So before getting into this a note of praise - you aren’t dumb if you don’t fully understand CUDA drivers. I’m a deep learning practitioner with software experience, and this was painful for me.

First some terminology:

• Nvidia X.Y Toolkit: what tutorials mean when they say “install CUDA X.Y”. The core versions of CUDA are 10.xx (past) and 11.yy (future)
• Nvidia Driver: the driver controlling defining GPU’s interface with your OS / Architecture.
• nvcc: the CUDA compiler driver

Nvidia Software Ecosystem at-a-glance.

Toolkit documentation:

Release Notes :: CUDA Toolkit Documentation

Also useful is the driver documentation. You can find it for your specific driver version at:

NVIDIA Accelerated Linux Graphics Driver README and Installation Guide

The Decision: picking your CUDA installation

If you read the release notes above, you’ll observe multiple installation methods for the CUDA driver / toolkit pair on linux. It can be deeply confusing. Do I go with the runfile or Debian installer? Should I just go with the newest versions of the CUDA toolkit and CUDA driver?

Nvidia makes it clear that there is no correct answer here. While there are some constraints (described below), just do whatever is right for your system.

CUDA Compatibility

Not every version of CUDA driver supports every version of the CUDA Toolkit. Before making a choice, refer to CUDA compatibility matrix, copied here for reference.

NVIDIA Driver Documentation

But wait! This is not the actual table you should refer to - this table just says that for CUDA Toolkit 11.x minor version compatibility (i.e. to even be able to support minor version “x”), you need at least driver ≥450.80.02. The actual table we will refer to is below.

This is the reference table we care about - it shows the minimum driver required for Linux 64 bit x86 architecture to support a given CUDA Toolkit version. Let’s say we decide on 11.2.0 GA for the global installation of CUDA Toolkit on our system. We will need at least CUDA driver 460.27.03 on our system. That’s it, we’ve solved it 🤝. We’re free to use a higher version of the CUDA driver, since CUDA drivers are always designed to be backwards compatible, but sticking with a lower (stable) version of driver is fine too.

Special Case: Forward Compatibility

Let’s quickly discuss a special use-case. The vast majority of users will go with a stable pair of CUDA toolkit and CUDA driver, but what about the case where we’re fixed to an old driver but want to use a newer version of CUDA.

As an example, let’s say we are fixed to CUDA 418.40.04, but need our system to use CUDA 11.2. The compatibility table says we are out of luck, since the minimum required version is 460.27.03 😩. Forward compatibility packages to the rescue! Referring to the table below, we see forward compability for the 418.40.04 driver is available for CUDA 10.2-11.6 🎉.

All we have to do is install the forward compatiblity package onto our system. On Ubuntu, prefer installing the compatibility package from network repositories. In highly specific use cases, can install the compatibility package manually from a runfile (.run) - but for most users just use apt-get to install.

sudo apt-get install -y cuda-compat-11-2

The package will install to /usr/local/cuda/compat . Now include the installed CUDA compability files. This code below also runs the CUDA Device Query utility which “enumerates the properties of the CUDA devices present in the system”.****

LD_LIBRARY_PATH=/usr/local/cuda/compat:$LD_LIBRARY_PATH samples/bin/x86_64/linux/release/deviceQuery

The Installation

Quoting from the Nvidia docs:

When installing CUDA on Ubuntu, you can choose between the Runfile Installer and the Debian Installer. The Runfile Installer is only available as a Local Installer. The Debian Installer is available as both a Local Installer and a Network Installer. The Network Installer allows you to download only the files you need. The Local Installer is a stand-alone installer with a large initial download. In the case of the Debian installers, the instructions for the Local and Network variants are the same.

As a heuristic, if you don’t know why you would choose one installation method over the other, choose the Debian Installer!

I’m not going to copy steps from the docs, but to summarize:

1. Go through pre-installation actions
   1. If going through a “local” installation (.deb or .runfile install), instead of network installation (.deb install only), download the installer to your system from the downloads page for the specific CUDA toolkit you want to install
2. Go through installation action
3. Go through post-installation actions

Hard Reset: clearing your system of installed CUDA packages

“Removing CUDA” is a popular topic online. Now that we understand CUDA is actually a collection of packages (Toolkit, Driver, forward-compat, etc.) we have a better idea of what it takes to uninstall CUDA. First, from the Nvidia Docs we can uninstall a CUDA deb installation on Ubuntu using:

sudo apt-get --purge remove cuda*

Remember that using deb installs files to /usr/local, so after this command ls /usr/local/ | grep cuda should show no matched files.

If we installed using a runfile instead of the deb installer, we need to uninstall the Driver and Toolkit with separate commands:

sudo /usr/local/cuda-X.Y/bin/cuda-uninstaller # Uninstall the Toolkit
sudo /usr/bin/nvidia-uninstall # Uninstall the Driver

Anaconda: the easiest way to manage CUDA versions across projects

The easiest way to manage CUDA for different DL projects is manage your environments with Anaconda.

Some highlights:

• Anaconda can manage multiple CUDA toolkit versions separately from your system’s base CUDA toolkit version
• Anaconda takes care of the overhead of installation and un-installation.
• Uninstall is as easy as conda remove cuda

Attempt #1

I had almost no idea what I was doing for this attempt. My only requirement was CUDA 11.7 because I wanted to run Jax from the base CUDA system install and Jax requires 11.7. Given this tutorial, I now know this is not smart. You can install a system-wide version of Toolkit + Driver that is stable to the system hardware (i.e. support suspend-to-RAM) and use conda to manage a higher CUDA version that supports Jax. Installed CUDA driver 515 and CUDA toolkit 11.7 via the Runfile installation method. deviceQuery sample returns expected output and nvidia-smi command also shows expected output. However, I consider this installation attempt a failure. Most important feature for my system is suspend-to-RAM - i.e. the ability to take GPU state at a given time, move it to RAM, and put the system into “suspend” mode without fully turning off the power. This is the most important feature to modern computer usage - i.e. shutting your laptop lid, opening back up and having all the applications you had open... still open. Having to do a full power cycle after every computer use significantly hampers usability 🤦🏼‍♂️. I’ll have to purge everything and try again.

Attempt #2 (ongoing)

Pre-notes: I’m going to try purging all CUDA from the entire system first. I’ve set in the BIOS that the system uses integrated graphics on the mobo, not the GPU, for graphics, so the system will boot fine without the GPU drivers installed. I have a feeling using the newest Nvidia driver (515) is the problem - going to try reverting to 470 since it comes with the Debian installer and is the base version installed on Ubuntu by default. I’m also hoping using the Debian installer fixes the “suspend-to-RAM” issue. If you read the release notes for the drivers (470 vs. 515) they handle power management differently 🤔. I’m sure 515 has more features + efficiency, but I want things to work.

Summary

Installing CUDA on your system is non-trivial. In general, the Nvidia docs are thorough and do contain all of the information you might need, but feel inscrutable to a first-time user. It would probably be helpful for Nvidia to put together more quick installation guides for common configurations. This guide is by no means complete (or even correct)! If I’m missing anything, please let me know.

What I’m amused by is how the first thing the media jumps to is zero days in our critical software infrastructure that is open source (the linux kernel, etc), and they’re right, that is scary. Because if all it takes is a security researcher saying “find me a zero day plz”, is 100 days really enough to patch everything? I truly doubt it, because the first thing red team will do when the model hits open access is ask:

“For each of the next 10000k non-famous, but still critical open source projects, find me zero days please”.

And if the Mythos claims are real, that WILL work with certainty. And if it doesn’t on the first pass, we’ll just keep doing it again and again and again telling the model to be more exhaustive. Because this is 100 days of wall clock time but Anthropic is obviously limiting the compute over those 100 days due to costs and resource constraints, and the red team will have infinite time after the model release.

As little as I care about using Defi myself, I’ll be watching the markets after this release. Think about it - what other markets immediately tie economic value to Mythos token. If Mythos tokens really are this magic vulnerability finding machine, then we’ll immediately see those tokens be used to extract economic value from the anonymous internet UNTIL the token values surpass the economic value to be extracted OR the TVL locked in these Defi protocols and contracts decreases. As I’m writing this the total Defi market cap is 77.21 BUSD - so if those protocols start getting breached due to the new omni-model, we’ll see the effects immediately, but I’m curious how quickly that will actually happen. As the low hanging vulnerability fruit is picked, how much token kerosene are we going to burn searching for new vulnerabilities that translate immediately to money?

Source: TradingView CRYPTOCAP:TOTALDEFI

But in the context of research / algorithms oriented software that has no uptime requirements, the bias coming out of academia is to get the math into code, these days Python, and put it in production. Python is dynamically typed and has no type inference... so the typing is optional. What Mypy users rarely discuss is the cognitive burden of maintaining the type system that is separate from the code. With a lot of tasks on your plate and new features you are waiting to start, sometimes it feels silly to type annotate Python. Is this really necessary?

def(a):
    print(a)
    return

# With types
def(a: int) -> None:
    print(a)
    return

The situation we experience more often is that the simple and obvious functions get type annotated, but the complex ones that deal with uncommon types, like declaring types for decorators F = TypeVar('F', bound=Callable[..., Any]), go un-typed. And we expect this! Catching subtle bugs via the mypy type-checking mechanism is a long game.

Highly effective teams know that type annotation is one of the best ways to make a codebase stronger. I know that's a vague and probably incorrect word to use in the context of software, but it really feels that way in a Python codebase. Without type annotations, it's the blind leading the blind - you need faith that variable names accurately describe the data that code will operate on. "Faith" is cold-comfort when we start talking SLA's.

def concatenate_str(x: str, y: str) -> str:
    return x + y

# the function name said '_str' but our data is 'int'
concatenate(1, 2) -> 3

prevent programmer's from making dumb mistakes in their work. I believe that well-designed code is a force multiplier

The following code will type check, even though the types are wrong. Mypy is a tool for type annotation; it's a document of what you want the types to be, and if the types are what you say they are, then the code is type safe.

Lessons

So if the investment in type annotations (Mypy) only pays off with a full commitment, the most common scientific libraries are still working on type stubs, and no one has the time or willingness to fully commit, the question is, why even bother?

1. Mindfulness is enough to prevent common mistakes

Even if they go completely unused in any formal way, just making developers write types prevents dumb mistakes. For Python, this is often Optional[T] vs [T]. This is not a joke. Errors like this have led to downtime and rollbacks for production services.

"operator '>' not defined for Int and None"

# We have a vague understanding that 'a' is an int
__init__(self, val=None):
    func(val)

# No types - explosion
def func(a):
    if a > 3:
        return a
    else:
        return -1
    
# Types annotations remind us that 'a' can be None.
# either handle the Optional case (None), or prevent
# ever invoking 'func'
def func(a: Optional[int]) -> int:
    if a is None or a < 3:
        return -1
    else:
        return a

2. Don't use Python for production services

If your service has SLA's or has multiple downstream services, don't use Python. To have any chance at service stability, you'll need to fully use MyPy, and have some CI hooks that check for type soundness and prevent non type-annotated code from deploying. Your engineers will need to have the MyPy docs open frequently to maintain the type system, and from experience, you'll get a lot of complaints.

"I know this code works - our test-suite passes, why am I wasting time getting MyPy to stop complaining about an unbound type?"

If you have to maintain a type system anyways, it's easier to choose a statically typed language where the compiler builds in type-checking. Also, most popular IDE's for statically typed languages have type hints - so by choosing Python you're just increasing the cognitive load for developers.

3. Configure MyPy to prevent returning Any

Developers are people, and people can be lazy. The laziest way to get around MyPy errors is to return type Any - the supertype of all types. This is like returning Any in Scala; as a last resort developers will do this to satisfy the type-checker so they can keep moving. Stop them, or at least make them annotate in code every time they really mean to return Any. Keep the MyPy file as concise as possible, but fail on any errors reported.

# file: mypy.ini

# global options:

[mypy]
warn_return_any = True

Takeways

There's truth to the criticisms of trying to use Python in production. I've personally found that types lead to better code, and try to avoid dynamically typed code without type annotations at all costs - the extra time it takes to reconcile types always leads to stronger production code. It isn't static vs. dynamic, but instead interpreted vs. compiled that is the main differentiator in developer productivity.

References

[1] Types for anyone who knows a programming language: https://www.destroyallsoftware.com/compendium/types?share_key=baf6b67369843fa2

"Fighting with a large army under your command is nowise different from fighting with a small one: it is merely a question of instituting signs and signals."

-Sun Tzu

Overview and Common Assumptions

The tactical lesson of the 2016 presidential election is that only a fool would blindly trust the polls. Listening to both sides' campaign managers the night before the election it was clear that paradoxically both Trump and Clinton were going to win the election. Somehow the same polls in the same states told the universe of strategists, spectators, wonks, and traders that election result margins were both "stable" and "erroneous", both "decisive" and "within the margin of error". Not to mention the classic campaign strategist spin that "internal polls have us ahead". I contend that media organizations and private companies have spent an inordinate amount of time, money, and effort trying to estimate the result of an election given a collection of polls. These individual polls are known to be flawed, but the reasoning goes that we can use statistics to adjust for all of the errors and estimate the right result. This reasoning is fast becoming an academic relic. Traditional land lines are disappearing are disappearing from American households and response rates are falling across all mediums of communication both traditional and non-traditional [3]. Those organizations that fail to adjust to this new reality will make incorrect predictions, and, more importantly, will provide incorrect analytics. Our group advocates a new estimation methodology and model, discussed in Appendix A, that centers on the assertion that we should estimate election results using larger and therefore more accurate polls from fewer counties.

I am not writing to provide the academic evidence nor the quantitative rational for every one of my statements. I will paint in broad strokes where I find it necessary, dig into the numbers when it is useful, and will offload nuanced in depth explanations to cited works.

Section 3 dives into the mathematical basis of polling. In section 7 I will describe the new polling model our team built and tested over the last two years.

Disclaimers

A key assumption of the model we now advocate is that American elections are binary races where voters have a choice between two candidates. Yes, the United States does indeed have more than two political parties. There are the Democrats, Republicans, Green Party, Libertarian, Socialists, etc. When more than 90% of Americans go to the ballot box they vote for either a Democrat or a Republican. Anyone else is an independent candidate - and independents do not win elections in the winner-take-all American system. The highest vote percentage ever achieved by an independent candidate was 19% by Ross Perot in 1992. If an independent candidate doesn't declare early, have significant momentum going into the election, or have a large campaign war chest, there are only two choices for president. Either a Democrat or Republican will win. This is the reason of course that Bernie Sanders joined the Democratic Party. The Independent candidate will play spoiler to the party whom he/she steals more votes from. It is a grim truth that we as Americans are reduced to picking between two parties that are meant to capture every aspect of our beliefs with patchwork platforms appearing to broad coalitions of voters, but it is the truth, and until this system changes we will continue to operate our model assuming two parties.

To my own chagrin, the conservative attack that the Media is biased contains shades of the truth. What the public seems to forget is that the television Media (CNN, MSNBC, FOX) is in the business of selling advertisement slots. While it may have been true that the first television news programs were run as a public service, this is no longer the case. There are very few news programs that offer unbiased news, if such a thing can even exist. CSPAN, ProPublica, and PBS are arguably excellent examples of nonpartisan news. CSPAN is also dry as dirt, boring and substantive like plain oatmeal. Americans want to see action! We like our politics like we like our sports: fast, exciting, and close until the end. No one likes the game that ends 10-0. We want the shoot-out. We want the nail-biter. It's hurts our democracy when the Media chooses to treat politics like sports, but they do it because that's what the market demands.

Polling Math

This section should be seen as medicine. I wrote it to be accessible to the average reader, and I hope you take the time to skim it to at least understand that polling isn't all hand-waving and voodoo.

Expectation of Random Variable

We have a random variable $X$. The random variable $X$ has expectation $\mathbb{E}[X] = \mu$. We use random variables to model random phenomena, like rolling dice, or flipping coins. From now on we will use the abbreviation "R.V." interchangeably with random variable. We compute the expectation in the finite case as follows:

Don't stop reading, this is actually an easy concept to grasp. The expectation or "expected value" of the random variable $X$ is calculated by adding up every possible value $X$ can take on, multiplied by the probability of $X$ taking on that value. Let's say the random variable $X$ is modeling a dice roll. You and I both know when we roll a six-sided dice it will land one of the values: $\{1, 2, 3, 4, 5, 6\}$. Assuming no one is cheating, we intuit that there is an equal chance of the dice showing one of the numbers. Using the formula above we an calculate the expectation of the random variable $X$.

I might not get around to showing the proof, but something called The Law of Large Numbers says that if we look at the value the random variable spits out many, many times, the difference between the average of those spit out values and the expectation we computed above will fall to zero. This needs to be very clear. I roll the dice once and it turns up a 3. The second time I roll it turns up a 2. The next time a 5. The next time 5. If I roll the dice a billion times, add up all the numbers I roll, and divide by a billion, the answer is all but guaranteed to be incredibly close to 3.5.

Markov's Inequality

So what was the point of this? We're getting to the good part. Here's the question you should be asking (that I borrowed [1]):

"What is the probability that the value of the random variable X, is not close to its expectation?"

I just told you that we can take a random variable and compute the value we expect as the average if we can 'roll the dice' an infinite number of times. But in the real world, you don't get to roll the dice billions of times, you often just get one roll of the dice. So we want to know how likely it is that the random variable spits out a value that isn't close at all to the expectation. First I'll give the formula, then we'll unpack it together, finally I'll give the proof.

Markov's Inequality: For a nonnegative random variable, $X: \Omega \rightarrow \mathbb{R}$, where $X(s) \geq 0$ for all $s \in \Omega$, for any positive real number $a > 0$:

Unpacking the formula

As promised, let's unpack the formula step-by-step. We already know that $X$ is a random variable, but what does $\Omega \rightarrow \mathbb{R}$ mean? $\Omega$ is a symbol that means "the probability space". So whenever you see the symbol $\Omega$, substitute the words "the probability space" in. Every time you see $P$ and then parentheses, substitute the words "the probability of". $P(\text{dessert for dinner})$ is the same as "the probability I will eat dessert for dinner".

The probability space is the space of everything that can happen in our probability world. In the world where $X$ represents a dice roll, $\Omega$ means every integer between $-\infty$ to $\infty$. This means that $\Omega$ includes the numbers 1, -57, 263, -1454, and so on. "But if I roll a dice, the number -57 will never show up?!". That's right! The symbols $\Omega \rightarrow \mathbb{R}$ means that $X$ is actually a function. It takes in some event from the probability space $\Omega$ and it spits out a real number $\mathbb{R}$ that is greater than or equal to zero. So in the the case where $X$ is a dice roll, here's a few examples of what $X: \Omega \rightarrow \mathbb{R}$ spits out.

The probability that $X$ takes on the value '1' is $\frac{1}{6}$:

The probability that $X$ takes on the value '-57' is zero. It will never happen:

The probability that $X$ takes on the value '4' is $\frac{1}{6}$:

The probability that $X$ takes on the value '7' is zero. It will never happen:

So that's pretty reasonable. $X$ takes in an event that belongs to the complete probability space $\Omega$ and spits out a number that is either zero, meaning the event will never happen, one, meaning the event will literally always occur, or something between zero and one, meaning the event has some probability of occurring. Got it? Onwards then!

$a$ is a constant that is greater than 0. We get to decide what the constant $a$ is. You want $a$ to be 0.00001. Okay. You want $a$ to be 100000. Okay. As long as $a$ isn't less than or equal to zero, you're all good. We'll see later that we can pick specific values of $a$ that let us do interesting things, but for now just remember that we decide what value $a$ takes on.

That's all there is to it! We know what everything means. The following section is the derivation of Chebyshev's Inequality from Markov's Inequality.

First the proof of Markov's Inequality (taken from [1]):

Proof. Let the event $A \subseteq \Omega$ be defined by: $A = \{s \in \Omega \mid X(s) \geq a\}$. We want to prove $P(A) \leq \frac{\mathbb{E}[X]}{a}$.

Thus, $P(X \geq a) \leq \frac{\mathbb{E}[X]}{a}$.

Markov's Inequality is a general statement that holds for any non-negative random variable. It is useful for defining a rough upper bound on the probability of an event. For the case of polling, remember that we wanted to answer the question of how likely it is that our estimation is far away from the truth. For that we need to use Chebyshev's inequality, which can be defined as a special case of Markov's Inequality.

Chebyshev's Inequality: For any random variable, $X: \Omega \rightarrow \mathbb{R}$, and let $r > 0$ be any positive real number.

Again, let's spend some time unpacking this formula. With Markov's Inequality we could compute an upper bound on the probability that our R.V $X$ spit out a value that was greater than a constant $a$. This information is useful, but what we really want is the probability that our estimation is between an upper and lower bound. Let's say we have a simple election between candidate A and candidate B. We say that RV $X$ outputs the percentage of the vote candidate A receives. That is, the random variable $X$ outputs a number between 0 and 1. If the actual result of the election is 0.5 (a tie), Markov's inequality may be useful to tell us how likely it is our poll actually tells us that the final vote will 0.5.

Closing Remarks on the Math

Markov's and Chebyshev's inequalities are truths. You can of course disagree with the conclusions I draw in this article, and I will even point out places where you should disagree, but the math is unimpeachable. If you have any faith in mathematics, then you must at least accept the definitions above because they all come from the same mathematical foundation. If you have no faith in mathematics, god help you.

Here we begin the process of translating from the impenetrable mathematics to a model we can use to predict and estimate events in the real world. Polling is complicated process expressly because we cannot apply Chebyshev's Inequality perfectly to our world of flesh and blood. The core assumption of Chebyshev's Inequality applied to polling is that we are able to collect a perfectly representative sample of the place we are trying to predict. If we want to predict the U.S. Presidential Election with a single poll, the slice of people we poll needs to perfectly represent the whole country. We need to represent the opinions of an impossibly diverse set of three hundred million people using the voices of only a few. Chebyshev's Inequality, while brilliant, doesn't solve for voting ethno-racial and socioeconomic strata, nor income inequality, aging populations, nor the rural/urban divide.

The Importance of Counties

Counties are the smallest voting increment when we talk about the U.S. Presidential election. Each county is responsible for administering the vote within its boundaries. If you live in, say, Cayahoga county, you may only vote in Cayahoga county, and your vote for the presidential election is counted by the county government of Cayahoga county. The county reports the vote total to the state government, and the state government reports the state vote total to the Federal Election Commission. This is true even in the case of absentee ballot voting. If you choose to vote absentee, you send in your ballot to the county where you permanently reside.

Our model relies heavily on U.S. Census data that the Constitution mandates occur every ten years. The U.S. Census data is provided with the finest resolution at the county level, so this a natural limit on the resolution of our model. Even if we wanted to build a model with finer resolution, what standardized unit could we possibly use? The concept of neighborhoods or locals is poorly defined and subject to rapid change. Therefore, we use counties as the finest resolution of our model.tw

Case Study: The 2016 U.S. Presidential Election

We're going to perform a thought experiment with the fictitious Middletown County. Middletown county is (self-identified) 70% white voters, 20% Latino voters, and 10% Black voters. Middletown County is home to 1 million people. We are going to make the assumptions that all citizens of Middletown County are eligible to vote and that all citizens of Middletown exercise their right to vote. These assumptions will be addressed later, but for now all they add up to is that all 1 million citizens of Middletown vote in the 2016 election. Continuing the thought experiment, let's say we know the end result - Clinton wins the vote in Middletown County with 51.0% of the votes to Trump's 49.0% by a margin of 1.0%. We said before that all of Middletown's citizens vote, therefore Clinton received 510,000 votes, Trump received 490,000 votes, and Clinton won by a margin of 20,000 votes.

Chebyshev's Inequality guarantees that as we poll more people, the probability that our estimation of the final vote percentage is inaccurate, decreases. Let's say it again. The more people we poll, the more likely it is we are going to estimate correctly. The beautiful thing is that we now know how the mathematical tools work, so we can come up with hard numbers and not just say words that sound correct. Let's calculate the exact number of voters we need to poll to all but guarantee that we are below a specific error bound. We decide months before the election that we want to poll the population in a way that guarantees we are accurate to within 2% (total spread), 99% of the time. That means we are within +/- 1% of the actual result with 99% confidence. We both know the actual results of the election, but here are a few more results that would satisfy the above condition of Chebyshev's Inequality.

• Clinton 50.5% to Trump 49.5% -- Clinton wins -- 1.0% total absolute error
• Clinton 50.9% to Trump 49.1% -- Clinton wins -- 0.2% total absolute error
• Clinton 50.001% to Trump 49.999% -- Clinton wins -- 1.998% total absolute error
• Clinton 50.0% to Trump 49.0% -- Clinton wins -- 0% total absolute error

We clearly did an excellent job of predicting the election if made any of the claims above! If our poll came up with any one of the results above, we not only predicted the winner of the election in Middletown County, but we did it with less than 2% absolute error in the spread. Keep in mind the power of the previous statement. If we could actually predict the outcome of elections with this accuracy, I wouldn't be writing this post. I would keep this a secret, start a polling company, and make money predicting the future with a high degree of accuracy. And here's the best part - the number of people we need to poll is a set number. It doesn't matter if the underlying population of Middletown is 1 million, 1 billion, or 1 trillion. There is a specific number of people that all but guarantees (99% is essentially a certainty) that we will predict the result to within +/- 1% accuracy. So what is the magic number?

Solving Chebyshev's Inequality for an absolute result error of +/-1% with 99% confidence gives that we need to poll 250,000 people. If we do that, we will estimate the election result to within +/-1%, 99% of the time (equivalent to 2% total spread).

You might be thinking that our work is done. We calculated the number of people we need to poll, and the math checks out. But of course it isn't this easy. The math oversimplifies the numerous and aggravating complexities of the situation. Remember that there are 1 million people in Middletown County. According to the numbers at the beginning, 200,000 of the citizens in Middletown are Latino, and 100,000 are Black. Let's go through a quick sanity check - do we think the Black or Latino communities are going to vote for Donald Trump? Certainly not. There may be a universe where Donald Trump will win the majority of Latino or Black votes, but this is not it. Donald Trump may "love Hispanics!", but they aren't going to turn out to vote for him. Hillary Clinton is essentially guaranteed as a Democrat to win more than 90% of the Black vote. She is also going to win at least 55% of the Latino vote. So if we went to predominately Black and Latino neighborhoods to conduct our poll, and managed to poll 250,000 people to respond, we would deduce that Hillary Clinton is going to win Middletown County in a landslide.

It doesn't require a deep background to see that modern polling rests on assumptions that are nearly impossible to enforce. Below I detail several of the dubious assumptions pollsters make when using traditional polling methodologies. I highly recommend reading the cited literature; each of these assumptions has been comprehensively assessed in other venues.

Classic Polling Assumptions

Voter Turnout

We decided at the beginning of the case study that every eligible voter in Middletown votes on election day. This is not the case in any real election. Voter turnout has rarely exceeded 60% of eligible voters. For instance, about 55% of eligible voters vote in the U.S. Presidential Election on average. As we move to down-ballot races voter turnout numbers drop dramatically. For local elections voter turnout hovers around single digits. That we naively expected every single person in Middletown County to vote now seems laughable.

Minority Voter Suppression

Voter turnout will also be lower than 100% because a raft of deliberate policies and systemic racial issues suppress the votes of minorities. Following the passage of the 13th amendment, a host of Southern states employed literacy tests, poll taxes, and blatant voter intimidation to stifle the African-American vote. Tactics of voter suppression continued through the Jim Crow era until then President LBJ passed the Voting Rights Act of 1964. Among other things, this legislation eliminated all literacy tests and poll taxes, and gave the Justice Department final say on any voting legislation produced in a select list of Southern states with a history of voter suppression tied to systemic racism. In the last thirty years we have seen a concerted behind-the-scenes effort by conservative lawyers to challenge the legality of the Voting Rights Act of 1964, paving the way for the current raft of voter I.D. laws [4]. The tactic of requiring increased identification to vote is a thinly veiled move to suppress minority voters, who are less likely to have the proper identification to satisfy the new criteria. Not coincidentally, voter I.D legislation is cropping up in states with burgeoning minority populations that tend to vote for Democrats. A curious thought is that minority voter suppression engenders a vicious cycle. Republicans at the state and county level push for voter suppression because minorities vote for Democrats by a wide margin, and minorities often vote for Democrats because they are pushing to end minority voter suppression.

Likely Voters

Knowing what candidate a citizen would vote for is unhelpful if the citizen doesn't exercise their right to vote. Traditional polls focus heavily on assigning a likelihood to each person surveyed often by simply asking: "How likely are you to vote on or before Election Day?". As an example, we conduct a poll for an upcoming county election with a voting eligible population (VAP) of 50,000. Our 1000 citizen poll results in 100 responses from Black voters, with a margin of $>90\%$ saying they would vote for the Democratic candidate. However, none of these poll respondents give unequivocal answers that they will vote on election day. We therefore assign each citizen a low probability of turning out to vote. Viewed in the aggregate, the poll tells us that the majority of Black citizens who vote will vote Democrat, but more importantly that many Black citizens will choose not to vote in our fictitious county election. Turnout among that segment of the population is predicted to be low. We can quickly identify issues with scaling this methodology. In a state with millions of voters, we extrapolate voter turnout based on limited data built on a racial classification system that is anything but nuanced. It would be difficult, say, to defend how the responses of 100 citizens who identify as 'Black' can accurately predict voter turnout of any states' Black population that is both ethnically diverse and geographically spread out.

Respondent Honesty

Every poll assumes that the majority of respondents are honest or conversely, that the number of dishonest respondents is non-negligible. Given the choice between Candidate A, Candidate B, and Undecided, poll respondents will provide an honest answer. There is a distinct lack of empirical evidence for this assumption because it is difficult to determine if a respondent's answer to a poll matched his/her actual voting behavior. This assumption deserves a lot of our attention. If we estimate that as low as 0.5% of our respondents will lie, we are already faced with awkward and perplexing choices in our polling methodology. Empirically we have that no racial or ethnic group has any predisposition towards lying that makes that group more likely to lie than any other racial or ethnic group. Therefore, each of our respondents is equally likely to be a liar. The making of a polling catastrophe is quite simple. Say we have a high non-response rate among uneducated White voters in a district where they are a marginal group, yet still represent enough votes to swing an election. We take whatever responses we get from uneducated White voters and extrapolate them to predict the behavior of the voting bloc. But say also that there is a liar among our uneducated White respondents, entirely possible given our uniform distribution of liars among racial and ethnic groups. We have now amplified a skewed result thereby corrupting the accuracy of our poll. If we are unfortunate enough to have lying respondents in our sample of minority groups, we can end up inordinately increasing our overall polling error.

A Polling Strategy

The polling establishment is now encountering the perfect storm of all of the issues described in the assumptions above. Response rates are falling making it more expensive to obtain large numbers of valid poll respondents, ethnic and racial groups are increasingly blurred and fragmented making prediction in these categories prone to error, and voter turnout is difficult to assess and impossible to quantify, Traditional polling may still prove effective in small district races, but it is ever more ineffective at the state and national levels. It is both comical and sad to remember that a key argument in favor of modern polling is that all the issues I described above are managed if the probability of respondents lying is low and we have a sizable population of respondents. The results of the most recent election make it safe to say that there were serious issues with the polling in this election cycle.

Hillygus explains succinctly in Public Opinion Quarterly [2]:

The political parties have built enormous databases that contain information about every registered voter in the United States. Statewide, electronic voter registration files?mandated by the 2002 Help America Vote Act?are the cornerstone of these databases. These files typically include a person's name, home address, turnout history, party registration, phone number, and other information, and are available to parties and candidates (and, in most states, anyone else who wants it). Consumer, census, political, and polling data are then merged into these files to better predict who is going to turn out, what their beliefs and attitudes are and, ultimately, how they are going to vote.

The old methods of polling are antiquated in the age of big-data and data mining. Rather, we should focus our efforts on building sophisticated statistical tools that have greater predictive power. This is already under way with several major statistical tools being applied to predictive polling. Researchers typically break polls down on based on ethnicity, age, and gender depending on the complexity and targeting of the poll. We assume that there is nothing that differentiates the kind of person who will respond to a poll from someone who won't respond to a poll within ethnic or socioeconomic strata. That is, one White female between the ages of 35-40 must be interchangeable from another woman in the same category. For polling to work, knowing how one woman votes must provide information on how women composing the same group will vote. While it sounds dehumanizing, statistics and history validate that this class of techniques has predictive power. It turns out that we can use Bayesian modeling to figure out how much a specific attribute of an individual contributes to their vote, and then use that to predict of an entire population. We can actually figure out to a precise degree how much being a man/woman, white non-white, old/young, and urban/rural effect your vote likelihood and chosen candidate. By polling thousands of people, I can figure out that in a specific county your likelihood of voting is "20% based on race, 20% based on age, 40% based on gender, and 20% based on a record of government or military service". Our unique characteristics as humans are being used as data-points in a machine that seeks to rob us of the free will we have the right to exercise in an election!

The key idea behind the model we built is that it is incredibly easy to screw up or bias a poll. Polls are very, very difficult execute without introducing error. Our group's model rests on the assumption that polls are highly effective in small geographic areas, and ineffective at predicting larger geographic areas. We contend that wide ranging polls contain too many places to introduce error, and that we can limit our prediction error by executing fewer targeted polls with high accuracy, and then reconstruct the entire voting map with signal estimation techniques. Recognizing that polling requires financial resources, we argue that those resources should be directed at getting highly accurate targeted polls in smaller geographic areas. The question then becomes what counties to select as a representative sample of all American counties. The elegant answer that downplays the difficulty is to find the counties that are the most predictive with respect to historical elections and current economic and demographic data. We built our mathematical model using a network theory approach. Our group has demonstrated that we can more accurately predict the outcome of historical US Presidential elections using highly accurate polls from approximately 50 counties rather than using traditional national polling techniques.

Notes on Theory

Refer to Appendix A for the mathematical description of the theory we utilize.

Our model computes multiple data sources to determine how counties voting patterns are correlated. Sources for our data include the CQ Elections database, official FEC Election Reports, and the US Census data. We attempt to only rely on official data to produce our county correlation matrix.

References

[1] Kousha Etessami. Markov and Chebyshev's Inequalities.

[2] D. Sunshine Hillygus. "The Evolution of Election Polling in the United States". In: Public Opinion Quarterly 75.5 (2011), pp. 962-981.

[3] Scott Keeter Michael Mokrzycki and Courney Kennedy. "Cell-Phone-Only Voters in the 2008 Exit Poll and Implications for Future Non-coverage Bias". In: Public Opinion Quarterly 73.5 (2009), pp. 845-865.

[4] Jeffrey Toobin. "Holder v. Roberts. The Attorney General Makes Voting Rights the Test Case of his Tenure." In: The New Yorker February (2014).

Appendix A: Theory

A brief overview of the core theory and requisite terminology used for this research is presented. Credit to Marques, Segarra, et. al for the presentation of this theory.

A graph $\mathcal{G}$ is defined as $\mathcal{G} = (\mathcal{N}, \mathcal{E}, \mathcal{W})$. The set of nodes $\mathcal{N}$ has size $N$, the set of edges $\mathcal{E}$ is such that edge $(i, j) \in \mathcal{E}$ if node $i$ is connected to node $j$, and every edge in the set $\mathcal{E}$ has a corresponding weight in the set $\mathcal{W}$. A signal $\mathbf{x} = \begin{bmatrix} x_1 & x_2 & \cdots & x_n \end{bmatrix}^{T}$ is defined on the graph $\mathcal{G}$ where $x_i$ is the value of the signal corresponding to node $i$. Formally, $\mathbf{x} \in \mathbb{C}^N$. We have defined the graph structure, as well as an arbitrary signal defined on the graph.

The graph $\mathcal{G}$ has a graph-shift-operator $\mathbf{S}$ defined as an $S \in \mathbb{R}^{N \times N}$ matrix satisfying $S_{ij} = 0$ for $i \neq j$ and $(i, j) \notin \mathcal{E}$. We assume that $\mathbf{S}$ is diagonalizable, so there exists an $N \times N$ eigenvector matrix $\mathbf{V}$ and an $N \times N$ eigenvalue matrix $\boldsymbol{\Lambda}$ that can be used to decompose $\mathbf{S}$ into $\mathbf{S} = \mathbf{V}\boldsymbol{\Lambda}\mathbf{V}^{-1}$. If $\mathbf{S}$ is a normal matrix $\mathbf{S}\mathbf{S}^{H} = \mathbf{S}^{H}\mathbf{S}$. This means that $\mathbf{V}$ is unitary and gives $\mathbf{V}^{-1} = \mathbf{V}^{H}$. This yields the decomposition of the graph shift operator $\mathbf{S} = \mathbf{V}\boldsymbol{\Lambda}\mathbf{V}^{H}$.

The graph shift operator allows for the representation of the signal in the frequency domain, understanding as such, as the use of a basis that is invariant to linear filtering. The Graph Fourier Transform (GFT) is defined as $\tilde{\mathbf{x}} = \mathbf{V}^{-1}\mathbf{x}$ where $\tilde{\mathbf{x}}$ are the frequency components of the signal. The inverse Graph Fourier Transform (iGFT) is therefore $\mathbf{x} = \mathbf{V}\tilde{\mathbf{x}}$. We say that the signal $\mathbf{x}$ is $K$-bandlimited on the graph shift operator $\mathbf{S}$ if its GFT has at most $K$ nonzero components. That is, $\mathbf{V}^{-1}\mathbf{x} = \tilde{\mathbf{x}} = \begin{bmatrix} \tilde{\mathbf{x}}_{K}^{T} & \mathbf{0}_{N-K}^{T} \end{bmatrix}^{T}$ where $\mathbf{0}_{N-K}$ is a vector of $N - K$ zeros. If this is the case then the original signal can be reconstructed from a sampled version of the complete signal. Define a $K \times N$ selection matrix $\mathbf{C}$ that samples $K$ of $N$ of the graph nodes. Observe that $\mathbf{C}$ is a selection matrix if it is binary, has exactly one nonzero entry per row and at most one nonzero entry per column. This sampled signal is defined as $\bar{\mathbf{x}} = \mathbf{C}\mathbf{x}$. The reconstruction can be carried out by

whenever a selection matrix $\mathbf{C}$ that satisfies $\operatorname{rank}\{\mathbf{C}\mathbf{V}_{K}\} = K$ is used.

The difficulties specific to this theory include generating a selection sampling matrix $\mathbf{C}$ that satisfies the above equation, working with a graph signal that is not cleanly bandlimited, and working with a graph shift operator that does not decompose to a full rank eigenvalue matrix.

The unexamined life is not worth living. - Socrates

This might make me the worst / most-SF person at every party, but my favorite question a few drinks in is to ask “what do you care about?”. The right time to ask is after two beers - first beer is to loosen everyone up, second is to get the honest answers. The first look I get is confused. People usually tilt their heads as if to say “Matt - we’re at a party that’s a stupid question”, god, family, football. But then I’ll press. Beyond your family and community and the obvious things that give life meaning, what matters to you, what do you believe? I think what shakes me is how few of us can put words to our beliefs or to things that we anchor to. In one way I get it - most people are entirely focused on the core things, because that’s how life falls. If you’re raising a kid, the kid is what matters - America’s competitiveness in rare earths matters too - but does it really? The thing I love about my friends is that to some degree everyone has sat down and come up with an answer - it may be a stupid and inane answer, it might just be a grift, but it’s an answer.