← Back

Repo-conditioned code prediction with local Markov models

Published: 2026-04-06

Before transformers and LLMs became popular, I think like a decade ago, there were a handful of viral posts of people using "AI" to generate TV show and movie scripts. Most popular among these was The Office. I even played around with it myself. Making scripts of some of my favorite shows from scripts I found online. It really seemed like magic to me, as a high school student learning software development, to give a small script a bunch of text and have it generate bits and situations that sounded a lot like Dwight Schrute or Michael Scott.

It sounded really cool, but just like LLMs now, it's just math. Take all the words, make a hash of them, and given some starting words you can generate a bunch of other words after them. They're called Markov chains. The viral moment came and went, then a few years later transformers and LLMs became common place.

With coding agents becoming popular, I came back to the idea, maybe Markov chains could write code. I sort of knew it wouldn't be very good, but if codex could show me how well it would do in a night, I'd be happy. This page showcases some of those against popular repos. A python, rails, and typescript repo. I had a hunch that typescript wouldn't be very good, but maybe python and rails could be decent since they sometimes have a tendency of "natural language".

Anyways, there's a couple below with some deeper explanation of what they do.

What to try

Pick a repo, click one of the example beginnings, and see where it goes. The left side shows a longer continuation while the right side shows the individual words or symbols it thinks are most likely to come next. You can also type your own beginning into the box.

I think the most interesting way to play with it is to give multiple repos the same beginning. Each model has only seen one codebase, so even a plain word like render or async can send them in very different directions. Turn up the continuation length if you want to give one enough rope to get properly weird.

The three codebases

• Mastodon is the Rails example. Try its controller-flavored beginnings like current_ and before_action.
• Frigate is the Python example. Its suggestions tend to orbit things like self., functions, cameras, and configuration.
• pi-mono is the TypeScript example. Starting with const, await, or async gives it a lot of possible paths to choose from.

What the demo is actually showing

This is not an LLM reasoning about the code. It is looking at short patterns that appeared in a repo and choosing a likely next piece. Sometimes that is enough to stumble into something recognizable. Other times it confidently turns into punctuation soup. Both outcomes are kind of the point.

The “top suggestions” slider changes how many possible next pieces the demo considers. Keeping it low makes the result more predictable and repetitive. Raising it allows stranger choices. “Continuation length” controls how far the demo keeps going after your starting text.

A second try with more Mastodon context

The first demo treats every repo as a stream of small code patterns. For the second one, I wanted to see what would happen if the Mastodon version paid extra attention to the names and relationships that make a Rails app feel like a Rails app.

Try current_account, before_action, and redirect_to, then compare the results with Mastodon in the first demo. It is still guessing from nearby patterns, but familiar project names get a louder voice in the guess. The question is not whether it suddenly writes good code. It is whether the nonsense becomes more recognizably Mastodon-flavored.

Play around with the continuation length on this one, too. Keep the starting text the same and move the slider back and forth. The shorter and longer settings do not always give you the same opening followed by more code; changing the length can change the path it takes from the beginning.

So, can a Markov chain write code?

Not especially well. It can imitate the surface of a codebase for a few steps, especially when the beginning is something the repo repeats often. It has no idea what the program is trying to do, though, and the longer it runs, the easier that is to see.

Still, I like how clearly the demos pick up a repo's vocabulary. Mastodon, Frigate, and pi-mono each leave a different fingerprint on the same tiny predictor. It is less like having an agent write code and more like hearing a codebase talk in its sleep.