TriTalk miniGPT

About

This is the second in a series of mini GPT experiments, inspired by Andrej Karpathy's writing on extremely small language models.

One of my hobbies is triathlon: swimming, cycling, and running. Increasingly, though, my interest in the sport includes a fourth discipline: talking about it on the triathlon forum tritalk.co.uk.

TriTalk has been running for years and has developed a distinctive flavour and style, while also capturing a great deal of information, opinion, and nonsense about the triathlon world.

I wanted to see whether a language model could be trained exclusively on posts from tritalk.co.uk, and nothing else.

The training corpus

Every language model needs to be trained on text. This model was trained on 75,759 forum posts, with 8,418 validation posts, from a deduplicated corpus of 84,177 public TriTalk posts. These were downloaded from the public forum data and organised into large human-readable text files with one post per line: about 20 MB of text.

The posts were deduplicated and then split at the post level into training and validation sets, so the model was evaluated on held-out posts rather than on fragments it had already seen.

The tokenizer

Each post is turned into a sequence of discrete tokens, which can then be mapped to numbers, embedded as vectors, and processed by the transformer.

A subword tokenizer is a useful middle ground between character-level tokenization and whole-word tokenization. It creates a medium-sized vocabulary that can represent common words compactly while still allowing unfamiliar terms to be built from smaller known pieces.

In this project, tokenization was implemented with a custom BPE (byte pair encoding) tokenizer. The tokenizer lowercased the forum text, split it into word-like units, and represented each word as a sequence of characters prefixed with a special space marker. It then repeatedly merged the most frequent adjacent symbol pairs, learning a vocabulary of common subword pieces such as prefixes, endings, and frequent whole tokens.

This process produced a 4,061-token subword vocabulary. Once the vocabulary and merge list had been learned, each training post was encoded into a sequence of subword token IDs, with special beginning-of-sequence and end-of-sequence markers added around each post. These token sequences were saved as PyTorch tensors, so the training code could reload ready-made numeric data directly without reprocessing the raw text each time.

The most common non-special subword tokens include ▁., ▁the, ▁,, ▁a, and ▁to. Mid-frequency tokens include cing, ▁blue, arr, ▁roth, and ▁adjust.

The transformer

The core of the model is a decoder-only transformer: a neural network designed to predict the next token in a sequence from the tokens that came before it.

Each token ID is first converted into a learned embedding vector. A separate positional embedding is then added so the model can distinguish between tokens appearing early or late in the sequence. These combined vectors are passed through a stack of transformer blocks, where the model repeatedly updates its internal representation of the text.

Inside each block, self-attention allows each token to look back at earlier tokens in the sequence and decide which ones are most relevant. This helps the model capture relationships such as sentence structure, related topics, and forum-style phrasing. After attention, a feed-forward neural network further transforms the representation before passing it to the next block.

In this browser version, the transformer has 6 layers, 6 attention heads, an embedding width of 192, and a context window of 128 tokens. At inference time, it generates text one token at a time, feeding each new token back into the model to continue the sequence.

Training and results

The model was trained to predict the next token in each sequence. During training, it repeatedly saw short excerpts from the corpus and adjusted its weights to become better at continuing them.

The final training run reached a validation loss of about 4.535, which corresponds to a perplexity of roughly 93. Perplexity is a rough measure of how uncertain the model is when predicting the next token: lower values indicate more confident, and usually more coherent, predictions.

In practice, the model works best when given prompt starters that sound like the beginning of a real forum post. Short, natural openings give it useful cues to continue from.

In the browser

The model was trained off-line in Python. The resulting checkpoint was then exported into static weight files that JavaScript can load directly from this site.

When you press generate, the browser tokenizes your prompt into subword pieces, runs the transformer layer by layer, samples the next token, appends it to the sequence, and then repeats the process.

No API calls are made during inference. Once the model has loaded, the entire generation loop runs locally in your browser.

Limitations

This is a small experimental model with a short context window and very limited reasoning ability. It can imitate some of the tone, topics, and phrasing of the training corpus, but it does not truly understand triathlon, physiology, or the wider world in the way a person does.

Its main strength is style imitation rather than factual reliability. It is better at producing something that sounds like the start of a forum reply than at giving accurate or useful advice. It may mash together half-remembered patterns from the training data, or drift away from the original prompt.

Because the model is small, it is also prone to repetition, abrupt topic shifts, odd phrasing, and occasionally getting stuck in familiar loops. Some subjects and recurring references from the forum appear disproportionately often, simply because they were easy statistical patterns for the model to learn.

The short context window also limits how much information it can keep in mind at once. It does not maintain a deep understanding of longer prompts, and it can lose track of details as generation continues. So while it can be amusing and sometimes surprisingly plausible, it is still a very lightweight toy language model rather than anything more useful.

That said, this language model has learned how to talk from first principles, and exclusively from looking at TriTalk forum posts. It knows basic rules of grammar, it knows you swim in water, it knows that shoes go on your feet. So I think it is a limited success.

Your welcome.