20.2 Tokenization and Tensors

20.1 placed weights and tensors on the native runtime’s side of the boundary, with Go passing only handles and small data across it. This section examines that small data itself: how text becomes the integers a model consumes and how integers become text again, what a tensor looks like in memory, and how quantization changes a tensor. The point most worth a Go programmer’s attention is that mainstream tokenizers work on UTF-8 bytes, which matches the string model discussed in 5.2, and that streaming output is correct only if the boundaries of UTF-8 encodings are handled properly.

20.2.1 Tokenization: The Translation Layer Between Text and Integers

A model does not process text directly. Its inputs and outputs are integers, the numbers of tokens in a vocabulary. The component that converts back and forth between text and these integers is called a tokenizer. It does two inverse things: it splits an input string into a sequence of token IDs (encoding), and it joins the token IDs generated by the model back into a string (decoding).

Most modern LLM tokenizers are based on byte pair encoding (BPE) or a variant of it. BPE was originally a data compression algorithm: repeatedly replace the most frequent pair of bytes in a sequence with a byte that does not occur in it [Gage, 1994]. Sennrich et al. adapted it into a method for segmenting words: instead of merging byte pairs, it merges characters or character sequences. The procedure is as follows. Initialize the symbol vocabulary with the character set, represent each word as a sequence of characters with an end-of-word symbol appended, then repeatedly count all adjacent symbol pairs and replace every occurrence of the most frequent pair (A, B) with a new symbol AB. The final vocabulary size equals the size of the initial character set plus the number of merge operations, and the number of merges is the algorithm’s only hyperparameter. For efficiency, pairs that span word boundaries are not counted, so the algorithm can run directly on a dictionary of words weighted by frequency [Sennrich et al., 2016].

The paper gives a Python implementation of about ten lines. Here is a Go version, with the same dictionary and the same number of merges as the paper:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
package main

import (
	"fmt"
	"strings"
)

type pair struct{ a, b string }

func main() {
	// The example dictionary from the paper: words split into characters, </w> marks the end of a word.
	words := []string{"l o w </w>", "l o w e r </w>", "n e w e s t </w>", "w i d e s t </w>"}
	freq := []int{5, 2, 6, 3}

	var merges []string
	for range 10 {
		// Count the weighted frequency of adjacent pairs, remembering first-seen order so ties go to the earlier pair.
		count := map[pair]int{}
		var order []pair
		for i, w := range words {
			s := strings.Fields(w)
			for j := 0; j+1 < len(s); j++ {
				p := pair{s[j], s[j+1]}
				if _, ok := count[p]; !ok {
					order = append(order, p)
				}
				count[p] += freq[i]
			}
		}
		best := order[0]
		for _, p := range order {
			if count[p] > count[best] {
				best = p
			}
		}
		merges = append(merges, best.a+best.b)

		// Replace every occurrence of this pair in each word with the merged symbol.
		for i, w := range words {
			s := strings.Fields(w)
			var out []string
			for j := 0; j < len(s); j++ {
				if j+1 < len(s) && s[j] == best.a && s[j+1] == best.b {
					out = append(out, best.a+best.b)
					j++
					continue
				}
				out = append(out, s[j])
			}
			words[i] = strings.Join(out, " ")
		}
	}
	fmt.Println(strings.Join(merges, "  "))
	for _, w := range words {
		fmt.Println(w)
	}
}

es  est  est</w>  lo  low  ne  new  newest</w>  low</w>  wi
low</w>
low e r </w>
newest</w>
wi d est</w>

The first line lists the new symbols produced by the ten merges in order; the other four lines show each word after merging. The suffix est</w> shared by newest and widest became one symbol within the first three steps; low and newest became complete symbols, while the rarer lower is still assembled from low, e, r and the end-of-word symbol. This is the property of BPE: common words become a single token, rare words split into subwords, the vocabulary size stays controllable, and any input can be represented. The merge order matches the output of the paper’s Python program. One detail is specific to Go: Python’s max returns the key that appears first in the dictionary when frequencies tie, whereas the Go specification states that the iteration order of a map is unspecified and not guaranteed to be the same between two iterations, so the program records the order in which pairs first appear in the slice order to make the result deterministic.

Training produces an ordered merge table. To encode, the tokenizer splits a word into initial symbols and repeatedly merges the adjacent pair with the highest rank in the merge table, until nothing can be merged. Forma’s tokenizer implements this rule, and a comment explains an easy mistake: when the same pair occurs at overlapping adjacent positions (for example the two ("e","e") in "eee"), the leftmost occurrence must be taken. Writing the condition as <= takes the rightmost one, which looks equally consistent with “take the lowest rank” but makes about one in every thirty short strings come out with different IDs [Ou, 2026].

Besides BPE, another common implementation is SentencePiece. Most existing subword tools assume the input has already been split into words by whitespace, which does not hold for languages such as Chinese, Japanese and Korean that do not separate words with spaces. SentencePiece trains directly on raw sentences and treats whitespace as an ordinary symbol, replacing it with the meta symbol ▁ (U+2581), so that segmentation can be undone losslessly. It implements two segmentation algorithms, BPE and the unigram language model [Kudo and Richardson, 2018].

20.2.2 Why Bytes

Mainstream BPE today is byte-level: the smallest unit is not a Unicode code point but a UTF-8 byte. The GPT-2 paper explains why. The BPE in the reference implementations is called “byte pair” but mostly operates on Unicode code points; to represent all Unicode strings, the base vocabulary would have to contain every Unicode symbol, which is over 130,000 entries before any multi-symbol token is added, while BPE typically uses vocabularies of only 32,000 to 64,000 entries. Byte-level BPE needs a base vocabulary of only 256. Running BPE directly on byte sequences produces suboptimal merges (for example dog., dog! and dog? each taking a token), so GPT-2 forbids merges across character categories, with an exception only for spaces [Radford et al., 2019].

GPT-2’s encoder also applies a mapping: it maps the 256 byte values one-to-one onto printable code points, so that the vocabulary becomes a set of ordinary strings. The 188 bytes that are printable ASCII or Latin-1 map to themselves, and the remaining 68 map, in byte order, to code points starting at U+0100 [OpenAI, 2019]. The space 0x20 is the 33rd of these and maps to U+0120, that is Ġ, which is why Ġ appears everywhere in many vocabularies. Forma implements the same mapping and stresses that it is a bijection on bytes, so decoding is defined for any byte sequence, including invalid UTF-8. When mapping bytes to code points it processes the string byte by byte, and a comment points out that it must not go through a []rune conversion, which would replace invalid UTF-8 with U+FFFD and break the bijection [Ou, 2026].

This is fully consistent with the string model discussed in Chapter 5. 5.2 pointed out that a Go string is an immutable sequence of bytes, that range yields code points and that indexing yields bytes [Pike, 2013]. A byte-level tokenizer handles exactly the bytes underneath a string, with no need for any abstraction of a “character” in between.

But the byte-level design brings a problem: token boundaries do not necessarily fall on UTF-8 encoding boundaries. A Chinese character takes 3 bytes in UTF-8 and an emoji usually takes 4, and byte-level BPE may well place those bytes in two adjacent tokens. For encoding this does not matter; for decoding token by token it causes errors. The token the model emits first may contain the bytes of only half a character, which is not yet a complete code point; the rest of the bytes must wait for the next token before the complete character can be assembled. If every token received is immediately converted to a string and sent, the half character is displayed as the replacement character U+FFFD (�). The correct approach is to keep a byte buffer, emit only the encodings that are complete, and leave the incomplete bytes at the end for the next token:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
package main

import (
	"fmt"
	"unicode/utf8"
)

// Decoder accumulates the bytes of tokens and emits only complete UTF-8
// encodings; an incomplete tail is held back until the next token arrives.
type Decoder struct{ buf []byte }

func (d *Decoder) Push(tok []byte) string {
	d.buf = append(d.buf, tok...)
	n := 0
	for n < len(d.buf) && utf8.FullRune(d.buf[n:]) {
		_, size := utf8.DecodeRune(d.buf[n:])
		n += size
	}
	out := string(d.buf[:n])
	d.buf = d.buf[:copy(d.buf, d.buf[n:])]
	return out
}

func main() {
	text := []byte("Go€→🙂")
	// A hypothetical split: the encodings of "€", "→" and "🙂" are all cut in the middle.
	toks := [][]byte{text[:3], text[3:7], text[7:10], text[10:]}

	var d Decoder
	all := ""
	for _, t := range toks {
		naive := string([]rune(string(t))) // invalid bytes become U+FFFD under the range rules
		out := d.Push(t)
		all += out
		fmt.Printf("%-11s  naive: %s  buffered: %s\n", fmt.Sprintf("% x", t), naive, out)
	}
	fmt.Println("joined: " + all)
}

47 6f e2     naive: Go�  buffered: Go
82 ac e2 86  naive: ����  buffered: €
92 f0 9f     naive: ���  buffered: →
99 82        naive: ��  buffered: 🙂
joined: Go€→🙂

The first token ends with e2, the first byte of “€”. Direct conversion gives Go�, while with buffering only Go is emitted and e2 is held back. The second token brings the last two bytes of “€”, so Decoder emits the complete “€” and holds back the first two bytes of “→”. In the naive column, every invalid byte became one U+FFFD: this is what for range does on invalid UTF-8, and a []rune conversion follows the same rule.

The key to the decision is utf8.FullRune. Its documentation says that FullRune reports whether the bytes in p begin with a full UTF-8 encoding, and that an invalid encoding is considered a full rune since it will convert as a width-1 error rune. So it returns false only for an encoding that is valid but truncated: truly invalid bytes are emitted immediately, since waiting longer will not make them valid, and only a tail that the next token might complete is held back. Production code uses the same test. Forma’s tokenizer.Decoder uses utf8.FullRune to compute the length of the prefix that can be emitted, and if bytes remain when the stream ends it emits one U+FFFD to show that the stream ended in the middle of a character; it also keeps this buffer separate from the one needed for stop-word matching, because the former waits for bytes to complete and the latter waits for text to match, and sharing one buffer would make both judgments wrong [Ou, 2026]. The MLX runner of Ollama has a function named flushValidUTF8Prefix that does the same thing [Ollama, 2026]. Go defines a string as a sequence of bytes and provides tools for incomplete encodings in unicode/utf8 [Go Authors, n.d.b], and streaming decoding needs exactly these two things.

20.2.3 Tensors: Flat Memory Beneath the Shape

Once token IDs enter the model they become tensors. Taken apart, a tensor is just three things: a flat contiguous block of memory, a shape, and a data type (dtype). An input tensor of shape [batch, seq] and type int32 is, underneath, one contiguous int32 array of length batch × seq, which in Go is a []int32.

1
2
3
4
5
A tensor of shape [2, 3], laid out as flat memory in row-major order:

  logical: | a b c |      flat: [ a b c d e f ]
           | d e f |
  strides [3, 1]: element (i, j) is at offset 3i + j

The stride of each dimension equals the product of the lengths of the dimensions after it. Operations such as transposing and slicing often change only the shape and strides without moving data, which is why an inference runtime can transform tensors efficiently over the same block of memory.

Handing a Go []int32 to a runtime in the same process means passing the pointer to the first element together with the length and shape; this is the same matter as the zero-copy discussed in 5.2 and the pointers crossing the boundary discussed in 18.3. The cgo pointer-passing rules apply here as well (see 15.6). The cmd/cgo documentation states [Go Authors, n.d.a]:

1
2
3
C code may not keep a copy of a Go pointer after the call returns,
unless the memory it points to is pinned with [runtime.Pinner] and the
Pinner is not unpinned while the Go pointer is stored in C memory.

Go memory passed to C is implicitly pinned only for the duration of the call. If the runtime uses the memory asynchronously, for example by placing it on a batch queue and reading it later, it must be pinned with runtime.Pinner until the runtime is done with it, or a runtime-allocated native buffer must be used instead. If the runtime is in another process (see 20.1.4), none of these rules apply, at the cost of serializing tokens across the process boundary.

20.2.4 Quantization: Tensors Return to Bytes

Model weights are originally stored as 32-bit floating-point numbers (float32). To fit into less memory and compute faster, inference commonly uses quantization: compressing the weights to 16-bit floats (float16/bfloat16), 8-bit integers, or even 4 bits. A quantized tensor is, underneath, a compact run of bytes plus the scale factors needed for dequantization.

Take Q4_0, the simplest 4-bit format in GGUF. Every 32 weights form a block that shares one float16 scale factor d, plus 16 bytes holding 32 4-bit integers; each block takes 18 bytes, an average of 4.5 bits per weight. When quantizing, d is the value with the largest absolute value in the block divided by −8. When dequantizing, each 4-bit integer q is restored as $(q - 8) \times d$ ; the low 4 bits of each byte correspond to the first 16 elements of the block and the high 4 bits to the last 16 [ggml-org, 2026]. This layout shows that the tensor is no longer a []float32 but a run of bytes that must be interpreted block by block, which is one more reason 20.1 stressed keeping the weights on the native side: only the runtime knows how to interpret them. Forma’s estimate is 2 bytes per parameter at f16, about 1.06 bytes at int8 and about 0.53 bytes at int4, where the excess over the integer width comes from the block-shared scale factors (for int8, one f16 scale factor per 32 weights); it chooses int4 automatically only when int8 does not fit, and when int4 is chosen the embedding table is still kept at int8 [Ou, 2026].

Quantization brings the inner loop back to integer arithmetic, which connects to the SIMD discussed in 18.4 and 19.3. The dot product of int8 values is exactly the kind of computation CPU SIMD is good at. Under GOEXPERIMENT=simd, Go 1.26 provides the simd/archsimd package, which exposes architecture-specific vector types; it currently supports only AMD64 and is not covered by the Go 1 compatibility promise. Go 1.27 adds a portable simd package, which provides vector types for each numeric type (such as Int8s and Float32s) and ternary operations such as Float32s.MulAdd, and likewise requires the experiment flag. For implementing the inner loop of quantized inference on the CPU in pure Go, these two packages supply the basic building blocks.

Summary

The tokenizer is the translation layer between text and the model’s integers. BPE originates in a data compression algorithm, which Sennrich et al. adapted into a subword segmentation method; GPT-2 pushed it down to the byte level, so the base vocabulary needs only 256 entries, fully consistent with Chapter 5’s “a string is an immutable sequence of bytes”. The price of the byte-level design is that token boundaries need not align with UTF-8 encoding boundaries, so streaming decoding must keep a byte buffer and use utf8.FullRune to emit only complete encodings. A tensor is a combination of flat memory, a shape and a data type; when passed to a runtime in the same process it follows the cgo pointer rules, and when it must be held asynchronously, runtime.Pinner or a native buffer is used. Quantization compresses tensors back into bytes interpreted block by block, and hands the inner loop back to SIMD.

The data flow of a single request is now clear, but a service must handle thousands of such flows at once. 20.3 returns to the concurrency Go does well, examining how batching amortizes device cost, how tokens stream back to clients, and how channels and context handle backpressure and cancellation.