Abstract: In this paper, we introduce an Optimized Byte Pair Encoding (OBPE) tokenizer where the algorithm is optimized for the South African languages, including Sesotho, Setswana, Xhosa, Xitsonga, ...

Add a description, image, and links to the byte-pair-encoding topic page so that developers can more easily learn about it.

We present the B-spline Encoded Action Sequence Tokenizer (BEAST), a novel action tokenizer that encodes action sequences into compact discrete or continuous tokens using B-splines. In contrast to ...