GigaToken: ~1000x faster Language model tokenization

160 points

by syrusakbary

2 hours ago

27 comments