Skip to content

Pull requests: openai/tiktoken

New pull request New
30 Open 100 Closed
30 Open 100 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Assigned to nobody Loading
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

Loosened Encoding.encode*_batch's input types
#455 opened Oct 9, 2025 by jamesbraza Loading…
Refactor encoding name resolution to improve model prefix matching logic
#452 opened Sep 30, 2025 by hrishabhn Loading…
Optimize _byte_pair_merge to o(m log n) using heap-based candidate selection
#442 opened Aug 27, 2025 by ArchishmanSengupta Loading…
Swift Binding Generation via UniFFI
#431 opened Aug 14, 2025 by narner Loading…
Update license field to use proper SPDX identifier
#420 opened Aug 1, 2025 by leifdreizler Loading…
Add support for building TikToken WoA
#412 opened Jul 4, 2025 by Greenie0701 Loading…
Update load.py
#399 opened Apr 23, 2025 by Shivam-Singh-Dev Loading…
Make decoder and sorted_token_bytes re-use existing memory - Unit Tests
#384 opened Feb 19, 2025 by CodeBeaverAI Loading…
In-code documentation update
#383 opened Feb 17, 2025 by louisbrulenaudet Loading…
1
Option to use TIKTOKEN_BPE_HOST environment variable for configurable BPE host URL
#357 opened Nov 4, 2024 by AitBaali-Hamza-bcg Loading…
2
make decoder and sorted_token_bytes re-use existing memory
#352 opened Oct 17, 2024 by tmm1 Loading…
use HashSet from rustc-hash too
#351 opened Oct 17, 2024 by tmm1 Loading…
Uses Regex instead of fancy-regex - 6x speedup
#331 opened Aug 5, 2024 by Majdoddin Loading…
3
Cache for Encoding - Runtime Boosted by 12%
#319 opened Jul 10, 2024 by Majdoddin Loading…
Add Terminal-Based Visualization Tool for Tokenized Data Points in Tiktoken Tokenizer
#314 opened Jun 18, 2024 by LVivona Loading…
1
Remove dependency on requests
#271 opened Mar 26, 2024 by tristan-jl Loading…
1
Optimize byte pair merge for really big tokens (40x faster for a 2500 token word)
#239 opened Jan 15, 2024 by l0rinc Loading…
26
Support enterprise network support for self-hosted encodings (UPDATED v0.6.0)
#238 opened Jan 12, 2024 by blaney83 Loading…
1
0) Add the jtokkit test suite examples to validate the cl100k_base, p50k_base & r50k_base encodings
#237 opened Jan 6, 2024 by l0rinc Loading…
1
feat: Add option TIKTOKEN_FORCE_CACHE
#170 opened Jul 28, 2023 by Kontrolix Loading…
4
✨ feat: Split code rust, for run as python lib and rust lib
#167 opened Jul 18, 2023 by Miuler Loading…
1
5
fix pycache folder being added instead of encoding info files
#152 opened Jun 23, 2023 by dheerajiiitv Loading…
1
9
Add support for fine-tuned models in encoding_for_model
#135 opened May 23, 2023 by thespino Loading…
2
Option to use env variable for BPE_FILE location
#128 opened May 15, 2023 by csnelsonchu Loading…
update license metadata to use SPDX identifier as per recommendation
#125 opened May 12, 2023 by imrehg Loading…
ProTip! Updated in the last three days: updated:>2025-10-07.