Contains Typescript and C# implementation of byte pair encoding (BPE) tokeniser for OpenAI LLMs, it is based on open sourced rust implementation in the OpenAI tiktoken.
Publisher’s description
What is Tokenizer?
Tokenizer is an open-source ai assistants & llm program for Windows from Microsoft Corporation. It is written in C#.
Installation and requirements
Official installers exist for 64-bit as Portable. It runs without installation (portable).
Popularity and file types
The source code on GitHub (microsoft/Tokenizer) has 212 stars. Similar programs include Codex CLI, Claude Code and ChatGPT.
Project description on GitHub: “Typescript and .NET implementation of BPE tokenizer for OpenAI LLMs.”
Pros / Cons
Pros
- Open source: the code can be inspected and built by anyone
- Portable: runs without installation
Cons
- No native ARM64 build
In short: Tokenizer is an open-source ai assistants & llm tool for 64-bit PCs.
Is Tokenizer safe to download?
- The file is served by github.com, the publisher’s own domain
- The download link uses HTTPS
- A SHA256 hash is published, so you can verify the file:
E0F039C6521FB7A4E3013386CD60ED64FB44AD9CAB74BBE7D744B60D1AFD5CE4· Open the VirusTotal report for this hash - The link has not been checked yet
- The source code is public
- ExeAtlas does not host, wrap or modify installers
Related guide: How to check a file’s SHA256 hash on Windows
Recent versions
All 1 versions ›| Version | Date | Size | Files |
|---|---|---|---|
| 1.3.3Latest | December 29, 2025 | — | 64-bit ZIP |
Install from the command line
Silent install ›winget install --id Microsoft.Tokenizer -eRelated guide: How to use winget: install and update apps
Information
- Publisher
- Microsoft Corporation
- Category
- AI assistants & LLMs
- Licence
- MIT
- Price
- Free
- Source code
- github.com/microsoft/Tokenizer
- Written in
- C#
- Requires
- Windows 11 / 10
- Architecture
- 64-bit
- Installer
- Portable
- Version
- 1.3.3
- Released
- December 29, 2025
- Added to the catalogue
- December 29, 2025
- Versions listed
- 1
- GitHub stars
- 212
- GitHub forks
- 36
- Website
- github.com
- winget ID
- Microsoft.Tokenizer
Ratings and reviews
No reviews yet. Be the first to rate it.
Alternatives to Tokenizer
See all ›-
Codex CLI0.161.0Codex CLI is an open‑source local coding agent that runs in your terminal, letting you write, edit, and understand code without leaving the command line.Open source16.6K ↓ -
Claude Code2.1.302Unleash Claude’s raw power directly in your terminal. Search million-line codebases instantly. Turn hours-long workflows into a single command. Your tools. Your workflow. Your codebase, evolving at thought speed.Proprietary67.1K ↓ -
Open source17.1K ↓ -
Open source8.7K ↓ -
Open source3.3K ↓ -
oh-my-pi18.12.6⌥ AI Coding agent for the terminal — hash-anchored edits, optimized tool harness, LSP, Python, browser, subagents, and moreOpen source654 ↓ -
Open source1.9K ↓ -
Open source14K ↓
Frequently asked questions
Is Tokenizer free?
Yes. Tokenizer is open-source software under the MIT licence.
Is Tokenizer safe to download?
The download comes from github.com, which belongs to the publisher; ExeAtlas does not host or change the file. Compare its SHA256 hash with the one shown here, or open the VirusTotal report for that hash.
How do I install Tokenizer with winget?
Open PowerShell or Windows Terminal and run the winget command shown on this page. winget downloads the official installer and checks its SHA256 hash before running it.
Does Tokenizer run on ARM (Windows on Arm)?
There is no ARM64 build; Windows 11 on Arm can run the 64-bit version through emulation.
Can I use Tokenizer without administrator rights?
Yes, it is portable: unpack and run it.
What are alternatives to Tokenizer?
Codex CLI, Claude Code, ChatGPT, GPT4All and Jan. The alternatives page compares them.
