Would you accept a DeepSeek-OCR example? I have a complete, working MLX port and I'd like to contribute it, but I want to check fit before opening a PR.
The model (baidu/Unlimited-OCR): a document-OCR VLM — SAM ViT-B + CLIP ViT-L encoders, linear projector, and a 12-layer DeepSeek-V2 MoE LLM (64 routed experts top-6 + 2 shared, use_mla=False). It does free OCR, layout parsing with grounding boxes, high-res tiling, and multi-page PDF → markdown.
Code: https://github.com/vignesh-kumar-v/mlx-ocr-unlimited
State of the port:
- Pure MLX at inference (PyTorch only for one-time weight conversion, like the llava example).
- Verified against the reference CUDA implementation: 0.9998 output similarity in fp32 (0.99 in fp16) on a full 15-page arXiv PDF.
- ~175 tok/s decode, ~6 GB in fp16 on an M5 Pro (24 GB).
- Unit tests (MoE batched-vs-loop equivalence, gating) plus a weight-gated end-to-end test.
Some pieces may be interesting beyond this one model: a batched gather_mm MoE, a KV cache that pins the image/prompt tokens while rotating a 128-token decode window (the model's decode-time attention pattern), and position-embedding resizing that is bit-identical to torch.nn.functional.interpolate (bicubic+antialias) via cached separable resize matrices.
Fit question: I'm aware mlx-vlm has DeepSeek-OCR support. My thinking is that a self-contained, readable example belongs here in the same spirit as llava (there's currently no OCR/document-parsing example), but if you'd rather point this elsewhere I'd like to know before I open the PR. If there's interest, I'll restructure to the flat llava-style layout (generate.py / model files / test.py / README.md / requirements.txt), black-formatted per CONTRIBUTING.md, and upload converted weights to mlx-community (per #155).
Would you accept a DeepSeek-OCR example? I have a complete, working MLX port and I'd like to contribute it, but I want to check fit before opening a PR.
The model (
baidu/Unlimited-OCR): a document-OCR VLM — SAM ViT-B + CLIP ViT-L encoders, linear projector, and a 12-layer DeepSeek-V2 MoE LLM (64 routed experts top-6 + 2 shared,use_mla=False). It does free OCR, layout parsing with grounding boxes, high-res tiling, and multi-page PDF → markdown.Code: https://github.com/vignesh-kumar-v/mlx-ocr-unlimited
State of the port:
Some pieces may be interesting beyond this one model: a batched
gather_mmMoE, a KV cache that pins the image/prompt tokens while rotating a 128-token decode window (the model's decode-time attention pattern), and position-embedding resizing that is bit-identical totorch.nn.functional.interpolate(bicubic+antialias) via cached separable resize matrices.Fit question: I'm aware mlx-vlm has DeepSeek-OCR support. My thinking is that a self-contained, readable example belongs here in the same spirit as
llava(there's currently no OCR/document-parsing example), but if you'd rather point this elsewhere I'd like to know before I open the PR. If there's interest, I'll restructure to the flat llava-style layout (generate.py/ model files /test.py/README.md/requirements.txt), black-formatted per CONTRIBUTING.md, and upload converted weights tomlx-community(per #155).