Getting Dwarfstar serving GLM 5.3 Flash over the network
The first thing we wanted to try on the Mac Studio was to spin up GLM 5.3 Flash, a model we've had some good success with in the cloud.
@antirez built a custom inference engine for DeepSeek and that has now been extended to include GLM 5.3 Flash. Why a custom inference engine when there's a number of standardised ones? Those standardised engines will run a wide variety of models, not just the handful that Dwarfstar supports. Two words: efficiency and speed. A custom engine that's tuned to specific models is able to eke out performance that more general approaches cannot.
So we installed Dwarfstar.
After setup we ran /download_model.sh glm53-q4 - this downloaded the 4-bit quant of GLM 5.3 Flash.
Quantization is a method for compressing a model, so that it's smaller and will use less memory. Models are usually created with 16-bit floating point precision, but this makes them very large and a lot of that precision has a minimal impact on the model's accuracy. By compressing to 8, 4, or 2 bit precision you can create a dramatically smaller model that uses a lot less memory. 4-bit quants are often regarded as a good compromise between accuracy and size. In this case, the 4-bit quant of GLM 5.3 Flash fits very easily into the 256GB memory on the Mac Studio, so was the natural choice.
We wanted the model to be accessible over the network, not just on the local machine. A bit of googling around and the command line option --host 0.0.0.0 makes the server listen on the network, so ./ds4-server --ctx 100000 --kv-disk-dir /tmp/ds4-kv --host 0.0.0.0 got the server up and running.
A quick ping from another machine with curl http://192.168.8.108:8000/v1/models brought back a json document showing that we have GLM 5.3 Flash ready to receive requests.
Interestingly, the response gave back three models, not the expected one. There's one model name that has thinking on, one with it off and one where you can set thinking via a parameter. This is useful in situations where the client can't set the thinking parameter - so you can just choose the appropriate model.
Another quick curl, this time:
curl http://192.168.8.108:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "glm-5.3-flash-chat",
"messages": [{"role": "user", "content": "Explain Redis streams in one paragraph."}]
}'
This asked a question of the model and a response was received. The server logs suggested we're getting 36 tokens/second. That's a perfectly usable speed. Not super-fast, but the M5 Ultra processor has only been in developers hands for a few days at the time of writing and nothing has been optimised for it yet - it's probable there are speed-ups yet to be achieved.
People read at roughly 4–8 tokens per second, so 36 tok/s is several times faster than a person can follow as the text is written. But in practice that speed would feel slow. GPT-6 Astra averages around 37 tok/s in ChatGPT, so we're at parity with a top cloud model — but running on a Mac Studio. That's a good start!
I then tried the 4-bit quant of Qwen 3.8 Flash Next - that got me 60 tokens/second.
This does highlight the trade-offs involved in local models. In theory we could have waited for the 512GB Mac Studio to become available (but it'll likely cost in the region of £20,000, so not sure the budget would have stretched!), so we could run larger models - e.g. the unquantified version of GLM 5.3 Flash. But a larger model will be a lot slower. Right now, 256GB of memory feels about the right compromise between model size and speed for the M5 Ultra.
Anyway, we now have a model we know works well running at a respectable speed on our Mac Studio, serving over the local network. Sovereign AI is real at Barnacle Labs!
