another AI question

[Edward Capriolo]

On Wed, Apr 8, 2026 at 12:17 PM Martin Cracauer <cracauer at cons.org> wrote:

> Martin Cracauer wrote on Tue, Apr 07, 2026 at 05:50:21PM -0400:
> > The situation with LLMs on FreeBSD is not totally catastrophic.
> >
> > The NVidia drivers are currently broken on my 5090, so I cannot
> > compare Vulkan/FreeBSD to Linux/Cuda.
>
> Made them work, you need loader.conf
> hw.nvidia.registry.EnableGpuFirmware=17
>
> Performance on bartowski/Qwen_Qwen3.5-27B-GGUF:Q6_K_L in llama.cpp is:
> - FreeBSD Vulkan 49 tokens/second
> - Linux CUDA 56 tokens/second
>
> Will get Linux/Vulkan numbers when I have a chance.
>
> But this is encouraging.  Windows was also 10% slower than Linux.
>
> Martin
>
> > But they work on my 2080ti with Vulkan and run both ollama and
> > llama.cpp, accelerated.
> >
> > On my laptop with "AMD Ryzen 7 PRO 4750U with Radeon Graphics" also
> > runs Vulkan and accelerates ollama (although only by a factor of 3
> > compared to CPU).  This combo does not run llama.cpp
> >
> > Now that NVidia drivers are running on at least one of my cards I'll
> > give it another go to run CUDA through Linuxulator.
>
> That go failed.  No CUDA on Linuxulator still.
>
> Martin
> --
> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
> Martin Cracauer <cracauer at cons.org>   http://www.cons.org/cracauer/


A while back I did mention I wanted to do a port but it slipped my mind.
Here you go: https://github.com/freebsd/freebsd-ports/pull/579

To get a taste for it:
${LOCALBASE}/bin/curl -fsS --no-buffer -H 'Content-Type: application/json' \
-d '{"model":"Qwen3-0.6B-JQ4","stream":true,"max_tokens":384,"chat_template_kwargs":{"enable_thinking":false},"messages":[{"role":"user","content":"Deliverance
is an AI Inference engine that runs on Java and can leverage native code
for optimizations. Tell me why FreeBSD is a great operating system to run
Deliverance."}]}' \
FreeBSD is a great operating system to run Deliverance, a powerful AI
inference engine, for several reasons:\n\n1. **Open-Source and
Community-Driven**: \n FreeBSD is an open-source operating system, which
means it's free for all users. This is a significant advantage for
developers who want to run Deliverance without additional licensing costs.
The community is active and supportive, which means there's a strong
support system for developers and users alike.\n\n2. **High Performance and
Scalability**: \n FreeBSD is known for high performance and scalability.
This makes it ideal for running large-scale AI inference systems like
Deliverance, which can handle high computational demands. The operating
system is optimized for speed and efficiency, which is crucial for
real-time applications.\n\n3. **Cross-Platform Compatibility**: \n FreeBSD
is a Unix-like operating system, which means it works well on a wide range
of hardware, including desktops, servers, and embedded systems. This makes
it a great choice for developers who want to run Deliverance on various
platforms.\n\n4. **Customization and Flexibility**: \n FreeBSD is highly
customizable, allowing users to tailor the system to their specific needs.
This flexibility is important for developers who want to build their own AI
inference engines or integrate Deliverance with existing systems.\n\n5.
**Security and Reliability**: \n FreeBSD is known for strong security
features, which is essential for an AI inference engine that may be used in
production environments. The operating system is secure and reliable,
ensuring that Deliverance can be deployed safely and efficiently.\n\nIn
summary, FreeBSD is a great choice for delivering AI inference engines like
Deliverance due to its open-source nature, high performance, cross-platform
compatibility, and strong security features.

The way this is built I spend the majority of my time playing with small
models. The output above is from the QWEN 3 0.6B. I think you get about 100
tokens per second (with that model and moderate hardware like a 16GB mac).
I run 4B quantized on the same gear. On some modest servers with ~32 RAM i
run mixtral (quantized) as it is mixture of experts each token is only
using an "experts" worth of the weights.

Java 25 has support for panama which is SIMD, yet portable. Interestingly
that allows you to mix-and-match and even run an ARM based SIMD pipeline on
an X86. You wouldn't want to do that but it is nice for testing. There are
two levels to the native library support. The first is SIMD not in Panama
but hand rolled c-code.For some sets of operations the memory bandwidth on
CPU is the biggest factory and the native will not yield huge benefits over
Panama, in other cases it does.

The second level of native is GPU. This is using web GPU/dawn. This hasn't
been a huge focus but some parts of the decode pipeline use the GPU. With
dawn you dont need say 4 GB of cuda libs then 4 more GB blast libs, somehow
a 9MB library lets you build shaders.

The way the project is structured it is #writeonceinferenceanyware the port
uses pure Panama, and you can build the other stuff if you are feeling up
to the task. You can have fun reading all of
https://www.reddit.com/r/freebsd/comments/11nqzo0/nvidia_gpu_on_freebsd/

So that is the hook, give the port a try. The longest part is probably
downloading the model.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.nycbug.org:8443/pipermail/talk/attachments/20260810/661f5baf/attachment-0001.htm>