This post contains affiliate links. This site receives commissions upon sales generated from such links. Thank you for your support!
A Different Direction
While I was initially able to get the NVIDIA RTX A6000 working with some AI models using LocalAI through Docker, over time I began to see more and more errors from the GPU:
[ 120.018289] NVRM: GPU 0000:01:00.0: RmInitAdapter failed! (0x23:0xffff:1552)
[ 120.018323] NVRM: GPU 0000:01:00.0: rm_init_adapter failed, device minor number 0
Thinking that I may have better luck with Debian and the Ansible role used to deploy NVIDIA drivers on my other machines, I decided to abandon Arch and revert to Debian. Unfortunately, this did not solve my issues and I ended up seeing the same errors on the new OS.
I have been using the Tower Plus’ integrated graphics ever since. While I have been pleasantly surprised by the performance of the integrated unit, it isn’t a comparison to a dedicated GPU. I haven’t even attempted running AI models without one. Having tried the A6000 in several machines over several years, I decided that it was time to try something new. so I replaced it with an RTX 4090. With NVIDIA drivers already installed, I was expecting this GPU to work with my system out-of-the box.
By specs, the 4090 is a fairly significant downgrade from the A6000. VRAM suffers a devastating reduction from 48GB to 24GB, which will have a significant impact on the model size and context window capacities of the GPU. Comparing the same AI model across cards, however, the reduction in speed of token generation is not particularly noticeable; further, the 4090 is arguably better suited for gaming and video production applications, which makes it an overall better fit for this Tower Plus PC. Unfortunately, things did not go quite so smoothly. The 4090 requires 3x8pin PCIe power cables, and the Tower Plus only has two. Luckily, I was able to get a refund for the 4090 and purchase a 5060 Ti instead.
A fairly significant downgrade; however, for gaming and video rendering the 5060 is still quite a powerful unit. I additionally purchased a second Tesla T4 to be installed in my PowerEdge servers, so these will take over hosting LocalAI through Kubernetes (which is a better way to host the service anyway).
I have decided to continue using Debian instead of Arch primarily because I did not enjoy having to configure everything in Arch as much as I had expected. Customization is a rabbit hole and Debian gives me everything I want with significantly less effort. I have, however, installed the Tiling Shell estension for GNOME, as my brief time with qtile reminded me of its fluidity and functionality. I’ve found this simple extension to be more than sufficient for my needs.
https://extensions.gnome.org/extension/7065/tiling-shell/
Using the RTX 5060 causes GNOME to boot using Xorg rather than Wayland. I would generally be fine with using X11 if it weren’t for lack of support for touchpad gestures. I currently connect both screens to the integrated GPU and only send specific tasks to the 5060, and was surprised to see that Xorg was still being used in this configuration. I haven’t decided what to do about this just yet.