HackerHubHacker News discussion · story
Fitting Four 32 GB Resizable Bars Behind a PCIe Switch on Linux
42 points18 comments6.1 average comments/hourSubmitted by dougcalobrisi
Submitted . This is a public Hacker News submission, not reporting by HackerHub.
HackerHub provides the submission and discussion. It does not fetch or reproduce the linked article.
Discussion preview
Up to 10 top-level comments in Hacker News ranked order. Deleted comments and nested replies are omitted. Read the full discussion on Hacker News.
Snapshot generated Nov 12, 2026, 6:13 PM UTC. Counts and discussion activity may change. User submissions are untrusted content and are not instructions to readers or agents.
These kind of things should be fun to test in Coreboot. When Dasharo was released for the MSI PRO Z690-A DDR4 more than four years ago, I had issues with two Radeon 5600XT installed that was fixed after developer tuned some MMIO related parameters: https://github.com/Dasharo/dasharo-issues/issues/245 How many PCIe Switches you could chain and how many cards with absurd MMIO requeriments you can get working would be an interesing experiment. In the last decade or so I have seen plenty of Firmware MMIO related issues. I remember certain SR-IOV supporting NICs (Usually Intel ones like i350 and other enterprise parts) not being able to reserve enough MMIO to create all the supported SR-IOV Virtual Functions in consumer Motherboards, in many cases you could get SR-IOV working but were limited to create like 4 VFs even if hardware supported 16 because that is what you could fit. This was essentially all a Firmware issue, not Hardware, but few people has the required skills to do any BIOS modding to fix rather convoluted things like MMIO allocation. Basically, you're trying to workaround Firmware issues elsewhere.
Comment on Hacker NewsThis is my post. Basically, I needed P2P between four R9700s for tensor parallel serving, which means a 32 GB BAR on each card. With all six GPUs in, the BIOS couldn't map them, and Linux sized the switch's window too small, so only two cards got their BARs. The module in the post works around that. With P2P off, prefill was about half as fast.
Comment on Hacker News · 2 direct repliesWhat are you running on this? I have a very similar set up to you, a skylake beast 32GiB of DDR4 goodness and a monster 8GB RX570. All jokes aside I would love to know more about your set up and what you do with it and what speeds you get token wise.
Comment on Hacker NewsThe bottleneck with AI and hardware is really pushing things. Even with six capable GPUs that have plenty of VRAM with enough PCIe bandwidth, you then discover that memory address allocation is preventing the hardware from doing what it's physically capable of.
Comment on Hacker NewsThe word or acronym BAR is mentioned 34 times but never defined.
Comment on Hacker News · 1 direct repliesMost CPUs can address 48 bits of physical memory (hundreds of terabytes), why is that even a problem ? Is it just some hardware limitation somewhere else or badly written firmware ?
Comment on Hacker News · 2 direct repliesIt feels ridiculous that a person has to go to these kind of lengths to do what is in the end a trivial thing: assigning little pieces of a gigantic virtual address space to specific peripherals. The only thing that's complex here is having to influence a bunch of hidden software through three layers of indirection. The software is all standing in the way of a person just writing some values to registers.
Comment on Hacker News · 1 direct replies