homepage - blogs - devices

macOS vfio KVM under Debian

MAKE SURE IOMMU AND VT-D ARE ENABLED IN YOUR BIOS

this was from like 12am to 6am, no idea.

First, you need a 2nd gpu. i'm going with this AMD Radeon 520

Take the case off your computer

Put in the GPU

If your computer posts, congrats you now have a 2nd gpu in your computer.

Boot into Linux

Rememeber that you installed wmaker and broke everything because X11 hates 2 monitors with 2 diffirent aspect ratios and/or reseloution.

Install xfce4.

Start WDM, xfce4 wont show. (also its on the WRONG MONITOR -- and sideways.)

Finally get into xfce4 via startx command in linux.

Fix audio issues because creative doesn't wana make soundblaster drivers for linux.

figure out why THE 2ND GPU WONT EVEN BE DETECTED

Turn off the computer, reseat the GPU and boot.

ok, now we can begin the macOS KVM guide.. sorry i was going insane there.

lets get the easy part out of the way, the macOS VM.

follow this - https://neocities.org/site_files/text_editor?filename=blogs%2Fblog5%2Fblog5.html

so i run the command for step 1, and a error for the AMD GPU POPS UP!!!!! WHY DOES THAT POP UP, IM JUST INSTALLING QEMU.... GAHHH.

WTF??? this is why i hate setting this up.

run the command sudo modprobe kvm; echo 1 | sudo tee /sys/module/kvm/parameters/ignore_msrs and the command that corrseponds to your CPU. once you get to the ./fetch-macOS-v2.py part, select Monterey. (if you have a intel cpu you can choose Ventura and higher)

it should boot into QEMU after you run ./OpenCore-Boot.sh (just select macOS base system, not EFI)

Partition the disk via Disk Utility.

Go Back and Start the Install.

it should now have rebooted, press macOS Installer (should alerady be selected so it auto boots)

yay it installed! press enter.

should reboot (again), press enter.

when it boots, press win + alt + f5 to bring up the accesability menu.

select increase contrast and press done, now it should be a bit faster so you can get through OOBE.

go through setup till you get here, select vison

ONLY SELECT REDUCE MOTION

go through setup, don't migrate data or setup a apple id (you can look up on what to do, i'm not gonna bother with putting screenshots and everything. sorry)

now you should be finished with OOBE.

click the apple at the top left corner and press Shut Down

now we will get the KVM in virt-manager.


follow the rest of the guide.

yeah this is where it gets hard, might lose my insanaity, its also 3:30 am, so.

open up nano and do cd OSX-KVM then sudo nano macOS-libvirt-Catalina.xml

the lines that say CHANGEME, rename that to your home user name

just do "Crtl + W" and type "CHANGEME" and press enter, then the change the line highlighted in yellow to your home user name, once theres only 1 "CHANGEME" left, press "Crtl + X" and then "Crtl + Y" to save. (xD there was a easier way, but i just wanted to make you suffer (P.S: "sed "s/CHANGEME/$USER/g" macOS-libvirt-Catalina.xml > macOS.xml")

then run "virsh --connect qemu:///system define macOS.xml" to put it into QEMU.

now open a terminal and type "virt-manager" (or open it via applications menu)

our first issue! fuck.

put virsh --connect qemu:///system define macOS-libvirt-Catalina.xml instead and it should now show.

make sure it boots.

gah

seems it has to do with this "NIC :e6:85:40" device, lets remove it and fuck shit up!

after you remove it, start the KVM. it should work now


now to passthrough a physical GPU.

open up a terminal and type lspci -nnk

locate your GPU (at the very end look for [1002:xxxx]/b>), see what kernel it uses (in this case its radeon, amdgpu) REMEBER THE [1002:xxxx] NUMBER, WE NEED IT FOR LATER!

do "sudo nano /etc/modprobe.d/blacklist.conf"

put the lines "blacklist amdgpu" and "blacklist radeon" in (or whatever kernel it showed it used) at the VERY BOTTOM OF THE PAGE (unless your using debian, like me.)

press "Crtl + X" and "Crtl + Y" to exit and save.

follow https://github.com/kholia/OSX-KVM/blob/master/notes.md#gpu-passthrough-notes

now do you still have that [1002:xxxx] number? get it.

follow these next commands

sudo nano /etc/default/grub

on the line "GRUB_CMDLINE_LINUX_DEFAULT

put "vfio-pci.ids=1002:XXXX,1002:XXXX kvm.ignore_msrs=1 video=vesafb:off,efifb:off" for intel CPUs

put "iommu=pt amd_iommu=on vfio-pci.ids=1002:XXXX,1002:aab0 kvm.ignore_msrs=1 video=vesafb:off,efifb:off" for AMD CPUs

REPLACE XXXX WITH THE 4 NUMBERS/LETTERS

Ctrl + X Crtl + Y to exit

now do "sudo nano /etc/modprobe.d/vfio.conf"

put in these commands: options vfio-pci ids=1002:XXXX,1002:aaf0 disable_vga=1 softdep radeon pre: vfio-pci softdep amdgpu pre: vfio-pci softdep nouveau pre: vfio-pci softdep drm pre: vfio-pci

REPLACE XXXX WITH YOUR 4 LETTERS/NUMBERS!

sudo update-grub2

sudo update-initramfs -k all -u

sudo reboot

we will check to make sure it works.

now after linux boots, do sudo dmesg | grep -i iommu

should say "DMAR: IOMMU enabled" and "Default domain type: Passthrough" and "Adding to iommu group X" if not, i have no idea. sorry.

do sudo dmesg | grep vfio

should say [ 0.526198] vfio-pci 0000:01:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem [ 0.543768] vfio_pci: add [1002:67df[ffffffff:ffffffff]] class 0x000000/00000000 [ 0.563765] vfio_pci: add [1002:aaf0[ffffffff:ffffffff]] class 0x000000/00000000 [ 3.384597] vfio-pci 0000:01:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem

do sudo lspci -nkk -d 1002:XXXX

should say: 01:00.0 0300: 1002:67df (rev ef) Subsystem: 1da2:e366 Kernel driver in use: vfio-pci Kernel modules: amdgpu

do ./scripts/list_iommu_groups.sh should say something like: IOMMU Group 0: 00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor Host Bridge/DRAM Registers [8086:191f] (rev 07) IOMMU Group 1: 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 07) 01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] [1002:67df] (rev ff) 01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590] [1002:aaf0] (rev ff)

if everything works, we can move on, if not, no idea, still moving on.

do sudo cp vfio-kvm.rules /etc/udev/rules.d/vfio-kvm.rules

seems the other 2 commands give me "File not Found" so im gonna skip that.

now open virt-manager and right click on the VM and press "Open" and click on the lightbulb at the top.

go to "add hardware" and click "PCI Host Device" and click the GPU "0000:05:00.1" and "0000:05:00.1"

press the play button at the top, you should get this immou error, time to install a new fucking kernel.

we are going to install https://github.com/benbaker76/linux-acs-override/ -- this will seprate the devices into their each own iommu group.

mkdir ACS

cd ACS

git clone https://github.com/benbaker76/linux-acs-override/

cd linux-acs-override/

compile the kernel with "./build-debian.sh" now- oh hang on i think THEIR FUCKING GITHUB IS DOWN BECAUSE THE PROJECT IS ABANDONED AHHHHHHHHHHHHHH, oh nevermind. this might take a while to compile...

while thats compiing, do "sudo /etc/default/grub" and put "pcie_acs_override=downstream,multifunction" in (GRUB_CMDLINE_LINUX_DEFAULT=" ") and save.

if you add "1>Ubuntu, with Linux 6.3.0-acso" to (GRUB_DEFAULT="") it will auto boot into the patched kernel. (would recommend)

after it compiles, follow the instructions on their github - https://github.com/benbaker76/linux-acs-override/

do "ls *.deb"

do "sudo dpkg -i *.deb" sudo reboot

like this, but with sudo before both commands.

ok, after rebooting it just freezes here

but i did get th kernel installed, and can go to the previous one if i need to.

i will try again later, it is 6:39am and i just pulled a all-nighter trying to figure this out. night.

ok, so. i WAS able to get it to boot into the patched kernel ONCE, and the iommu groups were seperated, BUT the gpu didn't show in virt-manager. so i'm going to have to give up (for now). it should work following these instructions.>

good luck.