This guide is for you, if you passed through your GPU to a VM and now you want to use it in your host and have an easy way to reboot and choose a grub entry to pass it through again.

Introduction

If you have a setup similar to mine, e.g. you want to passthrough your nvidia graphic card to your VM for gaming purposes, but you want to take advantage of your powerful GPU for your daily use in your host:

1. first of all configure your system so that the nvidia driver is working correctly;


2. then you can create a new entry in grub, so that when you boot you can choose it for gaming.

Assumptions

I assume that you have your system set already correctly to run a VM with GPU passthrough e.g. as explained in this very good guide: Play games in Windows on Linux! PCI passthrough quick guide.

If this is the case, vfio is loaded before any other modules, so that it can claim your GPU (if it's blacklisted), basically you should have these entries (this may vary a bit, if you use AMD or need other modules than I do) in your /etc/modules :

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
kvm
kvm_intel
apm power_off=1

1. Configure NVIDIA

If your system is set as in the assumptions mentioned above, now you need to "de-configure" your setup, so that can use your GPU in your host. Don't worry, later you will re-add all the options you deactivate, but in a new entry in grub (see step 2).

In my case, the nouveau and nvidia drivers were blacklisted in /etc/modprobe/blacklist.conf  so I deactivated the blacklist by commenting those lines:

#blacklist nouveau
#blacklist nvidia

I also had to comment the options I added in my /etc/modprobe.d/vfio.conf :

#options vfio-pci ids=10de:13c0,10de:0fbb,8086:a12f

To make sure that the nvidia installation would detect my graphic card, I rebooted. Maybe this was not really needed, but I preferred to be safe:

sudo reboot

Installing the nvidia driver for my GTX 980 GPU was as easy as running this command:

sudo apt install nvidia-384

Now reboot and check if your nvidia graphic card works as expected.

For testing you may want to install glmark2, a tool to benchmark OpenGL:

sudo apt install glmark2

You can install steam and play some games in linux. Why playing games in a VM if you can play them directly in your host?! Try it.

2. Create an entry in grub to boot your system with GPU passthrough

I was looking for this method for a while, I even had 2 linux distros installed: one for VMs, one for my daily use. With this option, this is not needed anymore. I can have just one linux distro, but a different entry in grub to boot it with the necessary options to use my VMs.

Important: never update /boot/grub/grub.cfg , as it gets overwritten when you run the update-grub command, instead, add custom entries, as follows:

sudo pluma /etc/grub.d/40_custom

In my case, I copied the entry from /boot/grub/grub.cfg and edited as follows, to include the vfio options and to blacklist nvidia. Make sure to change the vfio pci ids with the ones of your devices you want to passthrough:

menuentry "Bionic VMs" {
	set root='hd0,gpt1'
        linux	/vmlinuz-4.15.0-20-generic root=/dev/mapper/vg0-bionic ro acpi=force apm=power_off intel_iommu=on vfio-pci.ids=10de:13c0,10de:0fbb,8086:a12f modprobe.blacklist=nouveau,nvidia,nvidia_uvm,nvidia_drm,nvidia_modeset
	initrd	/initrd.img-4.15.0-20-generic
}

Save, update grub, reboot and enjoy!

sudo update-grub
sudo reboot

Troubleshooting

If you can't see your grub menu, or it's too fast when you boot, you may customize it a bit, here how my grub is configured for me in /etc/default/grub

#GRUB_DEFAULT=3
#GRUB_HIDDEN_TIMEOUT=0
GRUB_HIDDEN_TIMEOUT_QUIET=false
GRUB_TIMEOUT=1
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
#GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
GRUB_CMDLINE_LINUX_DEFAULT="acpi=force apm=power_off intel_iommu=on"
GRUB_CMDLINE_LINUX=""
GRUB_DISABLE_OS_PROBER=true

I kept some lines commented, as sometimes I need to play with these options. I kept my timeout very low, just one second, because I want my system to boot fast, increase it if it's too fast for you and comment the GRUB_HIDDEN_TIMEOUT line, as I did. When I need to boot differently, I just keep pressing the up/down arrow keys at boot time, until I see the grub menu.

Conclusions

I hope that this guide was helpful for you, if so, consider buying a gadget at banggood using my referral link. Like this (comparing to a donation) we both benefit, you get a gadget that may be useful for you and I get something too (a little commission, but the price for you is the same).