Boot-up Graphics in Linux

Boot-up Graphics 指在 Linux 系统启动时涉及到一些和图形显示相关的问题。

当主板上同时有多张PCI显卡时, 哪一个会做为默认显示输出呢？

一个最简单的场景，主板上有一个 Intel 的集显，还有一个 AMD 的独显，这两个 GPU 各自通过 HDMI 接口连接到同一个显示器上，那么在Linux 系统启动时（未进入桌面环境），是哪个 GPU 在显示呢？

Linux VGAArbiter

Linux kernel vgaarbiter 的创建

commit deb2d2ecd43dfc51efe71eed7128fda514da96c6
Author: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Date:   Tue Aug 11 15:52:06 2009 +1000

    PCI/GPU: implement VGA arbitration on Linux

    Background:
    Graphic devices are accessed through ranges in I/O or memory space. While most
    modern devices allow relocation of such ranges, some "Legacy" VGA devices
    implemented on PCI will typically have the same "hard-decoded" addresses as
    they did on ISA. For more details see "PCI Bus Binding to IEEE Std 1275-1994
    Standard for Boot (Initialization Configuration) Firmware Revision 2.1"
    Section 7, Legacy Devices.

    The Resource Access Control (RAC) module inside the X server currently does
    the task of arbitration when more than one legacy device co-exists on the same
    machine. But the problem happens when these devices are trying to be accessed
    by different userspace clients (e.g. two server in parallel). Their address
    assignments conflict. Therefore an arbitration scheme _outside_ of the X
    server is needed to control the sharing of these resources. This document
    introduces the operation of the VGA arbiter implemented for Linux kernel.

    Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
    Signed-off-by: Tiago Vignatti <tiago.vignatti@nokia.com>
    Signed-off-by: Dave Airlie <airlied@redhat.com>
    Signed-off-by: Jesse Barnes <jbarnes@virtuousgeek.org>

 drivers/gpu/Makefile     |    2 +-
 drivers/gpu/vga/Kconfig  |   10 +
 drivers/gpu/vga/Makefile |    1 +
 drivers/gpu/vga/vgaarb.c | 1206 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 drivers/pci/pci.c        |   44 ++++
 drivers/video/Kconfig    |    2 +
 include/linux/pci.h      |    2 +
 include/linux/vgaarb.h   |  195 +++++++++++++++++
 8 files changed, 1461 insertions(+), 1 deletion(-)

X Server的输出设备检测和驱动加载

Xorg抽象的输出设备

Linux下的显示输出设备一般要么是一个pci_device，要么是一个xf86_platform_device
Xorg定义了一个全局数组xf86_platform_devices, 这个数组的元素类型是xf86_platform_device, 存储是动态申请的，Xorg探测到一个PCI显卡时，就申请一个存储。

struct xf86_platform_device *xf86_platform_devices;

struct xf86_platform_device {
		struct OdevAttributes *attribs;
		struct pci_device *pdev;
		int flags;
};

struct OdevAttributes {
		char *path;
		char *syspath;
		char *busid;
		int fd;
		int major;
		int minor;
		char *driver;
}

OdevAttributes定义了一个PCI输出设备的属性。

• path

  kernel device node, /dev/dri/card0

• syspath

  system device path, /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1

• busid

  DRI 风格的Bus ID， 如 “pci:0000:04:00.0”

• fd

  文件描述符，open(/dev/dri/card0)返回值

• major

  主设备号

• minor

  次设备号

• driver

  kernel driver name, 如 “amdgpu”

PCI BusID

DRI-style

pci:0000:04:00.0

• 0000 PCI domain
• 04 PCI bus
• 00 PCI device
• 0 PCI function

Xorg 日志

• 来自 xorg-server-1.18.4 版本的 Xorg 日志

[     8.456] (--) PCI: (0:5:0:0) 1002:677b:174b:3000 rev 0, Mem @ 0x1040000000/268435456, 0x58600000/131072, I/O @ 0x00002000/256, BIOS @ 0x????????/131072

• 来自 xorg-server-1.20.0 版本的 Xorg 日志

[    40.891] (--) PCI: (10@0:0:0) 1a03:2000:1a03:2000 rev 65, Mem @ 0x60000000/16777216, 0x61000000/131072, I/O @ 0x00002000/128, BIOS @ 0x????????/65536

变更发生于 2017-06-19:

commit e905b19a53f96013c4417bec993a1dea5a3b0a5f
Author: Michel Dänzer <michel.daenzer@amd.com>
Date:   Mon Jun 19 19:05:29 2017 +0900

    xfree86: Print BusID stanza compatible bus IDs for found devices

    The PCI domain has to be specified like this:

     "PCI:<bus>@<domain>:<device>:<function>"

    Example before:

     (--) PCI:*(0:0:1:0) 1002:130f:1043:85cb [...]
     (--) PCI: (0:1:0:0) 1002:6939:1458:229d [...]

    after:

     (--) PCI:*(0@0:1:0) 1002:130f:1043:85cb [...]
     (--) PCI: (1@0:0:0) 1002:6939:1458:229d [...]

    Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
    Signed-off-by: Michel Dänzer <michel.daenzer@amd.com>

Xorg 配置文件 “Device” Section

上面两种格式都可以作为Device段里BusID的格式，X11规定的BusID的格式是

“bus[@domain]:device[:func]“

Xserver依赖下面的用户空间库来填写该结构，这个过程也体现了Xserver检测显卡及加载驱动的过程

• libpciaccess
• libudev
• libdrm

1. 通过libpciaccess的接口发现系统PCI设备，获取syspath, path
2. 通过open系统调用打开侦测到的drm_device，fd有了
3. 通过libdrm接口获取major, minor和driver

Device Detect Routines

Xserver提供了两种设备检测方法：

void xf86PlatformDeviceProbe(struct OdevAttributes *attribs);
void xf86PciProbe(void);

默认是使用xf86PlatformDeviceProbe， 当这个函数完成xf86_add_platform_device后，除了attribs, xf86_platform_device的其它成员还没有被填写，剩下的任务交由libudev, libpciaccess和libdrm的接口完成。

libudev

• udev_enumerate_add_match_subsys
• udev_enumerate_add_match_sysname

libpciaccess

• pci_device_probe
• pci_device_is_boot_vga

libdrm

• drmSetInterfaceVersion
• drmGetBusid
• drmGetVersion

Primary Bus

在多卡的情况下，Xserver启动后默认使用哪个显卡显示？

typedef struct _bus {
	BusType type;
	union {
		struct pci_device *pci;
		SbusBusId sbus;
		struct xf86_platform_device *plat;
	} id;
} BusRec, *BusPtr;

Xserver定义了一个全局的BusRec类型变量primaryBus

BusRec primaryBus = { BUS_NONE, {0} };

这个primaryBus将是Xserver启动后默认使用的显卡设备的唯一候选者。

Who Is The Lucky Boy?

Xserver有两个规则去确定primaryBus:

• config_udev_odev_probe
• pci_device_is_boot_vga

How config_udev_odev_probe Works?

config_udev_odev_probe唯一的参数是一个callback函数xf86PlatformDeviceProbe, config_udev_odev_probe要做的就是调用libudev的接口枚举出/dev文件系统里注册的drm设备，将它的path填到xf86_platform_device.attribs->path。所以这一步决定了xf86_platform_devices数组中platform device的顺序，按照此顺序最后那个pci_device_is_boot_vga返回True的显示设备将为primaryBus, 但这是在没有配置PrimaryGPU选项时的行为。下面的代码来自libudev

_public_ int sd_device_enumerator_add_match_sysname(sd_device_enumerator *enumerator, const char *sysname) {
        int r;

        assert_return(enumerator, -EINVAL);
        assert_return(sysname, -EINVAL);

        r = set_put_strdup(&enumerator->match_sysname, sysname);
        if (r <= 0)
                return r;

        enumerator->scan_uptodate = false;

        return 1;
}

这里参数sysname是一个正则表达式card[0-9]*

How pci_device_is_boot_vga Works?

pci_device_is_boot_vga是一个虚接口，在Linux下的实现是pci_device_linux_sysfs_boot_vga, 下面的代码来自libpciaccess

static int pci_device_linux_sysfs_boot_vga(struct pci_device *dev)
{
    char name[256];
    char reply[3];
    int fd, bytes_read;
    int ret = 0;

    snprintf( name, 255, "%s/%04x:%02x:%02x.%1u/boot_vga",
	      SYS_BUS_PCI,
	      dev->domain,
	      dev->bus,
	      dev->dev,
	      dev->func );

    fd = open( name, O_RDONLY | O_CLOEXEC);
    if (fd == -1)
       return 0;

    bytes_read = read(fd, reply, 1);
    if (bytes_read != 1)
	goto out;
    if (reply[0] == '1')
	ret = 1;
out:
    close(fd);
    return ret;
}

这里SYS_BUS_PCI被定义为/sys/bus/pci/devices, pci_device_is_boot_vga的返回值取决于显示设备的kernel driver如何实现sysfs文件系统中的/sys/bus/pci/devices/0000:05:00.0/boot_vga节点。

以上两点规则说明，在多卡系统中，不使用PrimaryGPU的情况下，当且仅当目标卡的drm设备节点号最大，而且/sys/bus/pci/devices/XXXX:XX:XX.X/boot_vga被实现为read它返回ASCII字符'1'. 这样目标卡才能做为primaryBus设备默认显示输出。

参考

• pci/vgaarb: make vga_is_firmware_default() arch independent
• discussion on ML