Hardware and Device Drivers
Transcription
Hardware and Device Drivers
Faculty of Computer Science Institute for System Architecture, Operating Systems Group Hardware and Device Drivers Björn Döbel Dresden, 2007-11-13 Outline • What's so different about device drivers? • Hardware access • Writing device drivers on L4 – Reuse of legacy drivers • Device virtualization • Device drivers and security TU Dresden, 2007-11-13 Device Drivers Slide 2 von 45 Why are Drivers so hard? • Typically need kernel privileges because of – Interrupt handling – Hardware access • data transfer • config registers • firmware – Special memory management – DMA • physical adresses • DMA vs. paging – Synchronization & Timing mechanisms • complex state machines • Unsexy solution: keep drivers in the kernel TU Dresden, 2007-11-13 Device Drivers Slide 3 von 45 Idea: User-level Drivers • Isolate components – device drivers (disk, network, graphic, USB cruise missiles, ...) – stacks (TCP/IP, file systems, ...) • Separate address spaces for each • More robust components – Crashing network device does not hurt the file system • HW multiplexing TU Dresden, 2007-11-13 Device Drivers Slide 4 von 45 System Layout • Devices connected by busses (USB, PCI, PCIx) • Host chipset (DMA logic, IRQ controller) connects busses and CPU CPU System Bus Memory Bus Memory PCI Bus Chipset USB Bus Network Card PCI Controller PCI Bus USB Host Controller Sound Card PCI Bridge PCI Bus PCI Wifi USB Coffee Maker TU Dresden, 2007-11-13 Device Drivers Slide 5 von 45 Real World Example Intel 925x Chipset (source: http://www.intel.com) TU Dresden, 2007-11-13 Device Drivers Slide 6 von 45 Buses and Devices • A long long time ago: – device architecture hard-coded • Problem: more and more devices – need means of dynamic device discovery • Probing: try out every driver to see if it works • Plug&Play: first try of dynamic system description – device manufacturers provide unique IDs • PCI: dedicated config space • ACPI: system description without relying on underlying bus/chipset TU Dresden, 2007-11-13 Device Drivers Slide 7 von 45 Buses: PCI • Peripheral Component Interconnect • Hierarchy of busses, devices and functions • Configuration via I/O ports • Address + data register (0xcf8-0xcff) Chipset Device 1 Device 2 Device 3 PCI-to-PCI Bridge Device 4 Func 1 Func 2 TU Dresden, 2007-11-13 Device Drivers Slide 8 von 45 Buses: PCI (2) • PCI config space • 64 byte header – – – – – Busmaster DMA Interrupt line I/O port regions I/O memory regions + 192 byte additional space • must be provided by every device function • must be managed to separate device drivers TU Dresden, 2007-11-13 Device Drivers Slide 9 von 45 Buses: USB • Intel, 1996 • One bus to rule them all? – Firewire has always been faster • Tree of devices – root = Host Controller (UHCI, OHCI, EHCI) – Device drivers use HC to communicate with their device via USB Request Blocks (URBs) – USB is a serial bus • HC serializes URBs • Wide range of device classes (input, storage, peripherals, ...) • Device classes allow generic drivers TU Dresden, 2007-11-13 Device Drivers Slide 10 von 45 Interrupts • Signal device state change • Programmable Interrupt Controller (PIC, APIC) – map HW IRQs to CPU's IRQ lines – prioritize interrupts System Bus CPU PCI Bus PIC INT INT A INT B INT C IDE NET USB Chipset Memory Bus Memory TU Dresden, 2007-11-13 Device Drivers Slide 11 von 45 Interrupts (2) • Handling interrupts involves – examine / manipulate device – program PIC • acknowledge/mask/unmask interrupts System Bus CPU PCI Bus PIC INT INT A INT B INT C IDE NET USB Chipset Memory Bus Memory TU Dresden, 2007-11-13 Device Drivers Slide 12 von 45 L4: Interrupt handling • IRQs mapped to IPC • threads call ipc_recv() at a dedicated interrupt kernel thread • exactly one thread can wait for exactly one interrupt – no IRQ sharing support in the kernel Device Driver l4_ipc_receive(irq_id,…) Kernel Software Hardware TU Dresden, 2007-11-13 Fiasco Microkernel CPU Chipset Device Drivers Device Slide 13 von 45 L4: Problems with interrupts • IRQ lines are limited – IRQ sharing is needed • Multiple user-level drivers need to access PIC – locking – we don't want to trust our drivers' • IRQ priorities vs. thread priorities -> priority inversion • IRQs are a resource and must be managed! • Some drivers need to disable interrupts. TU Dresden, 2007-11-13 Device Drivers Slide 14 von 45 IRQ management: Omega0 • Omega0 – Hides all interrupt logic from device drivers • one interface for all architectures – Provides sharing of interrupts – Provides interrupt hand-over to another thread – Prevents priority inversion – Implementation may require a user-level server, depending on the kernel APIs TU Dresden, 2007-11-13 Device Drivers Slide 15 von 45 Omega0 layout • x86: user-level management server • ack IRQs immediately at PIC -> prevent priority inversion Device Driver 1 Device Driver 2 IRQ A IRQ B IRQ A IRQ B Omega0 Kernel Fiasco Microkernel Software Hardware CPU PIC INT INT A INT B TU Dresden, 2007-11-13 Chipset Device Drivers Device 1 Device 2 Slide 16 von 45 Omega0: Multiple IRQs per Thread • e.g.: one IDE driver for both IDE ports Device Driver IRQ A IRQ B IRQ A IRQ B Omega0 Kernel Fiasco Microkernel Software Hardware CPU PIC INT INT A INT B Device 1 Device 2 Chipset TU Dresden, 2007-11-13 Device Drivers Slide 17 von 45 Omega0: IRQ sharing • e.g., USB storage and USB missile launcher on the same IRQ line Device Driver 1 IRQ A Device Driver 2 Omega0 Kernel IRQ A Fiasco Microkernel Software Hardware CPU PIC INT INT A Device 1 Device 2 Chipset TU Dresden, 2007-11-13 Device Drivers Slide 18 von 45 Disabling interrupts • CLI – only with IO Privilege Level (IOPL) 3 • Should not be allowed for every user-level driver – unstrusted drivers – security risk • Observation: drivers often don't need to disable IRQs globally, but only access to their own IRQ – in this case dropping pending IRQs at omega0 is enough, no need to mingle with others TU Dresden, 2007-11-13 Device Drivers Slide 19 von 45 Hardware access • Apart from IRQs, drivers need to access device hardware – data transfer – configuration • Approaches – memory-mapped I/O – I/O ports (x86-specific) TU Dresden, 2007-11-13 Device Drivers Slide 20 von 45 I/O ports • x86-specific feature • I/O ports define own I/O address space – Each device uses its own area within this address space • Special instruction to access I/O ports – in / out: I/O read / write – Example: read byte from serial port mov $0x3f8, %edx in (%dx), %al • Everyone who can access I/O ports of a device can manipulate the device • Need to restrict I/O port access – Allow device drivers access to I/O ports used by its device only TU Dresden, 2007-11-13 Device Drivers Slide 21 von 45 I/O Bitmap • Tasks have I/O privilige level (IOPL) • If IOPL > current PL, all accesses are allowed (kernel mode) #0xfff0 • Else: I/O bitmap is checked #0xffe0 • 1 bit per I/O port #0x0000 – 65536 ports -> 8kB • Controls port access (0 == ok, 1 == GPF) • L4: per-task I/O bitmap – Switched during task switch – Allows per-task grant/deny of I/O port access TU Dresden, 2007-11-13 Device Drivers 1111111111011011 1111111111111111 ... 1111111111111111 I/O Map Base Address TSS Slide 22 von 45 I/O Flexpages • Reuse kernel's map/grant mechanism for mapping I/O port rights -> I/O flexpages • Kernel detects type of flexpage and acts accordingly • Task with all I/O ports mapped is raised to IOPL 3 Map/Grant 1111 Base Port 0000 log2#Ports 0 L4.Fiasco I/O flexpage format TU Dresden, 2007-11-13 Device Drivers Slide 23 von 45 I/O Memory • Devices often contain on-chip memory (NICs, graphcis cards, ...) • Instead of accessing through I/O ports, drivers can map this memory into their address space just like normal RAM – no need for special instructions – increased flexibility by using underlying virtual memory management Driver Memory Kernel Software Hardware TU Dresden, 2007-11-13 Fiasco Microkernel CPU Chipset Device Memoy Drivers Device Memory Slide 24 von 45 I/O memory (2) • Device memory looks just like phys. memory • Chipset needs to – map I/O memory to exclusive address ranges – distinguish physical and I/O memory access Driver Kernel Software Hardware Fiasco Microkernel CPU Chipset Device Memory Memory TU Dresden, 2007-11-13 Device Drivers Slide 25 von 45 I/O memory in L4 • Like all memory, I/O memory is owned by sigma0 • Sigma0 implements protocol to request I/O memory pages • Abstraction: Dataspaces containing I/O memory Driver 1 Driver 2 Sigma0 Kernel Software Hardware Fiasco Microkernel CPU Chipset Memory TU Dresden, 2007-11-13 Device Drivers Device 1 Device 2 Slide 26 von 45 Direct Memory Access (DMA) • Bypass CPU by directly transferring data from device to RAM – improved bandwidth – relieved CPU • DMA controller either programmed by driver or by device's DMA engine (Busmaster DMA) CPU Chipset Device DMA Engine Controller Memory TU Dresden, 2007-11-13 Device Drivers Slide 27 von 45 Problems with DMA • Most architectures use physical addresses for DMA • Drivers need to know these addresses in order to correctly use buffers etc. • Buffers must not be paged out (pinned pages) during DMA – close interaction with memory management • DMA with phys. addresses bypasses VM management – Drivers can overwrite any phys. address • DMA is a security risk. TU Dresden, 2007-11-13 Device Drivers Slide 28 von 45 Idea: I/O MMU • Like traditional MMU maps virtual to physical addresses – implemented in PCI bridge – manages a pagetable – I/O-TLB • Drivers access buffers through virtual addresses – I/O MMU translates accesses – restrict access to phys. memory by only mapping certain DMA addresses into driver's address space TU Dresden, 2007-11-13 Device Drivers Slide 29 von 45 I/O MMUs • Per-Bus IOMMU already in 64bit architectures Chipset Memory I/O-MMU Device 1 Device 2 Device 3 • Per-device IOMMU not available yet Chipset I/O-MMU I/O-MMU I/O-MMU Device 1 Device 2 Device 3 Memory TU Dresden, 2007-11-13 Device Drivers Slide 30 von 45 I/O MMU Architecture • I/O MMU manages by yet another resource manager • Before accessing I/O memory, drivers use manager to establish a virt->phys mapping Client Application Driver Software Hardware Dataspace Manager CPU Device Manager Chipset I/O-MMU Manager I/O-MMU Device Memory TU Dresden, 2007-11-13 Device Drivers Slide 31 von 45 Summary: Driver support in L4 • Interrupts -> IPC • I/O ports and memory -> flexpage mappings • User-level resource manager -> L4IO Driver lib_l4io Driver lib_l4io lib_dm lib_dm Driver lib_l4io lib_dm L4IO Dataspace Manager - Omega0 - I/O Ports / Memory - PCI - Phys. Addresses - Pinned Memory Kernel Software Hardware Fiasco Microkernel CPU Chipset Devices Memory TU Dresden, 2007-11-13 Device Drivers Slide 32 von 45 Untrusted Device Drivers • Problem: How to enforce device access policies on untrusted drivers? • Solution: – I/O manager needs to manage device resources – external driver loader configures policy • General concept: Split policy configuration and enforcement I/O manager Driver loader - knows all devices - manages I/O resources - decide which drivers to load and which resources to assign configure policy Kernel Software Hardware TU Dresden, 2007-11-13 Fiasco Microkernel CPU Chipset Device Drivers Devices Slide 33 von 45 Implementing Device Drivers • Just like in any other OS: – Specify an interface (IDL in our case) – Implement interface, use the access methods provided by the L4 Environment • Highly optimized code possible • Hard to maintain, implementation time-consuming • Many vendors won't give you a spec. • Why reimplement drivers if there are already versions available? – Linux, BSD – Open Source – Windows – Binary drivers TU Dresden, 2007-11-13 Device Drivers Slide 34 von 45 Reusing legacy device drivers • Exploit virtualization: Device Driver OS Source: LeVasseur et. al.: "Unmodified Device Driver Reuse and Improved System Dependability via Virtual Machines”, OSDI 2004 TU Dresden, 2007-11-13 Device Drivers Slide 35 von 45 Reusing Legacy Device Drivers • More work-intensive idea: wrap legacy source code with a dedicated environment – implement Linux interface in a library (which maps this interface to L4) – Device Driver Environment (DDE) • Reasonable less overhead than using a complete VM for each driver Interface Code • Maintainable code Original • Reasonable effort – DDE can be reused Driver Code Glue Code Environment TU Dresden, 2007-11-13 Device Drivers Slide 36 von 45 Emulating Linux: dde_linux • Multiple L4 threads provide a Linux environment – Workqueues – SoftIRQs / Bottom Halves – Timers • Emulate SMP-like system (each thread assumed to be one processor) • Special synchronization mechanism: – irq_lock instead of CLI/STI (tamed Linux) TU Dresden, 2007-11-13 Device Drivers Slide 37 von 45 DDE Structure Client Application Client Library L4 Server Code Work Queues Linux Driver Source Code SoftIRQs Timer IRQs lib_l4io Emulation Library (dde_linux) lib_dm L4IO Dataspace Manager Kernel Software Hardware Fiasco Microkernel CPU Chipset Devices Memory TU Dresden, 2007-11-13 Device Drivers Slide 38 von 45 DDEKit – another abstraction layer • Next task: I want to reuse FreeBSD drivers, too. • Don't implement the same stuff from scratch. • Extract common mechanisms into another layer – DDEKit – and build DDEs upon it: – – – – – – – IRQ handling I/O port management PCI config space access Locks Threads Timers ... TU Dresden, 2007-11-13 Device Drivers Slide 39 von 45 DDEKit (2) • Implementation overhead for single DDEs gets much smaller • Performance overhead still reasonable – e.g., no visible increase of network latency in user-level ethernet driver • L4-specific parts (sloccount): – – – – standalone DDE Linux 2.4: DDEKit DDEKit-based DDE Linux 2.6: Standalone Linux VM: > ~ 3.000 LoC ~ 2.000 LoC ~ 1.000 LoC 500.000 LoC • Highly customizable: implement DDE base library and support libs (net, disk, sound, ...) TU Dresden, 2007-11-13 Device Drivers Slide 40 von 45 DDEKit (3) Client Application Client Library L4 Server Code Linux Driver Source Code Linux-specifics (SoftIRQs, KThreads) dde_linux Emulation Library (ddekit) L4IO Dataspace Manager Kernel Software Hardware Fiasco Microkernel CPU Chipset Devices Memory TU Dresden, 2007-11-13 Device Drivers Slide 41 von 45 DDE(Kit): disk driver server_init() write_block() driver_init() { /* find device */ /* init ISR */ } pci_find_device() request_irq() driver_write_block() { /* prog device */ /* wait4completion */ } driver_bh() { /* do work */ /* notify */ } driver_irq() add_wait_queue() schedule() wake_up() bh_thread() irq_thread() lib_ddekit lib_l4io TU Dresden, 2007-11-13 Device Drivers IRQ Slide 42 von 45 DDE(Kit): Use Cases • DDELinux2.4 – IDE Disk Driver – Virtual Ethernet Interface – USB Webcam – TCP/IP Network Stack – OSS sound server • DDELinux 2.6 – Virtual Ethernet Interface – ATA disk driver – ALSA sound server • DDEFreeBSD – ATA disk driver TU Dresden, 2007-11-13 Device Drivers Slide 43 von 45 Literature Omega0 – http://os.inf.tu-dresden.de/~jork/papers/omega0.pdf Safe interrupt handling – http://os.inf.tu-dresden.de/papers_ps/part98.pdf DMA and I/O MMU – http://os.inf.tu-dresden.de/papers_ps/tr-ioarch-2003.pdf – http://os.inf.tu-dresden.de/papers_ps/mehnert-phd.pdf DDE and DDE-based drivers – http://os.inf.tu-dresden.de/paper_ps/helmuth-diplom.pdf – http://os.inf.tu-dresden.de/papers_ps/griessbach-diplom.pdf – http://os.inf.tu-dresden.de/papers_ps/menzer-diplom.pdf – http://os.inf.tu-dresden.de/papers_ps/friebel-diplom.pdf Device Driver OS – http://i30www.ira.uka.de/research/documents/l4ka/2004/ .. .. LeVasseur04UnmodifiedDriverReuse.pdf TU Dresden, 2007-11-13 Device Drivers Slide 44 von 45 Coming soon • Tomorrow: Exercise on “The nucleus of a multiprogramming system” • Next week: Resource management TU Dresden, 2007-11-13 Device Drivers Slide 45 von 45