LWN Articles (Paywall Just Removed) About Kernel/Linux: KVM, Locking and Memory

-
KVM for Android [LWN]
A Google project aims to bring the Linux kernel virtualization mechanism, KVM, to Android systems. Will Deacon leads that effort and he (virtually) came to KVM Forum to discuss the project, its goals, and some of the challenges it has faced. Unlike some Android projects of the past, though, "protected KVM" is being worked on in the open, with code going upstream along the way.
Deacon is one of the maintainers of the arm64 architecture for the kernel, as well as a maintainer and contributor in various other parts of the kernel, including concurrency, locking, atomic operations, and tools for the kernel memory model. He has worked in the kernel for a long time, but not really on KVM; the closest he had come to that is maintaining the Arm IOMMU drivers. He started working on the Android Systems team at Google in 2019 "and found myself leading the protected KVM project", which is the KVM on Android effort.
The project is the top contributor to KVM for arm64 for the 5.9 and 5.10 kernels; KVM seems to be a "hot topic" right now, he said, and not just for arm64, but for other architectures as well. All of the project's work is being upstreamed as it goes, so what he was presenting was "very much a work in progress". He wants to avoid the trap of doing a bunch of work out of tree and then "throwing it over the wall", which does not lead to good solutions that are embraced by the community.
-
Migration disable for the mainline
The realtime developers have been working for many years to create a kernel where the highest-priority task is always able to run without delay. That has meant a long process of finding and fixing situations where high-priority tasks might be blocked from running; one of the persistent problems in this regard has been kernel code that disables preemption. One tool that the realtime developers have reached for is disabling migration (moving a process from one CPU to another) rather than preemption; this approach has not been entirely popular among scheduler developers, though. Even so, the solution would appear to be this migration-disable patch set from scheduler developer Peter Zijlstra.
One of the key scalability techniques used in the kernel is per-CPU data. System-wide locking is an effective way of protecting shared data, but it can kill performance in a number of ways, even if a given lock is itself not heavily contested. Any data structure that is only accessed by a single CPU does not need to be protected by system-wide locks, avoiding this problem. Thus, for example, the memory allocators maintain per-CPU lists of available memory that can be handed out without interference from the other CPUs on the system. But kernel code can only safely manipulate per-CPU data if it has exclusive access to the CPU; if some other process is able to jump in, it could find (or create) inconsistent per-CPU data structures. The normal way to prevent this from happening is to disable preemption when necessary; it is a cheap operation (setting a flag, essentially) that ensures that a given task will not be interrupted until its work is done.
Disabling preemption runs afoul of the goals of the realtime developers, who have put so much work into ensuring that any given task can be interrupted if a higher-priority task needs the CPU. As they have worked to remove preemption-disabled regions, they have observed that, often, all that is really needed is to keep tasks from being moved between CPUs while they are accessing per-CPU data, with perhaps some (normally CPU-local) locking as well. See, for example, the kmap_local() work. Disabling migration still allows a process to be preempted, so it does not interfere with the goals of the realtime project — or so those developers hope.
Disabling migration brings problems of its own, though. The kernel's CPU scheduler is tasked with making the best use of all of the CPUs in the system. If there are N CPUs available, they should be running the N highest-priority tasks at any given time. That goal cannot be achieved without occasionally moving tasks between CPUs; it would be nice if tasks just happened to land on the right processors every time, but the real world is not like that. Depriving the scheduler of the ability to migrate tasks, even for brief periods, thus takes away a tool that is crucial for the overall behavior and throughput of the system.
-
Atomic kmaps become local
A 32-bit processor will, unsurprisingly, use 32-bit pointers, which limits the amount of memory that can be addressed to 4GB. The resulting 4GB address space is split between user space and the kernel, with the kernel getting 1GB in the most common configurations; that space holds the kernel's code and data, memory-mapped I/O areas, and the "direct map" that gives the kernel access to physical memory. The direct map clearly cannot address a lot of memory; once the kernel's other needs are taken care of, there is room for significantly less than 1GB of mappings to physical memory.
As a result, any system with 1GB or more of physical memory will have to be managed without a direct mapping to some of that memory. The memory that lies above the range that can be directly mapped is called "high memory"; on many systems, most of the installed memory is high memory. User space can use high memory without noticing any difference, but the kernel side is a bit more complicated. Whenever the kernel must access a high-memory page (to zero out a page prior to giving it to user space, for example), it must first create a temporary mapping for that page. The kmap() interface exists to manage these mappings.
The kmap() function itself will map a given page into the kernel's address space, returning a pointer that can now be used to access the page's contents. Mappings created this way are expensive, though. They consume address space, and mapping changes must be propagated across all the CPUs of the system, which is costly. This work is necessary if a mapping must last for a relatively long time, but the bulk of high-memory mappings in the kernel are short-lived and only used in one place; the cost of kmap() is mostly wasted in such cases.
Thus, the kmap_atomic() API was added as a way of avoiding this cost. It, too, will map a high-memory page into the kernel's address space, but with some differences. It uses one of a small set of address slots for the mapping, and that mapping is only valid on the CPU where it is created. This design implies that code holding one of these mappings must run in atomic context (thus the name kmap_atomic()); if it were to sleep or be moved to another CPU, confusion and data corruption would be an almost certain result. Thus, whenever code running in kernel space creates an atomic mapping, it can no longer be preempted or migrated, and it is not allowed to sleep, until all atomic mappings have been released.
-

- Login or register to post comments
Printer-friendly version- 3253 reads
PDF version
More in Tux Machines
- Highlights
- Front Page
- Latest Headlines
- Archive
- Recent comments
- All-Time Popular Stories
- Hot Topics
- New Members
digiKam 7.7.0 is released
After three months of active maintenance and another bug triage, the digiKam team is proud to present version 7.7.0 of its open source digital photo manager. See below the list of most important features coming with this release.
|
Dilution and Misuse of the "Linux" Brand
|
Samsung, Red Hat to Work on Linux Drivers for Future Tech
The metaverse is expected to uproot system design as we know it, and Samsung is one of many hardware vendors re-imagining data center infrastructure in preparation for a parallel 3D world.
Samsung is working on new memory technologies that provide faster bandwidth inside hardware for data to travel between CPUs, storage and other computing resources. The company also announced it was partnering with Red Hat to ensure these technologies have Linux compatibility.
|
today's howtos
|








.svg_.png)
Content (where original) is available under CC-BY-SA, copyrighted by original author/s.

Recent comments
1 year 11 weeks ago
1 year 11 weeks ago
1 year 11 weeks ago
1 year 11 weeks ago
1 year 11 weeks ago
1 year 11 weeks ago
1 year 11 weeks ago
1 year 11 weeks ago
1 year 11 weeks ago
1 year 11 weeks ago