Language Selection

English French German Italian Portuguese Spanish

Kernel Planet

Syndicate content
Kernel Planet -
Updated: 1 day 23 hours ago

Rusty Russell: The HRF Bounty On Human-Readable Offers

Thursday 14th of September 2023 02:30:00 PM

At least two people have contacted me concerning the 2 BTC bounty:

2 BTC for a human-readable bolt 12 offer generator feature integrated into a popular iOS or android bitcoin wallet. “Human-readable” means something that can be used on feature phone without QR or copy/paste ability. For example, something that looks like LN address.

This, of course, is asking to solve Zooko’s Triangle, so one of decentralizationm, human readability, or security needs to compromise! Fortunately, the reference to LN address gives a hint on how we might proceed.

The scenario, presumably, is Bob wants to pay Alice, where Alice shows Bob a “Human Readable Offer” and Bob types it into his phone. Each one runs Phoenix, Greenlight, or (if their phone is too low-end) uses some hosted service, but any new third party trust should be minimized.

There are three parts we need here:

  1. Bob finds Alice’s node.
  2. Bob requests Alice’s node for invoice.
  3. If she wants, Alice can easily check Bob’s going to pay the right thing.
The Imagined Scenario

Consider the normal offer case: the offer encodes Alice’s nodeid and description (and maybe other info) about what’s on offer. Bob turns this into an invoice_request, sends an onion message to Alice’s node, which returns the (signed) invoice, which Bob pays. We need to encode that nodeid and extra information as compactly as we can.

Part 1: Finding Alice’s Node from a Human Readable Offer

The issue of “finding Alice’s node” has been drafted already for BOLT12, at (but it needs updating!). This means that if you say “” you can get a valid generic offer, either by contacting the webserver at “” or having someone else do it for you (important for privacy!), or even downloading a public list of common receivers.

Note that it’s easier to type * than @ on feature phones, so I suggest allowing both and RUSTY*BLOCKSTREAM.COM.

What’s Needed On The Server
  1. The BOLT 12 Address Format needs to be updated.
  2. It needs to be implemented for some Web server.
  3. Ideally, integrate it into BTC Payserver or the like.
Part 2: Getting the Invoice

Now, presumably, we want a specific invoice: it might be some default “donate to Alice”, but it could be a specific thing “$2 hot dog”. So you really want some (short!) short-code to indicate which invoice you want. I suggest a hash, followed by some randomly chosen alphanumeric string here (case-insensitive!): an implementation may choose to restrict themselves to numbers however, as that’s faster to enter on a feature phone.

What’s Needed On The Server
  1. We can put the short-code in the invreq_payer_note field in BOLT 12 or add a new odd field.
  2. We need to implement (presumably in Core Lightning):
    • A way to specify/assign a short-code for each offer.
    • A way of serving a particular invoice based on this short-code match.
Part 3: Checking the Invoice

So, did you even get the right node id? That’s the insecure part; you’re trusting! Checking the nodeid is hard: someone can grind out a nodeid with the same first 16 digits in a few weeks. But I think you can provide some assurance, by creating a 4-color “flag” using the node id and the latest bitcoin blocks: this will change every new block, and is comparable between Alice and Bob at a glance:

This was made using this hacky code which turns my node id 024b9a1fa8e006f1e3937f65f66c408e6da8e1ca728ea43222a7381df1cc449605 into an RGB color (by hashing the nodeid+blockhash).

For a moment, when a new block comes in, one image might be displaced, hence the number, but it’ll only be out by one.

Putting it All Together What’s Needed On Alice’s Client
  1. Alice needs to configure her BOLT12 Address with some provider when she sets up the phone: it should check that it works!
  2. She should be able to choose an existing offer (may be a “donation” by default), or create a new one on the fly (with a new short code).
  3. Display the BOLT12-ADDRESS # SHORT-CODE, and the current nodeid flag.
What’s Needed On Bob’s Client
  1. It needs to be able to convert BOLT12-ADDRESS into a bolt12 address request:
    • Either via some service (to be implemented!), or by directly query (ideally over Tor).
  2. It needs to be able to produce an offer from the returns bolt12 address response, by putting the SHORT-CODE into the invreq_payer_note.
  3. It needs to be able to fetch an invoice for this offer.
  4. It needs to be able to display the current nodeid flag for the invoice’s node id.
  5. Allow Bob to confirm to send payment.
Is There Anything Else?

There are probably other ways of doing this, but this method has the advantage of driving maturity in several different areas which we want to see in Bitcoin:

  1. bolt12 address to support vendor field validation for offers.
  2. Simple name support for bootstrapping.
  3. Driving Bitcoin to be more accessible to everyone!

Feel free to contact me with questions!

Matthew Garrett: Reconstructing an invalid TPM event log

Wednesday 13th of September 2023 09:02:44 PM
TPMs contain a set of registers ("Platform Configuration Registers", or PCRs) that are used to track what a system boots. Each time a new event is measured, a cryptographic hash representing that event is passed to the TPM. The TPM appends that hash to the existing value in the PCR, hashes that, and stores the final result in the PCR. This means that while the PCR's value depends on the precise sequence and value of the hashes presented to it, the PCR value alone doesn't tell you what those individual events were. Different PCRs are used to store different event types, but there are still more events than there are PCRs so we can't avoid this problem by simply storing each event separately.

This is solved using the event log. The event log is simply a record of each event, stored in RAM. The algorithm the TPM uses to calculate the PCR values is known, so we can reproduce that by simply taking the events from the event log and replaying the series of events that were passed to the TPM. If the final calculated value is the same as the value in the PCR, we know that the event log is accurate, which means we now know the value of each individual event and can make an appropriate judgement regarding its security.

If any value in the event log is invalid, we'll calculate a different PCR value and it won't match. This isn't terribly helpful - we know that at least one entry in the event log doesn't match what was passed to the TPM, but we don't know which entry. That means we can't trust any of the events associated with that PCR. If you're trying to make a security determination based on this, that's going to be a problem.

PCR 7 is used to track information about the secure boot policy on the system. It contains measurements of whether or not secure boot is enabled, and which keys are trusted and untrusted on the system in question. This is extremely helpful if you want to verify that a system booted with secure boot enabled before allowing it to do something security or safety critical. Unfortunately, if the device gives you an event log that doesn't replay correctly for PCR 7, you now have no idea what the security state of the system is.

We ran into that this week. Examination of the event log revealed an additional event other than the expected ones - a measurement accompanied by the string "Boot Guard Measured S-CRTM". Boot Guard is an Intel feature where the CPU verifies the firmware is signed with a trusted key before executing it, and measures information about the firmware in the process. Previously I'd only encountered this as a measurement into PCR 0, which is the PCR used to track information about the firmware itself. But it turns out that at least some versions of Boot Guard also measure information about the Boot Guard policy into PCR 7. The argument for this is that this is effectively part of the secure boot policy - having a measurement of the Boot Guard state tells you whether Boot Guard was enabled, which tells you whether or not the CPU verified a signature on your firmware before running it (as I wrote before, I think Boot Guard has user-hostile default behaviour, and that enforcing this on consumer devices is a bad idea).

But there's a problem here. The event log is created by the firmware, and the Boot Guard measurements occur before the firmware is executed. So how do we get a log that represents them? That one's fairly simple - the firmware simply re-calculates the same measurements that Boot Guard did and creates a log entry after the fact[1]. All good.

Except. What if the firmware screws up the calculation and comes up with a different answer? The entry in the event log will now not match what was sent to the TPM, and replaying will fail. And without knowing what the actual value should be, there's no way to fix this, which means there's no way to verify the contents of PCR 7 and determine whether or not secure boot was enabled.

But there's still a fundamental source of truth - the measurement that was sent to the TPM in the first place. Inspired by Henri Nurmi's work on sniffing Bitlocker encryption keys, I asked a coworker if we could sniff the TPM traffic during boot. The TPM on the board in question uses SPI, a simple bus that can have multiple devices connected to it. In this case the system flash and the TPM are on the same SPI bus, which made things easier. The board had a flash header for external reprogramming of the firmware in the event of failure, and all SPI traffic was visible through that header. Attaching a logic analyser to this header made it simple to generate a record of that. The only problem was that the chip select line on the header was attached to the firmware flash chip, not the TPM. This was worked around by simply telling the analysis software that it should invert the sense of the chip select line, ignoring all traffic that was bound for the flash and paying attention to all other traffic. This worked in this case since the only other device on the bus was the TPM, but would cause problems in the event of multiple devices on the bus all communicating.

With the aid of this analyser plugin, I was able to dump all the TPM traffic and could then search for writes that included the "0182" sequence that corresponds to the command code for a measurement event. This gave me a couple of accesses to the locality 3 registers, which was a strong indication that they were coming from the CPU rather than from the firmware. One was for PCR 0, and one was for PCR 7. This corresponded to the two Boot Guard events that we expected from the event log. The hash in the PCR 0 measurement was the same as the hash in the event log, but the hash in the PCR 7 measurement differed from the hash in the event log. Replacing the event log value with the value actually sent to the TPM resulted in the event log now replaying correctly, supporting the hypothesis that the firmware was failing to correctly reconstruct the event.

What now? The simple thing to do is for us to simply hard code this fixup, but longer term we'd like to figure out how to reconstruct the event so we can calculate the expected value ourselves. Unfortunately there doesn't seem to be any public documentation on this. Sigh.

[1] What stops firmware on a system with no Boot Guard faking those measurements? TPMs have a concept of "localities", effectively different privilege levels. When Boot Guard performs its initial measurement into PCR 0, it does so at locality 3, a locality that's only available to the CPU. This causes PCR 0 to be initialised to a different initial value, affecting the final PCR value. The firmware can't access locality 3, so can't perform an equivalent measurement, so can't fake the value.


Linux Plumbers Conference: Registration Currently Sold Out, We’re Trying to Add More Places

Sunday 3rd of September 2023 10:00:17 AM

Linux Plumbers is now sold out and in-person registration is closed.

This year it happened not as fast as in 2022, but the registration is still sold out long before the event.

We are setting up a waitlist for in-person registration (virtual attendee places are still available). Please fill in this form and try to be clear about your reasons for wanting to attend. This year we’re giving waitlist priority to new attendees and people expected to contribute content.

Linux Plumbers Conference: Containers and Checkpoint/Restore MC CFP

Sunday 3rd of September 2023 09:26:25 AM

The Containers and Checkpoint/Restore micro-conference focuses on both userspace and kernel related work. The micro-conference targets the wider container ecosystem ideally with participants from all major container runtimes as well as init system developers.

The microconference will be discussing recent advancements in container technologies with some of the usual candidates being:

  • CGroupV2 feature parity with CGroupV1
  • Emulation of various files and system calls through FUSE and/or Seccomp
  • Dealing with the eBPF-ification of the world
  • Making user namespaces more accessible
  • VFS idmap improvements

On the checkpoint/restore front, some of the potential topics include:

  • Restoring FUSE services
  • Handling GPUs
  • Dealing with restartable sequences

And quite likely a variety of other container and checkpoint/restore topics as things evolve between now and the event.

Past editions of this micro-conference have been the source of many developments in the Linux kernel, including:

Use LPC abstract submission page to submit your proposals and select “Containers and Checkpoint/Restart” track.

Dave Airlie (blogspot): Talk about compute and community and where things are at.

Friday 1st of September 2023 07:12:17 PM

 Sriram invited me to the oneAPI meetup, and I felt I hadn't summed up the state of compute and community development in a while. Enjoy 45 minutes of opinions!

Linux Plumbers Conference: Power Management and Thermal Control MC CFP

Thursday 31st of August 2023 11:37:57 AM

The Power Management and Thermal Control microconference focuses on power management and thermal control infrastructure, CPU and device power-management mechanisms, and thermal control methods.

In particular, we are interested in improving the thermal control infrastructure in the kernel to cover more use cases and utilizing energy-saving opportunities offered by modern hardware in new ways.

The goal is to facilitate cross-framework and cross-platform discussions that can help improve energy-awareness and thermal control in Linux.

The current list of topics proposed so far includes the following:


Matthew Garrett: Unix sockets, Cygwin, SSH agents, and sadness

Tuesday 29th of August 2023 06:57:29 AM
Work involves supporting Windows (there's a lot of specialised hardware design software that's only supported under Windows, so this isn't really avoidable), but also involves git, so I've been working on extending our support for hardware-backed SSH certificates to Windows and trying to glue that into git. In theory this doesn't sound like a hard problem, but in practice oh good heavens.

Git for Windows is built on top of msys2, which in turn is built on top of Cygwin. This is an astonishing artifact that allows you to build roughly unmodified POSIXish code on top of Windows, despite the terrible impedance mismatches inherent in this. One is that until 2017, Windows had no native support for Unix sockets. That's kind of a big deal for compatibility purposes, so Cygwin worked around it. It's, uh, kind of awful. If you're not a Cygwin/msys app but you want to implement a socket they can communicate with, you need to implement this undocumented protocol yourself. This isn't impossible, but ugh.

But going to all this trouble helps you avoid another problem! The Microsoft version of OpenSSH ships an SSH agent that doesn't use Unix sockets, but uses a named pipe instead. So if you want to communicate between Cygwinish OpenSSH (as is shipped with git for Windows) and the SSH agent shipped with Windows, you need something that bridges between those. The state of the art seems to be to use npiperelay with socat, but if you're already writing something that implements the Cygwin socket protocol you can just use npipe to talk to the shipped ssh-agent and then export your own socket interface.

And, amazingly, this all works? I've managed to hack together an SSH agent (using Go's SSH agent implementation) that can satisfy hardware backed queries itself, but forward things on to the Windows agent for compatibility with other tooling. Now I just need to figure out how to plumb it through to WSL. Sigh.


Linux Plumbers Conference: Confidential Computing MC CFP

Tuesday 29th of August 2023 04:48:16 AM

Confidential Computing is continuing to remain a popular topic in computing industry. From memory encryption to trusted I/O, hardware has been constantly improving and broadening. In the past years,  confidential computing microconferences have brought together developers working on various features in hypervisors, firmware, Linux kernel, low level userspace up to container runtimes. We have  discussed a broad range of topics, ranging from, hardware enablement to generic attestation workflows.

Just in the last year, we have seen support for Intel TDX and AMD SEV-SNP guests merged into Linux. Support for unaccepted memory has also landed in mainline. We have also had support for running as a CVM under Hyper-V partially merged into the kernel. However, there is still a long way to go before a complete Confidential Computing stack with open source software and Linux as the hypervisor becomes a reality. We invite contributions to this microconference to help make progress to that goal.

Topics of interest include

Please use the LPC CfP process to submit your proposals. Submissions can be made via the LPC abstract submission page. Make sure to select “Confidential Computing MC” as the track.

Linux Plumbers Conference: IoT MC CFP

Wednesday 23rd of August 2023 02:53:28 PM

The IoT Microconference is a forum for developers to discuss all things IoT. Topics include tools, telemetry, device drivers, and protocols in not only the Linux kernel but also Real-Time Operating Systems such as Zephyr.

Since last year, there have been a number of new technical topics with significant updates.

  • Opportunities in IoT and Edge computing with the Linux /dev/accel API
  • Using the Thrift RPC framework between Linux and Zephyr
  • Zephyr’s new HTTP Server (a GSoC project)
  • Rust in the Zephyr RTOS: Benefits, Challenges and Missing Pieces
  • BeagleConnect Freedom Updates, Greybus, and the Linux Interface
  • Linux-wpan updates on 6lowpan, 802.15.4 PAN coordinators and UWB

Current Problems that require attention (stakeholders):

  • IEEE 802.15.4 SubGHz improvement areas in Zephyr, Linux (Florian Grandel, Stefan Schmidt,
  • WpanUSB upstreaming in the Linux kernel, potentially dropping Zephyr support (Andrei Emeltchenko,
  • IEEE 802.15.4 Linux subsystem device association handling (Miquel Raynal, Alexander Aring, Stefan Schmidt)
  • Zephyr potentially dropping Bluetooth IPSP?

On a slightly less technical topic.

  • Reflections after Two Years of Zephyr LTSv2

We are pleased to announce that the IoT Microconference is now accepting proposals!

If you are interested in presenting an IoT-related topic involving the Linux kernel, userspace tools, firmware, Zephyr, or frameworks, please upload your submission before September 15th.

Submissions can be made via the LPC Call for Proposals, by selecting Internet of Things MC for your track.

Linux Plumbers Conference: VFIO/IOMMU/PCI MC CFP

Monday 21st of August 2023 08:41:26 AM

On behalf of the PCI sub-system maintainers, we would like to invite everyone to join the VFIO/IOMMU/PCI micro-conference (MC) this year.

We are hoping to bring together, both in person and online, everyone interested in the VFIO, IOMMU, and PCI space to talk about the latest developments and challenges in these areas.

The PCI interconnect specification, the devices that implement it, and the system IOMMUs that provide memory and access control to them are nowadays a de-facto standard for connecting high-speed components, incorporating more and more features such as:

These features are aimed at high-performance systems, server and desktop computing, embedded and SoC platforms, virtualisation, and ubiquitous IoT devices.

The kernel code that enables these new system features focuses on coordination between the PCI devices, the IOMMUs they are connected to, and the VFIO layer used to manage them (for userspace access and device passthrough) with related kernel interfaces and userspace APIs to be designed in-sync and in a clean way for all three sub-systems.

The VFIO/IOMMU/PCI MC focuses on the kernel code that enables these new system features, often requiring coordination between the VFIO, IOMMU and PCI sub-systems.

Following the success of LPC 2017, 2019, 2020, 2021, and 2022 VFIO/IOMMU/PCI MC, the Linux Plumbers Conference 2023 VFIO/IOMMU/PCI track will focus on promoting discussions on the PCI core but also current kernel patches aimed at VFIO/IOMMU/PCI sub-systems with specific sessions targeting discussions requiring the three sub-systems coordination.

See the following video recordings from 2022: LPC 2022 – VFIO/IOMMU/PCI MC

Older recordings can be accessed through our official YouTube channel at @linux-pci and the archived LPC 2017 VFIO/IOMMU/PCI MC web page at Linux Plumbers Conference 2017, where the audio recordings from the MC track and links to presentation materials are available.

The tentative schedule will provide an update on the current state of VFIO/IOMMU/PCI kernel sub-systems, followed by a discussion of current issues in the proposed topics.

The following was a result of last year’s successful Linux Plumbers MC:

Tentative topics that are under consideration for this year include (but are not limited to):

  • PCI
  • VFIO
    • Write-combine on non-x86 architectures
    • I/O Page Fault (IOPF) for passthrough devices
    • Shared Virtual Addressing (SVA) interface
    • Single-root I/O Virtualization(SRIOV)/Process Address Space ID (PASID) integration
    • PASID in SRIOV virtual functions
    • Device assignment/sub-assignment
    • /dev/iommufd development
    • IOMMU virtualisation
    • IOMMU drivers SVA interface
    • DMA-API layer interactions and the move towards generic dma-ops for IOMMU drivers
    • Possible IOMMU core changes (e.g., better integration with the device-driver core, etc.)

If you are interested in participating in this MC and have topics to propose, please use the Call for Proposals (CfP) process.

Otherwise, join us to discuss helping Linux keep up with the new features added to the PCI interconnect specification. We hope to see you there!

Proposals can be submitted here here by selecting Track “VFIO/IOMMU/PCI MC

Linux Plumbers Conference: Tracing MC CFP

Thursday 17th of August 2023 01:09:16 PM

The Linux kernel has grown in complexity over the years. Complete understanding of how it works via code inspection has become virtually impossible. Today, tracing is used to follow the kernel as it performs its complex tasks. Tracing is used today for much more than simply debugging. Its framework has become the way for other parts of the Linux kernel to enhance and even make possible new features. Live kernel patching is based on the infrastructure of function tracing, as well as BPF function hooks. It is now even possible to model the behavior and correctness of the system via runtime verification which attaches to trace points. There is still much more that is happening in this space, and this microconference will be the forum to explore current and new ideas.

Results and accomplishments from the last Tracing microconference (2021):

  • User events were introduced, and have finally made it into the kernel.
  • The discussion around trace events to handle user faults initiated the event probe work around to the problem. That was to add probes on existing trace events to change their types. This works on synthetic events that can pass the user space file name of the entry of a system call to the exit of the system call which would have faulted in the file and make it available to the trace event.
  • Dynamically creating the events directory with the eventfs patch set is queued to be accepted. This will save memory as the dentries and inodes will only be allocated when accessed.
  • The discussion about function tracing with arguments has helped inspire both fprobes and function graph return value tracing.
  • There’s still ongoing effort in unifying the return path tracers of function graph and kretprobes and fprobes.

Possible ideas for topics for this year’s conference:

  • Use of sframes. How to get user space stack traces without requiring frame pointers.
  • Updating perf and ftrace to extract user space stack frames from a schedulable context (as requested by NMI).
  • Extending user events. Now that they are in the kernel, how to make them more accessible to users and applications.
  • Getting more use cases with the runtime verifier. Now that the runtime verifier is in the kernel (uses tracepoints to model against), what else can it be used for.
  • Wider use of ftrace_regs in fprobes and rethook from fprobes because rethook may not fill all registers in pt_regs too. How BPF handles this will also be discussed.
  • Removing kretprobes from kprobes so that kprobe can focus on handling software breakpoint.
  • Object tracing (following a variable throughout each function call). This has had several patches out, but has stopped due to hard issues to overcome. A live discussion could possibly come up with a proper solution.
  • Hardware breakpoints and tracing memory changes. Object tracing follows a variable when it changes between function calls. But if the hardware supports it, tracing a variable when it actually changes would be more useful albeit more complex. Discussion around this may come up with a easier answer.
  • MMIO tracer being used in SMP. Currently the MMIO tracer does not handle race conditions. Instead, it offlines all but one CPU when it is enabled. It would be great if this could be used in normal SMP environments. There’s nothing technically preventing that from happening. It only needs some clever thinking to come up with a design to do so.
  • Getting perf counters onto the ftrace ring buffer. Ftrace is designed for fast tracing, and perf is a great profiler. Over the years it has been asked to have perf counters along side ftrace trace events. Perhaps its time to finally accomplish that. It could be that each function can show the perf cache misses of that function.
For more information, feel free to contact the MC Leads: Steven Rostedt <> Masami Hiramatsu <> Please follow the suggestions from this BLOG post when submitting a CFP for this track. Submissions are made via LPC submission system, selecting Track “Tracing MC”

Linux Plumbers Conference: Kernel Testing & Dependability MC CFP

Wednesday 9th of August 2023 03:40:34 PM

Once again The Kernel Testing & Dependability Micro-conference will be taking place at LPC 2023, to discuss testing and dependability related topics.

Please submit proposals for discussion via LPC submission system.

The Linux Plumbers 2023 Kernel Testing & Dependability track focuses on advancing the current state of testing of the Linux Kernel and its related infrastructure. The main purpose is to improve software quality and dependability for applications that require predictability and trust.

The goal of this micro-conference is making connections between folks working on similar projects, and help individual projects make progress.

This track is intended to promote collaboration between all the communities and people interested in the Kernel testing & dependability. This will help move the conversation forward from where we left off at the LPC 2022 Kernel Testing & Dependability MC.

We ask that any topic discussions focus on issues/problems they are facing and possible alternatives to resolving them. The Micro-conference is open to all topics related to testing on Linux, not necessarily in the kernel space.

Suggested topics:

  • KernelCI: Topics on improvements and enhancements for test coverage
  • Growing KCIDB, integrating more sources
  • Sanitizers
  • Using Clang for better testing coverage
  • How to spread KUnit throughout the kernel?
  • Building and testing in-kernel Rust code
  • Explore ways to improve testing framework and tests in the kernelwith a specific goal to increase traceability and code coverage
  • Explore how do SBOMs figure into dependability?

List of accomplishments this past year after LPC 2022:

  • Developed a new, modern API for KernelCI with Pub/Sub interface
  • Added Rust coverage in KernelCI
  • KCIDB is continuing to gather results from many test systems: KernelCI, Red Hat’s CKI, syzbot, ARM, Gentoo, Linaro’s TuxSuite etc. The current focus is on generating common email reports based  on this data and dealing with known issues.
  • KFENCE is continuing to aid in detecting Out-of-bound OOB accesses, use-after-free errors (UAF),  Double free and Invalid free and so on.
  • Clang: CFI, weeding out issues upstream, etc.
  • Kselftest continues to add coverage for new and existing features and subsystems.
  • KUnit is continuing to act as the standard for some drivers and a de facto unit testing framework in the kernel
  • The Runtime Verification (RV) interface from Daniel Bristot de Oliveira was merged.

Proposals can be submitted here, by August 20th:

MC leads can be reached for question and further information::
Shuah Khan (
Sasha Levin <>
Guillaume Tucker <>

Linux Plumbers Conference: Live Patching MC CFP

Wednesday 9th of August 2023 03:38:51 PM

After a three-year hiatus, the Live Patching Microconference is back for 2023.

Accomplishments post 2019 Microconference:

  • API enhancements: Livepatch pre/post (un)patch callback system state change tracking was added in v5.5. The new API enhances the safety of cumulative livepatch upgrades [v5.5]
  • KLP-relocations: To facilitate module_disable_ro() removal, arch-specific livepatch .klp.arg sections were deprecated. Special arch section KLP-relocations (like x86 jump labels) are still supported for vmlinux cases, and are now applied at the same time as normal relocations. [v5.8]
  • Documentation: Practical information on how to implement reliable stacktraces needed by the livepatching consistency model was added [v5.12]
  • Architecture: Implemented Power32 support [v5.18]
  • KLP-relocations: To support target module reloading, clear KLP-relocations in livepatch modules when their target module is unloaded. This satisfies a module loader sanity check when resolving relocations on the next target module load (x86_64 only) [v6.3]

Discussion Topics

The following topics have been proposed:

  • Shadow variables are considered a livepatching power-feature that can require careful management, especially across livepatch up and downgrades. Is garbage collection or a refactoring of callbacks a  better approach to manage these resources?
  • klp-relocations were originally introduced to resolve livepatch / kernel and module symbol scoping issues. Recent security features like CET and IBT suggest another use case and renewed interest in having an in-tree klp-relocation build support. Is a simple conversion utility sufficient, or does said tool require greater features?
  • The livepatching kselftests consist of test scripts under tools/testing/selftests and associated livepatch module code in lib/. Consolidating these under the former offers better flexibility in templating the livepatch modules as well as the benefits of building them out-of-tree.  Are there any outstanding blockers to implement these changes?
  • arm64 support is moving forward on several fronts: toolchain, reliable stack unwinding, user space, etc.  The Toolchains MC plans to address topics like CFG in ELF and handling of noinstr functions.  What issues remain in livepatching and the kernel at large to fully support arm64?
  • Rust looks to be a hot topic at this year’s LPC.  Its impact on kernel livepatching is relatively open ended as Rust code has only recently been merged in small parts.  That said, which features, problems, patchsets should we be paying attention to as we all learn more about this newly supported kernel language?

These potential discussion topics were selected from on-going livepatching mailing list threads, but additional livepatching related topics are welcome for consideration as well. For ideas on what makes for an ideal Microconference topic, checkout this post.

Matthew Garrett: Updating Fedora the unsupported way

Tuesday 8th of August 2023 05:54:01 AM
I dug out a computer running Fedora 28, which was released 2018-04-01 - over 5 years ago. Backing up the data and re-installing seemed tedious, but the current version of Fedora is 38, and while Fedora supports updates from N to N+2 that was still going to be 5 separate upgrades. That seemed tedious, so I figured I'd just try to do an update from 28 directly to 38. This is, obviously, extremely unsupported, but what could possibly go wrong?

Running sudo dnf system-upgrade download --releasever=38 didn't successfully resolve dependencies, but sudo dnf system-upgrade download --releasever=38 --allowerasing passed and dnf started downloading 6GB of packages. And then promptly failed, since I didn't have any of the relevant signing keys. So I downloaded the fedora-gpg-keys package from F38 by hand and tried to install it, and got a signature hdr data: BAD, no. of bytes(88084) out of range error. It turns out that rpm doesn't handle cases where the signature header is larger than a few K, and RPMs from modern versions of Fedora. The obvious fix would be to install a newer version of rpm, but that wouldn't be easy without upgrading the rest of the system as well - or, alternatively, downloading a bunch of build depends and building it. Given that I'm already doing all of this in the worst way possible, let's do something different.

The relevant code in the hdrblobRead function of rpm's lib/header.c is:

int32_t il_max = HEADER_TAGS_MAX;
int32_t dl_max = HEADER_DATA_MAX;

il_max = 32;
dl_max = 8192;

which indicates that if the header in question is RPMTAG_HEADERSIGNATURES, it sets more restrictive limits on the size (no, I don't know why). So I installed rpm-libs-debuginfo, ran gdb against, loaded the symbol file, and then did disassemble hdrblobRead. The relevant chunk ends up being:

0x000000000001bc81 <+81>: cmp $0x3e,%ebx
0x000000000001bc84 <+84>: mov $0xfffffff,%ecx
0x000000000001bc89 <+89>: mov $0x2000,%eax
0x000000000001bc8e <+94>: mov %r12,%rdi
0x000000000001bc91 <+97>: cmovne %ecx,%eax

which is basically "If ebx is not 0x3e, set eax to 0xffffffff - otherwise, set it to 0x2000". RPMTAG_HEADERSIGNATURES is 62, which is 0x3e, so I just opened in hexedit, went to byte 0x1bc81, and replaced 0x3e with 0xfe (an arbitrary invalid value). This has the effect of skipping the if (regionTag == RPMTAG_HEADERSIGNATURES) code and so using the default limits even if the header section in question is the signatures. And with that one byte modification, rpm from F28 would suddenly install the fedora-gpg-keys package from F38. Success!

But short-lived. dnf now believed packages had valid signatures, but sadly there were still issues. A bunch of packages in F38 had files that conflicted with packages in F28. These were largely Python 3 packages that conflicted with Python 2 packages from F28 - jumping this many releases meant that a bunch of explicit replaces and the like no longer existed. The easiest way to solve this was simply to uninstall python 2 before upgrading, and avoiding the entire transition. Another issue was that some data files had moved from libxcrypt-common to libxcrypt, and removing libxcrypt-common would remove libxcrypt and a bunch of important things that depended on it (like, for instance, systemd). So I built a fake empty package that provided libxcrypt-common and removed the actual package. Surely everything would work now?

Ha no. The final obstacle was that several packages depended on rpmlib(CaretInVersions), and building another fake package that provided that didn't work. I shouted into the void and Bill Nottingham answered - rpmlib dependencies are synthesised by rpm itself, indicating that it has the ability to handle extensions that specific packages are making use of. This made things harder, since the list is hard-coded in the binary. But since I'm already committing crimes against humanity with a hex editor, why not go further? Back to editing and finding the list of rpmlib() dependencies it provides. There were a bunch, but I couldn't really extend the list. What I could do is overwrite existing entries. I tried this a few times but (unsurprisingly) broke other things since packages depended on the feature I'd overwritten. Finally, I rewrote rpmlib(ExplicitPackageProvide) to rpmlib(CaretInVersions) (adding an extra '\0' at the end of it to deal with it being shorter than the original string) and apparently nothing I wanted to install depended on rpmlib(ExplicitPackageProvide) because dnf finished its transaction checks and prompted me to reboot to perform the update. So, I did.

And about an hour later, it rebooted and gave me a whole bunch of errors due to the fact that dbus never got started. A bit of digging revealed that I had no /etc/systemd/system/dbus.service, a symlink that was presumably introduced at some point between F28 and F38 but which didn't get automatically added in my case because well who knows. That was literally the only thing I needed to fix up after the upgrade, and on the next reboot I was presented with a gdm prompt and had a fully functional F38 machine.

You should not do this. I should not do this. This was a terrible idea. Any situation where you're binary patching your package manager to get it to let you do something is obviously a bad situation. And with hindsight performing 5 independent upgrades might have been faster. But that would have just involved me typing the same thing 5 times, while this way I learned something. And what I learned is "Terrible ideas sometimes work and so you should definitely act upon them rather than doing the sensible thing", so like I said, you should not do this in case you learn the same lesson.


Linux Plumbers Conference: Build Systems MC CFP

Saturday 5th of August 2023 09:48:18 AM

In the Linux ecosystems, there are many ways to build all the software used to put together a running system. Whether it’s building all the binary packages for a binary Linux distribution, using a source-based distribution, or building an embedded system from scratch, there are a lot of shared challenges which each system solves in its own way.

This microconference is a way to get people who work on disparate build systems to discuss common problems and possible shared solutions across the entire problem space. The kinds of topics we want to discuss are the following:
  • Bootstrapping the build system
  • Cross building software
  • Make, autoconf, and other similar software build tools
  • Package build systems, bitbake, emerge/portage, pacman, etc
  • Packaging formats
  • Managing software with language-specific package managers
  • Patch sharing
  • Building within a container
  • Build systems for building containers
  • License gathering and verification
  • Security updates
  • Software chain-of-trust
  • Repeatable builds
  • Documentation and education
  • Finding the next generation of maintainers
  • Build-system visibility within the wider Plumbers attendeesThis is not a definitive list, and you are free to post abstracts for other related topics.
Build Systems micorconference would like to gather representatives (developers and maintainers) from all the various build systems and related technologies. This is not a definitive list of possible attendees.
  • Android
  • Arch Linux
  • Buildroot
  • ChromeOS
  • Gentoo
  • OpenEmbedded
  • OpenWRT/LEDE
  • Yocto Project
  • Other traditional Binary Packaged distributions
For more information, feel free to contact the MC Leads: Behan Webster <> Philip Balister <> Please follow the suggestions from this BLOG post when submitting a CFP for this track. Submissions are made via LPC submission system, selecting Track “Build Systems MC”

Dave Airlie (blogspot): nvk: the kernel changes needed

Friday 4th of August 2023 10:26:49 PM

The initial NVK (nouveau vulkan) experimental driver has been merged into mesa master[1], and although there's lots of work to be done before it's application ready, the main reason it was merged was because the initial kernel work needed was merged into drm-misc-next[2] and will then go to drm-next for the 6.6 merge window. (This work is separate from the GSP firmware enablement required for reclocking, that is a parallel development, needed to make nvk useable). Faith at Collabora will have a blog post about the Mesa side, this is more about the kernel journey.

What was needed in the kernel?

The nouveau kernel API was written 10 years or more ago, and was designed around OpenGL at the time. There were two major restrictions in the current uAPI that made it unsuitable for Vulkan.

  1. buffer objects (physical memory allocations) were allocated 1:1 with virtual memory allocations for a file descriptor. This meant the kernel managed the virtual address space. For proper Vulkan support, the bo allocation and vm allocation have to be separate, and userspace should control the virtual address space.
  2. Command submission didn't use sync objects. The nouveau command submission wasn't wired up to the modern sync objects. These are pretty much a requirement for Vulkan fencing and semaphores to work properly.
How to implement these?

When we kicked off the nvk idea I made a first pass at implementing a new user API, to allow the above features. I took at look at how the GPU VMA management was done in current drivers and realized that there was a scope for a common component to manage the GPU VA space. I did a hacky implementation of some common code and a nouveau implementation. Luckily at the time, Danilo Krummrich had joined my team at Red Hat and needed more kernel development experience in GPU drivers. I handed my sketchy implementation to Danilo and let him run with it. He spent a lot of time learning and writing copious code. His GPU VA manager code was merged into drm-misc-next last week and his nouveau code landed today.

What is the GPU VA manager?

The idea behind the GPU VA manager is that there is no need for every driver to implement something that should essentially not be a hardware specific problem. The manager is designed to track VA allocations from userspace, and keep track of what GEM objects they are currently bound to. The implementation went through a few twists and turns and experiments. 

For a long period we considered using maple tree as the core of it, but we hit a number of messy interactions between the dma-fence locking and memory allocations required to add new nodes to the maple tree. The dma-fence critical section is a hard requirement to make others deal with. In the end Danilo used an rbtree to track things. We will revisit if we can deal with maple tree again in the future. 

We had a long discussion and a couple of implement it both ways and see, on whether we needed to track empty sparse VMA ranges in the manager or not,  nouveau wanted these but generically we weren't sure they were helpful, but that also affected the uAPI as it needed explicit operations to create/drop these. In the end we started tracking these in the driver and left the core VA manager cleaner.

Now the code is in tree we will start to push future drivers to use it instead of spinning their own.

What changes are needed for nouveau?

Now that the VAs are being tracked, the nouveau API needed two new entrypoints. Since BO allocation will no longer create a VM, a new API is needed to bind BO allocations with VM addresses. This is called the VM_BIND API. It has two variants

  1. a synchronous version that immediately maps a BO to a VM and is used for the common allocation paths.
  2. an asynchronous version that is modeled after the Vulkan sparse API, and takes in/out sync objects, which use the drm scheduler to schedule the vm/bo binding.
The VM BIND backend then does all the page table manipulation required. 
The second API added was an EXEC call. This takes in/out sync objects and a set of addresses that point to command buffers to execute. This uses the drm scheduler to deal with the synchronization and hands the firmware the command buffer address to execute.Internally for nouveau this meant having to add support for the drm scheduler, adding new internal page table manipulation APIs, and wiring up the GPU VA. Shoutouts:

My input was the sketchy sketch at the start, and doing the userspace changes to the nvk codebase to allow testing.

The biggest shoutout to Danilo, who took a sketchy sketch of what things should look like, created a real implementation, did all the experimental ideas I threw at him, and threw them and others back at me, negotiated with other drivers to use the common code, and built a great foundational piece of drm kernel infrastructure.

Faith at Collabora who has done the bulk of the work on nvk did a code review at the end and pointed out some missing pieces of the API and the optimisations it enables.

Karol at Red Hat on the main nvk driver and Ben at Red Hat for nouveau advice on how things worked, while he smashed away at the GSP rock.

(and anyone else who has contributed to nvk, nouveau and even NVIDIA for some bits :-)



Linux Plumbers Conference: Linux Plumbers Refereed Track – Upcoming Deadline – August 6, 2023

Wednesday 2nd of August 2023 04:03:17 PM

August is now upon us, and the deadline for refereed track submissions is August 6, which is right around the corner. We have already received some excellent submissions, for which we gratefully thank our submitters!

For those thinking about submitting, please polish off your ideas, and point your browsers at the call-for-proposals page. Looking forward to your submissions.

Reminder: we’ve got a tight deadline to prepare the submissions for the LPC program committee to review, so, as communicated last year, we will not be extending the deadline this year, please submit by August 6th, anywhere on earth.

Linux Plumbers Conference: Rust MC CFP

Sunday 30th of July 2023 07:05:27 AM

LPC 2023 will host the second edition of the Rust MC. This microconference intends to cover talks and discussions on both Rust for Linux as well as other non-kernel Rust topics. Proposals can be submitted via LPC submission system, selecting the Rust MC track.

Rust is a systems programming language that is making great strides in becoming the next big one in the domain. Rust for Linux is the project adding support for the Rust language to the Linux kernel.

Rust has a key property that makes it very interesting as the second language in the kernel: it guarantees no undefined behavior takes place (as long as unsafe code is sound). This includes no use-after-free mistakes, no double frees, no data races, etc. It also provides other important benefits, such as improved error handling, stricter typing, sum types, pattern matching, privacy, closures, generics, etc.

Possible Rust for Linux topics:

  • Rust in the kernel (e.g. status update, next steps…).
  • Use cases for Rust around the kernel (e.g. subsystems, drivers,
    other modules…).
  • Discussions on how to abstract existing subsystems safely, on API design, on coding guidelines…
  • Integration with kernel systems and other infrastructure (e.g. build system, documentation, testing and CIs, maintenance, unstable features, architecture support, stable/LTS releases, Rust versioning, third-party crates…).Updates on its subprojects (e.g. klint, pinned-init)

Possible Rust topics:

  • Language and standard library (e.g. upcoming features, stabilization of the remaining features the kernel needs, memory model…).
  • Compilers and codegen (e.g. rustc improvements, LLVM and Rust, rustc_codegen_gcc, Rust GCC…).
  • Other tooling and new ideas (bindgen, Cargo, Miri, Clippy, Compiler Explorer, Coccinelle for Rust…).
  • Educational material.
  • Any other Rust topic within the Linux ecosystem.

Last year was the first edition of the Rust MC and the focus was on showing the ongoing efforts by different parties (compilers, Rust for Linux, CI, eBPF…). Shortly after the Rust MC, Rust got merged into the Linux kernel. Abstractions are getting upstreamed, with the first major drivers looking to be merged soon: Android Binder, the Asahi GPU driver and the NVMe driver (presented in that MC).

Daniel Vetter: EOSS Prague: Kernel Locking Engineering

Friday 28th of July 2023 12:00:00 AM

EOSS in Prague was great, lots of hallway track, good talks, good food, excellent tea at meetea - first time I had proper tea in my life, quite an experience. And also my first talk since covid, pack room with standing audience, apparently one of the top ten most attended talks per LF’s conference report.

The video recording is now uploaded, I’ve uploaded the fixed slides, including the missing slide that I accidentally cut in a last-minute edit. It’s the same content as my blog posts from last year, first talking about locking engineering principles and then the hierarchy of locking engineering patterns.

Linux Plumbers Conference: Android MC CFP

Thursday 27th of July 2023 05:12:10 AM

The Android Microconference brings the upstream community and Android systems developers together to discuss issues and changes to the Android platform and their dependencies and interactions with the Linux  kernel, allowing for collaboration on solutions for upstream.

Since last year’s conference, there has been quite a bit of progress, specifically around:

Currently planned discussion topics for this year include:

  • 16k Pages
  • RISC-V
  • android-mainline on Pixel6
  • Updates on Binder
  • BPF usage w/ Android
  • Kernel and platform integration testing
  • Vendor Hook Usage
  • Building Modules for Android GKI Kernels
  • Resolving Priority Inversion w/ Proxy Execution
  • AOSP Devboards
  • And likely more…

People are encouraged to submit topics related to new  Android functionality as well as issues in getting that functionality upstream.

Please consider that the goal is to discuss open problems, preferably with patch set submissions already in discussion on LKML. The slots are very short (10-15 mins), and the main portion of the time should be given to the debate – thus, the importance of having an open and relevant problem, with people in the community engaged in the solution.

The CFP for the Android Micro-conference closes on Aug 15th, so get your topics in early!

Additionally, we already have a busy tentative schedule, but please submit your topics, and should it not fit, we hope to have additional discussion space in a follow-on BoF.

More in Tux Machines

digiKam 7.7.0 is released

After three months of active maintenance and another bug triage, the digiKam team is proud to present version 7.7.0 of its open source digital photo manager. See below the list of most important features coming with this release. Read more

Dilution and Misuse of the "Linux" Brand

Samsung, Red Hat to Work on Linux Drivers for Future Tech

The metaverse is expected to uproot system design as we know it, and Samsung is one of many hardware vendors re-imagining data center infrastructure in preparation for a parallel 3D world. Samsung is working on new memory technologies that provide faster bandwidth inside hardware for data to travel between CPUs, storage and other computing resources. The company also announced it was partnering with Red Hat to ensure these technologies have Linux compatibility. Read more

today's howtos

  • How to install go1.19beta on Ubuntu 22.04 – NextGenTips

    In this tutorial, we are going to explore how to install go on Ubuntu 22.04 Golang is an open-source programming language that is easy to learn and use. It is built-in concurrency and has a robust standard library. It is reliable, builds fast, and efficient software that scales fast. Its concurrency mechanisms make it easy to write programs that get the most out of multicore and networked machines, while its novel-type systems enable flexible and modular program constructions. Go compiles quickly to machine code and has the convenience of garbage collection and the power of run-time reflection. In this guide, we are going to learn how to install golang 1.19beta on Ubuntu 22.04. Go 1.19beta1 is not yet released. There is so much work in progress with all the documentation.

  • molecule test: failed to connect to bus in systemd container - openQA bites

    Ansible Molecule is a project to help you test your ansible roles. I’m using molecule for automatically testing the ansible roles of geekoops.

  • How To Install MongoDB on AlmaLinux 9 - idroot

    In this tutorial, we will show you how to install MongoDB on AlmaLinux 9. For those of you who didn’t know, MongoDB is a high-performance, highly scalable document-oriented NoSQL database. Unlike in SQL databases where data is stored in rows and columns inside tables, in MongoDB, data is structured in JSON-like format inside records which are referred to as documents. The open-source attribute of MongoDB as a database software makes it an ideal candidate for almost any database-related project. This article assumes you have at least basic knowledge of Linux, know how to use the shell, and most importantly, you host your site on your own VPS. The installation is quite simple and assumes you are running in the root account, if not you may need to add ‘sudo‘ to the commands to get root privileges. I will show you the step-by-step installation of the MongoDB NoSQL database on AlmaLinux 9. You can follow the same instructions for CentOS and Rocky Linux.

  • An introduction (and how-to) to Plugin Loader for the Steam Deck. - Invidious
  • Self-host a Ghost Blog With Traefik

    Ghost is a very popular open-source content management system. Started as an alternative to WordPress and it went on to become an alternative to Substack by focusing on membership and newsletter. The creators of Ghost offer managed Pro hosting but it may not fit everyone's budget. Alternatively, you can self-host it on your own cloud servers. On Linux handbook, we already have a guide on deploying Ghost with Docker in a reverse proxy setup. Instead of Ngnix reverse proxy, you can also use another software called Traefik with Docker. It is a popular open-source cloud-native application proxy, API Gateway, Edge-router, and more. I use Traefik to secure my websites using an SSL certificate obtained from Let's Encrypt. Once deployed, Traefik can automatically manage your certificates and their renewals. In this tutorial, I'll share the necessary steps for deploying a Ghost blog with Docker and Traefik.