NetBSD News Beat

NetBSD's packages collection knows the concept of "bulk builds" since the NetBSD 1.3 days, when I wrote the bulk build infrastructure in pkgsrc/mk/bulk to test if all packages build, and to rebuild (only) things that were changed. The infrastructure grew, Dan McMahill helped to optimize many points, and it was working for its purpose. A few drawbacks were inherent, though -- a long phase of scanning all Makefiles before the build and the inability to build on more than one machine/CPU were two of the more annoying ones.

In an attempt to solve those problems, and offer other features, Joerg Sonnenberger has worked on the "pbulk" system for some optimizations. The system is still in development, and documentation is somewhat spread over several places, which makes it not a first-hand replacement for the first system.

Due to this, Aleksey Cheusov has adapted pbulk and improved it into distbb, a tool for distributed bulk builds.

Aleksey compares his system to the other ones in a separate mail. Key items are the use of many separate tools for processing bits, with the core in /bin/sh and awk. A major point for forking pbulk was also the easier maintenance for Aleksey as his own project, with less communication overhead for coordination with the pbulk author.

I guess we'll see what system is the better one. For me, I'll define the winner as the system that gives me binary packages for a NetBSD release for all platforms that NetBSD supports.

The following kernel-related projects were raised in the past few weeks:

• Kernel Preemption: Andrew Doran has continued his work towards fine-grained locking, and he has proposed a patch to implement kernel preemption, i.e. that in a realtime environment, high-priority processes can interrupt system calls running inside the kernel.

  Handling the Floating Point Unit (FPU) was added later on -- the FPU needs special attention as saving and restoring is expensive, and doesn't need to be done in many cases. But if a program uses it, care must be taken to handle the case. The exact handling is explained by Christoph Egger.

  While there, Christoph also outlined the roadmap for getting realtime support in NetBSD - there are still a number of bits missing, but being able to preempt the kernel is a good first step!

• Fine-grained socket locking: In order to allow fine-grained locking (instead of blocking all other processes from entering the kernel, as is done in the "biglock" SMP approach), many kernel subsystems need to be changed. The socket system is the core part of interprocess communication, and Andrew Doran has changed it to use fine-grained locking now.

  In that context, the question of what code still runs with the biglock held, and Andrew gave an overview where more work is needed: some file systems (lfs, ext2fs, nfs), most of the drivers, protocols like TCP/IP, Veriexec, and some machine-dependent parts.

  Veriexec-Hacker Brett Lymn added details on the status of Veriexec with respect to its transition towards fine-grained locking.

• Kernel modules and ramdisk: A change in kernel modules was proposed some time ago, and Andrew Doran has used this scheme now to unify the way many ports handle the install media: There, the kernel loaded contains a ramdisk (miniroot) image inside the kernel, which is then used as root-filesystem for the kernel, containing the install tools.

  In order to split things and eventually use a stock GENERIC kernel for both running and installing, Andy has changed the x86 boot process to load the miniroot as a kernel module.

  When booting it may be useful to select one of several ramdisks: one for installing, and one for resuing the system, For this, the recently introduced boot.cfg file was extended to handle kernel modules in the boot menu.

  Izumi Tsutsui has made an ISO with all changes for testing available.

• Device File System (devfs): Another area of the kernel where a lot of work is currently being done by Matt Fleming is NetBSD's device driver infrastructure, esp. under aspects of dynamic attaching, detaching, and suspending (power management!). To talk to the various drivers, device nodes in the /dev directory are kept right now, but those are static and need to be updated when a new driver is added. Matt is working on a Device Filesystem (devfs) that dynamically created /dev from the list of devices inside the kernel. The fileysstem will also handle dynamic creation and deletion of nodes, and as an important case it will also keep permissions across reboots, if someone changes permissions manually.

  The work is at a very mature point right now and needs some testing - see Matt's mail to the tech-kern list for more information!

• Testing driver attachment: While talking about testing of device drivers, David Young has reminded driver developers to test individual drivers' detachment and re-attachment, suspension and resumption after changes. He has also posted a how-to for those tests, using drvctl(8). (The manpage needs some updating, sorry -- UTSL :-)

Leonardo Taccari had the idea to make a reference card for NetBSD and pkgsrc, and has posted about it. The card is available in english and italian language, and covers links, commands for audio, localization, managing users and groups, monitoring the system, the NetBSD rc.d startup system, starting and stopping of services, wscons, pkgsrc and related tools and more.

After the DaemonNews paper magazine is long gone, there's a new paper magazine BSD Magazine now! Dru Lavigne lists the contents in her blog article on the mag, shamelessly ripped off here:

• Dru Lavigne, FreeBSD 7.0 Installation and Configuration
• Michael Lucas, FreeBSD's bsnmp
• Jan Stedehouder, Pushing BSD an an Open Source Desktop
• Svetoslav Chukov, PC-BSD Overview
• Richard Bejtlich, Sguil 0.7.0 on FreeBSD 7.0
• Jay Kruizenga, How to Dual-Boot Vista with BSD
• Peter Hansteen, Keep Smiling, Waste Spammers' Time
• Henrik Lund Kramshoj, Defense in Depth and FOSS
• Donald Hayford, NetBSD on the NSLU2
• Girish Venkatachalam, OpenBSD pf
• Eric Schnoebelen, Instant Messaging with jabber/XMPP
• Federico Biancuzzi, Interview with FreeBSD Developer Jeff Roberson
• Mikel King, What is in a Certification
• Henrik Lund Kramshoj, Review of the Book of PF

I guess this is a very good thing, and an opportunity to show what NetBSD is and can do. I'm sure the publishers will be happy to receive more articles about NetBSD -- contribute!