ZFS as whole has 2 main ways it can be accessed. The first is ZVOL and the second is ZPL. In my first status update I said that I had ported ZVOL layer to NetBSD, and I was able to create and use ZFS Zpools and Zvols (Logical partitions exported from one disk storage pool called zpool).

Over the last few weeks I have worked on a ZPL port. ZPL is ZFS file system layer. I have ported zfs_vfsops.c file and zfs_vnops.c file to NetBSD. Today I have ZFS to state where I can mount ZFS data set, copy whole kernel source tree there and finally build NetBSD kernel on it.

$ su 
# modload /mod/solaris.kmod                                                                                                                       
# modload /mod/zfs.kmod                                                                                                                           
# zfs mount test/zfs
 # mount 
/dev/wd0a on / type ffs (local)
kernfs on /kern type kernfs (local)
ptyfs on /dev/pts type ptyfs (local)
/dev/zvol/dsk/test/zfs on /test/zfs type zfs (local)
# zfs list 
NAME        USED  AVAIL  REFER  MOUNTPOINT
test        391M  1.57G    18K  /test
test/tank    40M  1.61G    16K  -
test/zfs    351M  1.57G   351M  /test/zfs
test/zfs1    18K  1.57G    18K  /test/zfs1
# cd /test/zfs/src/sys/arch/i386/compile/GENERIC/
# make           
making sure the compat library is up to date...
`libcompat.a' is up to date.
making sure the kern library is up to date...
`libkern.o' is up to date.
#   compile  GENERIC/init_main.o
cc -ffreestanding -fno-zero-initialized-in-bss -O2 -std=gnu99 -fno-strict-aliasing -Werror -Wall -Wno-main -Wno-format-zero-length -Wpointer-arith -Wmissing-prototypes -Wstrict-prototypes -Wswitch -Wshadow -Wcast-qual -Wwrite-strings -Wno-unreachable-code -Wno-sign-compare -Wno-pointer-sign -Wno-attributes -Wextra -Wno-unused-parameter -Werror -Di386 -I. -I../../../../../common/include -I../../../../arch -I../../../.. -nostdinc -DMAXUSERS=64 -D_KERNEL -D_KERNEL_OPT -I../../../../lib/libkern/../../../common/lib/libc/quad -I../../../../lib/libkern/../../../common/lib/libc/string -I../../../../lib/libkern/../../../common/lib/libc/arch/i386/string -I../../../../dist/ipf -I../../../../external/isc/atheros_hal/dist -I../../../../external/isc/atheros_hal/ic -I../../../../../common/include -c ../../../../kern/init_main.c
#    create  vers.c
sh ../../../../conf/newvers.sh 
#   compile  GENERIC/vers.o
cc  -ffreestanding -fno-zero-initialized-in-bss  -O2 -std=gnu99 -fno-strict-aliasing   -Werror -Wall -Wno-main -Wno-format-zero-length -Wpointer-arith -Wmissing-prototypes -Wstrict-prototypes -Wswitch -Wshadow -Wcast-qual -Wwrite-strings -Wno-unreachable-code -Wno-sign-compare -Wno-pointer-sign -Wno-attributes -Wextra -Wno-unused-parameter  -Werror   -Di386 -I.  -I../../../../../common/include -I../../../../arch  -I../../../.. -nostdinc  -DMAXUSERS=64 -D_KERNEL -D_KERNEL_OPT -I../../../../lib/libkern/../../../common/lib/libc/quad -I../../../../lib/libkern/../../../common/lib/libc/string -I../../../../lib/libkern/../../../common/lib/libc/arch/i386/string   -I../../../../dist/ipf -I../../../../external/isc/atheros_hal/dist -I../../../../external/isc/atheros_hal/ic -I../../../../../common/include  -c vers.c
#      link  GENERIC/netbsd
ld -Map netbsd.map --cref -T ../../../../arch/i386/conf/kern.ldscript -Ttext c0100000 -e start -X -o netbsd ${SYSTEM_OBJ} ${EXTRA_OBJ} vers.o
NetBSD 5.99.14 (GENERIC) #1: Tue Jun 30 20:00:37 UTC 2009
   text    data     bss     dec     hex filename
8554455  407284  538396 9500135  90f5e7 netbsd

I tried to boot build kernel and it worked like a charm. There is still much work to do port ZFS snapshot support, properly implement security policies for ZFS access, test ZFS ACL support etc.

My work is accessible in my git repository at git://rachael.ziaspace.com/src.git in a branch called haad-zfs. You can easily clone this repo with command git clone git://rachael.ziaspace.com/src.git. To get haad-zfs branch checkout you need to use command git checkout -b haad-zfs origin/haad-zfs from src directory.

About the project

The GUID Partition Table is a new standard for disk partitioning. The GPT layout provides a set of advanced partitioning features including, but not limited to:

• modern logical block addressing (LBA)
• 64-bit LBA pointers, allowing partitions up to 8 Zbytes in size, and even bigger
• suitable for disks with sector size, other than 512 bytes
• by default up to 128 partitions per disk
• backup partition table.

The NetBSD already has support for GPT disks via dkwedges, but can't boot off a GPT partitioned disk. My GSoC project is to implement a GPT aware bootloader for the NetBSD operating system by extending its existing MBR/disklabel BIOS-based multistaged kernel loader.

Please note, although GPT comes from the EFI world and the loader somewhat mimics EFI bootstrap behavior, the project is not about writing a pure EFI loader. Instead, it will be based on BIOS interface and will technically look like an arbitrary MBR loader but with GPT parsing. Such an approach should allow the use of GPT partitioned disks on non-EFI systems as well. On EFI systems NetBSD can be bootstrapped via the Compatibility Support Module.

Current status

The project assumes development of MBR (or LBA0), Partition Boot Record, and adopting of 1st stage and 2nd stage loaders.

Currently, the MBR (LBA0) loader which parses GPT is in a beta stage. It implements the following features:

• locating GPT headers by looking at LBA1 and the last LBA of boot disk
• checking the GPT header for its signature and validating the CRC32
• walking through the partition table looking for a bootable partition
• reading the PBR from a bootable partition, testing its boot capability, and passing control to it if the test was successful.

Code cleanup and CRC32 calculation of GUID partition array (which wasn't included due to space constraints, but likely can be enabled after optimisation) are still required.

It was decided to store the primary loader on an EFI System Partition (which is FAT formatted by convention), and not use NetBSD partitions for bootstrapping. Such behavior will be similar to what EFI bootstrapping does, and the boot path will be as follows:

BIOS → LBA0 → PBR on EFI syspart → /boot → NetBSD kernel.

Fortunately, the "fatboot" (or bootxx_fat16) loader can do almost all of the required things in place of the PBR and 1st stage loader. The only missing thing is FAT lookup at an arbitrary offset on the disk (right now it assumes LBA0). It was easily implemented by having partition offset passed from GPT MBR or from any other loader through the ecx:ebx register pair. As we can't rely on all other loaders using this mechanism, the offset could be embedded into the PBR itself by installboot(8).

My current efforts are focused on getting gpt.S and fatboot.S (and corresponding management tools) in shape and ready to publish for beta testing. So the future work in terms of the GSoC project will include the following:

• code cleanup for gpt.S (LBA0) and fatboot.S
• review of bootloader installation tools and updating documentation
• adding GPT support into boot(8)
• more documentation updates and getting code in shape for CVS merge

Some additional information (including development schedule) can be found at the project home page at: http://netbsd-soc.sourceforge.net/projects/gpt/

Three new security advisories were published, covering OpenSSH, ntpd, ntpq and hack:

• NetBSD-SA2009-005 Plaintext Recovery Attack Against SSH
• NetBSD-SA2009-006 Buffer overflows in ntp
• NetBSD-SA2009-007 Buffer overflows in hack(6)

You can find more information about them on the Security and NetBSD page.

Project Description

The utility resize_ffs is a program intended to resize Berkeley Fast File Systems (FFS) by either growing or shrinking them. This filesystem is the standard filesystem for the NetBSD operating system -- a free, fast, secure, and highly portable Unix-like Open Source operating system.

There is already a preliminary utility in the NetBSD source tree, and this project will first and primarily thoroughly test it with a regression test suite and code review, and correct and fix any bugs encountered. After this is accomplished, and the utility is stable, this project will attempt to extend the utility with features such as "live" resizing, that is the ability to dynamically resize the filesystem without first having to unmount it.

More information is available on the project website.

Status Update

After a little bit of a slow start, the project is back on track. I am moving from the "just testing" phase to using that test information to hunt down bugs and also improve the program. One addition I have already added is the ability to choose the new filesystem size in kilo-, mega-, or gigabytes - as well as the default of sectors. However, it does currently assume a sector size of 512 KB in the conversion. I have identified several potential bugs, the worst of which is data corruption which seems limited to when the filesystem is growing.

At USENIX 2009 I talked about rump file systems. The paper (pdf, html) and slides are available. Additionally, USENIX members can view a video of the presentation.

paper abstract

When kernel functionality is desired in userspace, the common approach is to reimplement it for userspace interfaces. We show that use of existing kernel file systems in userspace programs is possible without modifying the kernel file system code base. Two different operating modes are explored: 1) a transparent mode, in which the file system is mounted in the typical fashion by using the kernel code as a userspace server, and 2) a standalone mode, in which applications can use a kernel file system as a library. The first mode provides isolation from the trusted computing base and a secure way for mounting untrusted file systems on a monolithic kernel. The second mode is useful for file system utilities and applications, such as populating an image or viewing the contents without requiring host operating system kernel support. Additional uses for both modes include debugging, development and testing.

The design and implementation of the Runnable Userspace Meta Program file system (rump fs) framework for NetBSD is presented. Using rump, ten disk-based file systems, a memory file system, a network file system and a userspace framework file system have been tested to be functional. File system performance for an estimated typical workload is found to be ±5% of kernel performance. The prototype of a similar framework for Linux was also implemented and portability was verified: Linux file systems work on NetBSD and NetBSD file systems work on Linux. Finally, the implementation is shown to be maintainable by examining the 1.5 year period it has been a part of NetBSD.

During this year’s Google Summer of Code I’m improving the performance of NetBSD’s regular expression library and adding support to it for wide characters.

NetBSD’s current regular expression library was initially written by Henry Spencer in 1986, when ASCII was the de facto standard. Moreover, Spencer described it as slow. So it makes real sense to replace it and seek for alternatives.

I decided to develop a new regular expression library at first. It seemed rather easy in the beginning, but turned out to be more difficult in all its details than I expected (everyone who read the regular expression specification in the POSIX standard will agree). Therefore, I was really happy when Ville Laurikari offered to release his regular expression library tre, which he wrote his master thesis about, under a 2-clause BSD licence, so that tre can become part of libc.

tre is a mature, strictly POSIX-conforming regular expression library with additional features, such as approximate matching, matching on binary data, thread safety, ABI compatibility and wide character support. tre also includes agrep, an approximate version of grep, which will be imported into NetBSD.

At the moment I’m preparing a performance benchmark, which is based on realistic regular expressions and data, to ensure that tre is verifiably efficient. If the benchmark reveals that tre is less efficient than NetBSD’s current regular expression library, I will try to address the performance issues. Afterwards tre will be integrated in libc.

Moreover, Glenn Fowler from AT&T Research developed a regression testing framework for regular expressions, which I will integrate with the atf to test and ensure POSIX-compliance.

Postfix 2.6.2, the latest stable version of the popular mail transport agent, was imported into NetBSD-current recently. The following features have been added since version 2.5.4:

• Multi-instance support introduces a new postmulti(1) command to create/add/remove/etc. additional Postfix instances. The familiar "postfix start" etc. commands now automatically start multiple Postfix instances. The good news: nothing changes when you use only one Postfix instance. See MULTI_INSTANCE_README for details.
• Multi-instance support required that some files be moved from the non-shared $config_directory to the shared $daemon_directory. The affected files are postfix-script, postfix-files and post-install.
• TLS (SSL) support was updated for elliptic curve encryption. This requires OpenSSL version 0.9.9 or later. The SMTP client no longer uses the SSLv2 protocol by default. See TLS_README for details.
• The Milter client now supports all Sendmail 8.14 Milter requests, including requests for rejected recipient addresses, and requests to replace the envelope sender address. See MILTER_README for details.
• Postfix no longer adds (Resent-) From:, Date:, Message-ID: or To: headers to email messages with "remote" origins (these are origins that don't match $local_header_rewrite_clients). Adding such headers breaks DKIM signatures that explicitly cover non-present headers. For compatibility with existing logfile processing software, Postfix will log ``message-id='' for email messages that have no Message-Id header.
• Stress-adaptive behavior is now enabled by default. This allows the Postfix SMTP server to temporarily reduce time limits and error-count limits under conditions of overload, such as a malware attack or backscatter flood. See STRESS_README for details.

Enjoy!

wake is a new command to send Wake-on-LAN frames over an ethernet to Wake-on-LAN capable machines, remote powering them up. This functionality is generally enabled in a machine's BIOS and can be used to power on machines from a remote system without having physical access to them.

wake is available in NetBSD-current. See the wake(8) manual page for details.

A couple of weeks ago, Guillaume Lasmayous and I threw the idea of interviewing NetBSD developers through our website, NetBSDfr, to promote the NetBSD Project, and to make their work known to the widest possible audience.

Today, we are discussing with Soren Jacobsen, snj@, NetBSD 5.0 release engineer.

NetBSDfr: First of all, let me thank you for this interview. We plan to run
several interviews of NetBSD developers, I think one every one or 2
months or so. You've been chosen as the first one. Thanks for accepting
to answer our questions.

snj: Thanks for asking me, and thanks for your efforts to strengthen the
community.  It's great to see efforts like this.
Sorry it took so long to get back to you.  I wanted to take some time to
think over my answers.  Please let me know if you think anything needs
clarification.

NetBSDfr: For the readers who don't know you, can you shortly introduce
yourself ?

snj: My name is Soren Jacobsen and I am a NetBSD developer and
release engineer.  I've been part of releng since late 2005, and was the
lead release engineer for the 5.0 release.

NetBSDfr: Why did you choose to run NetBSD ? How long have you been
using it ?

snj: My first experience with NetBSD was 1.5.3, which I guess was
around the second half of 2002.  Compared to a lot of people, I was a bit
late to the party.  At the time, I was frustrated with Linux in a number of
ways, and NetBSD was a solid operating system with a philosophy that
 made sense to me.  There wasn't any one particular thing about
NetBSD that grabbed my attention.  It was more that there was nothing
that bothered me.  It just seemed right.

NetBSDfr: Do you have an idea of the time you spend working on the
NetBSD project daily, weekly, monthly ?

snj: Like most volunteers, the amount I'm able to spend on NetBSD
depends on real life and my motivation level.  A year ago, at most I was
doing 8 hours a week.  More recently, 8 hours a day wasn't uncommon.
The week leading up to the release was especially hectic; I put in a
couple 16 hour days there.

NetBSDfr: How did you become a NetBSD developer ?

snj: When I started using NetBSD full time as my primary operating
system, I became heavily dependent on pkgsrc, our framework for
third-party packages.  There were a number of packages I wanted to
see updated, and I started sending patches against pkgsrc and various
small things in the base system.  At the end of 2003, I was offered an
account, and became a developer in January of 2004.

To anyone out there who is interested in helping the project: pkgsrc
is a great way to get involved.  There are so many packages out there,
and only a finite number of developers to import and maintain them.
Consider submitting PRs against pkgsrc and joining the pkgsrc-wip
project.  We can use all the help we can get in this area.

NetBSDfr: How did you become member of NetBSD release engineering
team ?
Are the members of the releng team elected ? chosen among
developers ? Does this require specific skills ?

snj: The release engineering team is rather informal.  There is no
election process, and no strictly defined leader of the group, although
people do step up to fill this position.  When we feel understaffed, we
 put out a call for volunteers, and usually a person or two will join.  

Release engineering is not as glorious as a lot of other activities, so we
don't normally have people begging to join releng.
It's hard to say about specific skills.  Like in most other areas, having
sound judgment and a good sense of perspective is essential.  These are
the only strictly required skills.  Plenty of others are useful, though:
patience, caution, attention to detail, etc.  Pretty basic stuff, for the
most part.  Of course, the more technical knowledge, the better, but this
is not as big of a deal as it might seem.  I'll explain below.

NetBSDfr: Can you explain us more in details how does NetBSD release
engineering process work ?

snj: There are two basic tasks:
- Pulling up changes into branches.
- Releasing.

The first one is rather straightforward, and it is something we do
constantly.  Developers request that certain changes be pulled up to a
given branch.  If we agree that the changes should be pulled up, we commit
them to the branch.  To help track the status of a branch, we maintain a
record of every change made to a branch over its lifetime.

Putting together a release is a much more complicated task requiring
coordination with a large number of people.  The main task is to decide
the point at which the remaining bugs can safely be ignored.  There are
lots of other tasks involved here, of course: managing the build cluster,
pulling up changes, preparing information about the release, testing,
making sure that important fixes find their way to the branch, begging
people to work on certain things, paying very close attention to the
mailing lists, etc.

NetBSDfr: How does the quality assurance process works ? Who decides
to include this or that patch to a branch ?  This is also linked I think to
the following question: how does the releng team decide how many
betas, RCs versions are required before declaring a branch is stable ?

snj: No one person can have thorough technical knowledge in all areas
of the source tree.  Therefore, it's sometimes hard for a person (or
small group) to be able to say for sure whether a particular change
should be pulled up to the release branch.  This is where one of the
most important qualities of a release engineer comes into place: the
ability to get sound advice from the right people.  To some extent, we
trust that developers submitting changes for pullup have carefully
considered the implications of the change.  But at the same time, we
try to be very cautious, and being able to effectively consult with other
developers is of vital importance. 

We start the release process when we think the tree is in pretty good
shape.  At this point, the branch is known as, e.g., 5.0_BETA.  At this
point, more users start running the code, and we are constantly watching
for feedback (positive and negative).  Intrusive changes are avoided
during all stages of the release process, but they are most acceptable at
this point.  The general rule is that the further along in the release
process we are, the more restrictive we are about changes.  Unfortunately,
it is impossible to have many hard rules about this.  If big nasty bug
comes up, you're faced with a choice between ignoring the problem or
taking a risk on an intrusive change.  Sometimes we take the risk,
sometimes we don't.  This is another case where no one particular person
can have all the answers.  To avoid one person making a bad judgment call,
members of releng discuss these issues with each other and with other
developers.

A release candidate, by definition, occurs when it is felt that the tree
is good enough to release.  In practice, this rarely happens.  There is
almost always a good reason to do at least one more release candidate.  In
some respects, they're more psychological than anything else.  I admit I
rushed a couple release candidates out with full knowledge that 5.0 could
not be released without further fixes.  But when important last-minute
changes are made, putting out a release candidate is a great way to get
those changes tested.  Take 5.0_RC3, for example.  At the time, we were
aware of a few show-stopping bugs, but there were significant WAPBL
changes that needed wider testing, so we felt that RC3 was justified.

In the end, deciding when to declare a release stable comes down to
compromise.  You have a hard list of requirements that the release must
meet.  Beyond that, you just have to get to a point where you feel that
the remaining bugs are outweighed by all the positives in the new release.
This is a hard thing to do, because when you've been tracking every
problem raised for months, you tend to be focused on negatives and can
easily forget about all the people who are having wonderful experiences.
Eventually you have to let go and realize that you'll never have a
bug-free release.  When you get to that point, you tag the final release
and hope that you haven't made a horrible mistake  :) 

NetBSDfr: As a conclusion, can you tell us how you imagine the future
of NetBSD?

snj: We've made a massive jump with 5.0, and 6.0 should continue this
trend.
NetBSD has been ignored by many people for years, but this is changing.
As for more specific predictions, I think we're going to see NetBSD usage
increase greatly in the server and embedded worlds.  I don't envision any
radical shifts in direction, but this is not a problem.  NetBSD has always
been a great operating system; we just need to get out there and introduce
more people to it.

Many thanks to Guillaume for his hard work on preparing, conducting and translating this interview and rendez-vous in a few weeks for the next one. In the meantime, any names of developers to interview, or questions you'd like to ask, are more than welcome.

Original post at NetBSDfr.

My work thusfar has consisted of updating the system to work on NetBSD 5.0 i386. I've updated the scripts given to me by my mentor, Phil Nelson of Western Washington University, and made the system's documentation more extensive as well as clarified some ambiguous parts. I've committed a "skeleton" for the system to the NetBSD Summer of Code repositories which at this time must be filled in by the user with packages and configuration files by hand as per the documentation.

The next step in my work is to make an interactive program to automatically fill in the skeleton system based on a question and answer style format. I will use the BSD curses library to create a type of simplistic, intuitive GUI for this process.

I will then expand the project to include support for the x64, spark, and shark architectures, and hopefully towards the end of the summer I will have support for automatically upgrading to the latest versions of installation packages as per a system administrator's request.

For more information regarding this project please visit the project's page here.

To follow the project as it progresses please subscribe to the blog here.

Four new security advisories were published:

• NetBSD-SA2009-001 PF firewall remote Denial Of Service attack
• NetBSD-SA2009-002 tcpdump multiple denial of service and arbitrary code execution issues
• NetBSD-SA2009-003 proplib crashes on reading bad XML data
• NetBSD-SA2009-004 NetBSD OpenPAM passwd(1) changing weakness

You can find more information about them on the Security and NetBSD page.

Summary

A NetBSD system, in order to tolerate disk failures, can use the software RAID driver raid(4). Currently, if that system is shut down uncleanly (e.g., loses power or crashes), then when it comes back up it will have to check the entire RAID set's redundancy information. This process can take many hours, during which it imposes a substantial load on the system. It is also a distinct disadvantage to using NetBSD in server applications, and the inclusion of a journaling filesystem in NetBSD 5 makes it all the more prominent.

The goal of my Summer of Code project is to shorten that check from hours to minutes.

More Detail

The problem is due to a fundamental limitation of RAID: if the system is abruptly halted in the middle of writing to several independent disks, there is no way to tell afterwards which of those write operations were actually performed short of reading the actual data involved. This is especially problematic for RAID-5 and similar, where a mismatch between data and parity will result in garbling part of that data when (not if) it has to be reconstructed after a disk failure.

The solution taken in the RAIDframe codebase used by NetBSD is simple: while a RAID device is configured and in use, the entire thing is marked as needing a parity rewrite — even though only a small part, or perhaps none at all, has write operations in progress at any given time.

Thus, my fix is to record a better approximation of what parts of the RAID are being written to at any given time. Except that now the RAID driver has to update that record as writes are done, rather than just once on boot and once at shutdown, and if this approximation is too fine-grained then that will noticeably reduce performance — all of the time, not just after unplanned reboots.

Status

At the time of this writing, I have a prototype implementation of just such a parity map; it's not fit for general consumption at this point, but the basics are working. Next up is more testing and performance evaluation, to refine the parity map algorithm. Eventually there will need to be an interface for the system administrator to configure this feature, and the code will need to correctly handle a RAID set being moved between kernels with and without parity map support, but these should be relatively straightforward and are scheduled for later in the summer.

Some additional minor improvements to the RAID system are also planned, time permitting, pertaining in particular to the handling of spare disks and reconstruction after a disk failure. In any case, by the end of the summer, a notable weakness of NetBSD will have been corrected.

Generic file system mounting introduction

File systems have their own "proprietary" mount protocol. Those protocols are defined as opaque containers called struct <fs>_args which contain different things from one file system to another. These need to be treated differently, as each addition of a file system implies adding specific code on every program which wants to use it.

The aim of this project is to replace them by a generic protocol, which could be used for every file system. Doing so, it could be possible for instance to mount a device without having to specify the file system type. To do this, I am using a prop_dictionary instead of the struct <fs>_args.

Deliverables

Mandatory (must-have) components:

• mount_generic command
• patches for in-tree file systems
• patch to add a new vfs operation

• up-to-date manpages

Status

A proof of concept has been written. It contains the conversion of some file systems so as to use a properties dictionary, the new vfs_probe function and some applications which use them. It was tested with rump(3) which allows to do the tests without having to rebuild a kernel and to debug as easily as if it were a userland application. I converted fs-utils applications to make them use rump syscalls instead of the ukfs library. Then I wrote a new mounting library which uses the properties library. This library will be used to write the mount_generic program.
The code of these applications, the patch of the file systems and the vfs layer are available here.

Remaining work

• Asking on tech-kern for comments
• Adding support for remaining file systems
• Completing the mount_generic command

How to test it

Building the patched rump :

# cd /usr/src/sys
# patch < ~/genfsmount.patch
# cd rump
# make && make install

# cd fsu_rsprop/lib
# make includes
# cd ..
# make && make install

Examples

Usage

$ fsu_ls -o ro -w ffs.xml ~/ffs.img
Trying ffs: probing success, mount succeed
CVS             Makefile.rump   TODO            include         net
Makefile        README.dirs     fs              librump

$ cat ffs.xml 
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
        <key>fspec</key>
        <string>/home/stacktic/ffs.img</string>
</dict>
</plist>

$ fsu_cat -f ffs.xml Makefile
Trying ffs: probing success, mount succeed
#       $NetBSD: Makefile,v 1.4 2008/11/17 10:21:44 pooka Exp $
#

SUBDIR= include librump fs net

.include <bsd.subdir.mk>

$ fsu_ls -o ro ~/i386cd.iso 
Trying ffs: probing failed (22)
Trying cd9660: probing success, mount succeed
bin             dev             lib             mnt2            stand           sysinstmsgs.es  targetroot
boot            etc             libexec         netbsd          sysinst         sysinstmsgs.fr  tmp
boot.cfg        i386            mnt             sbin            sysinstmsgs.de  sysinstmsgs.pl  usr

Using specific options

$ fsu_ls ~/msdos.img               
Trying ffs: probing failed (22)
Trying cd9660: probing failed (22)
Trying msdos: probing succeeded, mount succeeded
CVS             Makefile.rump   TODO            include         net
Makefile        README.dirs     fs              librump

$ fsu_ls -s shortname,uid=stacktic,gid=users ~/msdos.img -l
Adding: shortname = true
Adding: uid = 1000
Adding: gid = 100
Trying ffs: probing failed (22)
Trying cd9660: probing failed (22)
Trying msdos: probing succeeded, mount succeeded
total 288
d---------  1 stacktic  users  16384 Jun 12 11:30 cvs
d---------  1 stacktic  users  16384 Jun 12 11:30 fs
d---------  1 stacktic  users  16384 Jun 12 11:30 include
d---------  1 stacktic  users  16384 Jun 12 11:30 librump
----------  1 stacktic  users    118 Jun 12 11:30 makefile
----------  1 stacktic  users   2377 Jun 12 11:30 makefi~1.rum
d---------  1 stacktic  users  16384 Jun 12 11:30 net
----------  1 stacktic  users   1525 Jun 12 11:30 readme~1.dir
----------  1 stacktic  users    530 Jun 12 11:30 todo

Links

• Project description on NetBSD SoC project list
• Project on GSoC page
• Project on NetBSD SoC sourceforge page

My Google Summer of Code student, Nhat Minh Le, is working on a suite of simple, efficient, stream-oriented tools for processing XML on UNIX systems. Nhat Minh is making good progress on xmlgrep, a grep-alike program that understands XML syntax.

Read about Nhat Minh's progress on his blog.

Keep reading for my explanation of the niche where Nhat Minh's tools fit.

UNIX's versatile text-processing system consists of simple tools (awk, grep, join, sed, sort) that provide elementary text-processing functions, and tool-building facilities (pipelines and scripts) that let you assemble simple tools into more sophisticated tools. UNIX tools are well-suited to processing tables where there is one record per line and where each field in a record is delimited from the next by a reserved character or characters.

About a decade ago, XML began to show up on UNIX systems in the form of both configuration files and XHTML web pages. Some UNIX administrators grumbled about the introduction of XML, especially XML as a configuration file format. They were accustomed both to reading and writing configuration files, and to automating chores by processing configuration files with standard UNIX tools. Tabular configuration files were more suited to be processed with standard UNIX tools, and to be read and written by people, than XML files were. Some admins thought that the use of XML as a configuration file should stop and never be reconsidered. Others felt that while XML configuration files may have promise, to introduce them without elementary processing tools was premature; some of those admins waited for the analogues to awk, sed, et cetera for XML to appear.

And they waited, and waited. Today, there are no XML tools to equal the economy of implementation, flexibility, and ease of use of the UNIX text-processing tools. Complicated tools with weighty prerequisites, such as a Java virtual machine, are common.

Nhat Minh's XML tools project aims to deliver small, simple programs that can work together to perform sophisticated XML processing tasks, bringing UNIX text processing into the XML era. xmlgrep will extract text from XML documents. A tool called xmlsed will transform them. We are still discussing the merits of xmlsort and xmljoin, and a tool for "decorating" one XML tree with another that has no parallel in the traditional UNIX tools.

(A draft of this blog entry first appeared on an internal mailing list of OJC Technologies.)