NetBSD News Beat
— You heard it here —
|
![]() |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
2009-07-01ZFS as whole has 2 main ways it can be accessed. The first is ZVOL and the second is ZPL. In my first status update I said that I had ported ZVOL layer to NetBSD, and I was able to create and use ZFS Zpools and Zvols (Logical partitions exported from one disk storage pool called zpool). Over the last few weeks I have worked on a ZPL port. ZPL is ZFS file system layer. I have ported zfs_vfsops.c file and zfs_vnops.c file to NetBSD. Today I have ZFS to state where I can mount ZFS data set, copy whole kernel source tree there and finally build NetBSD kernel on it.
$ su
# modload /mod/solaris.kmod
# modload /mod/zfs.kmod
# zfs mount test/zfs
# mount
/dev/wd0a on / type ffs (local)
kernfs on /kern type kernfs (local)
ptyfs on /dev/pts type ptyfs (local)
/dev/zvol/dsk/test/zfs on /test/zfs type zfs (local)
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
test 391M 1.57G 18K /test
test/tank 40M 1.61G 16K -
test/zfs 351M 1.57G 351M /test/zfs
test/zfs1 18K 1.57G 18K /test/zfs1
# cd /test/zfs/src/sys/arch/i386/compile/GENERIC/
# make
making sure the compat library is up to date...
`libcompat.a' is up to date.
making sure the kern library is up to date...
`libkern.o' is up to date.
# compile GENERIC/init_main.o
cc -ffreestanding -fno-zero-initialized-in-bss -O2 -std=gnu99 -fno-strict-aliasing -Werror -Wall -Wno-main -Wno-format-zero-length -Wpointer-arith -Wmissing-prototypes -Wstrict-prototypes -Wswitch -Wshadow -Wcast-qual -Wwrite-strings -Wno-unreachable-code -Wno-sign-compare -Wno-pointer-sign -Wno-attributes -Wextra -Wno-unused-parameter -Werror -Di386 -I. -I../../../../../common/include -I../../../../arch -I../../../.. -nostdinc -DMAXUSERS=64 -D_KERNEL -D_KERNEL_OPT -I../../../../lib/libkern/../../../common/lib/libc/quad -I../../../../lib/libkern/../../../common/lib/libc/string -I../../../../lib/libkern/../../../common/lib/libc/arch/i386/string -I../../../../dist/ipf -I../../../../external/isc/atheros_hal/dist -I../../../../external/isc/atheros_hal/ic -I../../../../../common/include -c ../../../../kern/init_main.c
# create vers.c
sh ../../../../conf/newvers.sh
# compile GENERIC/vers.o
cc -ffreestanding -fno-zero-initialized-in-bss -O2 -std=gnu99 -fno-strict-aliasing -Werror -Wall -Wno-main -Wno-format-zero-length -Wpointer-arith -Wmissing-prototypes -Wstrict-prototypes -Wswitch -Wshadow -Wcast-qual -Wwrite-strings -Wno-unreachable-code -Wno-sign-compare -Wno-pointer-sign -Wno-attributes -Wextra -Wno-unused-parameter -Werror -Di386 -I. -I../../../../../common/include -I../../../../arch -I../../../.. -nostdinc -DMAXUSERS=64 -D_KERNEL -D_KERNEL_OPT -I../../../../lib/libkern/../../../common/lib/libc/quad -I../../../../lib/libkern/../../../common/lib/libc/string -I../../../../lib/libkern/../../../common/lib/libc/arch/i386/string -I../../../../dist/ipf -I../../../../external/isc/atheros_hal/dist -I../../../../external/isc/atheros_hal/ic -I../../../../../common/include -c vers.c
# link GENERIC/netbsd
ld -Map netbsd.map --cref -T ../../../../arch/i386/conf/kern.ldscript -Ttext c0100000 -e start -X -o netbsd ${SYSTEM_OBJ} ${EXTRA_OBJ} vers.o
NetBSD 5.99.14 (GENERIC) #1: Tue Jun 30 20:00:37 UTC 2009
text data bss dec hex filename
8554455 407284 538396 9500135 90f5e7 netbsd
I tried to boot build kernel and it worked like a charm. There is still much work to do port ZFS snapshot support, properly implement security policies for ZFS access, test ZFS ACL support etc. My work is accessible in my git repository at git://rachael.ziaspace.com/src.git in a branch called haad-zfs. You can easily clone this repo with command git clone git://rachael.ziaspace.com/src.git. To get haad-zfs branch checkout you need to use command git checkout -b haad-zfs origin/haad-zfs from src directory. About the projectThe GUID Partition Table is a new standard for disk partitioning. The GPT layout provides a set of advanced partitioning features including, but not limited to:
The NetBSD already has support for GPT disks via dkwedges, but can't boot off a GPT partitioned disk. My GSoC project is to implement a GPT aware bootloader for the NetBSD operating system by extending its existing MBR/disklabel BIOS-based multistaged kernel loader. Please note, although GPT comes from the EFI world and the loader somewhat mimics EFI bootstrap behavior, the project is not about writing a pure EFI loader. Instead, it will be based on BIOS interface and will technically look like an arbitrary MBR loader but with GPT parsing. Such an approach should allow the use of GPT partitioned disks on non-EFI systems as well. On EFI systems NetBSD can be bootstrapped via the Compatibility Support Module. Current statusThe project assumes development of MBR (or LBA0), Partition Boot Record, and adopting of 1st stage and 2nd stage loaders. Currently, the MBR (LBA0) loader which parses GPT is in a beta stage. It implements the following features:
Code cleanup and CRC32 calculation of GUID partition array (which wasn't included due to space constraints, but likely can be enabled after optimisation) are still required. It was decided to store the primary loader on an EFI System Partition (which is FAT formatted by convention), and not use NetBSD partitions for bootstrapping. Such behavior will be similar to what EFI bootstrapping does, and the boot path will be as follows: BIOS → LBA0 → PBR on EFI syspart → /boot → NetBSD kernel. Fortunately, the "fatboot" (or bootxx_fat16) loader can do almost all of the required things in place of the PBR and 1st stage loader. The only missing thing is FAT lookup at an arbitrary offset on the disk (right now it assumes LBA0). It was easily implemented by having partition offset passed from GPT MBR or from any other loader through the ecx:ebx register pair. As we can't rely on all other loaders using this mechanism, the offset could be embedded into the PBR itself by installboot(8). My current efforts are focused on getting gpt.S and fatboot.S (and corresponding management tools) in shape and ready to publish for beta testing. So the future work in terms of the GSoC project will include the following:
Some additional information (including development schedule) can be found at the project home page at: http://netbsd-soc.sourceforge.net/projects/gpt/ Karolina Lesisska sent out a cry for help: apparently the Polish BSD magazine will be closed unless sales go up really soon (by monday, worst). So be sure to go and get a copy from your nearest book store right now! What, you're still there?! Well, as a reaser on what you'll get, there are a bunch of articles available in PDF format, which include an article on Installing NetBSD by Patrick Pippen, and an interview with Simon Burge, Antti Kantee and Greg Oster by Federico Biancuzzi. Get your subscription today! 2009-06-30Three new security advisories were published, covering OpenSSH, ntpd, ntpq and hack:
You can find more information about them on the Security and NetBSD page. Project Description The utility resize_ffs is a program intended to resize Berkeley Fast File Systems (FFS) by either growing or shrinking them. This filesystem is the standard filesystem for the NetBSD operating system -- a free, fast, secure, and highly portable Unix-like Open Source operating system. There is already a preliminary utility in the NetBSD source tree, and this project will first and primarily thoroughly test it with a regression test suite and code review, and correct and fix any bugs encountered. After this is accomplished, and the utility is stable, this project will attempt to extend the utility with features such as "live" resizing, that is the ability to dynamically resize the filesystem without first having to unmount it. More information is available on the project website. Status Update After a little bit of a slow start, the project is back on track. I am moving from the "just testing" phase to using that test information to hunt down bugs and also improve the program. One addition I have already added is the ability to choose the new filesystem size in kilo-, mega-, or gigabytes - as well as the default of sectors. However, it does currently assume a sector size of 512 KB in the conversion. I have identified several potential bugs, the worst of which is data corruption which seems limited to when the filesystem is growing. At USENIX 2009 I talked about rump file systems. The paper (pdf, html) and slides are available. Additionally, USENIX members can view a video of the presentation. paper abstractWhen kernel functionality is desired in userspace, the common approach is to reimplement it for userspace interfaces. We show that use of existing kernel file systems in userspace programs is possible without modifying the kernel file system code base. Two different operating modes are explored: 1) a transparent mode, in which the file system is mounted in the typical fashion by using the kernel code as a userspace server, and 2) a standalone mode, in which applications can use a kernel file system as a library. The first mode provides isolation from the trusted computing base and a secure way for mounting untrusted file systems on a monolithic kernel. The second mode is useful for file system utilities and applications, such as populating an image or viewing the contents without requiring host operating system kernel support. Additional uses for both modes include debugging, development and testing. The design and implementation of the Runnable Userspace Meta Program file system (rump fs) framework for NetBSD is presented. Using rump, ten disk-based file systems, a memory file system, a network file system and a userspace framework file system have been tested to be functional. File system performance for an estimated typical workload is found to be ±5% of kernel performance. The prototype of a similar framework for Linux was also implemented and portability was verified: Linux file systems work on NetBSD and NetBSD file systems work on Linux. Finally, the implementation is shown to be maintainable by examining the 1.5 year period it has been a part of NetBSD.
New Security Advisories: NetBSD-SA2009-005 through
NetBSD-2009-007 NetBSD.org
News 2009-06-30 00:00
UTC
2009-06-29
Google Summer of Code: Efficient wide character
regular expressions NetBSD
Blog 2009-06-29 20:16
UTC
During this year’s Google Summer of Code I’m improving the performance of NetBSD’s regular expression library and adding support to it for wide characters. NetBSD’s current regular expression library was initially written by Henry Spencer in 1986, when ASCII was the de facto standard. Moreover, Spencer described it as slow. So it makes real sense to replace it and seek for alternatives. I decided to develop a new regular expression library at first. It seemed rather easy in the beginning, but turned out to be more difficult in all its details than I expected (everyone who read the regular expression specification in the POSIX standard will agree). Therefore, I was really happy when Ville Laurikari offered to release his regular expression library tre, which he wrote his master thesis about, under a 2-clause BSD licence, so that tre can become part of libc. tre is a mature, strictly POSIX-conforming regular expression library with additional features, such as approximate matching, matching on binary data, thread safety, ABI compatibility and wide character support. tre also includes agrep, an approximate version of grep, which will be imported into NetBSD. At the moment I’m preparing a performance benchmark, which is based on realistic regular expressions and data, to ensure that tre is verifiably efficient. If the benchmark reveals that tre is less efficient than NetBSD’s current regular expression library, I will try to address the performance issues. Afterwards tre will be integrated in libc. Moreover, Glenn Fowler from AT&T Research developed a regression testing framework for regular expressions, which I will integrate with the atf to test and ensure POSIX-compliance. 2009-06-28Postfix 2.6.2, the latest stable version of the popular mail transport agent, was imported into NetBSD-current recently. The following features have been added since version 2.5.4:
Enjoy! 2009-06-26wake is a new command to send Wake-on-LAN frames over an ethernet to Wake-on-LAN capable machines, remote powering them up. This functionality is generally enabled in a machine's BIOS and can be used to power on machines from a remote system without having physical access to them. wake is available in NetBSD-current. See the wake(8) manual page for details. 2009-06-25One might be tempted to think that, while parsing can be a complex task involving various competing models, printing is straightforward and should be the easy part of the job. This is not quite true however, as David and I have been in disagreement for some time already over the exact semantics and syntax of a potential XML printing subsystem of xmltools. This post is all about the design of the printing language in xmltools and the underlying model. I also present at the end my latest hybrid proposal, that I’ve submitted to David for approval earlier today, and which I hope will bring satisfaction to all parties. If you have some opinion on this topic, do not hesitate to e-mail me! (My e-mail is my first name followed by my last name with all spaces replaced by dots; it’s written in French in the footer so you might not be able to decode it.) The basic problem can be described as follows: we have a source tree, the input document and we want to transform it by adding or removing nodes in some places. Simple? Not quite. Select & transformThe first design I proposed is based on sed(1): we select some node in the tree using a tree pattern (same as in my xmlgrep) and then transform it using some actions: insert, append, replace, etc. Only these needed to be adapted to XML. My idea was then to use PYX/ESIS-inspired actions, which also happen to model closely my internal API. These consisted of a one-character prefix indicating the thing to print, followed by contents (depending on the prefix). Prefixes were:
This is a very simple approach but David objected that it was unnatural to write XML using some language which was not XML.
Decorate & pullAnother approach, which is used most notably by XSLT, is to have a template, which itself is written as XML, in which some elements from the source document are pulled. Let me say it upfront in case anybody is not clear on this: pulling is not a stream technique, so this model as is cannot be implemented easily (some subsets of XSLT as stream transformations have been researched, but since it is from the start not a stream-oriented technique, it’s complex and IMHO not worth it). David first suggested using a template as a way of
having printing-heavy script actually look like their
output. He observed plain XML could be written
directly while some special elements, e.g.
I criticized this idea based on the fact that reserving a namespace was intrusive and that using XML to represent scripting elements was too verbose. But the main point was that pull-style transformation was never an option.
Towards a hybrid model: XML syntax for select & transformGiven the stream-unfriendly nature of the
template-based model, David came up with the idea to
represent actions of the select & transform model
as XML: XML opening and closing tags could be used to
represent the corresponding printing actions
(respectively I was still very dissatisfied with the verbosity
of having to write
A hybrid modelToday, after more brainstorming, I finally decided on the following model, and unless David is really against it (which I seriously doubt), I think it will make it into xmlsed one way or the other: the idea is the same as that of the previous attempt, use a template which actually describes actions inferred from the structure of XML itself. Only, this time, we do not use special elements to
indicate actions anymore, everything is
inferred from the structure. The trick to do this is
to have a virtual element represent the current node;
it can have any name (it will be user-settable),
let’s call it The idea was inspired from xargs(1) and its
A couple of weeks ago, Guillaume Lasmayous and I threw the idea of interviewing NetBSD developers through our website, NetBSDfr, to promote the NetBSD Project, and to make their work known to the widest possible audience. Today, we are discussing with Soren
Jacobsen,
NetBSDfr: First of all, let me thank you for this interview. We plan to run
several interviews of NetBSD developers, I think one every one or 2
months or so. You've been chosen as the first one. Thanks for accepting
to answer our questions.
snj: Thanks for asking me, and thanks for your efforts to strengthen the
community. It's great to see efforts like this.
Sorry it took so long to get back to you. I wanted to take some time to
think over my answers. Please let me know if you think anything needs
clarification.
NetBSDfr: For the readers who don't know you, can you shortly introduce
yourself ?
snj: My name is Soren Jacobsen and I am a NetBSD developer and
release engineer. I've been part of releng since late 2005, and was the
lead release engineer for the 5.0 release.
NetBSDfr: Why did you choose to run NetBSD ? How long have you been
using it ?
snj: My first experience with NetBSD was 1.5.3, which I guess was
around the second half of 2002. Compared to a lot of people, I was a bit
late to the party. At the time, I was frustrated with Linux in a number of
ways, and NetBSD was a solid operating system with a philosophy that
made sense to me. There wasn't any one particular thing about
NetBSD that grabbed my attention. It was more that there was nothing
that bothered me. It just seemed right.
NetBSDfr: Do you have an idea of the time you spend working on the
NetBSD project daily, weekly, monthly ?
snj: Like most volunteers, the amount I'm able to spend on NetBSD
depends on real life and my motivation level. A year ago, at most I was
doing 8 hours a week. More recently, 8 hours a day wasn't uncommon.
The week leading up to the release was especially hectic; I put in a
couple 16 hour days there.
NetBSDfr: How did you become a NetBSD developer ?
snj: When I started using NetBSD full time as my primary operating
system, I became heavily dependent on pkgsrc, our framework for
third-party packages. There were a number of packages I wanted to
see updated, and I started sending patches against pkgsrc and various
small things in the base system. At the end of 2003, I was offered an
account, and became a developer in January of 2004.
To anyone out there who is interested in helping the project: pkgsrc
is a great way to get involved. There are so many packages out there,
and only a finite number of developers to import and maintain them.
Consider submitting PRs against pkgsrc and joining the pkgsrc-wip
project. We can use all the help we can get in this area.
NetBSDfr: How did you become member of NetBSD release engineering
team ?
Are the members of the releng team elected ? chosen among
developers ? Does this require specific skills ?
snj: The release engineering team is rather informal. There is no
election process, and no strictly defined leader of the group, although
people do step up to fill this position. When we feel understaffed, we
put out a call for volunteers, and usually a person or two will join.
Release engineering is not as glorious as a lot of other activities, so we
don't normally have people begging to join releng.
It's hard to say about specific skills. Like in most other areas, having
sound judgment and a good sense of perspective is essential. These are
the only strictly required skills. Plenty of others are useful, though:
patience, caution, attention to detail, etc. Pretty basic stuff, for the
most part. Of course, the more technical knowledge, the better, but this
is not as big of a deal as it might seem. I'll explain below.
NetBSDfr: Can you explain us more in details how does NetBSD release
engineering process work ?
snj: There are two basic tasks:
- Pulling up changes into branches.
- Releasing.
The first one is rather straightforward, and it is something we do
constantly. Developers request that certain changes be pulled up to a
given branch. If we agree that the changes should be pulled up, we commit
them to the branch. To help track the status of a branch, we maintain a
record of every change made to a branch over its lifetime.
Putting together a release is a much more complicated task requiring
coordination with a large number of people. The main task is to decide
the point at which the remaining bugs can safely be ignored. There are
lots of other tasks involved here, of course: managing the build cluster,
pulling up changes, preparing information about the release, testing,
making sure that important fixes find their way to the branch, begging
people to work on certain things, paying very close attention to the
mailing lists, etc.
NetBSDfr: How does the quality assurance process works ? Who decides
to include this or that patch to a branch ? This is also linked I think to
the following question: how does the releng team decide how many
betas, RCs versions are required before declaring a branch is stable ?
snj: No one person can have thorough technical knowledge in all areas
of the source tree. Therefore, it's sometimes hard for a person (or
small group) to be able to say for sure whether a particular change
should be pulled up to the release branch. This is where one of the
most important qualities of a release engineer comes into place: the
ability to get sound advice from the right people. To some extent, we
trust that developers submitting changes for pullup have carefully
considered the implications of the change. But at the same time, we
try to be very cautious, and being able to effectively consult with other
developers is of vital importance.
We start the release process when we think the tree is in pretty good
shape. At this point, the branch is known as, e.g., 5.0_BETA. At this
point, more users start running the code, and we are constantly watching
for feedback (positive and negative). Intrusive changes are avoided
during all stages of the release process, but they are most acceptable at
this point. The general rule is that the further along in the release
process we are, the more restrictive we are about changes. Unfortunately,
it is impossible to have many hard rules about this. If big nasty bug
comes up, you're faced with a choice between ignoring the problem or
taking a risk on an intrusive change. Sometimes we take the risk,
sometimes we don't. This is another case where no one particular person
can have all the answers. To avoid one person making a bad judgment call,
members of releng discuss these issues with each other and with other
developers.
A release candidate, by definition, occurs when it is felt that the tree
is good enough to release. In practice, this rarely happens. There is
almost always a good reason to do at least one more release candidate. In
some respects, they're more psychological than anything else. I admit I
rushed a couple release candidates out with full knowledge that 5.0 could
not be released without further fixes. But when important last-minute
changes are made, putting out a release candidate is a great way to get
those changes tested. Take 5.0_RC3, for example. At the time, we were
aware of a few show-stopping bugs, but there were significant WAPBL
changes that needed wider testing, so we felt that RC3 was justified.
In the end, deciding when to declare a release stable comes down to
compromise. You have a hard list of requirements that the release must
meet. Beyond that, you just have to get to a point where you feel that
the remaining bugs are outweighed by all the positives in the new release.
This is a hard thing to do, because when you've been tracking every
problem raised for months, you tend to be focused on negatives and can
easily forget about all the people who are having wonderful experiences.
Eventually you have to let go and realize that you'll never have a
bug-free release. When you get to that point, you tag the final release
and hope that you haven't made a horrible mistake :)
NetBSDfr: As a conclusion, can you tell us how you imagine the future
of NetBSD?
snj: We've made a massive jump with 5.0, and 6.0 should continue this
trend.
NetBSD has been ignored by many people for years, but this is changing.
As for more specific predictions, I think we're going to see NetBSD usage
increase greatly in the server and embedded worlds. I don't envision any
radical shifts in direction, but this is not a problem. NetBSD has always
been a great operating system; we just need to get out there and introduce
more people to it.
Many thanks to Guillaume for his hard work on preparing, conducting and translating this interview and rendez-vous in a few weeks for the next one. In the meantime, any names of developers to interview, or questions you'd like to ask, are more than welcome. Read the following for security issues: 2009-06-23My work thusfar has consisted of updating the system to work on NetBSD 5.0 i386. I've updated the scripts given to me by my mentor, Phil Nelson of Western Washington University, and made the system's documentation more extensive as well as clarified some ambiguous parts. I've committed a "skeleton" for the system to the NetBSD Summer of Code repositories which at this time must be filled in by the user with packages and configuration files by hand as per the documentation. The next step in my work is to make an interactive program to automatically fill in the skeleton system based on a question and answer style format. I will use the BSD curses library to create a type of simplistic, intuitive GUI for this process. I will then expand the project to include support for the x64, spark, and shark architectures, and hopefully towards the end of the summer I will have support for automatically upgrading to the latest versions of installation packages as per a system administrator's request. For more information regarding this project please visit the project's page here. To follow the project as it progresses please subscribe to the blog here. Four new security advisories were published:
You can find more information about them on the Security and NetBSD page. 2009-06-22Netbsd 5.0 has been successfully installed with specified packages on i386 machines! I will be posting these results as well as committing what I have so far at http://blog.netbsd.org/ later today! Edited 22.06. Incompatible syntax changes and more examples. Since Matthew Sporleder on tech-userlevel has (implicitly) suggested that xmlgrep could be used to dissect Atom feeds, but was a bit lost at how to do it exactly, I thought I’d post a little demo of a console session playing with an Atom file and xmlgrep (as well as some other command-line tools). First, let’s fetch the feed:
As a side effect of xmlgrep, we might want to indent the XML
file to make it human-readable:
List all posts in the NetBSD category with their IDs:
Those of you yet unfamiliar with the syntax might
have some trouble understanding. The previous pattern
could be read "select a text child of an id or title
element, itself a child of an entry element, which
contains a category element which has an attribute
child term equal to NetBSD." Step by step, you should
notice that Now, let’s select a post by ID:
Or, select a post by title and view its contents using w3m:
As a side note, I should mention that up to now we
have used subpatterns quite a lot. This is because
the Atom feed specification does not force an order
(or does it?) on the children of Let’s print all entry titles which date from
March 2009 using the fact that we know the
A friend of mine told me it would be useful to have
arithmetic predicates. I think they will feature in xmltools sooner
or later, but even without them, it is still possible to do some
simple statistics, by combining the results with awk(1), for
example. The following one-liner counts the number of posts that
have no older than March:
That’s it; I hope this will help people who want to get started with xmlgrep. If you have other good examples you’d want me to elaborate, do not hesitate to send me a mail! SummaryA NetBSD system, in order to tolerate disk failures, can use the software RAID driver raid(4). Currently, if that system is shut down uncleanly (e.g., loses power or crashes), then when it comes back up it will have to check the entire RAID set's redundancy information. This process can take many hours, during which it imposes a substantial load on the system. It is also a distinct disadvantage to using NetBSD in server applications, and the inclusion of a journaling filesystem in NetBSD 5 makes it all the more prominent. The goal of my Summer of Code project is to shorten that check from hours to minutes. More DetailThe problem is due to a fundamental limitation of RAID: if the system is abruptly halted in the middle of writing to several independent disks, there is no way to tell afterwards which of those write operations were actually performed short of reading the actual data involved. This is especially problematic for RAID-5 and similar, where a mismatch between data and parity will result in garbling part of that data when (not if) it has to be reconstructed after a disk failure. The solution taken in the RAIDframe codebase used by NetBSD is simple: while a RAID device is configured and in use, the entire thing is marked as needing a parity rewrite — even though only a small part, or perhaps none at all, has write operations in progress at any given time. Thus, my fix is to record a better approximation of what parts of the RAID are being written to at any given time. Except that now the RAID driver has to update that record as writes are done, rather than just once on boot and once at shutdown, and if this approximation is too fine-grained then that will noticeably reduce performance — all of the time, not just after unplanned reboots. StatusAt the time of this writing, I have a prototype implementation of just such a parity map; it's not fit for general consumption at this point, but the basics are working. Next up is more testing and performance evaluation, to refine the parity map algorithm. Eventually there will need to be an interface for the system administrator to configure this feature, and the code will need to correctly handle a RAID set being moved between kernels with and without parity map support, but these should be relatively straightforward and are scheduled for later in the summer. Some additional minor improvements to the RAID system are also planned, time permitting, pertaining in particular to the handling of spare disks and reconstruction after a disk failure. In any case, by the end of the summer, a notable weakness of NetBSD will have been corrected.
New Security Advisories: NetBSD-SA2009-001 through
NetBSD-2009-004 NetBSD.org
News 2009-06-22 00:00
UTC
2009-06-21Generic file system mounting introductionFile systems have their own "proprietary" mount protocol. Those protocols are defined as opaque containers called struct <fs>_args which contain different things from one file system to another. These need to be treated differently, as each addition of a file system implies adding specific code on every program which wants to use it. The aim of this project is to replace them by a generic protocol, which could be used for every file system. Doing so, it could be possible for instance to mount a device without having to specify the file system type. To do this, I am using a prop_dictionary instead of the struct <fs>_args. DeliverablesMandatory (must-have) components:
StatusA proof of concept has been written. It contains
the conversion of some file systems so as to use a
properties dictionary, the new vfs_probe function and
some applications which use them. It was tested with
rump(3) which allows to do the tests without having
to rebuild a kernel and to debug as easily as if it
were a userland application. I converted fs-utils
applications to make them use rump syscalls instead
of the ukfs library. Then I wrote a new mounting
library which uses the properties library. This
library will be used to write the mount_generic
program. Remaining work
How to test itBuilding the patched rump : # cd /usr/src/sys # patch < ~/genfsmount.patch # cd rump # make && make installBuilding fs-utils libraries and applications : # cd fsu_rsprop/lib # make includes # cd .. # make && make install ExamplesUsage
$ fsu_ls -o ro -w ffs.xml ~/ffs.img
Trying ffs: probing success, mount succeed
CVS Makefile.rump TODO include net
Makefile README.dirs fs librump
$ cat ffs.xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>fspec</key>
<string>/home/stacktic/ffs.img</string>
</dict>
</plist>
$ fsu_cat -f ffs.xml Makefile
Trying ffs: probing success, mount succeed
# $NetBSD: Makefile,v 1.4 2008/11/17 10:21:44 pooka Exp $
#
SUBDIR= include librump fs net
.include <bsd.subdir.mk>
$ fsu_ls -o ro ~/i386cd.iso
Trying ffs: probing failed (22)
Trying cd9660: probing success, mount succeed
bin dev lib mnt2 stand sysinstmsgs.es targetroot
boot etc libexec netbsd sysinst sysinstmsgs.fr tmp
boot.cfg i386 mnt sbin sysinstmsgs.de sysinstmsgs.pl usr
Using specific options$ fsu_ls ~/msdos.img Trying ffs: probing failed (22) Trying cd9660: probing failed (22) Trying msdos: probing succeeded, mount succeeded CVS Makefile.rump TODO include net Makefile README.dirs fs librump $ fsu_ls -s shortname,uid=stacktic,gid=users ~/msdos.img -l Adding: shortname = true Adding: uid = 1000 Adding: gid = 100 Trying ffs: probing failed (22) Trying cd9660: probing failed (22) Trying msdos: probing succeeded, mount succeeded total 288 d--------- 1 stacktic users 16384 Jun 12 11:30 cvs d--------- 1 stacktic users 16384 Jun 12 11:30 fs d--------- 1 stacktic users 16384 Jun 12 11:30 include d--------- 1 stacktic users 16384 Jun 12 11:30 librump ---------- 1 stacktic users 118 Jun 12 11:30 makefile ---------- 1 stacktic users 2377 Jun 12 11:30 makefi~1.rum d--------- 1 stacktic users 16384 Jun 12 11:30 net ---------- 1 stacktic users 1525 Jun 12 11:30 readme~1.dir ---------- 1 stacktic users 530 Jun 12 11:30 todo LinksThose who have been keeping an eye on my Git
repository have surely noticed the recent update of
the The main advantage of this approach is
vectorization. Such operations are easily
done on a single unit (the type of which is defined
in The results are not quite satisfactory and the
branch was not merged into
On the chart, GCC has vectorization disabled by
default while ICC always has vectorization (even
though I have not specified
I must say I can’t explain why operations on
native integers (the Anyway, this was just a side experiment of mine; work on the main tools continue: plans for more features in the matcher (on request, case insensitivity, arithmetic predicates, maybe some other things) as well as the introduction of xmlsed are underway. 2009-06-20My Google Summer of Code student, Nhat Minh Le, is working on a suite of simple, efficient, stream-oriented tools for processing XML on UNIX systems. Nhat Minh is making good progress on xmlgrep, a grep-alike program that understands XML syntax. Read about Nhat Minh's progress on his blog. Keep reading for my explanation of the niche where Nhat Minh's tools fit. UNIX's versatile text-processing system consists of simple tools (awk, grep, join, sed, sort) that provide elementary text-processing functions, and tool-building facilities (pipelines and scripts) that let you assemble simple tools into more sophisticated tools. UNIX tools are well-suited to processing tables where there is one record per line and where each field in a record is delimited from the next by a reserved character or characters. About a decade ago, XML began to show up on UNIX systems in the form of both configuration files and XHTML web pages. Some UNIX administrators grumbled about the introduction of XML, especially XML as a configuration file format. They were accustomed both to reading and writing configuration files, and to automating chores by processing configuration files with standard UNIX tools. Tabular configuration files were more suited to be processed with standard UNIX tools, and to be read and written by people, than XML files were. Some admins thought that the use of XML as a configuration file should stop and never be reconsidered. Others felt that while XML configuration files may have promise, to introduce them without elementary processing tools was premature; some of those admins waited for the analogues to awk, sed, et cetera for XML to appear. And they waited, and waited. Today, there are no XML tools to equal the economy of implementation, flexibility, and ease of use of the UNIX text-processing tools. Complicated tools with weighty prerequisites, such as a Java virtual machine, are common. Nhat Minh's XML tools project aims to deliver small, simple programs that can work together to perform sophisticated XML processing tasks, bringing UNIX text processing into the XML era. xmlgrep will extract text from XML documents. A tool called xmlsed will transform them. We are still discussing the merits of xmlsort and xmljoin, and a tool for "decorating" one XML tree with another that has no parallel in the traditional UNIX tools. (A draft of this blog entry first appeared on an internal mailing list of OJC Technologies.) 2009-06-19Of course NFS is done by now, and I'm almost done setting up a NetBSD 5.0 directory tree. I ran into a bit of a snag with fdisk, and I'm looking into it now. 2009-06-16After spending some days fixing bugs in the xmlgrep matcher code, I took some time again to look at the performance issues. The previous measurements showed xmlgrep was slower than xmlstarlet, even on patterns without subpatterns, which are theoretically optimal for my implementation. That really bothered me. Profiling seemed to show xmlgrep spent most of its
time in bit vector operations – that much was
expected. But my vain attempt (Git branch
Then I had a talk with David Young, my mentor, and we concluded that both the input mechanism and the memory allocation pattern were suboptimal. About the input problem, I’ve replaced the old input feeder, which used fgets(3), with direct read(2) calls, and this has indeed shaved some seconds off the counter. This is especially noticeable for patterns without subpatterns, since I/O accounts for more in that case. The memory allocation issue was more serious.
Basically, a lot of bit vectors and subcontext frames
were needlessly allocated then almost immediately
freed afterwards. This is due to the following
construct commonly found in patterns (maybe I should
optimize this one case specifically): patterns which
start with an anchored subpattern, typically
something like I’ve switched to pool allocators for both internal tree nodes and bit vectors. This has drastically reduced the number of allocations made during a run. David also pointed out that I could use a pre-allocated vector instead of a linked list for the subcontext stack, and I remembered one of the nice properties of my implementation: the number of subcontext is bounded by the maximal nesting depth of subpatterns. He also suggested I should use copy-on-write bit vectors, but that proved unnecessary as I now use pointer swapping instead of copying then erasing (which was quite stupid in the first place, I agree). All these changes were developed in the
2009-06-13I'm almost done with my NFS server, the final stage that will allow me to test my work on the guinea pig computers. It's been a rocky road, setting up this slew of servers, but I've learned a lot of applicable things. I'll be doing work over the weekend and hope to have everything ready to roll on Monday. 2009-06-12As xmlgrep is approaching a pre-pre-alpha release (something that is still very experimental, but nonetheless working, somewhat :), I’ve been doing some basic tests on it, including one simple benchmark. The results are, IMHO, encouraging. The benchmark pits xmlgrep against equivalent existing software:
Since each tool accepts its own patterns and treats them differently, I’ve tried my best to get meaningful results with each, but obviously, the behaviors obtained differ somewhat. The
The goal was to retrieve the
The first two lines correspond to two different invocations of xmlgrep, which do not do exactly the same thing, and with the first being the good one, the other inefficiently abusing the look-ahead mechanism. Yet, it was included for comparison purposes and maybe to be fair to the XPath-based alternatives, which have to rely on a predicate. Results follow; the second and third pictures are just zooms which omit one or another of the candidates. I know these results don’t mean much, so don’t take them too seriously. I’ve only tested a single simple pattern. Actually, I didn’t really intend to make a comparison, at first, I just wanted to make sure xmlgrep doesn’t leak or otherwise misuse memory. But since the sampling code was there, I thought it’d be fun to do some additional measurements.
As expected, xmlstarlet, which uses a full DOM model requires a lot of memory to do its work (more than 400M!). But it is the fastest. My xmlgrep, when used right, is still nearly 42% slower than xmlstarlet (from above 4s to below 6s), but I think the times remain reasonable; memory-wise, it is also the lightest, with a constant 2.9M in use throughout the whole run. Even GNU grep requires about 2.8M. Surprisingly, the Twig xmlgrep is not too big a memory killer, though its memory usage increases over time, by steps (though it looks linear on the long run). This appears somewhat strange to me; why some nodes should be pruned from its internal DOM-like tree while others seem to remain for the duration of the program. However, on the speed side, Twig is predictably very slow; it takes nearly two minutes to produce the results comparable to those of its C counterparts. That’s it for tonight; as one friend of mine told me, more interesting benchmarks will probably come from actual users. 2009-06-11Having completed two Spring quarter finals today and one yesterday, I can finally put my entire effort into GSOC! I'm definitely looking forward to it. 2009-06-10xmltools is a project whose ultimate goal is to bring Unix command-line efficiency to the XML world (and not, as many might think, to bring XML into the Unix world, which is being done already, though not by my fault). The idea is to treat XML as a generic tree representation format. This seems very straightforward; in fact, one could argue that XML was designed from the start to be a generic tree representation. However, XML has a number of problems that make it hard to use in this role. The first such problems is doctype declarations. While the bare XML structure, which we see most of the time, is very simple, doctype declarations and their semantics are both unneeded and unwanted in tree (syntax) manipulation programs. It would be fine if doctype support was optional for non-validating XML processors, but as I understand it, the XML standard mandates support for a portion of it: entity declarations and references, and default attribute values. The second issue is with white space handling: XML
does not include any default behavior regarding white
space. XML parsers must pass all white space to the
application, which is responsible for some
sensible default processing. The
This all led me to develop a set of compatibility rules which precisely describe what one should expect from xmltools, no matter which backend is used (in case we move away from expat, which I plan to do, eventually). This subset is sufficient to describe any tree-like data set. The rules will be maintained in the
2009-06-09My apologies for not having written in a while, I
will be updating the site daily from now on. Recently, I’ve been working with expat extensively, as part of my Google Summer of Code project, and have come to the realization that there is, IMHO, no good well-maintained lightweight stream-oriented XML parser. Of course, one would think expat, at this point, and I thought so too, when I started, only to find myself now locked with an expat-like design, which is both inefficient and unnecessarily complicated (for what I want to do). So, what is this expat design I am talking about? It is the event-driven (callback-oriented) model, what the XML guys out there call push parsers, though I personally think it doesn’t make much sense to call it that way, so I won’t. While event-driven parsing is good for some tasks — awk(1) is a really clever tool — it really fails on complex parsing tasks such as those required by my xmltools that rely on partial reprocessing of the document tree. Reprocessing of the document tree basically means we have to keep an in-memory copy of the parts we want to traverse more than once. Of course, expat can (help us) do that. Using expat, the basic idea is to have two input sources which both call the event handlers. Then, whenever we need to reprocess some part of the tree, we just traverse our in-memory copy and feed the information to the handlers as if they were receiving these from expat. Now, this seems good at first, but the fact is we now have two parsers: expat and our tree parser. Moreover, they don’t share an identical behavior: on the expat side, we may need to build new nodes to put in the partial tree (for future reprocessing), while in the in-memory tree parser, we may need to free some nodes which are no longer needed. But even in the expat handling code there may be nodes we have to keep (usually the current branch, from the root to the current node); this in turn implies we sometimes have to free some nodes as well. Now we have bits and pieces of the in-memory tree management code everywhere. Things start to look messy… Besides, when do we call the surrogate parser? The
issue here is we don’t want expat to feed new
data while we are in the middle of (or more precisely
before we can even start) a reprocessing operation.
The expat manual points at the
But there is a more elegant solution: the idea is to have the internal tree management module drive the process. It maintains the in-memory tree and calls the event code on its nodes whenever necessary, possibly going through loops or any patterns that suit its needs, for that matter. Then, when it requires a node that is not present in the tree, it calls upon the input module which returns the next node in the stream. This is the so-called pull method. There really is nothing special about it; in fact, it’s the way most I/O libraries out there are built. Why XML had to be different, I don’t know. As I’ve said, to the extent of my knowledge, there is no lightweight and well-maintained (I mean not some two-year-old experiment by some obscure hacker, though I respect these things, I don’t have time right now to play with such codes) C library which implements the full XML standard and this pull metaphor. If anyone out there knows, do not hesitate to let me know. 2009-05-27
CaveatsI found a few issues with quotas-
quota(1) - check your own quota repquota(8) - check everyone's quota quotaon(8) - enable quotas edquota(8) - edit the quota file quotacheck(8) - see if a user is exceeding his quota 2009-05-24Okay, so this is basically a post about
procrastination as I avoid working on
my first pseudo device. One thing that struck me
was how accessible this part of the internal working
of NetBSD were. I mean, seriously, if I can get a
working device then anyone can. 2009-05-17
CVS' activity digest - Issue 000 2009-05-10I spent some time today working out the kinks of
making a directory tree for NetBSD 5.0 which only
recently was released. Understanding
scripts/importing packages comprised most of my day
today. 2009-05-09Update! This whole article has proven
mostly worthless thanks the following two entries in
fstab: 2009-05-04The NetBSD Project recently launched a new official blog for NetBSD. From here, I'd like to invite you to visit it and subscribe to it. It's only with your support (through reading and, specially, commenting) that developers will post more entries! Enjoy :-) 2009-05-03So after re-installing NetBSD on my soekris I got
to thinking about jumpstart
for netbsd and about the whole pxe boot in
general. After doing a little reading of the pretty
excellent
pxeboot man page (I mean, NetBSD really does have
some great documentation) I wanted to give some of
the more flexible options a try. So I'm definitely interested to see how the above-mentioned summer of code project decides to use these options, if he decides to use MULTIBOOT to pass an arbitrary list of options into the boot sequence (my-ctrl-file=tftp:macaddress.xml) or who knows what. Abandon CDROM booting! :) (my dhcpd.conf is mostly from the pxeboot man page and I'm testing by connecting my eee to my powerbook with a normal patch cable because modern interfaces will automagically act like crossovers)
2009-05-02I just upgraded my soekris from NetBSD 3.1 to
NetBSD 5.0. I actually did a fresh install following
my own docs at soekris.html and
am still working on getting all of my packages
installed now. So far the systems seems fine,
although I'm having a little more trouble fitting it
onto 512MB. :) 2009-05-01Today I moved some personal equipment into my workspace. Things will be going slowly until next Monday (May 4) due to a midterm project for my Game Programming class. 2009-04-29Today I prepared the work area for the project. I
installed the new NetBSD 5.0 on my x86 server machine
as well as procured a couple of SPARC machines for
use in testing. 2009-04-28
patch_* tools are announced in an email
describing the beginnings of a tool which is like
what mt_diff
was supposed to be. 2009-04-27First and foremost, I would like to express my
excitement at having been selected to work with
Google, NetBSD, and Dr. Philip Nelson. I feel
extremely honored to have been given this
opportunity. This blog will follow major project
events for those interested in its completion. 2009-04-19As
noted in another blog, wasabi systems has gone
away. It would seem that they donated WAPBL to NetBSD
in a final act of charity before being forced to
close their doors. Now that they're gone, I wonder if
they have any other technologies they would be able
to give to NetBSD or even sell to the netbsd
foundation. Old hardware is probably too easy to sell
for bill-paying, or else that would probably be nice
too. 2009-04-13pkgsrc-dashboard is now, basically, functionally
complete for its initial goals: 2009-04-06So in IRC I brought up the idea of a web dashboard
for managing a network of pkgsrc servers and liked it
so much I started a google code project for it. So this weekend I actually got some code out on
pkgsrc-dashboard
and I'm pretty happy with it. Well.. almost. :) 2009-04-02The Google Summer of Code 2009 application
deadline for students is tomorrow and NetBSD has got
very few applications so far. If you have the
interest in working on a cool operating system
project, where almost any project idea can fit, take
the time to read our
proposals and apply! New, original ideas not
listed there will also be considered. 2009-03-28Normally I would do a whole How-To on this topic,
but it was so easy that it really only qualifies as a
blog post. Everything pretty much just works on my
eeepc 701 so there isn't a whole lot to document. The
main accomplishment was just figuring out how to turn
off the screen. 2009-03-25Jibbed The NetBSD livecd has been updated to a new version. 2009-03-20Okay, so
an email came through the mailing lists recently
about systrace being removed and how to handle it.
This got me thinking about
kauth and
secmodel. Two ideas came to mind: 2009-03-14We had an interesting discussion on EFNet/#netbsd
this morning about some possible enhancements to acpi
and some other facilities being improved/introduced
in NetBSD 5. The main idea was this: given a set of
criteria, could netbsd automatically turn off
hardware components to save on power? 2009-03-10I wanted to give a little press to mk-configure
by Aleksey Cheusov because I think it's an
interesting project and because the auto tools are a
regular source of heartache for developers and users.
(you might know them as the standard ./configure
&& make && make install cycle) 2009-03-07Interview with Andrew Doran from the NetBSD Project. We talk about the upcoming 5.0 release. File Info: 22Min, 10MB. Ogg Link: http://cisx1.uma.maine.edu/~wbackman/bsdtalk/bsdtalk171.ogg 2009-02-27tech-userlevel might not seem like the most
exciting mailing list, but it usually has some good
stuff. 2009-02-24I'm not sure why this wasn't announced yet, but it
looks like the desktop-gnome
meta-pkg was added. This is a big step for the
Desktop NetBSD project. 2009-02-21Brian Seklecki just posted a new wiki page about how to run a netbsd source mirror. I have helped him with some of this stuff (although not very much), so I thought I would advertise the page here. 2009-01-02I reinstalled NetBSD-current recently on my shark
(Digital DNARD) and, out of curiosity, I wanted to
see if the new-style kernel modules worked fine on
this platform. To test that, I attempted to load the
puffs module and failed with an error message saying
something like "kobj_reloc: unexpected relocation
type 1". Similarly, the same error appeared when
running the simpler regression tests in
/usr/tests/modules. Feeds
|
Hosted by Global Wire Oy
|