2008-11-22

HowTo: Making a bootable ISO w/ miniroot.kmod etc. hubertf's NetBSD blog 2008-11-22 18:40 UTC

I've asked how building a bootable ISO works now that we have kernel modules and a bootloader that can load kernels before the kernel, e.g. for a ramdisk. After some digging, I've managed to fit the pieces together, get things going, and I've posted a write-up to tech-install so others that are intestested don't have to wade through that swamp again.

The posting describes how the build process was in pre-5.0/current times (e.g. 4.0), what things change in 5.0/current now, and what doesn't change.

g4u 2.4alpha3 is available for testing hubertf's NetBSD blog 2008-11-22 17:53 UTC

After a lot of time, a lot of distractions, and a lot of changes to NetBSD's install system, I've finally found time again to adjust g4u to the current world order, and I'm happy to finally have g4u 2.4alpha3 available for testing.

What is g4u? ``g4u ("ghosting for unix") is a NetBSD-based bootfloppy/CD-ROM that allows easy cloning of PC harddisks to deploy a common setup on a number of PCs using FTP. The floppy/CD offers two functions. The first is to upload the compressed image of a local harddisk to a FTP server, the other is to restore that image via FTP, uncompress it and write it back to disk. Network configuration is fetched via DHCP. As the harddisk is processed as an image, any filesystem and operating system can be deployed using g4u. Easy cloning of local disks as well as partitions is also supported.''

Get it now:

What's new in g4u 2.4alpha3:
  • Support Bluetooth keyboard and USB hubs
  • ACPI support is enabled by default. SMP is, too, but that's not really used in g4u; They can both be disabled form the bootloader in case of problems.
  • Moved away from GENERIC kernel back to customized kernel, to avoid bloat through too many drivers that aren't needed in g4u (audio, video, ...)
  • The build structure was adjusted to NetBSD's latest framework, which loads the kernel from CD, then loads the ramdisk, too, before starting the kernel. No more kernel+ramdisk for now. (Need to re-evaluate situation for PXE)
  • Drivers for:
    • AMD CS5535/CS5536 and Intel PIIX4 PCI bridges
    • Areca SATA PCI RAID controllers
    • Workbit NinjaATA-32 CardBus IDE controllers
    • Realtek 8139C+/8169/8169S/8110S cardbus wlan cards
    • Geode CSC and ScanLogic SL811HS USB controllers
    • Intel PRO/Wireless 4965AGN
    • JMicron JMC2[56]0 ethernet
    • Atheros L2 Fast-Ethernet
    • Realtek 8139C+/8169/8169S/8110S cardbus network cards
  • ... and any driver updates, bug fixes and other enhancements from NetBSD-current
  • This version of g4u was again cross-compiled on Mac OS X. To get your own build, grap NetBSD-current sources from yesterday (20081122), unpack the g4u source .tgz then "cd .../src ; sh g4u-build" and report what happens.
I'd like to hear any reports if this version works better or worse than any previous alpha version or release, esp. under the light that this version is (another...) attempt to switch to ACPI, which is on by default in NetBSD now. Let me hear if it works, or if it hangs for you!

2008-11-18

NetBSD 5.0 release cycle - status update hubertf's NetBSD blog 2008-11-18 22:53 UTC

OK, it's en vogue to hype unreleased software these days, so here is an status update of the NetBSD 5.0 release. Citing from a posting by Alistair Crooks:

  • release engineering would really like to hear the results of your testing. It's you that can help us make 5.0 even better than it is right now. Even something simple like "Installed on $X computer and is currently running great!". We can't track everything ourselves, and direct reports from individuals are very helpful in assessing the state of the tree.

    No official way on reporting has been proposed (yet), but we're looking forward to involve our community into the release process. YOU can make a difference! (And we're a volunteer project, too, after all ;-)

  • progress in the last week has been good - some great pullups to the branch have been made, including pullups to improve FP stability on amd64, and to point to binary packages on the project ftp server, amongst many, many others. Please see the source-changes mailing list for a more detailed list of these pullups.
  • We still need to knock some critical PRs on the head - if you could help out with any of those, please see the PR list; if you could, we'd be very grateful (see community involvement, above :)
  • The latest binary snapshots are available from the releng build status page
Happy testing!

2008-11-16

How to crossbuild on Cygwin hubertf's NetBSD blog 2008-11-16 15:05 UTC

For those poor people that are forced to use Windows and who still want to (cross)build NetBSD, there's help now: Izumi Tsutsui has posted instructions on how to crossbuild NetBSD on Cygwin. It assumes Windows XP and Cygwin 1.5.25.

Basically the venture boils down to a simple ``sh build.sh -m i386 -U release'', but the real art is in setting up the build environment with all those packages that you have installed on a decent Unix system by default.[1]

[1] Speaking of a 'decent' Unix system: I've had the joy to setup pkgsrc on a minimal Debian installation the other day, and it's amazing what list of standard Unix tools it's lacking: cpp, (n)curses, gmake, cvs, g++ (compiler plus headers; gcc was there, but without headers), bc, and bsd-mail / mailx.

2008-11-14

12/Oct/2008 Jeremy C. Reed's blog 2008-11-14 01:57 UTC

Went to the NetBSD Developer Summit -- at least the second half on Friday , October 10th. My first time in New York. A few NetBSD developers were there and I was very glad to meet them. It was mostly a social time with various random discussions. Later that night met with many NYCBSDCon attendees at a restaurant. Great to meet many BSD people.

Attended the NYCBSDCon conference on Saturday. I took many notes as I attended all the sessions. Adrian Chadd discussed POSIX AIO, mmap, sendfile, and squid IO issues. Jason Wright discussed many issues where hardware manufacturers make stupid decisions make it hard for developers and even actually shipping hardware that is limited or broken. Matt Dillon discussed his DragonFly HAMMER file system -- designed for terabyte disks and constant snapshots for a forever history of a disk. Anders Magnusson talked about his PCC work and then Mickey Shalayeff talked about porting PCC. Ike (the MC) handed out my Peter Salus The Daemon, The Gnu, and the Penguin book. Julio M. Merino Vidal introduces his ATF -- Automated Testing Framework; it was good he showed some examples; now I need to add some tests. Then I gave a lecture introducing DNSSEC. Many questions and discussion and went overtime. Then Jason Dixon gave his comical presentation about BSD versus GPL. That night we had dinner at same restaurant, Havana Central.

Sunday morning's first lecture was by Theo Schlossnagle about Reconnoiter; I have used Nagios a lot for two years to monitor hundreds of devices and services -- I need to try the alternative: Reconnoiter. Pawel Jakub Dawidek gave two lectures: a short fun intro to ZFS and then a more detailed, look at ZFS. Kurt Miller introduced Position Independent Executables (PIE) and his OpenBSD work on that. I had to leave after lunch. So I missed the last two lectures.

NYCBSDCon was a great opportunity. I met a CERT coordinator; met Dru Lavigne and Jim Brown (my co-board members) in person for the first time; had two BSD Certificate group meetings: board and infrastructure; and met many others. It was great to meet in person many that I have had much online work and discussions with, like perry, jmmv, ragge, and others. Also took some photos with all the NetBSD developers who were there on Sunday. George and Ike and the rest of the NYCBSDCon team did a great job. George and Ike constantly introduced people to each other -- this made the event a lot better as they build and encouraged relationships. (Because of this, NYCBSDCon 2008 was many times better than BSDCon 2000.) I took some photos. I will upload them later.

13/Nov/2008 Jeremy C. Reed's blog 2008-11-14 01:08 UTC

Started a webpage to start listing my many NetBSD questions. I constantly have more NetBSD questions that I can't quickly find answers for from searching man pages and even from quickly searching source. This list of questions is hosted at http://reedmedia.net/~reed/netbsd/. If you have any answers or can point me to the specific docs, please let me know at reed AT reedmedia DOT net. In return I will make sure the manual pages or any corresponding official documentation is clear about it.

Catching up - what happened in NetBSD-land between mid-August and mid-November hubertf's NetBSD blog 2008-11-14 00:47 UTC

OK, I've been slacking^Wbusy for the past weeks, but I hope things will get a bit better now. For a start, here's a catch-up of the things that accumulated in my inbox in the past ~two months:

  • Google Summer of Code is over for some time, but apparantly no final report has emerged so far (shame!). Still, a number of individual status reports came by on the official lists:

    I know of at least one other project (uvc) that has completed but that I didn't see a report here - maybe I've missed it. Anyways, GSoC was another big success this year. Thanks, Google!

  • Speaking of Adam Hamsik and Logical Volume Management (LVM), Adam has continued his work in that are, and he has written a device mapping library that interacts with his kernel driver. This allows to interact with his GSoC project without using any GPL code! See Adam's posting for more details.
  • Force 10 Networks, producer of 10gbit switches that use an operating system based on NetBSD, have added a new feature as part of their FTOS operating system: VirtualView, which provides virtualization of Force 10 based equipment. From the xchange article: ``Force10 Networks Inc. this week introduced VirtualView software for benchmaking, troubleshooting and managing virtualized environments based on Force10 gear.''

    More information is available from the Force 10 Networks homepage, plus in articles by fibresystems.org, light reading, Zycko, and SmartBrief.

  • Following the latest hype in portable computers, NetBSD has created a netbook page that intends to list models and the extent to which they are supported. Your contributions are most welcome here! (Contact me for sending updates and hardware :-)
  • Zafer Aydogan has made RSS feeds available for CVS commits to single files - see his mail to netbsd-users for more details.
  • New security advisory were released that I've missed in my last update:
  • A project that's been ongoing for quite some time is the move from "old-school" loadable kernel modules (LKMs) to new-style kernel modules. Important changes include the fact that modules can be either linked into the kernel at build time, or loaded into the kernel at runtime from the same file. Also, the bootloader was modified to load modules after the kernel, e.g. for a RAM-disk like the one that is used by the INSTALL kernel.

    In the same line, some parts are starting to be moved out of the GENERIC kernel, and installed as modules that can be loaded by the new framework then. The start is made by POSIX semaphores as a first step and proof-of concept, even if some details are still under hot debate, e.g. what the file system layout for modules is, and if the belong to the kernel and its build process, or to the userland.

  • While talking about splitting the kernel into modules, Antti Kantee has continued his work to move parts of the kernel into userspace, in particular running file system code as userland in his RUMP, and puffs and (Re)FUSE works. The idea is to provide the interfaces that file systems need in the userland, and the result is that you can run code that used to run inside the kernel in userland now.

    Another subsystem running in the kernel that could be moved to userland by providing appropriate interfaces with the rest of the kernel is the network stack, and Antti has moved just that to the userland. See Antti's mail to tech-net@ for more information on this impressive work.

  • NetBSD has shipped XFree in previous releases, and people who wanted to use X.org had to install it from pkgsrc. That's all fine, but to get a modern X, one had to compile things, as no precompiled binary packages are made available for many platforms. This is changing now, and NetBSD is getting X.org integrated via a reachover infrastructure which is also enabled for crosscompiling.

    The "user interface" for this is still in flux, but after some detour ("build.sh -V MKXORG=yes", without -x), "build.sh -x" now builds whatever X is considered the default for the platform. Some platforms already default to use X.org as X, and more will come, as changes that were made to NetBSD's copy of XFree are adopted to X.org.

    Platforms that use X.org by default now are macppc (see here and here), sparc sparc64, shark, amd64 and i386.

    As X.org is at Revision 7 now, it's installed in /usr/X11R7, which will lead to a lot of interesting effects. pkgsrc is already prepared for the new layout, but there are still many minor details that will need adjusting to the new directory. If you find one, post your patches to tech-x11.

  • Besides the GNU C compiler, there's the BSD-licensed Portable C Compiler around for some time now. It doesn't offer the same support as its GNU cousin yet, but this may change now: The BSD Fund is currently doing a fund drive to get money to enhance PCC. The goal is to raise $12,000US to improve support for core compiler functionality as well as support for C99, gcc compatibility and the amd64 architecture. See the project page for further details.
  • The NetBSD 5.0 release cycle has started! There's a netbsd-5 branch in CVS, daily binaries are available for testing, and some of the highlights of the upcoming release include file system journalling for FFS via WAPBL, and X.org.

    To help testing of NetBSD on Cobalt machines, Izumi Tsutsui has made a NetBSD 5.0_BETA based version of the Cobalt restore CD available. Enjoy!

  • As the final point today, a word on NUMA support from Christoph Egger. Non-Uniform Memory Access is needed in massive parallel systems where some nodes have RAM more tightly associated than others, where the RAM is further away, resulting in different access times for different regions of memory. In order to support this, Christop Egger has made first steps.

    His example implementation uses information from ACPI, and shows some heavy dmesg-pr0n from a 16-core machine with four sockets. Yumm!

So much for today. With the NetBSD 5.0 release cycle started, I'd like to encourage everyone to test the release branch, report errors, send patches as well as beer and choccolate to make this the best release that we've ever had.

2008-11-13

NetBSD TV: The Media Server What Do You Want? 2008-11-13 20:16 UTC

As I mentioned recently I bought a Playstation 3 (PS3) to use as a streaming video client in conjunction with the MediaTomb UPnP media server. Here is my recipe for installing and configuring the server package under NetBSD. The instructions apply to version 0.11.0nb2 or newer of the package, older versions provide neither a startup script nor a dedicated user account.

  1. Use make install or pkg_add(1) to install the package.
  2. Start the MediaTomb server manually as user root with the following command:
    /usr/pkg/share/examples/rc.d/mediatomb start
    Wait about 5 seconds and stop the server with this command afterwards:
    /usr/pkg/share/examples/rc.d/mediatomb stop
    The server should have created a configuration file located under /var/mediatomb/config.xml. Please use your favourite editor to make the changes suggested below.
  3. Change <accounts enabled=”no” …> to <accounts enabled=”yes” …> and add a password to the account definition one line below. This will protect the web user interface by password and stop unauthorized users from browsing your file systems.
  4. You can define a name for your media server by changing the <name>MediaTomb</name> line.
  5. If you want to use a firewall to restrict access to MediaTomb you need to put a fixed port number above 49152 into the <ports>0</ports> line. MediaTomb will otherwise choose a port number dynamically.
  6. Change the <protocolInfo extend=”no”/> to <protocolInfo extend=”yes”/> to enable support for protocol extensions required by the Playstation 3.
  7. Further down in the mappings section you will find an entry to enable DivX support for the PS3. Uncommenting that line isn’t really necessary to watch DivX files. But it will enable shiny DivX icons for all .avi files in the PS3’s video menu.
  8. Start the server again and and look at the logfile /var/log/mediatomb.log. You should find two lines likes these at the end of the logfile:
    2008-11-11 01:02:03 INFO: MediaTomb Web UI can be reached by following this link:
    2008-11-11 01:02:03 INFO: http://192.0.2.1:49152/
    Please visit this URL with a web browser and login into the web user interface using the account mediatomb and the password you defined previously.
  9. Use the Filesystem option of the web user interface to select directories with media content (audio or video) that should be exported to the PS3.
  10. Finally enable autostart of the server by copying /usr/pkg/share/examples/rc.d/mediatomb to /etc/rc.d and adding mediatomb=YES to /etc/rc.conf.

You should now be able to enjoy your audio and video files on your Playstation 3.

2008-11-10

NetBSD TV: The Streaming Video Client What Do You Want? 2008-11-10 23:27 UTC

For quite some time I’ve been looking for a streaming video client that would allow me to watch the video files that are stored on my NetBSD server on the TV in the sitting room. I thought that my requirements for such a client were pretty basic:

  1. Decent analog video (preferably via a SCART connector) and digitial audio output.
  2. An HDMI connector for future use.
  3. Support for popular video file formats like DivX and MP4.
  4. Doesn’t require a proprietary server software.
  5. A good WAF.

But I was wrong. I couldn’t find any streaming video client that met these demands in over a year. When I recently learned that Sony’s Playstation 3 (PS3) supports DivX in newer versions of its firmware my interest was sparked. After a bit of research I found a number of facts in favour of the PS3:

  1. The PS3 supports UPnP AV and works fine with MediaTomb, an open source UPnP MediaServer.
  2. The PS3 has all the video and audio connectors that I wanted.
  3. As the PS3 can also play DVDs it could replace my DVD player. That would not only avoid an increase in the number of devices in the sitting room but also prevent a shortage of SCART ports on the TV.
  4. The case of the PS3 is well designed and shiny.
  5. In addition to all that the PS3 is also a powerful game console and a Blu-ray Disc player. And I was keen to play Assassin’s Creed anyway.

Based on the above evaluation I came up with a profound business case which was approved by the secretary of domestic affairs straight away (the WAF was even better than anticipated). I bought a Playstation 3 online the next day.

2008-07-26

Re-bootstrapping pkgsrc unsigned long geek = random(); 2008-07-26 10:43 UTC

pkgsrc doesn’t officially support re-bootstrapping once it’s been installed (ie updating the core packages by doing a fresh bootstrap). There is a way to force one by removing a few core packages first though:
pkg_delete -ff bootstrap-mk-files bmake tnftp pax pkg_install
cd /usr/pkgsrc/bootstrap
./bootstrap

As long as the fresh bootstrap is done using the same parameters as used originally (prefix, database directory, etc), everything should work as before.

I tested this a short while ago on an OS X 10.5 system using a pkgsrc -current tree and it worked perfectly.

2007-07-18

Build a better mail server with NetBSD, part 2 unsigned long geek = random(); 2007-07-18 08:10 UTC

In the first part of this short series, I detailed the reasoning behind my need for a new mail server. In this second part, I’m going to detail my mail architecture as well as the software choices I made and why.

All mail for my various domains is delivered to the primary MX, which is a hosted server sitting in the US running Debian GNU/Linux (unfortunately my hosting provider doesn’t support NetBSD). It runs postfix and makes use of a variety of checks within postfix itself as well as postgrey for greylisting. I use a fairly conservative list of RBLs and, in conjunction with greylisting, they stop most spam from being accepted. Why postfix? Well, I stopped using sendmail over ten years ago, and although I’ve had good results with Exim in the past, these days I’m just most comfortable with postfix and it suits my needs perfectly.

The RBLs I use:

  • zen.spamhaus.org
  • cbl.abuseat.org
  • list.dsbl.org

Once mail has been received by my primary MX, it is delivered to local mailboxes, one per user. None of the users read their mail from the US server, however. All the mail is downloaded to the local mail server via SSL-secured POP3 and accessed here, either locally or via IMAP. The local mail server is a Sun Ultra 2 running NetBSD/sparc64 3.1_STABLE.

Software I’m using on the local mail server:

  • postfix
  • dovecot for IMAP (over SSL) access. There are a number of IMAP/POP3 servers available, but I chose Dovecot because of its clean design, good security record and flexible support for mail storage, amongst other things.
  • amavisd-new with spamassasin (with razor and Bayesian filtering enabled) and clamav for content filtering
  • getmail to download mail from the US server
  • mailgraph for simple reporting

All of the above are available in pkgsrc. As I have already done any RBL-based checks on the MX, I don’t do any of them locally.

2007-07-16

pkgsrc on FreeBSD/sparc64 6.2 unsigned long geek = random(); 2007-07-16 17:49 UTC

Although FreeBSD is a platform supported by pkgsrc, only i386 supported is explicitly mentioned and the most recent bootstrap binaries are for FreeBSD/i386 5.3. Being the adventurous chap I am, I decided to bootstrap from source on my dual CPU Ultra 60 running FreeBSD/sparc64 6.2. In true pkgsrc style, it Just Worked. Since bootstrapping, I’ve build a number of fairly large packages without problem: zsh, ncurses, vim, postfix, perl.

Why pkgsrc and not ports on a FreeBSD system? Well, I have a finely tuned pkgsrc environment that builds packages with my set of defaults (MIT Kerberos support, for example), so I decided to stick with what works well for me.

2007-06-13

Network booting FreeBSD on sparc64 systems unsigned long geek = random(); 2007-06-13 14:20 UTC

I’ve been network booting SPARC systems for a while now, ever since my last run in with a faulty floppy drive on a SPARCstation 2. NetBSD makes it easy - the standard installation includes a diskless client filesystem which can simply be extracted onto the boot server. It wasn’t quite so straightforward with FreeBSD, so here are a few pointers…

My boot server is running NetBSD 3.1 and I booted FreeBSD 6.2, so if you’re using different software you may need to make some adjustments. YMMV.

  1. On the boot server, configure rarpd(8) as usual, adding the entry for your machine’s MAC address to /etc/ethers. For example (for a machine called test02):
    08:00:20:b2:2f:b6 test02
  2. Extract the FreeBSD base fileset to the appropriate location on your boot server (for example, /export/install/fb62_sp64).
  3. Extract the FreeBSD GENERIC kernel fileset to boot/ in your diskless filesystem.
  4. Within your diskless root, symlink boot/GENERIC to boot/kernel (boot/GENERIC is a directory that contains the kernel and its modules).
  5. Put boot/loaders from the diskless filesystem into your tftp root directory and symlink it to your machine’s IP address in hex. For example, the filename for 192.168.1.92 is C0A8015C.
  6. Export your diskless root filesystem via NFS and add the necessary dhcpd.conf stanza. For example:
    host test02.pimp.org.za {
    hardware ethernet 08:00:20:b2:2f:b6;
    fixed-address 192.168.1.92;
    option host-name "test02";
    option root-path "/export/install/fb62_sp64";
    }
  7. Boot your machine - “boot net” from the PROM should do it.

A few tips:

  • It’s normally a good idea to update the machine’s OpenBoot PROM to the latest release. Old PROMs often have subtle bugs.
  • Extracting FreeBSD filesets is simple:
    cat 6.2-RELEASE/base/base.* > /tmp/base.tar.gz
    tar -xzvpf /tmp/base.tar.gz -C /export/install/fb62_sp4

Update: Fixed incorrect command to extract sets (thanks John Messenger!)

Feeds

Feed RSS Last update Next update
#NetBSD Community Blog XML 2008-11-23 11:15 2008-11-23 13:15
bsdtalk XML 2008-11-23 11:15 2008-11-23 13:15
freshmeat.net announcements (Global) XML 2008-11-23 11:15 2008-11-23 13:15
http://ndis-netbsd.blogspot.com/ XML 2008-11-23 11:15 2008-11-23 13:15
hubertf's NetBSD blog XML 2008-11-23 11:15 2008-11-23 13:15
i summon one kim XML 2008-11-23 11:15 2008-11-23 13:15
Implementality XML 2008-11-23 11:15 2008-11-23 13:15
Jeremy C. Reed's blog XML 2008-11-23 11:15 2008-11-23 13:15
Latest Secunia Advisories XML 2008-11-23 11:15 2008-11-23 13:15
NetBSD.org News XML 2008-11-23 11:15 2008-11-23 13:15
OSNews XML 2008-11-23 11:15 2008-11-23 13:15
rhaen daily XML 2008-11-23 11:15 2008-11-23 13:15
Seebach Exhibit 7 XML 2008-11-23 11:15 2008-11-23 13:15
start XML 2008-11-23 11:15 2008-11-23 13:15
The Julipedia XML 2008-11-23 11:15 2008-11-23 13:15
unsigned long geek = random(); XML 2008-11-23 11:15 2008-11-23 13:15
What Do You Want? XML 2008-11-23 11:15 2008-11-23 13:15
Yareev's schmonz.com - Technology XML 2008-11-23 11:15 2008-11-23 13:15