RUMPUSER(3)             NetBSD Library Functions Manual            RUMPUSER(3)

     rumpuser -- rump kernel hypercall interface

     rump User Library (librumpuser, -lrumpuser)

     #include <rump/rumpuser.h>

     The rumpuser hypercall interfaces allow a rump kernel to access host
     resources.  A hypervisor implementation must implement the routines
     described in this document to allow a rump kernel to run on the host.
     The implementation included in NetBSD is for POSIX-like hosts (*BSD,
     Linux, etc.).  This document is divided into sections based on the func-
     tionality group of each hypercall.

     Since the hypercall interface is a C function interface, both the rump
     kernel and the hypervisor must conform to the same ABI.  The interface
     itself attempts to assume as little as possible from the type systems,
     and for example off_t is passed as int64_t and enums are passed as ints.
     It is recommended that the hypervisor converts these to the native types
     before starting to process the hypercall, for example by assigning the
     ints back to enums.

     A hypercall is always entered with the calling thread scheduled in the
     rump kernel.  In case the hypercall intends to block while waiting for an
     event, the hypervisor must first release the rump kernel scheduling con-
     text.  In other words, the rump kernel context is a resource and holding
     on to it while waiting for a rump kernel event/resource may lead to a
     deadlock.  Even when there is no possibility of deadlock in the strict
     sense of the term, holding on to the rump kernel context while performing
     a slow hypercall such as reading a device will prevent other threads
     (including the clock interrupt) from using that rump kernel context.

     Releasing the context is done by calling the hyp_backend_unschedule()
     upcall which the hypervisor received from rump kernel as a parameter for
     rumpuser_init().  Before a hypercall returns back to the rump kernel, the
     returning thread must carry a rump kernel context.  In case the hypercall
     unscheduled itself, it must reschedule itself by calling

     int rumpuser_init(int version, struct rump_hyperup *hyp)

     Initialize the hypervisor.

     version          hypercall interface version number that the kernel
                      expects to be used.  In case the hypervisor cannot pro-
                      vide an exact match, this routine must return a non-zero

     hyp              pointer to a set of upcalls the hypervisor can make into
                      the rump kernel

   Memory allocation
     int rumpuser_malloc(size_t len, int alignment, void **memp)

     len              amount of memory to allocate

     alignment        size the returned memory must be aligned to.  For exam-
                      ple, if the value passed is 4096, the returned memory
                      must be aligned to a 4k boundary.

     memp             return pointer for allocated memory

     void rumpuser_free(void *mem, size_t len)

     mem              memory to free

     len              length of allocation.  This is always equal to the
                      amount the caller requested from the rumpuser_malloc()
                      which returned mem.

   Files and I/O
     int rumpuser_open(const char *name, int mode, int *fdp)

     Open name for I/O and associate a file descriptor with it.  Notably,
     there needs to be no mapping between name and the host's file system
     namespace.  For example, it is possible to associate the file descriptor
     with device I/O registers for special values of name.

     name             the identifier of the file to open for I/O

     mode             combination of the following:

                      RUMPUSER_OPEN_RDONLY   open only for reading

                      RUMPUSER_OPEN_WRONLY   open only for writing

                      RUMPUSER_OPEN_RDWR     open for reading and writing

                      RUMPUSER_OPEN_CREATE   do not treat missing name as an

                      RUMPUSER_OPEN_EXCL     combined with
                                             RUMPUSER_OPEN_CREATE, flag an
                                             error if name already exists

                      RUMPUSER_OPEN_BIO      the caller will use this file for
                                             block I/O, usually used in con-
                                             junction with accessing file sys-
                                             tem media.  The hypervisor should
                                             treat this flag as advisory and
                                             possibly enable some optimiza-
                                             tions for *fdp based on it.
                      Notably, the permissions of the created file are left up
                      to the hypervisor implementation.

     fdp              An integer value denoting the open file is returned

     int rumpuser_close(int fd)

     Close a previously opened file descriptor.

     int rumpuser_getfileinfo(const char *name, uint64_t *size, int *type)

     name             file for which information is returned.  The namespace
                      is equal to that of rumpuser_open().

     size             If non-NULL, size of the file is returned here.

     type             If non-NULL, type of the file is returned here.  The
                      options are RUMPUSER_FT_DIR, RUMPUSER_FT_REG,
                      for directory, regular file, block device, character
                      device or unknown, respectively.

     void rumpuser_bio(int fd, int op, void *data, size_t dlen, int64_t off,
     rump_biodone_fn biodone, void *donearg)

     Initiate block I/O and return immediately.

     fd               perform I/O on this file descriptor.  The file descrip-
                      tor must have been opened with RUMPUSER_OPEN_BIO.

     op               Transfer data from the file descriptor with
                      RUMPUSER_BIO_READ and transfer data to the file descrip-
                      tor with RUMPUSER_BIO_WRITE.  Unless RUMPUSER_BIO_SYNC
                      is specified, the hypervisor may cache a write instead
                      of committing it to permanent storage.

     data             memory address to transfer data to/from

     dlen             length of I/O.  The length is guaranteed to be a multi-
                      ple of 512.

     off              offset into fd where I/O is performed

     biodone          To be called when the I/O is complete.  Accessing data
                      is not legal after the call is made.

     donearg          opaque arg that must be passed to biodone.

     int rumpuser_iovread(int fd, struct rumpuser_iovec *ruiov, size_t iovlen,
     int64_t off, size_t *retv)

     int rumpuser_iovwrite(int fd, struct rumpuser_iovec *ruiov,
     size_t iovlen, int64_t off, size_t *retv)

     These routines perform scatter-gather I/O which is not block I/O by
     nature and therefore cannot be handled by rumpuser_bio().

     fd               file descriptor to perform I/O on

     ruiov            an array of I/O descriptors.  It is defined as follows:
                            struct rumpuser_iovec {
                                    void *iov_base;
                                    size_t iov_len;

     iovlen           number of elements in ruiov

     off              offset of fd to perform I/O on.  This can either be a
                      non-negative value or RUMPUSER_IOV_NOSEEK.  The latter
                      denotes that no attempt to change the underlying objects
                      offset should be made.  Using both types of offsets on a
                      single instance of fd results in undefined behavior.

     retv             number of bytes successfully transferred is returned

     int rumpuser_syncfd(int fd, int flags, uint64_t start, uint64_t len)

     Synchronizes fd with respect to backing storage.  The other arguments

     flags            controls how synchronization happens.  It must contain
                      one of the following:

                      RUMPUSER_SYNCFD_READ      Make sure that the next read
                                                sees writes from all other
                                                parties.  This is useful for
                                                example in the case that fd
                                                represents memory to write a
                                                DMA read is being performed.

                      RUMPUSER_SYNCFD_WRITE     Flush cached writes.

                      The following additional parameters may be passed in

                      RUMPUSER_SYNCFD_BARRIER   Issue a barrier.  Outstanding
                                                I/O operations which were
                                                started before the barrier
                                                complete before any operations
                                                after the barrier are per-

                      RUMPUSER_SYNCFD_SYNC      Wait for the synchronization
                                                operation to fully complete
                                                before returning.  For exam-
                                                ple, this could mean that the
                                                data to be written to a disk
                                                has hit either the disk or
                                                non-volatile memory.

     start            offset into the object.

     len              the number of bytes to synchronize.  The value 0 denotes
                      until the end of the object.

     The hypervisor should support two clocks, one for wall time and one for
     monotonically increasing time, the latter of which may be based on some
     arbitrary time (e.g. system boot time).  If this is not possible, the
     hypervisor must make a reasonable effort to retain semantics.

     int rumpuser_clock_gettime(int enum_rumpclock, int64_t *sec, long *nsec)

     enum_rumpclock   specifies the clock type.  In case of
                      RUMPUSER_CLOCK_RELWALL the wall time should be returned.
                      In case of RUMPUSER_CLOCK_ABSMONO the time of a mono-
                      tonic clock should be returned.

     sec              return value for seconds

     nsec             return value for nanoseconds

     int rumpuser_clock_sleep(int enum_rumpclock, int64_t sec, long nsec)

     enum_rumpclock   In case of RUMPUSER_CLOCK_RELWALL, the sleep should last
                      at least as long as specified.  In case of
                      RUMPUSER_CLOCK_ABSMONO, the sleep should last until the
                      hypervisor monotonic clock hits the specified absolute

     sec              sleep duration, seconds.  exact semantics depend on clk.

     nsec             sleep duration, nanoseconds.  exact semantics depend on

   Parameter retrieval
     int rumpuser_getparam(const char *name, void *buf, size_t buflen)

     Retrieve a configuration parameter from the hypervisor.  It is up to the
     hypervisor to decide how the parameters can be set.

     name             name of the parameter.  If the name starts with an
                      underscore, it means a mandatory parameter.  The manda-
                      tory parameters are RUMPUSER_PARAM_NCPU which specifies
                      the amount of virtual CPUs bootstrapped by the rump ker-
                      nel and RUMPUSER_PARAM_HOSTNAME which returns a prefer-
                      ably unique instance name for the rump kernel.

     buf              buffer to return the data in as a string

     buflen           length of buffer

     void rumpuser_exit(int value)

     Terminate the rump kernel with exit value value.  If value is
     RUMPUSER_PANIC the hypervisor should attempt to provide something akin to
     a core dump.

   Console output
     Console output is divided into two routines: a per-character one and
     printf-like one.  The former is used e.g. by the rump kernel's internal
     printf routine.  The latter can be used for direct debug prints e.g. very
     early on in the rump kernel's bootstrap or when using the in-kernel rou-
     tine causes too much skew in the debug print results (the hypercall runs
     outside of the rump kernel and therefore does not cause any locking or
     scheduling events inside the rump kernel).

     void rumpuser_putchar(int ch)

     Output ch on the console.

     void rumpuser_dprintf(const char *fmt, ...)

     Do output based on printf-like parameters.

     A rump kernel should be able to send signals to client programs due to
     some standard interfaces including signal delivery in their specifica-
     tions.  Examples of these interfaces include setitimer(2) and write(2).
     The rumpuser_kill() function advises the hypercall implementation to
     raise a signal for the process containing the rump kernel.

     int rumpuser_kill(int64_t pid, int sig)

     pid              The pid of the rump kernel process that the signal is
                      directed to.  This value may be used as the hypervisor
                      as a hint on how to deliver the signal.  The value
                      RUMPUSER_PID_SELF may also be specified to indicate no
                      hint.  This value will be removed in a future version of
                      the hypercall interface.

     sig              Number of signal to raise.  The value is in NetBSD sig-
                      nal number namespace.  In case the host has a native
                      representation for signals, the value should be trans-
                      lated before the signal is raised.  In case there is no
                      mapping between sig and native signals (if any), the
                      behavior is implementation-defined.

     A rump kernel will ignore the return value of this hypercall.  The only
     implication of not implementing rumpuser_kill() is that some application
     programs may not experience expected behavior for standard interfaces.

     As an aside,the rump_sp(7) protocol provides equivalent functionality for
     remote clients.

   Random pool
     int rumpuser_getrandom(void *buf, size_t buflen, int flags, size_t *retp)

     buf              buffer that the randomness is written to

     buflen           number of bytes of randomness requested

     flags            The value 0 or a combination of RUMPUSER_RANDOM_HARD
                      (return true randomness instead of something from a
                      PRNG) and RUMPUSER_RANDOM_NOWAIT (do not block in case
                      the requested amount of bytes is not available).

     retp             The number of random bytes written into buf.

     int rumpuser_thread_create(void *(*fun)(void *), void *arg,
     const char *thrname, int mustjoin, int priority, int cpuidx,
     void **cookie)

     Create a schedulable host thread context.  The rump kernel will call this
     interface when it creates a kernel thread.  The scheduling policy for the
     new thread is defined by the hypervisor.  In case the hypervisor wants to
     optimize the scheduling of the threads, it can perform heuristics on the
     thrname, priority and cpuidx parameters.

     fun              function that the new thread must call.  This call will
                      never return.

     arg              argument to be passed to fun

     thrname          Name of the new thread.

     mustjoin         If 1, the thread will be waited for by
                      rumpuser_thread_join() when the thread exits.

     priority         The priority that the kernel requested the thread to be
                      created at.  Higher values mean higher priority.  The
                      exact kernel semantics for each value are not available
                      through this interface.

     cpuidx           The index of the virtual CPU that the thread is bound
                      to, or -1 if the thread is not bound.  The mapping
                      between the virtual CPUs and physical CPUs, if any, is
                      hypervisor implementation specific.

     cookie           In case mustjoin is set, the value returned in cookie
                      will be passed to rumpuser_thread_join().

     void rumpuser_thread_exit(void)

     Called when a thread created with rumpuser_thread_create() exits.

     int rumpuser_thread_join(void *cookie)

     Wait for a joinable thread to exit.  The cookie matches the value from

     void rumpuser_curlwpop(int enum_rumplwpop, struct lwp *l)

     Manipulate the hypervisor's thread context database.  The possible opera-
     tions are create, destroy, and set as specified by enum_rumplwpop:

     RUMPUSER_LWP_CREATE    Inform the hypervisor that l is now a valid thread
                            context which may be set.  A currently valid value
                            of l may not be specified.  This operation is
                            informational and does not mandate any action from
                            the hypervisor.

     RUMPUSER_LWP_DESTROY   Inform the hypervisor that l is no longer a valid
                            thread context.  This means that it may no longer
                            be set as the current context.  A currently set
                            context or an invalid one may not be destroyed.
                            This operation is informational and does not man-
                            date any action from the hypervisor.

     RUMPUSER_LWP_SET       Set l as the current host thread's rump kernel
                            context.  A previous context must not exist.

     RUMPUSER_LWP_CLEAR     Clear the context previous set by
                            RUMPUSER_LWP_SET.  The value passed in l is the
                            current thread and is never NULL.

     struct lwp * rumpuser_curlwp(void)

     Retrieve the rump kernel thread context associated with the current host
     thread, as set by rumpuser_curlwpop().  This routine may be called when a
     context is not set and the routine must return NULL in that case.  This
     interface is expected to be called very often.  Any optimizations per-
     taining to the execution speed of this routine should be done in

     void rumpuser_seterrno(int errno)

     Set an errno value in the calling thread's TLS.  Note: this is used only
     if rump kernel clients make rump system calls.

   Mutexes, rwlocks and condition variables
     The locking interfaces have standard semantics, so we will not discuss
     each one in detail.  The data types struct rumpuser_mtx, struct
     rumpuser_rw and struct rumpuser_cv used by these interfaces are opaque to
     the rump kernel, i.e. the hypervisor has complete freedom over them.

     Most of these interfaces will (and must) relinquish the rump kernel CPU
     context in case they block (or intend to block).  The exceptions are the
     "nowrap" variants of the interfaces which may not relinquish rump kernel

     void rumpuser_mutex_init(struct rumpuser_mtx **mtxp, int flags)

     void rumpuser_mutex_enter(struct rumpuser_mtx *mtx)

     void rumpuser_mutex_enter_nowrap(struct rumpuser_mtx *mtx)

     int rumpuser_mutex_tryenter(struct rumpuser_mtx *mtx)

     void rumpuser_mutex_exit(struct rumpuser_mtx *mtx)

     void rumpuser_mutex_destroy(struct rumpuser_mtx *mtx)

     void rumpuser_mutex_owner(struct rumpuser_mtx *mtx, struct lwp **lp)

     Mutexes provide mutually exclusive locking.  The flags, of which at least
     one must be given, are as follows:

     RUMPUSER_MTX_SPIN     Create a spin mutex.  Locking this type of mutex
                           must not relinquish rump kernel context even when
                           rumpuser_mutex_enter() is used.

     RUMPUSER_MTX_KMUTEX   The mutex must track and be able to return the rump
                           kernel thread that owns the mutex (if any).  If
                           this flag is not specified, rumpuser_mutex_owner()
                           will never be called for that particular mutex.

     void rumpuser_rw_init(struct rumpuser_rw **rwp)

     void rumpuser_rw_enter(int enum_rumprwlock, struct rumpuser_rw *rw)

     int rumpuser_rw_tryenter(int enum_rumprwlock, struct rumpuser_rw *rw)

     int rumpuser_rw_tryupgrade(struct rumpuser_rw *rw)

     void rumpuser_rw_downgrade(struct rumpuser_rw *rw)

     void rumpuser_rw_exit(struct rumpuser_rw *rw)

     void rumpuser_rw_destroy(struct rumpuser_rw *rw)

     void rumpuser_rw_held(int enum_rumprwlock, struct rumpuser_rw *rw,
     int *heldp)

     Read/write locks provide either shared or exclusive locking.  The possi-
     ble values for lk are RUMPUSER_RW_READER and RUMPUSER_RW_WRITER.  Upgrad-
     ing means trying to migrate from an already owned shared lock to an
     exclusive lock and downgrading means migrating from an already owned
     exclusive lock to a shared lock.

     void rumpuser_cv_init(struct rumpuser_cv **cvp)

     void rumpuser_cv_destroy(struct rumpuser_cv *cv)

     void rumpuser_cv_wait(struct rumpuser_cv *cv, struct rumpuser_mtx *mtx)

     void rumpuser_cv_wait_nowrap(struct rumpuser_cv *cv, struct rumpuser_mtx

     int rumpuser_cv_timedwait(struct rumpuser_cv *cv,
     struct rumpuser_mtx *mtx, int64_t sec, int64_t nsec)

     void rumpuser_cv_signal(struct rumpuser_cv *cv)

     void rumpuser_cv_broadcast(struct rumpuser_cv *cv)

     void rumpuser_cv_has_waiters(struct rumpuser_cv *cv, int *waitersp)

     Condition variables wait for an event.  The mtx interlock eliminates a
     race between checking the predicate and sleeping on the condition vari-
     able; the mutex should be released for the duration of the sleep in the
     normal atomic manner.  The timedwait variant takes a specifier indicating
     a relative sleep duration after which the routine will return with
     ETIMEDOUT.  If a timedwait is signaled before the timeout expires, the
     routine will return 0.

     The order in which the hypervisor reacquires the rump kernel context and
     interlock mutex before returning into the rump kernel is as follows.  In
     case the interlock mutex was initialized with both RUMPUSER_MTX_SPIN and
     RUMPUSER_MTX_KMUTEX, the rump kernel context is scheduled before the
     mutex is reacquired.  In case of a purely RUMPUSER_MTX_SPIN mutex, the
     mutex is acquired first.  In the final case the order is implementation-

     All routines which return an integer return an errno value.  The hypervi-
     sor must translate the value to the native errno namespace used by the
     rump kernel.  Routines which do not return an integer may never fail.


     Antti Kantee, "Flexible Operating System Internals: The Design and
     Implementation of the Anykernel and Rump Kernels", Aalto University
     Doctoral Dissertations, 2012, Section 2.3.2: The Hypercall Interface.

     For a list of all known implementations of the rumpuser interface, see .:

     The rump kernel hypercall API was first introduced in NetBSD 5.0.  The
     API described above first appeared in NetBSD 7.0.

NetBSD 9.0                      August 24, 2014                     NetBSD 9.0

You can also request any man page by name and (optionally) by section:


Use the DEFAULT collection to view manual pages for third-party software.

©1994 Man-cgi 1.15, Panagiotis Christias
©1996-2019 Modified for NetBSD by Kimmo Suominen