Installing Open Solaris as a Xen domU

This particular dom0 is running Ubuntu 8.10 on an amd64 kernel using an lvm volume group called ‘vg0’. Adjust accordingly. Note that the credentials for this particular live cd are:

unprivileged username: jack
unprivileged password: jack

privileged username: root
privileged password: opensolaris

I have been meaning to add a Solaris machine to my network for a while. I stopped by #opensolaris on freenode, and was helped a great deal by a guy that goes by the nick “pino42.” The gist is this:

  1. Download the ISO:
    cjac@dom0:~$ sudo mkdir /var/xen/osol && cd /var/xen/osol
    cjac@dom0:/var/xen/osol$ sudo wget -O osol-0906-ai-x86.iso
  2. Mount it on /mnt/osol/:
    cjac@dom0:/var/xen/osol$ sudo losetup /dev/loop0 osol-0906-x86.iso
    cjac@dom0:/var/xen/osol$ sudo mkdir /mnt/osol
    cjac@dom0:/var/xen/osol$ sudo mount /dev/loop0 /mnt/osol
  3. Copy the live kernel and ram disk from the iso so that Xen can boot from them:
    cjac@dom0:/var/xen/osol$ sudo mkdir /boot/osol
    cjac@dom0:/var/xen/osol$ sudo cp /mnt/osol/platform/i86xpv/kernel/amd64/unix /boot/osol/
    cjac@dom0:/var/xen/osol$ sudo cp /mnt/osol/boot/amd64/x86.microroot /boot/osol/
  4. Create the target lv to which we’ll install the OS:
    cjac@dom0:/var/xen/osol$ sudo lvcreate /dev/vg0 -n solaris0-disk -L 4G
  5. Create the xen domU .cfg file for the live CD/installer:
    cjac@dom0:/var/xen/osol$ sudo -s
    root@dom0:/var/xen/osol# cat > /etc/xen/solaris-live.cfg
    name        = "solaris-live"
    vcpus       = 1
    memory      = "768" # 512 will work, but it's slow                                                                                           
    kernel      = "/boot/osol/unix"
    ramdisk     = "/boot/osol/x86.microroot"
    extra       = "/mnt/osol/platform/i86xpv/kernel/amd64/unix -kd - nowin -B install_media=cdrom"
    disk        = ['phy:/dev/loop0,6:cdrom,r','phy:vg0/solaris0-disk,0,w']
    vif         = ['bridge=net,mac=00:16:3e:aa:b9:bc']
    on_shutdown = "destroy"
    on_reboot   = "destroy"
    on_crash    = "preserve"
    root@dom0:/var/xen/osol# exit
  6. Make sure you’ve got a DHCP server on the bridge you decide to use for ‘vif’. Setting this up is left as an exercise for the reader.
  7. All should be set to launch the live installer. Do so:
    cjac@dom0:/var/xen/osol$ sudo xm create -c solaris-live.cfg
  8. At the kernel debugger prompt, continue:
    Started domain solaris-live
                               Loading kmdb...
    Welcome to kmdb
    Loaded modules: [ unix krtld genunix ]
    [0]> :c
  9. After selecting your keyboard layout and desktop language, log in:
    opensolaris console login: jack
    Password: jack
  10. You will know that the DHCP server has assigned you an address when you see a message similar to the following on the console:
    Jun 12 10:21:07 opensolaris in.routed[688]: route --> nexthop is not directly connected

    The message will probably be different if you happen to have a RIP server functioning on the same link as the DHCP server. But I do not.

    To discover the IP address which has been assigned by the DHCP server, do the following:

    jack@opensolaris:~$ ifconfig -a
    lo0: flags=2001000849 mtu 8232 index 1
            inet netmask ff000000 
    xnf0: flags=1004843 mtu 1500 index 2
            inet netmask ffffff00 broadcast
    lo0: flags=2002000849 mtu 8252 index 1
            inet6 ::1/128 
    xnf0: flags=2004841 mtu 1500 index 2
            inet6 fe80::216:3eff:feaa:b9bc/10
  11. Since the installer is graphical, we’ll need to attach to the X server using vnc. On the solaris machine, you need to start the vnc server. You will be prompted for a password, which must be at least 6 characters in length:
    jack@opensolaris:~$ mkdir ~/.vnc
    jack@opensolaris:~$ cp ~/.Xclients ~/.vnc/xstartup
    jack@opensolaris:~$ vncserver
    You will require a password to access your desktops.
    Password: password
    Verify: password
  12. On a machine with an X display, connect to a machine which can communicate with the host and establish a VNC session:
    cjac@laptop:~$ echo $DISPLAY && ssh -AY dhcp-server
    cjac@dhcp-server:~$ sudo apt-get install xvnc4viewer
    cjac@dhcp-server:~$ echo $DISPLAY && vncviewer
    Password: password
  13. You should now have an X window with a view of the remote Solaris installer:
    OpenSolaris live installer
    OpenSolaris live installer

  14. Double-click on the "Install OpenSolaris" icon and proceed through the configuration. DO NOT reboot the system on completion, as we've got some Xen-specific mucking to do after the installer has completed.
  15. Return to your xm console shell and elevate your priviliges to root:
    jack@opensolaris:~$ su -
    Password: opensolaris
    Jun 12 10:50:24 opensolaris su: 'su root' succeeded for jack on /dev/console
    Sun Microsystems Inc.   SunOS 5.11      snv_111b        November 2008
  16. I slightly modified the script I found here to get a copy of the running kernel and boot archive:
    root@opensolaris:~# cat >
    zdb rpool > /tmp/zdb.out # this takes 5+ minutes
    bootpath=`grep phys_path /tmp/zdb.out | awk -F\' '{print $2}'`
    dataset_id=`grep 'Dataset rpool/ROOT/opensolaris ' /tmp/zdb.out | awk -F, '{print $2}' | cut -d' ' -f 3`
    root=`pfexec beadm list -H |  grep ';N*R;' | cut -d \; -f 1`
    mkdir /tmp/root
    pfexec beadm mount $root /tmp/root 2>/dev/null
    mount=`pfexec beadm list -H $root | cut -d \; -f 4`
    pfexec bootadm update-archive -R $mount
    echo "Kernel and ramdisk for $root should now be copied to $dom0:$dompath"
    echo "${mount}${unixfile} -> $dom0:$dompath/kernel.$root"
    echo "$mount/platform/i86pc/amd64/boot_archive -> $dom0:$dompath/ramdisk.$root"
    # pfexec beadm umount $root 2>/dev/null"
    echo "Kernel cmdline should be:"
    echo "$unixfile -B zfs-bootfs=rpool/$dataset_id,bootpath=$bootpath"
    root@opensolaris:~# chmod u+x
  17. Run the script:
    root@opensolaris:~# ./ dom0 /boot/osol amd64
  18. Go get a cup of coffee or something. When you return, you should see something like this:
    Kernel and ramdisk for opensolaris should now be copied to dom0:/boot/osol
    /tmp/root/platform/i86xpv/kernel/amd64/unix -> dom0:/boot/osol/kernel.opensolaris
    /tmp/root/platform/i86pc/amd64/boot_archive -> dom0:/boot/osol/ramdisk.opensolaris
    Kernel cmdline should be:
    /platform/i86xpv/kernel/amd64/unix -B zfs-bootfs=rpool/42,bootpath=/xpvd/xdf@0:a
  19. Copy the kernel and ramdisk to the dom0. Your dom0 is probably not on the same network segment as your dhcp server, but I'll assume that you can ssh from the dom0 to the dhcp server:
    jack@opensolaris:~$ scp /tmp/root/platform/i86xpv/kernel/amd64/unix cjac@dhcp-server:/tmp/kernel.opensolaris
    jack@opensolaris:~$ scp /tmp/root/platform/i86pc/amd64/boot_archive cjac@dhcp-server:/tmp/ramdisk.opensolaris
    cjac@dom0:~$ sudo scp cjac@dhcp-server:/tmp/ramdisk.opensolaris /boot/osol/
    cjac@dom0:~$ sudo scp cjac@dhcp-server:/tmp/kernel.opensolaris /boot/osol/
  20. shut the live domU down:
    jack@opensolaris:~$ jack@opensolaris:~$ pfexec halt
    Jun 12 22:00:46 opensolaris halt: initiated by jack on /dev/console
  21. Create your final Xen domU config file

    cjac@dom0:~$ sudo -s
    root@dom0:~# cat > /etc/xen/solaris0.cfg
    name    = 'solaris0'
    vcpus   = 1
    memory  = "512"
    vif     = ['bridge=loc,mac=00:16:3e:aa:b9:bc']
    disk    = [ 'phy:/dev/vg0/solaris0-disk,0,w' ]
    kernel  = '/boot/osol/kernel.opensolaris'
    ramdisk = '/boot/osol/ramdisk.opensolaris'
    extra   = '/platform/i86xpv/kernel/amd64/unix -B zfs-bootfs=rpool/42,bootpath=/xpvd/xdf@0:a'
    on_shutdown = 'destroy'
    on_reboot = 'destroy'
    on_crash = 'destroy'
    root@dom0:~# exit

  22. Instantiate the domU

    cjac@dom0:~$ sudo xm create -c solaris0.cfg
    Using config file "/etc/xen/solaris0.cfg".
    Started domain solaris0
                           v3.2-1 chgset 'unavailable'
    SunOS Release 5.11 Version snv_111b 64-bit
    Copyright 1983-2009 Sun Microsystems, Inc.  All rights reserved.
    Use is subject to license terms.
    Hostname: solaris0
    Reading ZFS config: done.
    Mounting ZFS filesystems: (6/6)
    solaris0 console login: 

, , , , ,

6 responses to “Installing Open Solaris as a Xen domU”

  1. CJ

    Thanks so much for posting this!

    I was struggling with getting NexentaStor Community Edition 3.1.1 running in XEN-4.0 with Debian Squeeze Dom0 until I found your post.

    Performance during the install phase is still kind of awful. I’m passing Nexenta a linux software raid-1 set consisting of two partitions on my boot drives, and I think this may be the issue.

    I’ll be passing a whole controller through as well (once I figure that out). My hope is that I can either
    a) figure out why the rpool (syspool) performance seems so bad.
    b) discover that the root volume performance doesn’t really matter for my virtual NAS appliance.

    Either way, I’m leaps and bounds ahead of where I was prior to finding your blog!

Leave a Reply