Nexenta: Ubuntu Server with ZFS goodness

OSAlert has been reporting on the Debian/Ubuntu/GNU/Opensolaris hybrid for several years. But for those of you who’ve never looked more closely at this interesting OS, a Nexenta developer has laid out some of its more noteworthy features and advantages.

The Nexenta project, started in 2005, has had 6 releases (NexentaOS and NexentaCore), and is preparing for the upcoming NexentaCore Platform 2 release. If you are a Debian/Ubuntu developer, consider taking a little time to take a look at an emerging platform that provides a feature rich developer environment.

Nexenta has built a small but vibrant community. It has been downloaded over half a million times, and is being used in around 10,000 systems, mostly data-centers.

Nexenta uses Debian’s dpkg packaging system, and provides everything provided by Debian/Ubuntu environment. However, these are only a subset of the features in Nexenta’s toolchain. We’ll take a look at the additional features and tools provided by Nexenta.

The Opensolaris technologies: ZFS

The open source world has gone gaga over the next generation file system from the Opensolaris community. It has been ported to Apple’s OSX and FreeBSD. Licensing incompatibilities have held up a Linux port. It is a file system and volume manager built into one. And with lightning fast snapshots and automatic integrity and error checking, this is the filesystem of the future. Lets take a look at how you as a developer can benefit by the use of ZFS:

Never have to worry about data loss, or the system and packaging system going into a mangled state

ZFS’s quick and painless snapshot feature allows you to take incremental backups of your entire filesystem or the packaging related data. So if the last kernel upgrade botches up the system or renders it unbootable, easily revert back to the earlier working state

Nexenta provide a tool called apt-clone which is a wrapper over apt-get. Roughly translated

apt-clone = system clone + apt-get

so a command apt-get install apache2 is equivalent to

zfs clone beforeapache2 rootfs
#add grub lines to boot to ‘beforeapache2’
apt-get install apache2

If things go wrong and apache2 isn’t installed right, you can revert back to the snapshot “beforeapache2” and the system reverts to the original configuration. Further information in the apt-clone manpage [0].

Use snapshot capabilities to create restore points

..all along your development/build cycle allowing for easy reversion to an earlier state

Another use of ZFS for package maintainers is the capability to easily take a snapshot of the environment (like the current working directory) and make those deadly changes. There no longer a need to worry or limit your changes for the fear of “ruining this working state” or “losing all these changes I’ve made”. Use ZFS like a quick and dirty versioning system for those short development sessions.

Increase build environment storage as and when required

If you’ve run into situations where you had dedicated space for a certain need, but later found yourself out of space. This could be for your package development needs or for the music you play when you develop. With ZFS, you no longer have to decide between removing package A or sources B. You can have both the Rock and the Blues collection. A new hard disk or a partition plus a simple command will take care of all your space needs.

ZFS pools and RAID-Z

..for superior volume and data redundancy management

ZFS filesystems are laid out on ‘pools’ which are storage spaces. This is a different from the traditional ‘disks’ and ‘partitions’ way of thinking about laying out filesystems. You can now add any type of storage (disk, network attached storage, a file) to the pool, and everything in the pool is made available to the filesystems that are created from the pool. This allows us to very easily increase the size of a pool by simply giving it more storage.

To create a pool from a disk

zpool create
zpool create diskpool c1d0p0

To increase the size of the pool, simply add another disk to it

zpool add
zpool add diskpool c2d0p0

ZFS supports various raid configurations out of the box, including RAID-Z, which is a better implementation of RAID5 [1]. It implements features into the filesystem that previously required specialized hardware. A comparison of ZFS with Linux LVMs can be seen at [2].

This allows you to setup your storage in mirrored disks by simply (eg)

zpool create mypool mirror c1d0p2 c2d0p2

Creating a zfs filesystem on our newly created pool is done via

zfs create mypool/myroot
zfs create mypool/john

Both myroot and john will use the mypool storage space created by zpool. Set the mount points for these via

zfs set mountpoint=/space/john mypool/john
zfs set mountpoint=

A complete documentation of the zpool and zfs commands can be found at docs.sun.com [3][4].

ZFS supports quotas and reservations for each dataset. You can thus allocate varying amounts of space for various purposes. The built in checksumming of everything does away within the filesystem allows you to breathe easy while ZFS does everything that can be done to keep your data error free. ZFS also supports compression and encryption (currently under review by the ZFS community and to be added to the Opensolaris kernel shortly)

The Opensolaris technologies: Zones

(..and ZFS zones with Debian environment)

Zones are lightweight virtual instances of the operating system. The main (or parent) instance is known as the global zone. A global zone can have as many local zones as resources like memory and disk space allow for.

Zones are a boon to package maintainers. they allow for quick and easy setup and generation of development environment. Once the tasks are done, the zone can be done away with, until required later. Thus you do not have to allocate a specific amount of your resources towards a build setup.

Nexenta provides a set of tools in a package called devzone, which allow for easy creation of development zones for package maintainers. These can be easily tuned to other specific needs. A typical zone takes less than a minute to setup and use.

As each zone is self-contained, you can provide root access to them to those you may not fully trust. This setup can be used in build machines where “potential” developers to a project can be given access to their own development zone to play around with. This setup is used by the Nexenta Project during its hackathon events. further information at [links].

A zone can be created using

devzone_create

Now, enter the zone using

devzone_enter

and poof!.. you are now root in the new virtual instance of the machine. you would now typically setup your /etc/apt/sources.lst and start installing and building packages.

You can reconnect and disconnect the zone as many times as you like. Once you’re done with using a zone, simply run the following command in the global zone

devzone_free

to kill the zone, and free up it’s resources.

Devzones provide the following advantages to developers

  • Better share resources: As development zones can be created on the fly, you can easily share resources. Developers in different parts of the world can access the machine at different times.
  • Peace of mind: If you have a dedicated build machine, easily allow multiple developers access to it, without worrying about interfering with each other, or one’s mistake bringing down the system.

To see how you can build Debian/Ubuntu packages in a devzone, take a look at the screencast at [5].

Simplified service management using SMF

SMF, or Service Management Framework, is the Opensolaris replacement for init.d scripts. Services need to be run by a system, and SMF does this intelligently.

Traditionally, we’ve used /etc/init.d method of service management, where the services are started upon init depending on the init state. The scripts in the rc.d directory are run one after the other, and care must be taken to place the important core services are listed before the others that depend on it.

Like apt, SMF services are maintained as a tree of dependencies. The ssh service, for example, has network/physical as its dependency. There would be no use of running the the ssh service without the network services. Thus, any SMF service when started, first starts all its dependencies, and then itself. Starting/stopping a service is as simple as

svcadm enable ssh
svcadm disable ssh

This allows for very easy administration of services. In the above example, you dont have to worry about starting its dependencies first: SMF will take care of it. If one of the dependencies failed (eg: due to a disk failure), this is reported to the system via the Fault Management Architecture (FMA).

Another advantage of SMF is the speed of service startups upon bootup. Traditional /etc/init.d starts services one after the other depending on their numbering in the rc.d directory. But this is not efficient as a slow service wastes CPU time. SMF, with its knowledge of dependencies, will start 2 independent parts of the tree in parallel. This the CPU is optimally used, and leads to faster system bootup and services startup

More information on the Opensolaris SMF and FMA technologies, with comparison of equivalent /etc/init.f commands can be gleaned at the Sun documentation [6]

Dtrace: Probe the nervous system

Dtrace is another unique and powerful technology present in Nexenta. It provides a developer with a “read-only” interface into each and every function of the operating system. Set breakpoints (or probes) and run your script as and when they’re fired. And it is not just limited to the kernel. More and more packages are adding Dtrace probes, allowing you to easily process the function call paths in the application, and collecting various statistics about the running system/application.

For example, the AMP stack have probes enabled in them[7]. This allows you easily monitor your web services and applications, and tune them as necessary. Using tcpdump and snoop to look at the network traffic? Why not a line in Dtrace?[8]

dtrace -qn ‘ip:ip:*:receive{ printf(“Packet recieved from %s: %d byte packetn”, args[2]->ip_saddr, args[4]->ipv4_length ); }

Take a look at hundreds of other one-liners. Dtrace has a full fledged scripting language, which is similar to C. Many Dtrace resources are linked from here. Take a look at screencasts introducing Dtrace.

Example: Setting up a Webserver

If you have prior experience with setting up a webserver with Debian or Ubuntu, then you’ve already learnt setting one up on Nexenta.

First, download apache2, mysql and php

apt-get install apache2 mysql php5

Now setup the configuration files as required. they’ll be available at /etc/apache2/httpd.conf

Once done, restart apache to update the configuration changes using

apache2ctl restart

and your webserver is up and running. Content goes under /var/www

Example: Setting up a CIFS server

CIFS is a protocol that allows for request of files and services over the network. It is the default file sharing service in Windows, and has been implemented by projects like Samba.

Nexenta makes it very easy via the ZFS filesystem to setup a file server.

First let us enable the SMF service, and setup the domain in which the server belongs (let us assume the domain is win.nexenta.org)

#svcadm enable -r smb/server
#smbadm join -u Administrator win.nexenta.org

Now, lets setup the ZFS to export a particular dataset

#zfs set sharesmb=on data/myshare
#zfs set sharesmb=name=myshare data/myshare

You might also want to set the case sensitivity of files if accessing from other windows boxes.

#zfs set casesensitivity=mixed data/myshare

Restart the smb/server and we’re done.

#svcadm restart smb/server

For a complete step by step procedure take a look at the article here[9].

Give it a try!

Nexenta is available at www.nexenta.org. The developer community meets via IRC on #nexenta@freenode or the web interface at http://www.nexenta.org/chat. We’re currently porting packages from Ubuntu’s hardy repository to NexentaCore.

Join in here and request for a login to our build machines to test Nexenta live, and try out the devzone environments.

References

[0] http://www.nexenta.org/os/apt-clone
[1] http://blogs.sun.com/bonwick/entry/raid_z
[2] http://www.unixconsult.org/zfs_vs_lvm.html
[3] http://docs.sun.com/app/docs/doc/819-2240/zpool-1m
[4] http://docs.sun.com/app/docs/doc/819-2240/zfs-1m
[5] http://www.gulecha.org/2008/08/22/getting-started-with-nexenta-development-part-2/
[6] http://www.sun.com/bigadmin/content/selfheal/SMF-quickstart.jsp
[7] http://blogs.sun.com/angelo/entry/dtrace_meet_the_amp_apache
[8] http://www.cuddletech.com/blog/pivot/entry.php?id=952
[9] http://jmlittle.blogspot.com/2008/03/step-by-step-cifs-server-setup-with.html

About the author:
Anil Gulecha is Nexenta developer. He maintains a blog at gulecha.org. He tries to play a little guitar in his free time.

20 Comments

  1. 2008-09-11 4:45 pm
    • 2008-09-11 5:01 pm
      • 2008-09-11 11:10 pm
        • 2008-09-11 11:26 pm
          • 2008-09-12 9:45 am
    • 2008-09-11 5:37 pm
    • 2008-09-11 6:42 pm
      • 2008-09-11 11:03 pm
      • 2008-09-12 6:08 am
        • 2008-09-12 2:49 pm
          • 2008-09-13 9:00 am
        • 2008-09-12 9:20 pm
  2. 2008-09-11 5:47 pm
  3. 2008-09-11 7:35 pm
  4. 2008-09-12 12:07 pm
    • 2008-09-12 12:52 pm
  5. 2008-09-12 2:05 pm
  6. 2008-09-12 2:15 pm
  7. 2008-09-15 6:36 pm