zfs « WordPress.com Tag Feed

Encrypted ZFS

agolon — Thu, 26 Nov 2009 10:28:07 +0000

Instead of the lot of encrypting hack found browsing around the internet, I’ve found a real elegant solution here. It is not negligible, that this solution can be done very easy because of the nature of GEOM, the disk handling layer of FreeBSD.

If we want to take this serious, here is an other solution, where the whole thing is done with the root filesystem, and the needed boot record and keys are stored on a USB stick.

Here is a quick and dirty test on a virtual machine according to the tutorials mentioned. I’ve created two ZFS pools, one is based on a “normal” disk device, and another GEOM-ELI encrypted one (ad3.eli here).

freebsd# zpool status
  pool: crypted
 state: ONLINE
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        crypted     ONLINE       0     0     0
          ad3.eli   ONLINE       0     0     0

errors: No known data errors

  pool: plain
 state: ONLINE
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        plain       ONLINE       0     0     0
          ad1       ONLINE       0     0     0

errors: No known data errors

After a reboot the encrypted device isn’t initialized yet, that’s why the crypted pool doesn’t come up.

freebsd# zpool list
NAME                    SIZE    USED   AVAIL    CAP  HEALTH     ALTROOT
crypted                    -       -       -      -  FAULTED    -
plain                   123M    322K    123M     0%  ONLINE     -

In real life this is what an unauthorized person would see when trying to use the disk.

freebsd# zpool status crypted
  pool: crypted
 state: FAULTED
status: One or more devices could not be used because the label is missing
        or invalid.  There are insufficient replicas for the pool to continue
        functioning.
action: Destroy and re-create the pool from a backup source.
   see: http://www.sun.com/msg/ZFS-8000-5E
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        crypted     FAULTED      0     0     0  corrupted data
          ad3       UNAVAIL      0     0     0  corrupted data

Dekriptáljuk a kulccsal és a jelszóval, létrejön egy .eli device. Természetesen beállítható, hogy bootoláskor bekérje a jelszót.

freebsd# ls /dev/ad3*
/dev/ad3

freebsd# geli attach -k /root/AD3.KEY /dev/ad3
Enter passphrase:
GEOM_ELI: Device ad3.eli created.
GEOM_ELI: Encryption: AES-CBC 128
GEOM_ELI: Crypto: software

freebsd# ls /dev/ad3*
/dev/ad3        /dev/ad3.eli

The two device (which has basicly the same physical disk behind) contains different data. (I’m not sure that is safe enouugh, but we should trust the developers.)

freebsd# od -N 64 /dev/ad3
0000000    013656  063203  126627  124130  123623  172041  031633  017602
0000020    031074  161743  066227  176053  134323  043301  014373  107567
0000040    115637  060745  175115  106175  044662  041262  044142  065625
0000060    021002  160304  045367  047763  065401  063457  026315  073572
0000100
freebsd# od -N 64 /dev/ad3.eli
0000000    050742  113732  071701  074476  135444  055035  170031  155754
0000020    150122  143035  154627  104534  121114  101434  167634  110345
0000040    013172  113341  001635  133547  022354  035751  072373  144310
0000060    122771  156712  175330  033173  164006  177125  061412  166472
0000100

Let’s see if the pool is alive! If yes we can use it.

freebsd# zpool list
NAME                    SIZE    USED   AVAIL    CAP  HEALTH     ALTROOT
crypted                 123M   86.7M   36.3M    70%  ONLINE     -
plain                   123M    322K    123M     0%  ONLINE     -

freebsd#  zpool status crypted
  pool: crypted
 state: ONLINE
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        crypted     ONLINE       0     0     0
          ad3.eli   ONLINE       0     0     0

errors: No known data errors

So we are happy now, let’s take some test. We are using dd to put some random data to the pools. The bad IO speed is because the test is running on VirtualBox over a poor old portable. Actually we do not care the speed, but the comparison of the speed of the pools.

Writing speed on unencrypted device:

freebsd# dd if=/dev/random of=/plain/testfile bs=1M count=85 & zpool iostat plain 1 100
[1] 1030
               capacity     operations    bandwidth
pool         used  avail   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
plain        451K   123M      0      3  1.66K   180K
plain        451K   123M      0     73      0  9.21M
plain       15.4M   108M      0    132      0  10.5M
plain       27.9M  95.1M      0    143      0  12.3M
plain       38.4M  84.6M      0    134      0  11.6M
plain       54.4M  68.6M      0    183      0  13.3M
plain       65.4M  57.6M      0    159      0  9.28M
plain       76.0M  47.0M      0    232  63.8K  13.4M
plain       83.5M  39.5M      0    251      0  7.42M
85+0 records in
85+0 records out
89128960 bytes transferred in 8.286649 secs (10755730 bytes/sec)
^C
[1]  + Done                          dd if=/dev/random of=/plain/testfile bs=1M count=85

The same on the encrypted device:

freebsd# dd if=/dev/random of=/crypted/testfile bs=1M count=85 & zpool iostat crypted 1 100
[1] 1034
               capacity     operations    bandwidth
pool         used  avail   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
crypted     1.72M   121M      0      0  2.73K  51.0K
crypted     1.72M   121M      1     43   125K  5.48M
crypted     16.7M   106M      0    114  63.5K  9.76M
crypted     16.7M   106M      0     71  63.9K  8.87M
crypted     29.1M  93.9M      0     86  62.0K  5.91M
crypted     39.4M  83.6M      0    105      0  8.85M
crypted     48.0M  75.0M      0    129  63.9K  9.29M
crypted     48.0M  75.0M      0     45      0  5.62M
crypted     55.1M  67.9M      0     78      0  4.30M
crypted     66.0M  57.0M      0    157      0  9.49M
crypted     70.1M  52.9M      0     95      0  4.60M
crypted     76.5M  46.5M      0    151  63.9K  7.53M
crypted     80.8M  42.2M      0    128      0  5.51M
crypted     83.8M  39.2M      0    204  63.2K  5.71M
85+0 records in
85+0 records out
89128960 bytes transferred in 13.812837 secs (6452618 bytes/sec)
^C
[1]  + Done                          dd if=/dev/random of=/crypted/testfile bs=1M count=85

Of course this test above can not simulate a real life environment, I was only curious, how much resource does need the GELI layer in the middle. So I can not decide from this test wheter this values are tolerable or not. We were not getting clever, we know what we already knew, the encryption slows down the system, and needs more resources.

The benefit of the solution is the transparency. The created .eli device can be used for any filesystem which is supported by the operating system itself. Without the key and the password, the data on the disk is unusable.

On security side, I like the key/pass system. We can put the key on a safe place, and if somehow it is stolen, a password is still there to protect or privacy.

Insights on Discharge

iccink — Wed, 25 Nov 2009 00:16:07 +0000

Ed Branigan

Another topic we get asked on a regular basis is that concerning discharge. Many printers would like to use it, but have either heard that it is smelly, or complicated to use. Here is Ed Branigan with some insights on discharge:

“Discharge printing is enjoying a resurgence thanks to the trend towards soft hand vintage style t-shirts that’s going on right now. People are still spooked by the discharge process though and the choice to use it seems always to be taken warily. They imagine thick vapors of formaldehyde floating in clouds around their print shops while everyone has to wear a space suit just to get the job done. But things have changed in recent years as far as discharge printing systems go and for the better.

The technology which applies to discharge printing has been around for over 100 years. Discharge, also known as extraction printing because the result of the application is the extraction of the dye from the fabric, was not applicable to garments at first because printed garments needed to be steamed during the curing process which didn’t make it a very attractive option. The development of chemcially reactive discharge made it much easier to use for t-shirt printers because the garment could now be cured in an oven. That still left the formaldehyde.

Discharge has had a reputation for its smell and many printers will offer this by way of an explanation for being wary of using it: “I hear that it has a really bad smell” or “I used to use it but stopped because of the smell “. The short answer is that it does have a smell but thanks to newer developments in discharge ink technology, this smell has been greatly reduced.

The cause of the smell is Zinc Formaldehyde Sulfate (ZFS). ZSF is the discharge agent and is typically added to a clear discharge base at about 5%-6% by weight.

The activated base is printed just like a regular ink. During the curing process the dye in the shirt covered by the printed area is discharged or extracted returning it back to its original cotton color. It is also during the curing stage that formaldehye is released and where the smell comes from. As I mentioned already though, there have been improvements. The formaldehyde content has been greatly reduced and much less is gassed off than used to be. Thanks to the Clean Air Act and other legislation passed in the US in recent decades, stricter controls have been placed on VOC’s (Volatile Organic Compounds) being released into the air from screen printing inks and this has spurred technological advances.

Discharge is no longer the demon that it used to be. While it still requires care with handling and printing, it is by far way easier to manage than previous generations.

While both plastisol and waterbased discharge systems available on the market today the waterbased are more popular.

Waterbased discharge can be tinted using non corrosive pigments or inks to make color. The pigment or ink should be added to the base first and mixed before adding the ZFS. Once mixed it has a shorter shelf life, so be sure to only mix what you need and to follow the instructions.

One last thing to mention: Not every shirt is dischargeable. It is important to test the garment or fabric first. Garment fabric dyes are fickle and don’t always behave themselves. Do a test print. If the discharge doesn’t work, don’t fight it. It’s not going to and you can be sure that every shirt that was dyed in that particular lot won’t work either. So when ordering shirts make sure to ask for dischargeable ones.”

FYI, International Coatings carries a waterbased discharge system containing of the 3707 Discharge Base and the 3709 Discharge Activator. Our GEN IV waterbased ink series can also be used to tint the waterbased discharge. The above-mentioned products can be purchased through our distributor network. To find International Coatings distributors in your area, you can go to our distributor locator page here.

Discharge on black shirt

Tinted discharge

Special Discharge Effect

Snapshots en UFS de FreeBSD

ignacio — Wed, 18 Nov 2009 02:38:27 +0000

Hace unos días descubrí los snapshots para filesystems UFS(2) de FreeBSD (lo se, lo se, soy un desastre) y si bien se implementa también para ZFS le di caña a los snapshots para UFS (default de FreeBSD).
Para probarlo, lo que hice fue crear y montar una imagen en un archivo (dd, mkfs, mdconfig, etc…) quedando algo así:

/home/ignacio $ mount | grep md0
/dev/md0 on /usr/home/ignacio/md0 (ufs, local)

Lo primero que hay que hacer, es notar la existencia del directorio .snap dentro de la recién montada imagen. Es en la raíz de cada partición UFS es donde está el directorio .snap.
Yo lo que hice fue instalar el frontend snapshot(8) mediante su port sysutils/freebsd-snapshot.

En la página citada, donde Ralf S. Engelschall explica la motivación y la forma de uso la verdad que está muy buena y simple.
Básicamente el comando snapshot, consta de cuatro acciones básicas: list, make, mount y umount.

Con snapshot list podemos ver la lista completa de los snapshots creados para alguna partición, en este caso, aún no hemos creado ninguno por lo que el listado, estará vacío:

/home/ignacio $ snapshot list /home/ignacio/md0
Filesystem          User   User%     Snap   Snap%  Snapshot

Vemos el listado actual del directorio actual:

/home/ignacio $ cd md0
/home/ignacio/md0 $ ls -l
drwxrwxr-x  2 root  operator      512 Nov 17 14:06 .snap
-rw-r--r--  1 root  wheel          12 Nov 10 16:23 ignacio
-rw-r--r--  1 root  wheel     1048576 Nov 10 17:37 test_file.txt

y creamos nuestro primer snapshot:

/home/ignacio/md0 $ snapshot make -g4 /home/ignacio/md0:snaps

Donde, -g4 indica que guardaremos 4 generaciones (cuatro vistas al pasado), /home/ignacio/md0 es (donde está montada) la partición y snaps es un tag que le asignamos.

Tiremos ahora un nuevo list:

/home/ignacio $ snapshot list /home/ignacio/md0
Filesystem          User   User%     Snap   Snap%  Snapshot
/home/ignacio/md0      2MB    2.9%    224KB    0.2%  snaps.0

Para jugar un poco borramos el archivo test_file.txt por error y ahora, para salvar el día, recuperamos el snapshot que creamos:

/home/ignacio/md0 $ ls -l
total 4
drwxrwxr-x  2 root  operator  512 Nov 17 14:09 .snap
-rw-r--r--  1 root  wheel      12 Nov 10 16:23 ignacio
/home/ignacio/md0 $

Para recuperar el snapshot debemos montarlo indicando que generación queremos recuperar, en nuestro caso, la 0:

/home/ignacio $ snapshot mount /home/ignacio/md0:snaps.0 /mnt
WARNING: opening backing store: /usr/home/ignacio/md0/.snap/snaps.0 readonly

Buscamos en el /mnt los archivos:

/home/ignacio $ cd /mnt
/mnt $ ls
.snap           ignacio         test_file.txt
/mnt $

Devolvemos test_file.txt a la normalidad:

/mnt $ cp -v test_file.txt /home/ignacio/md0
test_file.txt -> /home/ignacio/md0/test_file.txt
/ $ snapshot umount /mnt
/ $ cd /home/ignacio/md0
/home/ignacio/md0 $ ls -l
total 1044
drwxrwxr-x  2 root  operator      512 Nov 17 14:09 .snap
-rw-r--r--  1 root  wheel          12 Nov 10 16:23 ignacio
-rw-r--r--  1 root  wheel     1048576 Nov 17 14:14 test_file.txt

Con lo de arriba, copiamos el archivo que habiamos perdido por error, desmontamos el snapshot y verificamos como quedó nuestra partición.

En fin, terrible herramienta! Es recomendado revisar además la man page periodic-snapshot(8) para hacer snapshots periodicos usando el cron y periodic.

Estoy comenzando a ser un usuario feliz de este tipo de filesystems. Claro que ya tengo configurado el cron para que genere los snapshots de manera automatizada para todas las particiones de mi disco y también espero jamás en mi vida necesitarlos.

סרטון נהדר להצגת המערכת.

cloudvc — Mon, 16 Nov 2009 15:11:13 +0000

HowTo: Mirror the ZFS rpool

Martin — Sun, 15 Nov 2009 12:00:06 +0000

Let’s use the shiny ZFS to have a mirrored pool which we can use to boot in case one of the hard disks fails. This involves several steps :

Install a second disk in your box
format the disk so that we can actually use it
make it the same layout as the disk we already run on
attach it to the pool in question

Now point (3) and (4) need some explanation since I’m experimenting with the root pool (nothing better than a little thrill not being able to boot if things go wrong) and my computer uses the good old BIOS it cannot boot EFI labeled disks. For this reason I have to use slices, I’ll probably rip out some old storage box in the next weeks and see how much of this process is left when using whole disks.

I assume you are indeed capable of physically installing a second, third or n^th disk in your computer so let’s jump to point (2).

Formatting the disk(s)

Simply use “format”

$ pfexec format
Searching for disks...done

AVAILABLE DISK SELECTIONS:
 0. c8t0d0 <DEFAULT cyl 30398 alt 2 hd 255 sec 63>
 /pci@0,0/pci103c,3035@1f,2/disk@0,0
 1. c8t2d0 <ATA-ST3250310AS-C cyl 30398 alt 2 hd 255 sec 63>
 /pci@0,0/pci103c,3035@1f,2/disk@2,0
Specify disk (enter its number): 1
selecting c8t2d0
[disk formatted]

FORMAT MENU:
 disk       - select a disk
 type       - select (define) a disk type
 partition  - select (define) a partition table
 current    - describe the current disk
 format     - format and analyze the disk
 fdisk      - run the fdisk program
 repair     - repair a defective sector
 label      - write label to the disk
 analyze    - surface analysis
 defect     - defect list management
 backup     - search for backup labels
 verify     - read and display labels
 save       - save new disk/partition definitions
 inquiry    - show vendor, product and revision
 volname    - set 8-character volume name
 !<cmd>     - execute <cmd>, then return
 quit

No go “fdisk” it and delete all the partitions on the disk. Then create a new partition, make it a Solaris partition and be done with that part.

format> fdisk
Total disk size is 30401 cylinders
 Cylinder size is 16065 (512 byte) blocks

 Cylinders
 Partition   Status    Type          Start   End   Length    %
 =========   ======    ============  =====   ===   ======   ===
 1       Active    Solaris2          1  30400    30400    100

SELECT ONE OF THE FOLLOWING:
 1. Create a partition
 2. Specify the active partition
 3. Delete a partition
 4. Change between Solaris and Solaris2 Partition IDs
 5. Exit (update disk configuration and exit)
 6. Cancel (exit without updating disk configuration)
Enter Selection: 3
Specify the partition number to delete (or enter 0 to exit): 1
Are you sure you want to delete partition 1? This will make all files and
programs in this partition inaccessible (type "y" or "n").
y
Partition 1 has been deleted. This was the active partition.
Enter Selection: 1
Select the partition type to create:
 1=SOLARIS2  2=UNIX        3=PCIXOS     4=Other
 5=DOS12     6=DOS16       7=DOSEXT     8=DOSBIG
 9=DOS16LBA  A=x86 Boot    B=Diagnostic C=FAT32
 D=FAT32LBA  E=DOSEXTLBA   F=EFI        0=Exit? 1
Specify the percentage of disk to use for this partition
(or type "c" to specify the size in cylinders). 100
Should this become the active partition? If yes, it  will be activated
each time the computer is reset or turned on.
Please type "y" or "n". y
Partition 1 is now the active partition.
Enter Selection: 5

Duplicate the disk layout on both disks

Now OpenSolaris provides us with 2 nice tools to make one disk have exactly the same layout as another disk. Use them like below:

$ pfexec prtvtoc /dev/rdsk/c8t0d0s2 # first disk
$ pfexec prtvtoc /dev/rdsk/c8t2d0s2 # second disk
$ pfexec prtvtoc /dev/rdsk/c8t0d0s2 | pfexec fmthard -s - /dev/rdsk/c8t2d0s2 # make the 1st disk the same as the 2nd

Attach it to the pool in question

Let’s have a quick look at the existing pool:

$ zpool status
 pool: rpool
 state: ONLINE
 scrub: scrub completed after 0h26m with 0 errors on Wed Nov 25 10:52:26 2009
config:

 NAME        STATE     READ WRITE CKSUM
 rpool       ONLINE       0     0     0
 c8t0d0s0    ONLINE       0     0     0

errors: No known data errors

Nice we now see what our original device name is and what our pool name is, we need both for the next command. Remember in my example the new disk slice (the BIOS vs. EFI thing) is c8t2d0s0. We should be all fine to attach the slice to the pool and see it start recovering.

$ pfexec zpool attach rpool c8t0d0s0 c8t2d0s0
invalid vdev specification
use '-f' to override the following errors:
/dev/dsk/c8t2d0s0 overlaps with /dev/dsk/c8t2d0s2

Ouch, didn’t quite work, so let’s do as instructed, and right thereafter let’s investigate the staus of the pool:

$ pfexec zpool attach -f rpool c8t0d0s0 c8t2d0s0
Please be sure to invoke installgrub(1M) to make 'c8t2d0s0' bootable.
$ zpool status
 pool: rpool
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
 continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
 scrub: resilver in progress for 0h0m, 0.18% done, 3h28m to go
config:

 NAME          STATE     READ WRITE CKSUM
 rpool         ONLINE       0     0     0
 mirror        ONLINE       0     0     0
 c8t0d0s0      ONLINE       0     0     0
 c8t2d0s0      ONLINE       0     0     0  113M resilvered

errors: No known data errors

Looks all nice, now let’s do the grub stuff so that we can boot from both disks:

$ pfexec installgrub /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c8t2d0s0
stage1 written to partition 0 sector 0 (abs 16065)
stage2 written to partition 0, 271 sectors starting at 50 (abs 16115)

A last check on the status of our pool:

$ zpool status
 pool: rpool
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
 continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
 scrub: resilver in progress for 0h2m, 4.89% done, 0h53m to go
config:

 NAME          STATE     READ WRITE CKSUM
 rpool         ONLINE       0     0     0
 mirror        ONLINE       0     0     0
 c8t0d0s0      ONLINE       0     0     0
 c8t2d0s0      ONLINE       0     0     0  2.95G resilvered

errors: No known data errors

To my untrained OpenSolaris eye this looks all nice and shiny now.

Sun Adds De-Duplication to ZFS

solori — Tue, 03 Nov 2009 22:24:43 +0000

Yesterday Jeff Bonwick (Sun) announced that deduplication is now officially part of ZFS – Sun

Multiple Boot Environments

Wendy — Mon, 02 Nov 2009 20:47:06 +0000

One of features I really like on OpenSolaris is cloning your system. Especially when doing a system upgrade. There is nothing better than feeling you can safely upgrade your up-and-running fully configured system to newer version and when something went wrong then simply reboot to previous one. Wicked!

This feature is called Boot Environment (BE). You can have as many BE as you wish. To display all your environments, type beadm list:

wendy@honey:~$ beadm list BE Active Mountpoint Space Policy Created -- ------ ---------- ----- ------ ------- opensolaris - - 4.10G static 2009-02-18 08:35 devel - - 5.45G static 2009-04-24 15:01 opensolaris-2 - - 6.84G static 2009-09-01 23:16 opensolaris-3 NR / 9.02G static 2009-11-02 19:01

For my destructive tasks I use separated BE so it is almost sate to do anything… Almost…

Any way. If you want play with your system and don’t be afraid of changes you will be doing, create new BE, activate it and reboot.

pfexec beadm create testing pfexec beadm activate testing pfexec init 6

If something bad happens, boot some old BE, activate other BE and reboot. Simply shit happens.

For more information see man pages or on-line manual at http://dlc.sun.com/osol/docs/content/IPS/ggfxy.html

PS: Do you have same story how BE saved your ass? Please, share it in comments.

Testing out FreeBSD 8.0-RC2

koitsu — Mon, 02 Nov 2009 18:34:14 +0000

Those who haven’t read about my 8.0-RC1 experience should do so first:

Testing out FreeBSD 8.0-RC1

Basically, my experience with 8.0-RC2 was identical to that of RC1, except some of the bugs/issues I experienced are now gone (hooray!).

Fixes/improvements:

The issue I experienced with the Boot Manager selection phase of installation has been fixed. Also, Standard is now the default option (first choice).
The geometry does not match label problem has been addressed by fixing the FreeBSD slice editor in sysinstall/sade; see below.
libdisk has been modified to work properly with GEOM; the FreeBSD slice editor links to libdisk — thanks to Randi Harper for tracking this commit down! Commonly when installing FreeBSD on a box, people go into the slice editor and press “a” to use the entire disk. Previously, users would end up with a disk where the first 63 sectors were unused (probably for the PBR/MBR and overall alignment), then the FreeBSD slice, and a third “unused” portion of the disk (which, if I remember correctly, was done solely for alignment reasons). Example:
```
Offset       Size(ST)        End     Name  PType       Desc  Subtype    Flags

         0         63         62        -     12     unused        0
        63  390716802  390716864    ad8s1      8    freebsd      165    A
 390716865       5103  390721967        -     12     unused        0
```
Starting with RC2, this is what you’ll see:
```
Offset       Size(ST)        End     Name  PType       Desc  Subtype    Flags

         0         63         62        -     12     unused        0
        63  390721905  390721967    ad8s1      8    freebsd      165    A
```
Note the lack of the last “unused” section.

Sadly, this also means people will need to reinstall FreeBSD (specifically, deleting the slice and re-creating it) to benefit from this. As far as I know, you can’t fix this without a full reinstallation.
The EOF issue for ttys (re: ^D being shown) has been fixed and committed to CURRENT (FreeBSD 9.0), but hasn’t been MFC’d to RELENG_8 yet. Yes, it’s scheduled to be (in about 2 weeks). Big thanks to Ed Schouten for fixing this!
There were some ZFS commits which happened between RC1 and RC2 which may indicate that the ARC exhausting all available kmem is no longer possible. I have not been able to confirm/deny whether this fix works, but looking at the code, it may be sufficient. I’d need to get in touch with Kip Macy to confirm/deny.

Issues that are still pending:

bsdlabel still behaves incorrectly (“Class not found”). Instead, users should use gpart to write the bootstraps as follows: gpart bootcode [disk], where [disk] is ad4 or similar. Note that you pick the disk itself now, not the slice like in bsdlabel (unless you were using dangerously dedicated disks ).
The ZFS notice pertaining to vfs.zfs.prefetch_disable when the system has less than 4GB RAM available has been re-worded again, but still is vague/unclear. A little bit of ego here — the person committing these changes should really consider changing the message to what I proposed.
I still haven’t received a reply to my request for clarification on ZFS stabilisation. Is /boot/loader.conf tuning for kmem-related parameters still required? We still need an official statement on this matter.

I also want to take a moment to send a shout-out to John Baldwin, who has been working incredibly hard on the FreeBSD kernel (specifically VM and ACPI) over the past 4 weeks. John, I’ve seen/followed your commits, and I appreciate the improvements! Thank you!

UNIX mail format annoyances

koitsu — Thu, 29 Oct 2009 16:09:39 +0000

For many years now I’ve been dealing with an ongoing issue which still to this day has no real solution: classic UNIX mailboxes (called mbox) comparing the files’ mtime to its atime to determine if there’s new mail inside of the mailbox (if the mtime is greater than the atime, there’s new mail. If the mtime is smaller than the atime, new mail has been read/there is no new mail). “Classic mail spools” (e.g. /var/mail or /var/spool/mail) are mbox.

Why is this a problem? Because those of us who use mutt/alpine/etc. on our UNIX machines, who also do backups using things like tar/cp/rsync (more on rsync in a moment) end up with mailboxes with a lost/clobbered atime after the backup takes place. The end result: our mail clients no longer tell us there’s new mail in that mailbox, which can be detrimental in many respects.

The most common rebuttal is “shut up and use Maildir“. What Maildir advocates don’t care to acknowledge is that there are many problems with the Maildir concept, particularly when used on a filesystem like ZFS. With classic mbox, your multi-megabyte mailboxes loads quickly — but with Maildir, since it uses a single file per mail, the end result is a mail client that takes forever to load due to the one-file-per-mail concept. ZFS does not perform well when it comes to massive numbers of small/terse files.

UFS/UFS2, ext2fs/ext3fs, and other filesystems don’t have this problem, but let’s pull our heads out for a moment (since tunnel vision/ostrich syndrome is what got us here in the first place!) — we’re entering year 2010 and ZFS is already being used heavily by Solaris/OpenSolaris and FreeBSD users across the globe; ZFS is here to stay, end of discussion. There are some proposed solutions such as making use of ZFS’s semi-new L2ARC to add an additional layer of caching using dedicated low-latency devices (specifically SSDs), but there’s been no actual evidence this improves things with Maildir. And besides, who in their right mind is going to go out and drop hundreds of dollars on an Intel X25-M per machine just to solve this problem? Seriously.

And let’s not forget administrators who mount their filesystems with the noatime mount flag for added performance benefits, especially on a journalled filesystem.

One workaround proposed for mutt users involves recompiling mutt to use Oracle/SleepyCat DB, GDBM, or Tokyo Cabinet to maintain a cache of mailbox headers (using the header_cache directive), thus speeding up the process. Does this help? Yes, there’s a decent improvement, but anyone who uses this method (such as me) can tell you that it’s still no where near as fast as classic mbox, especially when you’ve got a mailbox with a couple hundred new mails in it.

Does the saga end here? Not even close.

There’s a new mailbox format, called MIX, which is being used within alpine. This format is more or less a combination of mbox and Maildir, and performs much better than Maildir. Sounds great, right? Except those of us who use mutt are out of luck — unsupported, and there’s been absolutely no discussion of it since February 2007. Even the author of mutt, Michael Elkins, had nothing useful to say other than snide comments. Oh, and MIX isn’t supported in procmail or Sieve either — double whammy. But MIX does sound like the way to go — too bad it isn’t getting the attention it should.

Some administrators using ZFS are using ZFS snapshots to do their backups instead of something like rsync, which is great except that they’re hit-or-miss (reliability-wise) on FreeBSD — or at least that’s what I last read 6-9 months ago — while rsync is filesystem-independent. Most folks I know who run into snapshot problems revert back to rsync.

So what now? With all the above in mind,I decided to poke at rsync, because there’s been many discussions in the past on the mailing lists about getting rsync to preserve file atime. rsync out-of-the-box will preserve ctime and mtime when using the --times flag. However, there’s a patch called atimes.diff which comes with the rsync-patches tarball that provides a --atimes flag that supposedly solves this. Sounds great… except there’s one problem…

The flag does cause the atimes of the source file to be copied to the destination, but the atimes of the source file are lost! And here’s a more recent confirmation.

If that’s not enough, here’s final confirmation. Note that I’m using non-zero-byte files intentionally; rsync behaves differently when the files are zero bytes.

rsync -a:

$ echo "hello" > source
$ stat -x source
Access: Wed Oct 28 06:27:05 2009
Modify: Wed Oct 28 06:27:05 2009
Change: Wed Oct 28 06:27:05 2009
$ rsync -a source dest
$ stat -x source
Access: Wed Oct 28 06:27:29 2009
Modify: Wed Oct 28 06:27:05 2009
Change: Wed Oct 28 06:27:05 2009
$ stat -x dest
Access: Wed Oct 28 06:27:29 2009
Modify: Wed Oct 28 06:27:05 2009
Change: Wed Oct 28 06:27:29 2009
$ rm source dest

Above, we see that after the rsync, the atime in the source file is lost, and the ctime in the destination file does not match that of the source — only the mtime is retained.

rsync -a --atimes:

$ echo "hello" > source
$ stat -x source
Access: Wed Oct 28 06:32:50 2009
Modify: Wed Oct 28 06:32:50 2009
Change: Wed Oct 28 06:32:50 2009
$ rsync -a --atimes source dest
$ stat -x source
Access: Wed Oct 28 06:34:06 2009
Modify: Wed Oct 28 06:32:50 2009
Change: Wed Oct 28 06:32:50 2009
$ stat -x dest
Access: Wed Oct 28 06:32:50 2009
Modify: Wed Oct 28 06:32:50 2009
Change: Wed Oct 28 06:34:06 2009

Above, we see the atime and the mtime in the source file is retained in the destination. However, again, the atime in the source file is lost and the ctime doesn’t match that of the source.

cp -p:

$ echo "hello" > source
$ stat -x source
Access: Wed Oct 28 06:37:56 2009
Modify: Wed Oct 28 06:37:56 2009
Change: Wed Oct 28 06:37:56 2009
$ cp -p source dest
$ stat -x source
Access: Wed Oct 28 06:38:27 2009
Modify: Wed Oct 28 06:37:56 2009
Change: Wed Oct 28 06:37:56 2009
$ stat -x dest
Access: Wed Oct 28 06:37:56 2009
Modify: Wed Oct 28 06:37:56 2009
Change: Wed Oct 28 06:38:27 2009

With cp -p, we see identical behaviour to that of rsync -a --atimes.

Some may be wondering: “is it even possible to solve this problem?” Of course it is. The logic flow should be pretty obvious at this point:

stat(2) or fstat(3) the source file and save (in memory) the atime, mtime, and ctime. Neither call modifies the atime
Copy the source file to the destination file
Set the atime, mtime, and ctime of the destination file using utimes(3) with the previously-obtained values
Set the atime and mtime of the source file using utimes(3) with the previously-obtained values

You can accomplish the same thing with touch.

And let’s not forget that FreeBSD lacks the O_NOATIME GNU extension for open(2), which was proposed in 1998.

So is there a solution to all of this? As far as I’ve been able to tell, no, there isn’t. Using filesystem-level snapshots appears to be the only way to “solve” this issue. I’d be much happier if the --atimes patch for rsync did what it was supposed to… but it’s 23KB, and I’m not familiar with the rsync code (it’s not as black-and-white as one may think).

We UNIX folks should be ashamed of this whole debacle. There isn’t a better way to say it: what a clusterfuck.

Mac port of ZFS discontinued | MacWorld's view on the issue

Bruce Kraus — Tue, 27 Oct 2009 15:05:46 +0000

ZFS - The Last Word in File Systems?

MacWorld has decided to wade through the murky waters of Apple’s discontinuation of porting Sun Microsystems ZFS to Mac OS X Snow Leopard. Read more here:

Mac port of ZFS discontinued

Posted using ShareThis

Kecewa, Apple menutup project open source ZFS

luthfi.emka — Tue, 27 Oct 2009 05:55:05 +0000

dotexe.wordpress.com – Apa yang sebenarnya ada dibenak manajemen Apple Computer, mereka secara resmi melakukan konferensi pers dan mengumumkan bahwa project file sistem ZFS untuk Mac OS yang dilepas untuk komunitas open source secara resmi ditutup. Menurut announcement dengan enteng tercantum “Project ZFS secara resmi dihentikan. Mailing list dan repository akan segera dihapus”.

Memang ada beberapa alasan yang mungkin membuat Apple untuk mengakhiri project ini. Satu alasan yang cukup besar adalah isu tentang hak paten. ZFS sendiri telah digugat oleh NetApp namun bukan kepada Apple tapi kepada Sun. Alasan lain yang mungkin adalah masalah akuisisi Sun oleh Oracle yang sedikit banyak akan mempengaruhi kerjasama Apple dengan Sun terkait ZFS dan btrfs yang dimiliki dan dibesarkan oleh Oracle.

Kemungkinan lain adalah ketidakcocokan budaya ZFS dengan budaya pengguna Mac yang kebanyakan setia menggunakan satu harddisk dan bukan banyak device seperti keunggulan ZFS.

Semoga penutupan ini tidak dilanjutkan dengan penghentian banyak project open source yang telah besar dari Apple seperti Darwin dan ZFS, sehingga open source tak terlalu jauh dengan Mac.

Apple abandona ZFS en OSX por licencias

Sr. Luzbel — Tue, 27 Oct 2009 04:05:42 +0000

El sistema de archivos de Sun, según ellos “la última palabra en sistemas de archivos”,

Goodbye ZFS, we barely knew ya..

YK — Mon, 26 Oct 2009 03:17:15 +0000

ZFS was one of the new ‘features’ of Snow Leopard (Server), even though Snow Leopard is supposed to have 0 new features. We all know that 0 new features was an epic marketing lie, but that is for another day.

ZFS is a new file system by Sun, and I guess it was meant to replace the aging HFS+. ZFS has a lot of useful features, like pooling, where all your drives can be pooled into a single, logical volume. This means you can have 10 hard drives, with varying sizes and RAID configurations, and you see it as a single drive on your Mac. It will dynamically balance the usage of the drives within the pool, meaning if you have a SSD hard drive with a RAID-Z volume in the pool, it will intelligently use the SSD for caching (your data will be written there first, because it is faster) and after which the data will be moved into the RAID-Z volume for redundancy.

RAID-Z is also a new form of RAID, only for ZFS. It is Sun’s version of RAID-5, which solves the problem of data corruption thanks to ZFS’s copy-on-write feature.

Apple also promised better performance with ZFS on Snow Leopard, like data integrity and redundancy, and other nifty features I can’t find references to right now.

These features are all but gone as on Friday, Apple posted this message on Mac OS Forge:

The ZFS project has been discontinued. The mailing list and repository will also be removed shortly.

This obviously means ZFS will never see the light of day on OS X for the foreseeable future.

The real reason behind this was probably a licencing issue:

Apple can currently just take the ZFS CDDL code and incorporate it (like they did with DTrace), but it may be that they wanted a “private license” from Sun (with appropriate technical support and indemnification), and the two entities couldn’t come to mutually agreeable terms. -Jeff Bonwick, ZFS Lead Developer, Sun

This saddens me as ZFS had a huge potential to replace HFS+, an aging file system format used since Mac OS 8.1. Apple had to hack features into HFS+ for Snow Leopard, like file system compression, when ZFS already had the feature cooked straight in with no performance penalties.

Apple better has something up their sleeves. I don’t expect to be still on HFS+ when OS XI drops in the future, though I won’t be surprised.

If you still want some ZFS goodness, do not fret, as there are other OSes that use ZFS, like FreeNAS. Not so much so for desktop usage, but more for your server needs, like a media server.

The loss of ZFS

chimac — Sun, 25 Oct 2009 17:31:55 +0000

This is from a programmers point of view. Gets into nice details about why this might not have been a good fit for Apples needs. Click here to read more.

The Loss of ZFS

Bruce Kraus — Sun, 25 Oct 2009 15:56:59 +0000

ZFS - The Last Word in File Systems?

Louis Gerbarg has posted an excellent summation of the Apple & Sun ZFS Controversy. Read more here:

The loss of ZFS.

Apple Halts ZFS Project?

Bruce Kraus — Sat, 24 Oct 2009 03:41:29 +0000

Based on a post on slashdot.org,

“… Apple has replaced its ZFS project page with a notice that ‘The ZFS project has been discontinued. The mailing list and repository will also be removed shortly.’ Apple originally touted ZFS as a feature that would be available in Snow Leopard Server. A few months before release, all mention of ZFS was removed from the Apple web site and literature, and ZFS was notably absent from Snow Leopard Server at launch…”

Is the highly touted 128-bit ZFS file system, originally developed by Sun Microsystems, now defunct in the Apple camp? Read more here.

הצגת הארכטיקטורה - Sun VDI3.1

cloudvc — Thu, 22 Oct 2009 19:58:19 +0000

Sun VDI 3.1

View more presentations from mprove.

Sun VDI 3.1 - חידושים ועדכונים

cloudvc — Thu, 22 Oct 2009 17:07:05 +0000

אני שמח להודיעכם על שחרור גרסת הניסויי לVDI 3.1 מבית סאן מיקרוסיסטמס. גרסה זו כבר הותקנה בארץ וניתן לקבוע הדגמות. כיודע לכם עוד מגרסת ה-VDI הקודמת, מדובר בסל מוצרים אשר מאוגדים תחת ממשק ניהול אחיד, רק שהפעם בנוסף לשינויים המינורים ותיקוני הבאגים, המוצר קיבל את כל החידושים האפשריים, כדוגמת – SUNRAY 5, SSGD 4.5, VBOX, OVF, 3D GPU, vCPU, וכד’.

תחילה ברצוני לציין שמדובר בגרסת ביטא הכי מוצלחת מאי פעם שיצא לי לעבוד איתה. חלו מספר שינויים בתפיסה הכוללת של המוצר ובעיקר מרבית הדגשים הושמו בצד הניהולי של המוצר. שזה כולל גם שליטה ובקרה טובה יותר הן בצד תחנות וירטואליות והן בצד הטיפול ותחזוקת התבניות (templates).

ההפתעה הגדולה ביותר של VDI3.1 הינה התממשקות ותמיכה דווקא במוצרי מייקרוסופט. למשל, בפנל הניהול ישנו מודול שלם אשר מטפל בכל הפרמטרים של פרוטוקל ה-RDP. ואפילו תמיכה מלאה ב – SYSPREP של מייקרוספט.

במודול ה-HOST PROVIDERS נוספו שני מודולים חדשים לתמיכה ב HYPER-V ו TS2008 של מייקרוסופט ובכך יתמוך המוצר בסביבות הוירטואליזציה הבאות:

המשמעות: המוצר SUN VDI3.1 בגרסתו החדשה משמש כשכבת ניהול לכל סוגי שכבות הוירטואלזציה בעת ובעונה אחת ככלי ניהול אחיד ואפקטיבי אשר מחבר בין שכבת תחנות הקצה (UTC, TC, PC, PDA) לספק המשאבים הוירטואלים (HOST PROVIDER ) ללא תלות בייצרן.

הפתעה נוספת, למי שבוחר לעבוד בוירטואליזציה החינמית של סאן (VBOX), גרסת הVBOX בVDI3.1 תומכת OVF ובכך תתאפשר אופציה של יבוא ויצוא המכונות הוירטואליות ללא תלות ביצרן הדיסק הוירטואלי. להזכירכם, מדובר בפורמט פתוח שהפך לסטנדרט ונמצא בשימוש בכל מוצרי הוירטואלזציה הקיימים היום בשוק.

גרסת ה-VDI3.1 מבוססת קונטרולר חדש לחלוטין – שימו לב SUNRAY 5 כבר כאן. כמה דגשים לגבי ה – SRSS 5.

מולטימדיה, פלאש, תמיכה ב-VPN`s/VLAN – כל אלו כבר אינם נחשבים לאתגר לתחנות הרזות של סאן SUNRAYS. בגרסת ה-SRSS5 ניתן לצפות בסרטוני הפלאש באמצעות תחנות עבודה רזות SUNRAY 2, 2FS, 270, כנ”ל גם לגבי תחנות עבודה רזות אחרות ואפילו תחנות PC אשר מתחברות דרך פרוטוקול ה- RDP סטנדרטי לחלוטין. כן, תחנות PC רגילות עם סאן ראיי– ראיתם נכון.

SRSS 5 – לא מחייב לעבוד עם תחנות עבודה של SUNRAY. הקונטרולר החדש משלב בתוכו את גרסת ה-SOFT או במילים אחרות אמולציה ל SUNRAY. מה שמאפשר לארגונים מעבר הדרגתי לVDI 3.1. כלומר, אין יותר צורך להשקיע בעמדות UTC החל מיום הראשון בתחילת הפרוייקט, המעבר ל- SUNRAYS מתבצע בהדרגה. כל עמדת PC אשר יוצאת ממחזור החיים, מוחלפת ב- SUNRAY.

שרידות וזמינות (HIGH availability ). כפי שאתם כבר מבינים VDI3.1 איננו עוד מוצר צד שלישי שיש לו ערך מוסף להפעלת המסופים ו/או ניהול הסיישנים ו/או בורר עומסים, מצפין תעבורה ברשת וכד’… למרות שהמוצר אכן מטפל בכל הפונקציות הנ”ל למעשה מדובר במוצר ליבה אשר מפעיל את כלל מערך התשתיות הוירטואליות של הארגון. החל מרמת שולחן העבודה אל תוך שליטה במשאבים פיסים בשרתים הארגוניים. היות ומדובר במוצר ליבה, בסאן השקיעו לא מעט שעות פיתוח לאבטחת השרידות וליצירת המשכיות בכל מערכי הוירטואלזציה אשר פועלים ומופעלים באמצעות VDI3.1 ובניהם:

שרידות וזמינות ל-HOST PROVIDERS, על בסיס n+1. Active/Active

שרידות הליבה המרכזית. (VDI CORE Cluster), על בסיס n+1 Active/Active

שרידות מערכי הדיסקים הוירטואליים, מבלי תלות בספק ה-storage.

יכולת הפעלה ושימוש בטכנולוגיות מבית סאן כדוגמת ZFS על NETAPPו/או EMC.

להורדת הגרסה, ליותר פרטים ו/או להדגמת המוצר הינכם מוזמנים לצור קשר עם מנהלי תיק לקוח שלכם בסאן, או ישירות איתנו.

טכנולוג, יזם, מייסד ומנכ”ל חב’ פורנט פלוס בע”מ. מתמחה במחשוב עננים, VDI, HPC, עולמות וירטואליים ופלטפורמות עטירות ביצועים.

Dr. StrangeRAID, or, how I learned to stop worrying and love ZFS

Leigh — Wed, 14 Oct 2009 11:23:00 +0000

So now that we’ve chosen to go with Solaris and use NFS for our filesystem, let’s consider how we’re going to use ZFS to best suit our needs.

So let’s say that I have four 1Tb disks lying around which I want to thow into my new file server. At some point in the future, I’d like to upgrade my capacity by replacing one or all of the disks with larger ones (say, 2Tb).

How do we go about it?

Before we start, there’s a few things that we need to be mindful of. Rules of ZFS, if you will. They are:

A vdev cannot have any additional devices added to it, but its devices can be replaced.
A vdev cannot be removed from a pool. You have to destroy the pool and start over.
A raidz vdev must contain disks of equal size.
A raidz vdev can be expanded, but some trickery is required.

Essentially when you create a group of disks in a zpool, you’re creating what’s known as a vdev. A vdev can be a single disk, an N-way mirrored array, a RaidZ1 array or a RaidZ2 array. It can also be a logging disk, a cache disk or a spare disk, but we’ll get on to them down the track.

For my experiment, I’ll be using files instead of disks, but creating 1Tb files is plain silly, so I’ll use 1Gb files instead. So wherever you see a space reading in Gigabytes, assume that it’s Terabytes. To create the files, we’ll first create a folder for them to live in, then make a few 1Gb “disks” for each scenario we want to experiment with.

root@Zues:~# mkdir /zdev root@Zues:~# cd /zdev root@Zues:/zdev# mkfile 1g flat1a && mkfile 2g flat2a && mkfile 1g 1b.... and so on. 49Gb of disk images total!

Scenario 1: Flat dynamically striped array (“Dynamic” RAID0)

The reason ZFS striped arrays are called “Dynamic” is because… well, they are. Adding or replacing disks is simple and frighteningly quick – most times involving only a single command, and no reboots to make it take effect.

To create a dynamically striped array is as simple as:

root@Zues:/zdev# zpool create dynamic /zdev/flat_1a /zdev/flat_1b /zdev/flat_1c /zdev/flat_1d

Which gives us:

root@Zues:/zdev# zpool list dynamic NAME SIZE USED AVAIL CAP HEALTH ALTROOT dynamic 3.97G 79.5K 3.97G 0% ONLINE - root@Zues:/zdev# df -h |grep dynamic dynamic 4.0G 21K 4.0G 1% /dynamic

Four gigs of space. Not bad. Replacing our disks in the future happens like this:

root@Zues:/zdev# zpool replace dynamic /zdev/flat_1a /zdev/flat_2a root@Zues:/zdev# zpool replace dynamic /zdev/flat_1b /zdev/flat_2b root@Zues:/zdev# zpool replace dynamic /zdev/flat_1c /zdev/flat_2c root@Zues:/zdev# zpool replace dynamic /zdev/flat_1d /zdev/flat_2d

root@Zues:/zdev# zpool list dynamic NAME SIZE USED AVAIL CAP HEALTH ALTROOT dynamic 3.97G 104K 3.97G 0% ONLINE - root@Zues:/zdev# df -h |grep dynamic dynamic 4.0G 21K 4.0G 1% /dynamic

So It’s as easy as pie to replace disks to increase the size of your array, and although I did them all at the same time, it’s quite simple to do them one at a time.

Let’s see how it performs:

Pros:

Maximum storage
Maximum simplicity
Great performance
Replace any disk at any time to expand the array

Cons:

Zero Redundancy

Conclusion:
Best if you want something really simple, with lots of storage space and very little redundancy.

Scenario 2: Mirroring (RAID1)

Building a mirrored array is a little tricker to get your head around. The first way to do it is to add disks in pairs and let the system figure it out. My preferred way, however, is to create a standard ZFS array of two disks, then attach the other two disks as hot mirrors of the two first ones. I’ll have to call this array ‘reflect’ because the words mirror and mirroring are reserved by ZFS.

root@Zues:/zdev# zpool create reflect /zdev/mirror_1a /zdev/mirror_1b

and now we’ll attach the mirror disks:

root@Zues:/zdev# zpool attach reflect /zdev/mirror_1a /zdev/mirror_1c root@Zues:/zdev# zpool attach reflect /zdev/mirror_1b /zdev/mirror_1d root@Zues:/zdev# zpool status reflect pool: reflect state: ONLINE scrub: resilver completed after 0h0m with 0 errors on Wed Oct 14 21:27:04 2009 config: NAME STATE READ WRITE CKSUM reflect ONLINE 0 0 0 mirror ONLINE 0 0 0 /zdev/mirror_1a ONLINE 0 0 0 /zdev/mirror_1c ONLINE 0 0 0 mirror ONLINE 0 0 0 /zdev/mirror_1b ONLINE 0 0 0 /zdev/mirror_1d ONLINE 0 0 0 55.5K resilvered errors: No known data errors

The available space of this array, as you would imagine, is 2Gb thanks to their being two 1Gb disks available.

To upgrade a mirrored array, you simply need to detach the mirror disk, replace the primary, then attach a new mirror disk. For example:

root@Zues:/zdev# zpool detach reflect /zdev/mirror_1c root@Zues:/zdev# zpool replace reflect /zdev/mirror_1a /zdev/mirror_2a root@Zues:/zdev# zpool attach reflect /zdev/mirror_2a /zdev/mirror_2c root@Zues:/zdev# zpool status reflect pool: reflect state: ONLINE scrub: resilver completed after 0h0m with 0 errors on Wed Oct 14 21:30:16 2009 config: NAME STATE READ WRITE CKSUM reflect ONLINE 0 0 0 mirror ONLINE 0 0 0 /zdev/mirror_2a ONLINE 0 0 0 /zdev/mirror_2c ONLINE 0 0 0 34K resilvered mirror ONLINE 0 0 0 /zdev/mirror_1b ONLINE 0 0 0 /zdev/mirror_1d ONLINE 0 0 0

errors: No known data errors

But hang on a second! Even though we’ve upgraded one of the mirrors, we’re still only showing 2Gb of disk space! We should have at least 3Gb avaiable!

root@Zues:/zdev# zpool list reflect NAME SIZE USED AVAIL CAP HEALTH ALTROOT reflect 1.98G 96K 1.98G 0% ONLINE -

Don’t panic, this is standard operating procedure. In order to make the new disk space available to the pool, we need to export it and re-import it.

root@Zues:/zdev# zpool export reflect root@Zues:/zdev# zpool import -d /zdev/ reflect root@Zues:/zdev# zpool list reflect NAME SIZE USED AVAIL CAP HEALTH ALTROOT reflect 2.98G 94.5K 2.98G 0% ONLINE -

Note the -d slash and the file path are only necessary when you’re working with files. Were these actual disks, I could just type “zpool import <poolname>” and get the same result.

As for performance, mirroring is definitely the way to go if you’re concerned about throughput. Over a 100 megabit network, this array gives perfect performance:

Pros:

Excellent Redundancy
Excellent Performance
Still fairly simple
Can upgrade the array mirror at a time
Can roll back to dynamic array at any time by detaching mirrors and adding them to the pool

Cons:

Very expensive

Conclusion:
Great mix of redundancy and flexibility, but not for the faint of wallet.

Scenario 3: RaidZ1 (RAID5)

A Z1 array is essentially a RAID5 array, but much, much sexier. Why? Because it has a Z in front of it!

Seriously, though, the RAIDZ array sits pretty much half way between naked disks and mirrored disks, providing n-1 redundancy so that while you’re secure, you can still have a good amount of storage without breaking the bank. Let’s go ahead and set one up.

root@Zues:/zdev# zpool create z1 raidz /zdev/raidz1_1a /zdev/raidz1_1b /zdev/raidz1_1c /zdev/raidz1_1d root@Zues:/zdev# zpool status z1 pool: z1 state: ONLINE scrub: none requested config:

NAME STATE READ WRITE CKSUM z1 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 /zdev/raidz1_1a ONLINE 0 0 0 /zdev/raidz1_1b ONLINE 0 0 0 /zdev/raidz1_1c ONLINE 0 0 0 /zdev/raidz1_1d ONLINE 0 0 0

errors: No known data errors

So now that the RaidZ array has been created (note that the vdev specifies that it’s raidz1), let’s look at how much space we’ve actually got to play with.

root@Zues:/zdev# zfs list z1 NAME USED AVAIL REFER MOUNTPOINT z1 96.5K 2.92G 31.4K /z1

Remember, n-1 spacing means that even though we’ve got 4Gb of disks, we only get 3Gb of space, and then 1/64th of that space is reserved by ZFS.

Now let’s say it’s a year down the track, we’ve mostly filled our array with files, and we want to replace these disks with larger ones. First, you’ll have to make sure that the “Expand” option is enabled on the array, otherwise it won’t automatically grow when we add the new disks.

root@Zues:/zdev# zpool get expand z1 NAME PROPERTY VALUE SOURCE z1 autoexpand off default root@Zues:/zdev# zpool set expand=on z1 root@Zues:/zdev# zpool get expand z1 NAME PROPERTY VALUE SOURCE z1 autoexpand on local

Excellent. Now we can go ahead and replace out our disks with the new 2Gb models we just bought.

root@Zues:/zdev# zpool replace z1 /zdev/raidz1_1a /zdev/raidz1_2a root@Zues:/zdev# zpool replace z1 /zdev/raidz1_1b /zdev/raidz1_2b root@Zues:/zdev# zpool replace z1 /zdev/raidz1_1c /zdev/raidz1_2c root@Zues:/zdev# zpool replace z1 /zdev/raidz1_1d /zdev/raidz1_2d root@Zues:/zdev# zpool status z1 pool: z1 state: ONLINE scrub: resilver completed after 0h0m with 0 errors on Wed Oct 14 21:54:17 2009 config:

NAME STATE READ WRITE CKSUM z1 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 /zdev/raidz1_2a ONLINE 0 0 0 /zdev/raidz1_2b ONLINE 0 0 0 /zdev/raidz1_2c ONLINE 0 0 0 /zdev/raidz1_2d ONLINE 0 0 0 31K resilvered

errors: No known data errors

root@Zues:/zdev# zfs list z1
NAME USED AVAIL REFER MOUNTPOINT
z1 123K 5.87G 31.4K /z1

Excellent. Just under 6Gb of space to play with, although I shudder to think how much four of those disks cost. Let’s see how it performs.

Ok, so performance is a little bit degraded because of the fact that the system needs to calculate parity blocks during the write process. It also increases CPU utilization for the same reason.

Pros:

Single disk Redundancy
Can also add spares which can kick in if a drive fails
Halfway between naked drives and mirroring for price

Cons:

Not so stellar performance
To upgrade, all drives must be replaced

Conclusion:
Not for the faint of heart when it comes to upgrade paths, but definitely one of the most versitile options

Scenario 4: RaidZ2 (RAID6)

Essentially this is the ZFS equivalent of RAID6, and offers two disk redundancy – i.e. two disks can fail before you need to start worrying about data. The only downside is that you’ll lose two disks to parity information, so it tends to get rather pricey. First, let’s create a RaidZ2 array:

root@Zues:/zdev# zpool create z2 raidz2 /zdev/raidz2_1a /zdev/raidz2_1b /zdev/raidz2_1c /zdev/raidz2_1d root@Zues:/zdev# zpool status z2 pool: z2 state: ONLINE scrub: none requested config:

NAME STATE READ WRITE CKSUM z2 ONLINE 0 0 0 raidz2 ONLINE 0 0 0 /zdev/raidz2_1a ONLINE 0 0 0 /zdev/raidz2_1b ONLINE 0 0 0 /zdev/raidz2_1c ONLINE 0 0 0 /zdev/raidz2_1d ONLINE 0 0 0

errors: No known data errors

and the available space?

root@Zues:/zdev# zfs list z2 NAME USED AVAIL REFER MOUNTPOINT z2 101K 1.95G 31.4K /z2

Ew. Not much space to play with, considering we just threw 4 1G disks at it, although we had the same sort of thing with Mirroring. Upgrading this array is the same as a RaidZ1 array, in that we need to enable exand, then upgrade all four disks at once.

root@Zues:/zdev# zpool get expand z2 NAME PROPERTY VALUE SOURCE z2 autoexpand off default root@Zues:/zdev# zpool set expand=on z2 root@Zues:/zdev# zpool replace z2 /zdev/raidz2_1a /zdev/raidz2_2a root@Zues:/zdev# zpool replace z2 /zdev/raidz2_1b /zdev/raidz2_2b root@Zues:/zdev# zpool replace z2 /zdev/raidz2_1c /zdev/raidz2_2c root@Zues:/zdev# zpool replace z2 /zdev/raidz2_1d /zdev/raidz2_2d root@Zues:/zdev# zpool status z2 pool: z2 state: ONLINE scrub: resilver completed after 0h0m with 0 errors on Wed Oct 14 22:17:12 2009 config:

NAME STATE READ WRITE CKSUM z2 ONLINE 0 0 0 raidz2 ONLINE 0 0 0 /zdev/raidz2_2a ONLINE 0 0 0 /zdev/raidz2_2b ONLINE 0 0 0 /zdev/raidz2_2c ONLINE 0 0 0 /zdev/raidz2_2d ONLINE 0 0 0 56.5K resilvered

errors: No known data errors
root@Zues:/zdev# zfs list z2
NAME USED AVAIL REFER MOUNTPOINT
z2 126K 3.91G 31.4K /z2

So with four 2Gb disks, we now have just shy of 4Gb of space which is double redundant. Let’s see what its performance is like:

Another small drop thanks to the extra parity we have to write, but nothing out of the ordinary.

Pros:

Double disk Redundancy
Can also add spares which can kick in if a drive fails

Cons:

Underwhelming performance
To upgrade, all drives must be replaced
Lots of disk space lost to Parity

Conclusion:
In a massive array, RaidZ2 would be very worthwhile, but in our example, two disk redundancy can be performed by Mirroring just as easily and have the added bonus of being much easier to expand.

So there you have it, folks. I’ll admit that this look into ZFS and its various options isn’t exactly in-depth, but hopefully it gives you an idea of what is available.

Final Conclusion.

I can’t tell you which style of zpool layout to go for. For most people, a flat dynamically striped array is probably all they need, as all they really care about is the size of the array. Others might baulk at the idea of putting so much data in the hands of the gods – after all, a flat array is horribly susceptible to disk failure.

Having weighed up all the options, I can say that the most likely one I’ll go for is a four-disk mirrored array. The fact that I can upgrade two of the disks and leave the other two alone is a massive bonus, as it means that while I enjoy the luxury of double redundant disks, I also don’t have to buy four new disks when it comes time to upgrade. Two (or even one) new 2Tb disks will bump my storage array to 3Tb – more than enough to keep me trucking.

If you’d like to simulate any of these options, all you need to do is grab a copy of the Opensolaris LiveCD, whack it in to your computer (or your virtual machine software of choice) and give it a go.

Enjoy!

Testing out FreeBSD 8.0-RC1

koitsu — Mon, 12 Oct 2009 17:45:33 +0000

EDIT: Those interested in the upcoming release of FreeBSD 8.0 should read both the below, as well as my Testing out FreeBSD 8.0-RC2 post (which notes that many, but not all, of these problems have been fixed).

Yesterday I took the plunge and upgraded my home FreeBSD amd64 box from RELENG_7 to FreeBSD 8.0-RC1, which going forward I will refer to as RELENG_8 (yes, it has been tagged!). I did a complete reinstall, like I always do when migrating between major FreeBSD releases. Said box consists of a Supermicro X7SBA motherboard, Intel Core2Duo E8400 CPU, 4GB of RAM, and 3 SATA disks connected via the on-board Intel ICH9 + AHCI — one for the OS, and two in a ZFS mirror pool. Relevant dmesg information:

CPU: Intel(R) Core(TM)2 Duo CPU     E8400  @ 3.00GHz (2992.52-MHz K8-class CPU)
real memory  = 4294967296 (4096 MB)
avail memory = 4112478208 (3921 MB)
atapci0: <Intel ICH9 SATA300 controller> port 0x1c50-0x1c57,0x1c44-0x1c47,0x1c48-0x1c4f,0x1c40-0x1c43,0x18e0-0x18ff mem 0xdc000800-0xdc000fff irq 17 at device 31.2 on pci0
atapci0: [ITHREAD]
atapci0: AHCI called from vendor specific driver
atapci0: AHCI v1.20 controller with 6 3Gbps ports, PM supported
ata4: <ATA channel 2> on atapci0
ata4: [ITHREAD]
ata5: <ATA channel 3> on atapci0
ata5: [ITHREAD]
ata7: <ATA channel 5> on atapci0
ata7: [ITHREAD]
ad8: 190782MB <WDC WD2000JD-00HBB0 08.02D08> at ata4-master SATA150
ad10: 953869MB <WDC WD1001FALS-00J7B1 05.00K05> at ata5-master SATA300
ad14: 953869MB <WDC WD1001FALS-00J7B1 05.00K05> at ata7-master SATA300

And ZFS-related details:

  pool: storage
 state: ONLINE
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        storage     ONLINE       0     0     0
          mirror    ONLINE       0     0     0
            ad10    ONLINE       0     0     0
            ad14    ONLINE       0     0     0

errors: No known data errors

The first thing worth noting is that I performed the installation entirely using the FreeBSD 8.0-RC1 amd64 memstick image on a SanDisk Cruzer Micro 2GB USB drive. If you’re going to try this, make sure you have a 2GB or larger USB drive, since the memstick image is larger than 1GB.

Writing the .img file to the USB drive required the use of another FreeBSD box (someone is going to have to address that fact eventually), and was achieved per Ken Smith’s original instructions:

dd if=8.0-RC1-amd64-memstick.img of=/dev/da0 bs=10240 conv=sync

The entire dd took about 4 minutes. I then booted it directly without any issues. Installation went as expected, with the exception of choosing a different installation medium than usual; there’s a new USB menu item near the bottom of the installation medium selection list. I should also note that I deleted the existing FreeBSD partition and re-created it during the sysinstall phase, and during the Boot Manager selection phase, I chose Standard like I always do. You’ll understand why I’ve noted these two things in a moment.

After rebooting + booting off the main OS hard disk, the first thing I saw which was different/anomalous was that I was being shown the F1/F5/F6 FreeBSD boot manager menu — as if I had selected BootMgr and not Standard. Options shown were F1 for FreeBSD, F5 for Floppy, and F6 for PXE (that’s a new one!). So there’s definitely a bug/regression somewhere with regards to the boot manager you choose; or maybe it was because I was installing from a USB drive? Not sure.

The FreeBSD box booted fine, but the following kernel message caught my eye:

GEOM: ad8s1: geometry does not match label (255h,63s != 16h,63s).

This indicated that what sysinstall wrote to the actual on-disk BSD label, as far as drive geometry went, didn’t match what GEOM expected — GEOM expecting 16 heads, the drive label containing 255 heads. I’m not using GPT (at least not knowingly; if sysinstall does it without telling you, then someone needs to work out where the actual problem lies). I first looked at gpart show ad8 and gpart show ad8s1 to see what it claimed:

# gpart show ad8
=>       63  390721905  ad8  MBR  (186G)
         63  390716802    1  freebsd  [active]  (186G)
  390716865       5103       - free -  (2.5M)

# gpart show ad8s1
=>        0  390716802  ad8s1  BSD  (186G)
          0    4194304      1  freebsd-ufs  (2.0G)
    4194304   16777216      2  freebsd-swap  (8.0G)
   20971520   33554432      4  freebsd-ufs  (16G)
   54525952   16777216      5  freebsd-ufs  (8.0G)
   71303168  319413634      6  freebsd-ufs  (152G)

Next, and more conclusive, I used bsdlabel -e -A ad8s1 and this is what I got:

# /dev/ad8s1:
type: ESDI
disk: ad8s1
label:
flags:
bytes/sector: 512
sectors/track: 63
tracks/cylinder: 255
sectors/cylinder: 16065
cylinders: 24321
sectors/unit: 390721968
rpm: 3600
interleave: 1
trackskew: 0
cylinderskew: 0
headswitch: 0           # milliseconds
track-to-track seek: 0  # milliseconds
drivedata: 0

8 partitions:
#        size   offset    fstype   [fsize bsize bps/cpg]
  a:  4194304        0    4.2BSD        0     0     0
  b: 16777216  4194304      swap
  c: 390716802        0    unused        0     0         # "raw" part, don't edit
  d: 33554432 20971520    4.2BSD        0     0     0
  e: 16777216 54525952    4.2BSD        0     0     0
  f: 319413634 71303168    4.2BSD        0     0     0

So yes, sysinstall is definitely doing the Wrong Thing(tm) here. One of the biggest problems with this is that users cannot easily fix this — it requires booting a fixit or LiveFS image and editing the disk label, as well as users having to re-calculate sectors/cylinder and cylinders by hand. And no, I’m not talking out of my ass. Also note that the post to freebsd-current is dated March 2009 — 7 months ago. So this issue has existed for quite some time without proper attention.

Upon exiting the editor session inside of bsdlabel (using :q!), I was given the following error:

bsdlabel: partition c doesn't cover the whole unit!
bsdlabel: An incorrect partition c may cause problems for standard system utilities
bsdlabel: Class not found
re-edit the label? [y]: n

Whoa whoa whoa, even more insanity going on here! Also, what’s with the Class not found error? Wow, this is pretty jacked, and apparently I’m not the only one who’s noticed. Note this post is dated July 2009 — 3 months ago. The easy solution seems to be to use gpart(8) to create all of the slices… except it’s obvious no one has fixed sysinstall to do this. There’s also a patch mentioned which fixes the problem, but that obviously wasn’t committed before the RELENG_8 tagging, nor backported since.

But the root of the problem does appear to be sysinstall not doing the Right Thing(tm) any longer.

Next up was my attempt to fix the Boot Manager oddities. Historically, re-writing the boot blocks on a disk consisted of doing the appropriate equivalent of bsdlabel -B ad8s1. However, I was greeted with the exact same “Class not found” error as above. Hmmm… This doesn’t bode well. Presumably I can use gpart(8) to re-write the boot blocks, but given that GEOM is complaining about disk geometry errors, I don’t dare mess with it.

All of this is a pretty major bug. The kernel message is going to catch a lot of user attention if it’s not fixed by the time 8.0-RELEASE is announced. I plan on sending Robert Watson and Ken Smith an Email about the issue after I get done writing this.

At this point I decided to make appropriate modifications to /etc/rc.conf, specifically the addition of zfs_enable="yes", so that I could get access to my ZFS filesystems. After doing so, and running /etc/rc.d/hostid start then /etc/rc.d/zfs start, I was greeted with the usual ZFS kernel messages — but with an unexpected surprise:

ZFS NOTICE: system has less than 4GB and prefetch enable is not set... disabling.

Given how familiar I am with FreeBSD and ZFS at this point, this message caught me off-guard.

First of all, grammatically this sentence is confusing as hell — there is no “prefetch enable” tunable; the tunable is actually called vfs.zfs.prefetch_disable, and it defaults to 0 (off, e.g. prefetching enabled), so why would I exclusively enable something which is enabled by default? And why’s it getting disabled? Secondly, my system has 4GB of RAM installed… so what’s going on here?!

I dug around in the relevant CVS commit logs and found numerous changes to this file, specifically the message that was printed. Apparently the existing message was reviewed by 5 separate people (revision 1.23) to “Improve wording”.

   3554 #ifdef _KERNEL
   3555         if (TUNABLE_INT_FETCH("vfs.zfs.prefetch_disable", &zfs_prefetch_disable))
   3556                 prefetch_tunable_set = 1;
   3557
   ...
   3565         if ((((uint64_t)physmem * PAGESIZE) < (1ULL << 32)) &&
   3566             prefetch_tunable_set == 0) {
   3567                 printf("ZFS NOTICE: system has less than 4GB and prefetch enable is not set"
   3568                     "... disabling.\n");
   3569                 zfs_prefetch_disable=1;
   3570         }
   3571 #endif

Ahh, now we have a much better idea of what’s going on. There are two reasons why this message got printed on my machine:

1) I had not done any tuning of /boot/loader.conf at this point, so vfs.zfs.prefetch_disable hadn’t been set. The above code basically says “if someone has administratively set vfs.zfs.prefetch_disable to something in loader.conf, set prefetch_tunable_set to 1″. You can set the tunable to whatever you want (enabled or disabled) and the message won’t get printed. If you don’t set the tunable, the following applies:

2) physmem is actually the amount of memory in pages that’s available to the kernel when it loads. The multiplication is actually hw.availpages * hw.pagesize. The 1ULL << 32 statement may look ugly but it’s a bitshift equivalent of 2^32, e.g. 4294967296.

Let’s work out the math:

# sysctl hw.pagesize hw.availpages
hw.pagesize: 4096
hw.availpages: 1046201
# expr 1046201 "*" 4096
4285239296

4285239296 is indeed less than 4294967296. Wait a minute… where’s that extra memory going?

Well, it’s going to two places on the X7SBA: 1) on-board video (which has an 8MB framebuffer), and 2) the AHCI BIOS which takes up an unknown amount of RAM, but I’d guess about 1-2MB. So let’s do the math:

# expr 4294967296 - 4285239296
9728000

With all of this information kept in mind, the kernel message really should be re-worded to say the following:

ZFS NOTICE: System has less than 4294967296 bytes (4GB) of usable memory,
ZFS NOTICE: and vfs.zfs.prefetch_disable has not explicitly been defined
ZFS NOTICE: in loader.conf.  Setting vfs.zfs.prefetch_disable="1"...

I’m also questioning the logic behind why prefetching is disabled on systems with less than 4GB of available memory; I’d like to know what the reasoning is there. Is it in regards to stability? Performance? I don’t know. I can’t find an answer on the mailing lists either.

Finally, I found an unexpected oddity with the new tty/pty/pts code with regards to EOF. All other operating systems, including RELENG_7 and earlier, behave as follows when EOF is pressed on a terminal. This is regardless of shell, by the way:

bash$ cat
{press Control-D here}bash$

While on RELENG_8, the literal Control-D (^D) character is shown on-screen:

bash$ cat
{press Control-D here}^Dbash$

I’ve already mailed Ed Schouten about this, and he agrees it’s a bug which he’ll work on fixing, hopefully tonight.

Everything else past this point was peachy keen. No odd problems building ports, no system lock-ups or odd experiences, and so on. It’s all worked great so far. I’m looking forward to upgrading our production servers to RELENG_8 when it comes out.

Solaris 10 10/09 (aka Solaris 10 Update 8) Released

wa4zko — Sat, 10 Oct 2009 15:56:05 +0000

Sun Microsystems has just released Solaris 10 10/09. This is the 8th major update to the Solaris 10 operating system. Downloads can be found here.

Solaris 10 10/09 (aka Update includes many updates, fixed, and additions such as, but not limited to:

1. Numerous well tested updates and fixes are included. Think of it as Solaris 10 with “Service Pack 8″ integrated. Saves you time when building up a new server as you need to download/install fewer updates as compared to an installation done with older media. Microsoft could learn some lessons here, especially as they continue to provide fewer and fewer service packs for their products. SP2 is it for Windows Server 2003, you gotta be kidding me? Sun supports their OS products for a much longer and reasonable time frames. Thus every time you turn around you are are not being forced into OS upgrades to perfectly functional production systems.

2. Support for newer hardware and a continued focus on improving/extending power management features. How fast are your data center electric and cooling bills growing? Getting ugly are they not? Judging by the increasingly idiotic energy policy ideas coming out of Washington, unnecessary and skyrocketing energy costs are likely in the near future.

3. ZFS support for Solid State Drives. Improvements towards more fine grained file system resource limits.

4. More streamlined system installation and management. Improvements to system patching/updating. Parallel patching of Solaris Containers (very nice). New automated framework for patching in single user mode (nice). Considerably (up to 4x) faster patching and package installs in many scenarios.

And more, check it out.

10 Reasons You Need to Look at ZFS

madhavkobal — Thu, 08 Oct 2009 17:51:29 +0000

Sun’s open-source ZFS file system has some amazing features. It was originally designed for Solaris and unveiled in 2005, but you’ll also find it in OpenSolaris and related distributions. In the future it may well become a popular file system to run with Linux and BSD as well.

Here are the top 10 features that ZFS fans find insanely great:

1. Checksums in Metadata for Data Integrity

Data integrity is of paramount importance in ZFS, and is the driver for many ZFS features.

The file system uses a 256-bit checksum, which is stored as metadata separate from the data it relates to, when it writes information to disk. Unlike a simple disk block checksum, this can detect phantom writes, misdirected reads and writes, DMA parity errors, driver bugs and accidental overwrites as well as traditional “bit rot.”

2. Copy on Write

ZFS ensures that data is always consistent on the disk using a number of techniques, including copy-on-write. What this means is that when data is changed it is not overwritten — it is always written to a new block and checksummed before pointers to the data are changed. The old data may be retained, creating snapshots of the file system through time as changes are made. File writes using ZFS are transactional — either everything or nothing is written to disk.

3. Data Snapshots With Time Slider

The latest version of OpenSolaris illustrates the power for ZFS’s snapshots capability with a small graphical application called TimeSlider. ZFS can be configured to take a snapshot of the file system (or a section of it, such as just a user’s home folder) on a regular basis — every 15 minutes, or every hour, and so on. These snapshots are very small and efficient, as only the deltas from the previous snapshot are stored.

TimeSlider offers a view of the file system (or a home folder), with a slider which can be moved back along a timeline to earlier snapshot times. As this is done, the view changes to show they state of the file system or the contents of a folder at the corresponding snapshot time. Recovering a file that has been overwritten by mistake or rolling back the system after an unsuccessful update is then just a matter of moving the slider back to the appropriate snapshot time.

Snapshots can also be made writable to create clones of existing file systems.

4. Pooled Data Storage

ZFS takes available storage drives and pools them together as a single resource, called a zpool. This can be optimized for capacity, or I/O performance, or redundancy, using striping, mirroring or some form of RAID. If more storage is needed then more drives can simply be added to the zpool — ZFS sees the new capacity and starts using it automatically, balancing I/O and maximizing throughput.

5. RAIDZ and RAIDZ2

RAID 5 has a well known flaw called the RAID 5 write hole. This causes a problem when a data block is written to a stripe but a power failure occurs before the corresponding parity block can be written. As a result the data and parity for the stripe will be inconsistent. If a disk then fails, the RAID reconstruction process will result in incorrect data. The only way out of this is if an entire stripe happens to be overwritten, thus generating a correct parity block.

RAIDZ gets around this problem by using a variable width stripe, so every write is effectively a full stripe write. This, together with ZFS’s copy on write characteristic (mentioned above,) eliminates the RAID 5 write hole completely. RAIDZ2 works in a similar way, but can tolerate the loss of two disks in the array using double parity.

Setting up a RAIDZ (or RAIDZ2) array is very easy and involves issuing one command

6. SSD Hybrid Storage Pools

High performance SSDs can be added to a storage pool to create a hybrid storage pool. When these are configured as high performance cache disks, ZFS uses them to hold frequently accessed data to improve performance. It also uses a technology called L2 ARC (adaptive replacement cache) to write data that has to be stored immediately. This can slowly be moved over to conventional hard drives for more permanent storage when time and resources allow.

7. Capacity

ZFS is a 128-bit file system, which means that in theory it could store 256 quadrillion ZB (a ZB is a billion TB.) In practice this is larger than would ever be necessary, for the foreseeable future at least.

8. Data Scrubbing

ZFS can be made to scrub all the data in a storage pool, checking each piece of data with its corresponding checksum to verify its integrity, detect any silent data corruption, and to correct any errors in encounters where possible.

When the data is stored in a redundant fashion — in a mirrored or RAID-type array — it can correct any corrupt data it detects invisibly and without any administrator intervention. Since data corruption is logged, ZFS can bring to light defects in memory modules (or other hardware) which cause data to be stored on hard drives incorrectly.

Scrubbing is given low I/O priority so that it has a minimal effect on system performance, and can operate while the storage pool that is being scrubbed is in use.

9 . Simple, Efficient Administration

Using ZFS commands you can administer a system with short, efficient commands. For example, a five disk RAIDZ array could be set up with the single command:

zpool create poolname raidz c1t0d0 c2t0d0 c3t0d0 c4t0d0 c5t0d0

10. More on the Way

ZFS is still evolving, and new features will appear regularly. The roadmap for 2009 includes encryption for increased security, and data deduping to increase storage efficiency.

If you are interested in trying ZFS out then the easiest way is to get started is to download OpenSolaris or related distro from http://opensolaris.org/os/downloads/. There is also a ZFS for FUSE/Linux project, and you can watch ZFS: A hands on tutorial.

Paul Rubens

FreeBSD and ZFS -- is it truly stable?

koitsu — Thu, 08 Oct 2009 14:48:56 +0000

There’s an “age old question” that has been floating around with regards to ZFS on FreeBSD — is it stable? “Stable” in this case means: do I risk losing my data, will it cause kernel panics or other oddities, and do I need to tune it?

The answer, still, may be yes.

I’ve taken the initiative — that is to say, get an official response to these type of questions, specifically with regards to kernel panics. I’m incredibly surprised no one — not even the user community — has responded at this point. It’s not a trick question either; FreeBSD users really do need an answer to this.

People are continually comparing FreeBSD’s ZFS to that of Solaris 10 and OpenSolaris’ ZFS. Given that my day job involves heavy use of Solaris 10 on massive numbers of servers across the United States, I can safely say without a doubt ZFS on Solaris behaves better and won’t crash your system due to kernel memory exhaustion.

Hadoop World: NYC 2009 | Cloudera

Visitor Blogs — Thu, 08 Oct 2009 14:02:25 +0000

Steve Jones from SQLServerCentral.com

I tried to read through the quickstart on Apache’s site for the common core installation and walk through a few examples, but it’s a little hard to tell what exactly the buzz is about. Wikipedia was more help, pointing me to the MapReduce papers that Google published. I’ll see if I can work through them at some point. Hadoop is available under a free license and the list of companies using it for large data set processing is impressive: Yahoo!, Amazon, Facebook, and more.

So what’s the purpose?

Hadoop appears to allow clusters of servers to perform data processing very efficiently. It’s built on it’s own distributed file system that scales to handle petabytes of data. That might seem like more data than you and I will ever need to work with, but I remember when it was a challenge to get enough disk drives together to assemble a terabyte in a server. Now I have 1.5TB in my desktop, with room for more.

It’s an interesting project, and with data volumes constantly growing, I wonder when we’ll see a similar technology in Microsoft’s data processing platform. They already purchased a search technology company based on Hadoop, and we might see this used in Bing.

I expect this type of processing, and others like the StreamInsight features in SQL Server 2008 R2, to complement, rather than supplant the traditional SQL database engine.