The open source community is coming up with a new OS almost every month !!! For me the urge to try out a new OS is insatiable, I want to migrate/test other variants as soon as they are available for download. Having multiple OS’s has it own problems, you need to cut your  HDD into multiple partitions and allocate it to the new OS, this creates a much bigger problem of data-isolation. User data available on one OS is invisible to the other, what if there was a solution wherein we could have plug and play OS and user data on a  file-server which can be accessed through any OS. I am trying to achieve this (codenamed : Project Alice) as per the plan below:

Step 1:   Build a low cost PC

Hardware has already been arranged, need to buy some old disk controllers and hard drives for the file-server. Hard drives are going to be inexpensive ones with lesser storage space (40 GB max), so as to minimize extent of damage and cost in case of a disk crash. If someone has some old unused disks in working condition, please let me know.

Step 2:  Make it a file-server and store all data on it

This is perhaps the easy part, configuring the file-server. I am planning to use some generic storage filesystems on it, like Veritas volume manager or something else, preferably open source and spread the data on it using RAID 5, that solves the problem of backing up data on a periodic basis and also restoration is a lot easier. The file-server can then be hooked on to the network.

Step 3 :  Create USB bootable copies of the OS and run it from the USB itself

And this is the most interesting part of the project Create bootable OS versions on 4GB flash drives. OS’s now a days are quite light and 4GB is more than enough for them. You may need to add an external swap if the system is too overloaded. The USB version of OS also allows you to make persistent changes. I have already built two USB drives with Ubuntu and  OpenSolaris on it. You just need to plug them in and restart the system, your new OS would be UP and running in minutes. To change the OS, Shutdown -> insert new OS flash drive -> reboot, simple.

Step 4: Access all user data from Fileserver irrespective of the OS

Right this is the trickiest part. Almost all Linux based OS’s use the same file-systems (ext2/ext3) while windows uses NTFS/FAT32. I need a method/driver to access (read and write) on both the filesystems. In simpler terms I can store a file on a windows machine and access it from a machine which is running Linux !!! As of now I dont know how to achieve it , but have certain theories. One of them is to share the data by using  Samba for Windows machines and NFS for Unix based machines. The other is to use some 3rd party tool to allow me reading/writing data across multiple filesystems, I am not sure if such tool is available, if anyone is working on or aware of such tools, please let me know. I am eager to try out the second option.

I have downloaded the Google Chrome OS virtual machine image and fired it up in Parallels on my Mac. While somewhat underwhelming perhaps that is its best feature. When I think of my mom as a use case this would be a perfect solution. She has already embraced gmail, google calendar, google docs and picasa.

All of her data is in the cloud today except her photos. What will be interesting to see is how google enables digital cameras and the cloud. Perhaps a diect upload to Picasa, with a new online photo editor? This will be something to watch.

For example if you fire up Chrome OS in a Virtual Machine you and you insert a camera card into the internal reader on the iMac the ChromeOS shows a new panel with the the mounted filesystem:

You can then reference this mounted filesystem in Gmail or Gtalk for example.  I suspect we will see advanced tools where the web system auto detects the content on the flash card as a Photo Card and imports directly.  Imagine a web version of Picasa for example.

Hace unos días descubrí los snapshots para filesystems UFS(2) de FreeBSD (lo se, lo se, soy un desastre) y si bien se implementa también para ZFS le di caña a los snapshots para UFS (default de FreeBSD).
Para probarlo, lo que hice fue crear y montar una imagen en un archivo (dd, mkfs, mdconfig, etc…) quedando algo así:

/home/ignacio $ mount | grep md0
/dev/md0 on /usr/home/ignacio/md0 (ufs, local)

Lo primero que hay que hacer, es notar la existencia del directorio .snap dentro de la recién montada imagen. Es en la raíz de cada partición UFS es donde está el directorio .snap.
Yo lo que hice fue instalar el frontend snapshot(8) mediante su port sysutils/freebsd-snapshot.

En la página citada, donde Ralf S. Engelschall explica la motivación y la forma de uso la verdad que está muy buena y simple.
Básicamente el comando snapshot, consta de cuatro acciones básicas: list, make, mount y umount.

Con snapshot list podemos ver la lista completa de los snapshots creados para alguna partición, en este caso, aún no hemos creado ninguno por lo que el listado, estará vacío:

/home/ignacio $ snapshot list /home/ignacio/md0
Filesystem          User   User%     Snap   Snap%  Snapshot

Vemos el listado actual del directorio actual:

/home/ignacio $ cd md0
/home/ignacio/md0 $ ls -l
drwxrwxr-x  2 root  operator      512 Nov 17 14:06 .snap
-rw-r--r--  1 root  wheel          12 Nov 10 16:23 ignacio
-rw-r--r--  1 root  wheel     1048576 Nov 10 17:37 test_file.txt

y creamos nuestro primer snapshot:

/home/ignacio/md0 $ snapshot make -g4 /home/ignacio/md0:snaps

Donde, -g4 indica que guardaremos 4 generaciones (cuatro vistas al pasado), /home/ignacio/md0 es (donde está montada) la partición y snaps es un tag que le asignamos.

Tiremos ahora un nuevo list:

/home/ignacio $ snapshot list /home/ignacio/md0
Filesystem          User   User%     Snap   Snap%  Snapshot
/home/ignacio/md0      2MB    2.9%    224KB    0.2%  snaps.0

Para jugar un poco borramos el archivo test_file.txt por error y ahora, para salvar el día, recuperamos el snapshot que creamos:

/home/ignacio/md0 $ ls -l
total 4
drwxrwxr-x  2 root  operator  512 Nov 17 14:09 .snap
-rw-r--r--  1 root  wheel      12 Nov 10 16:23 ignacio
/home/ignacio/md0 $

Para recuperar el snapshot debemos montarlo indicando que generación queremos recuperar, en nuestro caso, la 0:

/home/ignacio $ snapshot mount /home/ignacio/md0:snaps.0 /mnt
WARNING: opening backing store: /usr/home/ignacio/md0/.snap/snaps.0 readonly

Buscamos en el /mnt los archivos:

/home/ignacio $ cd /mnt
/mnt $ ls
.snap           ignacio         test_file.txt
/mnt $

Devolvemos test_file.txt a la normalidad:

/mnt $ cp -v test_file.txt /home/ignacio/md0
test_file.txt -> /home/ignacio/md0/test_file.txt
/ $ snapshot umount /mnt
/ $ cd /home/ignacio/md0
/home/ignacio/md0 $ ls -l
total 1044
drwxrwxr-x  2 root  operator      512 Nov 17 14:09 .snap
-rw-r--r--  1 root  wheel          12 Nov 10 16:23 ignacio
-rw-r--r--  1 root  wheel     1048576 Nov 17 14:14 test_file.txt

Con lo de arriba, copiamos el archivo que habiamos perdido por error, desmontamos el snapshot y verificamos como quedó nuestra partición.

En fin, terrible herramienta! Es recomendado revisar además la man page periodic-snapshot(8) para hacer snapshots periodicos usando el cron y periodic.

Estoy comenzando a ser un usuario feliz de este tipo de filesystems. Claro que ya tengo configurado el cron para que genere los snapshots de manera automatizada para todas las particiones de mi disco y también espero jamás en mi vida necesitarlos.

I created my own rootfile system for linux.Opps!! I am not able to ping to my own system but I could ping others

Posted by mohammadthalif on November 12, 2009

This happens in the embedded very often when you are desingning your target file system fom scratch or modifing the exsisting one. I came accross same situation. and thinking why and finally figured out root for problem.

#ifconfig eth0 192.168.1.5 netmask 255.255.255.0 up

Now my interface is up. So to check it I do a ping to some system ( my network gateway )

# ping 192.168.1.1

Ping is sucessful. Now ping my own IP 192.168.1.5

#ping 192.168.1.5

Now it fails…. what could be the problem for this. ? I was looking around the kernel configs, driver etc,. but no solution.

And finaly I did type just ifconfig alone in the prompt and found there is no loopback interface up. I brought up the interface up and now I am able to ping by own IP.

#ping 192.168.1.5

Now passed.

Recently when I made an exasperated posting about lengthy ext3 check times and looking forward to btrfs, Siobhán Ellis pointed out that there was already a filesystem available for Linux that met a lot of my needs – particularly in the backup space, where I’m after:

• Being able to create large filesystems that don’t take exorbitantly long to check
• Being able to avoid checks on abrupt system resets
• Speeding up the removal of files when staging completes or large backups abort

That filesystem of course is XFS.

I’ve recently spent some time shuffling data around and presenting XFS filesystems to my Linux lab servers in place of ext3, and I’ll fully admit that I’m horribly embarrassed I hadn’t thought to try this out earlier. If anything, I’m stuck looking for the right superlative to describe the changes.

Case in point – I was (and indeed still am) doing some testing where I need to generate >2.5TB of backup data from a Windows 32-bit client for a single saveset. As you can imagine, not only does this take a while to generate, but it also takes a while to clear from disk. I had got about 400 GB into the saveset the first time I was testing and realised I’d made a mistake with the setup so I needed to stop and start again. On an ext3 filesystem, it took more than 10 minutes after cancelling the backup before the saveset had been fully deleted. It may have taken longer – I gave up waiting at that point, went to another terminal to do something else and lost track of how long it actually took.

It was around that point that I recalled having XFS recommended to me for testing purposes, so I downloaded the extra packages required to use XFS within CentOS and reformatting the ~3TB filesystem to XFS.

The next test that I ran aborted due to a (!!!) comms error 1.8TB through the backup. Guess how long it took to clear the space? No, seriously, guess – because I couldn’t log onto the test server fast enough to actually see the space clearing. The backup aborted, and the space was suddenly back again. That’s a 1.8TB file deleted in seconds.

That’s the way a filesystem should work.

I’ve since done some (in VMs) nasty power-cycle mid-operation tests and the XFS filesystems come back up practically instantaneously – no extended check sessions that make you want to cry in frustration.

If you’re backing up to disk on Linux, you’d be mad to use anything other than XFS as your filesystem. Quite frankly, I’m kicking myself that I didn’t do this years ago.

From time to time I find myself needing to replace certain strings globally (eg: in a directory tree), or deleting lines based on a match. I find sed extremely annoying and I don’t manage to get around its syntax. So ruby comes in handy!

replacer.rb

exp_search = '<%=launchBean\.getCustomVersion\(\)%>'
exp_replace = '${xsession.info.launchBean.customVersion}'
directory = '**/'
file_mask = '*.jsp'

Dir.glob(directory+file_mask).each do |file|
  File.open(file, 'r+') do |f|
    content = f.read
    if /#{exp_search}/ =~ content
      s = content.gsub(/#{exp_search}/, exp_replace)
      f.rewind
      f.write s
      puts "#{file}: #{content.split(/#{exp_search}/).size-1} matches"
    end
  end
end

deleter.rb

exp_search = 'useBean.*launchBean'
directory = '**/'
file_mask = '*.jsp'

Dir.glob(directory+file_mask).each do |file|
  count = 0
  content = ""
  File.open(file, 'r+') do |f|
    while line = f.gets
      if /#{exp_search}/ =~ line
        count += 1
      else
        content = content << line
      end
    end
  end
  if count > 0
    File.open(file, 'w') do |f|
      f.write content
      puts "#{file}: #{count} lines deleted"
    end
  end
end

These commands are especially useful if you have to refactor a codebase that doesn’t provide direct support with refactoring tools. In this case, I was refactoring a few hundred JSPs of a legacy system.

Credits to RSF & Ruby One-Liners for most of the code & ideas. rcscript seems to be a bit too much for me at this time, I’m perfectly happy modifying replacer.rb and deleter.rb when I need it.

Here’s a curious clash of virtual filesystems. Am trying to access my own processes’ /proc/-pid-/fd directory to investigate a file descriptor leak, and am unable to do so:

$  ps -ef | grep db2sysc | grep peeter | grep -v grep | tail -1
peeterj   9318  9316 99 12:09 ?        01:03:14 db2sysc 0
$  cd /proc/9318/fd
bash: cd: /proc/9318/fd: Permission denied
$  cd /proc/9318
$  ls
/bin/ls: cannot read symbolic link cwd: Permission denied
/bin/ls: cannot read symbolic link root: Permission denied
/bin/ls: cannot read symbolic link exe: Permission denied
attr  cmdline  cwd      exe  loginuid     maps  mounts     oom_adj    root     smaps  statm   task
auxv  cpuset   environ  fd   mapped_base  mem   numa_maps  oom_score  seccomp  stat   status  wchan

I'd actually seen this before because we have code in our product that tries to access /proc/-pid-/stat stuff, and it doesn't work properly (sometimes and mysteriously).  Even odder, I can't even get at this as root

# ps -o pid -o ruid -o euid -o suid -o fsuid -o fname -a | grep $$
21861     0     0     0     0 sh
# cd /proc/9318/fd
sh: cd: /proc/9318/fd: Permission denied
# cd /proc/9318
# ls
attr  cmdline  cwd      exe  loginuid     maps  mounts     oom_adj    root     smaps  statm   task
auxv  cpuset   environ  fd   mapped_base  mem   numa_maps  oom_score  seccomp  stat   status  wchan
# ls -l
ls: cannot read symbolic link cwd: Permission denied
ls: cannot read symbolic link root: Permission denied
ls: cannot read symbolic link exe: Permission denied
total 0
dr-xr-xr-x   2 peeterj pdxdb2 0 2009-10-27 12:11 attr
-r--------   1 peeterj pdxdb2 0 2009-10-27 12:10 auxv

Something funny is happening in the kernel, since my session does appear to have sufficient root-ish behaviour (even the linux filesystem fsuid is set right). Turns out that this is some kind of clash between the clearcase version control virtual filesystem and the /proc virtual filesystem. When I am in my view, even as root:

# /usr/atria/bin/cleartool pwv
Working directory view: ** NONE **
Set view: peeterj_o26
#

I have no access to much of /proc/, but running as any old user when there is no trouble

$  /usr/atria/bin/cleartool pwv
Working directory view: ** NONE **
Set view: ** NONE **
$  pwd
/proc/9318/fd

What a bizarre quirk! Glad to have this figured out … now back to the file descriptor leak.

Courtsey: http://tldp.org/LDP/Linux-Filesystem-Hierarchy/html/lostfound.html
As was explained earlier during the overview of the FSSTND, Linux should always go through a proper shutdown. Sometimes your system might crash or a power failure might take the machine down. Either way, at the next boot, a lengthy filesystem check (the speed of this check is dependent on the type of filesystem that you actually use. ie. ext3 is faster than ext2 because it is a journalled filesystem) using fsck will be done. Fsck will go through the system and try to recover any corrupt files that it finds. The result of this recovery operation will be placed in this directory. The files recovered are not likely to be complete or make much sense but there always is a chance that something worthwhile is recovered. Each partition has its own lost+found directory. If you find files in there, try to move them back to their original location. If you find something like a broken symbolic link to ‘file’, you have to reinstall the file/s from the corresponding RPM, since your file system got damaged so badly that the files were mutilated beyond recognition. Below is an example of a /lost+found directory. As you can see, the vast majority of files contained here are in actual fact sockets. As for the rest of the other files they were found to be damaged system files and personal files. These files were not able to be recovered.

total 368
-rw-r–r– 1 root root 110891 Oct 5 14:14 #388200
-rw-r–r– 1 root root 215 Oct 5 14:14 #388201
-rw-r–r– 1 root root 110303 Oct 6 23:09 #388813
-rw-r–r– 1 root root 141 Oct 6 23:09 #388814
-rw-r–r– 1 root root 110604 Oct 6 23:09 #388815a
-rw-r–r– 1 root root 194 Oct 6 23:09 #388816
srwxr-xr-x 1 root root 0 Oct 6 13:00 #51430
srwxr-xr-x 1 root root 0 Oct 6 00:23 #51433
-rw——- 1 root root 63 Oct 6 00:23 #51434
srwxr-xr-x 1 root root 0 Oct 6 13:00 #51436
srwxrwxrwx 1 root root 0 Oct 6 00:23 #51437
srwx—— 1 root root 0 Oct 6 00:23 #51438
-rw——- 1 root root 63 Oct 6 13:00 #51439
srwxrwxrwx 1 root root 0 Oct 6 13:00 #51440
srwx—— 1 root root 0 Oct 6 13:00 #51442
-rw——- 1 root root 63 Oct 6 23:09 #51443
srwx—— 1 root root 0 Oct 6 10:40 #51445
srwxrwxrwx 1 root root 0 Oct 6 23:09 #51446
srwx—— 1 root root 0 Oct 6 23:09 #51448

This is from a programmers point of view.  Gets into nice details about why this might not have been a good fit for Apples needs.  Click here to read more.

A very simple question.

Where is the “Enterprise Filesystem” licensed in a pure BSD way?

I know ZFS has been implemented by OpenSolaris (well yeah…that’s the origin of ZFS) as well as FreeBSD but the CDDL is not without critique in the BSD community. The OpenBSD people consider the CDDL even more restrivtive than GNU/GPL.

So a question to all the guys and girls who are able to create a filesystem (I can’t my coding skills are nowhere near to be able to create one), where is the BSD licensed filesystem with all the new features that make a hype?

I’m thinking about stuff like this (thou maybe not complete, somebody may have an absolutely usefull new idea that obsoletes all these features

• Snapshots (BTRFS, ZFS)
  • Continous Snapshots (Versioning File System) (NILFS)
• COW (Copy on Write) (BTRFS, ZFS)
• Volume Management built-in (BTRFS, ZFS)
• RAID (like) features built-in the filesystem (BTRFS, ZFS)
• Really large maximum filesizes and^*
• Really large maximum filesystem sizes^*

^*This may not be a Problem now, but I do get why Sun chose to make ZFS a 128-bit filesystem

Le direttive presenti in un file di configurazione possono essere applicate a tutto il server, oppure ristrette a certe directory, file, host o URL. In questo articolo verranno descritte le sezioni , come utilizzarle al meglio per personalizzare e rendere efficae la configurazione del server Web Apache.

Ecco un primo elenco di tali sezioni:

• IfDefine , racchiude direttive che vengono considerate solo nel caso in cui il comando apache2 viene lanciato con l’appropriato parametro.

  Esempio:

<IfDefine ClosedForNow>
Redirect / http://otherserver.example.com/
</IfDefine>

In questo caso la re direzione ha effetto solo se apache viene avviato con:

apache2ctl -DclosedForNow

• IfModule, è molto simile alla sezione precedente, l’unica differenza è che viene applicata solo nel caso in cui il modulo specificato è presente sul server

• IfVersion, viene considerata solo nel caso in cui la versione del server Web coincide con il parametro.

  Esempio:

  • <IfVersion >= 2.1>
# Si applica solo la versione è superiore alla 2.1
</IfVersion>

Le sezioni maggiormente utilizzate sono quelle che modificano il comportamento del web server in base alla posizione relativa al filesystem o allo spazio web. Prima è però necessario comprendere la differenza fra questi due concetti:

1. Filesystem

   In questo caso si considera una vista del server web sul filesystem, così come viene effettuata dal sistema operativo

2. Spazio Web

   In questo caso si considera solo la porzione di disco visibile ad apache, ovvero dalla sua root in poi. In Ubuntu solitamente dalla cartella /var/www

La differenza più importante fra queste due categorie di viste è che lo spazio web non è mappato direttamente sul file system, ma è in grado di gestire pagine create dinamicamente, per esempio estratte da un database.

Parliamo ora di Sezioni relative al Filesystem

• Directory, le direttive contenute si applicano a quella directory presente su filesystem.

  Esempio:

  • <Directory /var/web/dir1>
Options +Indexes
</Directory>

• Files, in questo caso le direttive si applicano a ogni singolo file con nome congruente (attraverso tutte le directory). Questo comando può essere combinato con quello precedente, in modo tale da applicare la/e direttiva solo a determinati file contenuti in una determinata directory

  Esempio:

  • <Directory /var/web/dir1>
<Files private.html>
Order allow,deny
Deny from all
</Files>
</Directory>

Per quanto riguarda le Sezioni relativa allo Spazio Web

• Location, le direttive contenute in questa sezione si applicano a tutti gli url che coincideranno con quello spazio web. Nell'esempio mostrato poco sotto qualsiasi pagina richiamata tramite URL che contiene /private non verrà visualizzata (http://mysite/private e http://mysite/private123/dir/ciao.html )

  Esempio

  • <Location /private>
Order Allow,Deny
Deny from all
</Location>

In combinazione con le sezioni Location, Directory e Files si possono utilizzare delle wildcard simili a quelle utilizzabili via shell per indicare:

• * → qualsiasi sequenza di caratteri

• ? → ogni singolo carattere

• etc ..

Oppure si possono utilizzare le più flessibili: DirectoryMatch, FilesMatch e LocationMatch che permettono di utilizzare espressioni regolari stile Perl.

La scelta fra un a sezione a livello di filesystem ed una a livello di spazio web è semplice, si deve scegliere la prima opzione nel caso in cui si vogliano gestire delle direttive legate al filesystem (oscurare una directory fisica), utilizzare la seconda quando si vuole gestire qualcosa che non è legato al filesystem (come già detto una pagina generata dinamicamente).

Altre sezioni sono:

• VirtualHost, sono direttive applicate a specifici host. Tale sezione viene utilizzata quando su un unico server Web risiedono più siti. Ritorneremo in seguito su questa sezione, dato che ha una certa rilevanza in serer multi dominio.

• Proxy, tale sezione applica le proprie direttive solo verso quei siti acceduti tramite il modulo mod_proxy , che deve matchare l'URL fornita, ecco un esempio:

  • <Proxy http://mysite.org/*>
Order allow,deny
Deny from all
</Proxy>

In ultima analisi affrontiamo lo scope delle sezioni, ovvero quali direttive hanno precedenza?

Quali vengono applicate in caso di conflitto?

Ecco una lista ordinata per grado di visibilità crescente (dall'alto verso il basso):

1. Directory e .htaccess

2. DirectoryMacth

3. Files e FilesMatch

4. Location e LocationMatch

Tutte le direttive vengono processate così come sono ordinate nel file di configurazione, tranne quelle presenti nelle sezioni Directory, che vengono applicate in base alla profondità dell'albero, ovvero prima <Directory /var/web/a> poi <Directory /var/web/a/suba>

Le considerazioni sullo scope non sono banali e sono da comprendere a fondo, una errata configurazione può compromettere il livello di sicurezza di apache, eccone un esempio:

<Location />
Order deny,allow
Allow from all
</Location>

# Attenzione , questa directory non ha effetto, per ciò che sta scritto sopra!!
<Directory />
Order allow,deny
Allow from all
Deny from badguy.example.com
</Directory>

Tutti avranno accesso alla root del mio server anche chi proviene da badguy.example.com

]]> http://madhavkobal.wordpress.com/2009/10/08/10-reasons-you-need-to-look-at-zfs/ Thu, 08 Oct 2009 17:51:29 +0000 madhavkobal http://madhavkobal.wordpress.com/2009/10/08/10-reasons-you-need-to-look-at-zfs/

Sun’s open-source ZFS file system has some amazing features. It was originally designed for Solaris and unveiled in 2005, but you’ll also find it in OpenSolaris and related distributions. In the future it may well become a popular file system to run with Linux and BSD as well.

Here are the top 10 features that ZFS fans find insanely great:

1. Checksums in Metadata for Data Integrity

Data integrity is of paramount importance in ZFS, and is the driver for many ZFS features.

The file system uses a 256-bit checksum, which is stored as metadata separate from the data it relates to, when it writes information to disk. Unlike a simple disk block checksum, this can detect phantom writes, misdirected reads and writes, DMA parity errors, driver bugs and accidental overwrites as well as traditional “bit rot.”

2. Copy on Write

ZFS ensures that data is always consistent on the disk using a number of techniques, including copy-on-write. What this means is that when data is changed it is not overwritten — it is always written to a new block and checksummed before pointers to the data are changed. The old data may be retained, creating snapshots of the file system through time as changes are made. File writes using ZFS are transactional — either everything or nothing is written to disk.

3. Data Snapshots With Time Slider

The latest version of OpenSolaris illustrates the power for ZFS’s snapshots capability with a small graphical application called TimeSlider. ZFS can be configured to take a snapshot of the file system (or a section of it, such as just a user’s home folder) on a regular basis — every 15 minutes, or every hour, and so on. These snapshots are very small and efficient, as only the deltas from the previous snapshot are stored.

TimeSlider offers a view of the file system (or a home folder), with a slider which can be moved back along a timeline to earlier snapshot times. As this is done, the view changes to show they state of the file system or the contents of a folder at the corresponding snapshot time. Recovering a file that has been overwritten by mistake or rolling back the system after an unsuccessful update is then just a matter of moving the slider back to the appropriate snapshot time.

Snapshots can also be made writable to create clones of existing file systems.

4. Pooled Data Storage

ZFS takes available storage drives and pools them together as a single resource, called a zpool. This can be optimized for capacity, or I/O performance, or redundancy, using striping, mirroring or some form of RAID. If more storage is needed then more drives can simply be added to the zpool — ZFS sees the new capacity and starts using it automatically, balancing I/O and maximizing throughput.

5. RAIDZ and RAIDZ2

RAID 5 has a well known flaw called the RAID 5 write hole. This causes a problem when a data block is written to a stripe but a power failure occurs before the corresponding parity block can be written. As a result the data and parity for the stripe will be inconsistent. If a disk then fails, the RAID reconstruction process will result in incorrect data. The only way out of this is if an entire stripe happens to be overwritten, thus generating a correct parity block.

RAIDZ gets around this problem by using a variable width stripe, so every write is effectively a full stripe write. This, together with ZFS’s copy on write characteristic (mentioned above,) eliminates the RAID 5 write hole completely. RAIDZ2 works in a similar way, but can tolerate the loss of two disks in the array using double parity.

Setting up a RAIDZ (or RAIDZ2) array is very easy and involves issuing one command

6. SSD Hybrid Storage Pools

High performance SSDs can be added to a storage pool to create a hybrid storage pool. When these are configured as high performance cache disks, ZFS uses them to hold frequently accessed data to improve performance. It also uses a technology called L2 ARC (adaptive replacement cache) to write data that has to be stored immediately. This can slowly be moved over to conventional hard drives for more permanent storage when time and resources allow.

7. Capacity

ZFS is a 128-bit file system, which means that in theory it could store 256 quadrillion ZB (a ZB is a billion TB.) In practice this is larger than would ever be necessary, for the foreseeable future at least.

8. Data Scrubbing

ZFS can be made to scrub all the data in a storage pool, checking each piece of data with its corresponding checksum to verify its integrity, detect any silent data corruption, and to correct any errors in encounters where possible.

When the data is stored in a redundant fashion — in a mirrored or RAID-type array — it can correct any corrupt data it detects invisibly and without any administrator intervention. Since data corruption is logged, ZFS can bring to light defects in memory modules (or other hardware) which cause data to be stored on hard drives incorrectly.

Scrubbing is given low I/O priority so that it has a minimal effect on system performance, and can operate while the storage pool that is being scrubbed is in use.

9 . Simple, Efficient Administration

Using ZFS commands you can administer a system with short, efficient commands. For example, a five disk RAIDZ array could be set up with the single command:

zpool create poolname raidz c1t0d0 c2t0d0 c3t0d0 c4t0d0 c5t0d0

10. More on the Way

ZFS is still evolving, and new features will appear regularly. The roadmap for 2009 includes encryption for increased security, and data deduping to increase storage efficiency.

If you are interested in trying ZFS out then the easiest way is to get started is to download OpenSolaris or related distro from http://opensolaris.org/os/downloads/. There is also a ZFS for FUSE/Linux project, and you can watch ZFS: A hands on tutorial.

Paul Rubens

Depois de há tempos ter oferecido a mim mesmo a minha prenda de Natal antecipada, o Egreat 34A NMT, uma fabulosa caixinha a correr GNU/Linux que permite tudo isto….

Hoje quando vou a copiar os dois primeiros episódios do Stargate Universe, qual não é o meu espanto quando a pen usb de 4Gb que costumo usar, apesar do $mount dizer que a tinha tentado e teria montado como Read/Write, ao copiar ficheiros para ela, recebo erro a indicar que esta está Read Only.

Experimento fazer #mount -o remount,rw /dev/sda1 para ver o que acontece e ela é novamente montada como Read/Write mas continua a dar o mesmo erro, não se consegue escrever voltando a ficar Read only.

Dei olhada ao #less /var/log/messages e realmente havia lá qualquer coisa que não batia certo…

#tail -f /var/log/messages
Oct  7 14:42:25 solaris kernel: [17476.557699] lost page write due to I/O error on sda1 Oct  7 14:42:25 solaris kernel: [17476.557705] lost page write due to I/O error on sda1
Oct  7 14:42:25 solaris kernel: [17476.557711] lost page write due to I/O error on sda1 Oct  7 14:42:25 solaris kernel: [17476.557718] lost page write due to I/O error on sda1
Oct  7 14:42:25 solaris kernel: [17476.557724] lost page write due to I/O error on sda1 Oct  7 14:42:25 solaris kernel: [17476.557732] lost page write due to I/O error on sda1
Oct  7 14:42:25 solaris kernel: [17476.557738] lost page write due to I/O error on sda1 Oct  7 14:42:26 solaris kernel: [17477.703770] sd 1:0:0:0: [sda] 8060927 512-byte hardware sectors: (4.12 GB/3.84 GiB)

A pen já vinha formatada com VFAT e nunca até hoje havia modificado nada, mas parece que está na altura de colocar lá um filesystem a sério, #aptitude update&&aptitude install gparted e segue a venerável ext2 para bingo!

Também se poderia ir pelo caminho mais rápido e leve da CLI, linha de comandos.

#fdisk -l
# mkfs.ext2 /dev/sda1

#e2label /dev/sda1 pen4gb
#chown -R username:username /media/pen4gb

Numa googlada rápida há quem diga que isto pode acontecer devido a limitações do filesystem VFAT da microsoft….

Powered by ScribeFire.

sudo fdisk -ls

In Mac OS X 10.6 Snow Leopard, Apple stopped using creator codes to decide with which application to open a file. (For those who don’t know, creator codes are four-character strings stored in the metadata of files on the Mac OS.)

I don’t know much about how they work, but to me the idea of deciding with which application to open a file based on the file’s metadata seems intrinsically flawed.

A User Preference

The classic argument for creator metadata goes something like this: Person creates file in Photoshop. Person saves file. The next day, Person opens file again. It opens in Photoshop, because they presumably want to continue working on the Photoshop file.

This argument works fine, as long as Person opens the document. If Leprechaun opens the file, it is completely unreasonable to assume they want to open it in slow heavy Photoshop instead of fast light Preview. It would be like forcing someone to get a pen before opening a letter.

Of course, some people say that if you wanted to preview the file, you would press Space to activate Quick Look. Well, you might. However, as a user, I reserve the right to not behave in a completely predictable manner. Additionally, QuickLooking is impossible if the file type in question isn’t supported.

(Now is a good time to reiterate that I do not know all the details of how creator codes worked. And don’t complain to me about split infinitives. I like split infinitives.)

Creator, Editor, Viewer

Creator codes identify the application which created the file. So imagine if somebody was doing a piece of art on their computer using a photograph as a starting point. It would be perfectly reasonable to copy the file to the desired location, open it in Photoshop and start working on it.

At the end of the day, they close the file, ready to resume working on it in the morning. The next day, they double-click the file, expecting it to open in Photoshop. But instead it opens in Preview, because it doesn’t have the required creator metadata. Photoshop didn’t create the file, it only edited it, so it falls back on the default application for public.jpeg.

As far as I can see, any preference for opening files which is tied to the file, not the user, is broken. The Snow Leopard solution (i.e. nothing) isn’t very good, and I do think that file extensions are icky. However, this is and must always be a user preference.

I have no solution. I just have the problem.

If you have played around with any 2 of the 3 major operating systems (Windows, OS X and Linux) and start transferring files to and from external hard disk you will probably run into an issue with the type of filesystem you choose for your hard disk. Windows likes NTFS, OS X likes HFS and Linux likes ext. As you can see none of them are the same. Being different is not so much an issue, but being compatible and accessible to all is.

Based on default system settings:

• NTFS is readable on all operating systems, but not writable on OS X. Most modern Linux distributions can write to NTFS drives
• HFS is readable on OS X and modern Linux distributions, and not writable on Windows or Linux
• ext is only readable on Linux and not writable on Windows or OS X.

or to put it based on operating system

• Windows can only read and write to NTFS, nothing else
• OS X can read and write to HFS and read NTFS
• Linux can read and write to ext and NTFS and read HFS

Take note that there is also the older FAT32 filesystem that is fully supported for reading and writing by all operating systems but due to limitations of FAT32, I rather not consider this. Basically the main issues with FAT32 is that the maximum file size is 4GB and the maximum partition size is 32GB. If these are limitations you can deal with, for the sake for interoperability stick with FAT32.

Now to solve the problem of support for each filesystem in each operating system:

NTFS:

• OS X: NTFS-3G + MacFUSE
• Linux: NTFS-3G
• I’ve been using NTFS-3G in Linux for many years and haven’t had any problems with it and so far it’s working well with OS X too
• On another note if the NTFS drive is not unmounted properly or there are some issues with the file system integrity, it is necessary to use Windows scan disk to rectify the problem. Thus this requires a copy of Windows to fix the filesystem.

ext:

• Windows: Ext2 Installable File System for Windows
• OS X: Mac OS X Ext2 filesystem
• I’ve had issues of only being able to mount an ext2 partition in Linux and it gave a mount error in Windows and OS X and was due to an inode issue as new Linux distributions create the file system with inodes of 256 bytes but Ext2 fs only supports the older version with 128 bytes. And the only solution is to back up the files, and reformat partition with inodes of 128 bytes (-I 128) and restore the files.
• Filesystem integrity issues should be able to be fixed with “fsck” from a Linux distribution / live CD. The great thing about this is that you can get a Linux distribution for free and this recovery can be done with out any strings attached.

HFS:

• Windows: MacDrive (US$50, read and write), Paragon HFS for Windows (read only)
• Linux: Enabling HFS writing in Ubuntu
• Note: I have not personally tested these so I cannot give first hand experience of how well it works or what issues can be had with this.
• I believe that HFS+ journal
• I would believe any filesystem repairs would have to be done in OS X (similar to NTFS and Windows) and if so this enforces that you have OS X at hand, and in order to have OS X you must have Apple hardware or a Hackintosh either way this is very restrictive.

So it is pretty easy to get full read and write support of all 3 default file systems on the 3 major OS’s but there are issues. So far I’m inclined to stick with ext2/ext3 just due to the fact that it has no restrictions in terms of filesystem repair. I’ve had many NTFS issues related to damaged filesystems that required Windows and the inconvenience of taking the drive out of my box to find a Windows box was too much.

An sich hat man ja keinen Grund NTFS am Mac zu benutzen, gibt es doch weitaus bessere Filesysteme.

Wenn man allerdings Festplatten mit Windows Nutzern teilen möchte, und nicht von der 4GB Grenze eingeschränkt werden möchte, dann führt kaum ein weg daran vorbei. Vor allem haben ja so gut wie alle XP und Vista Installationen NTFS laufen und wenn man mal auf die Windows Festplatte zugreifen muss, bleibt einem keine andere Möglichkeit also um NTFS zu nutzen. Also vor allem beim Umstieg von Windows auf Mac sehr brauchbar um seine Daten rüber zu kopieren.

Nun ist es kein Problem das NTFS nicht beim Mac OS mitgeliefert wird, denn es gibt gleich diverse funktionierende Lösungen.

Snow Leopard NTFS

Die einfachste Möglichkeit für Snow Leopard Nutzer ist es die mitgelieferte NTFS Unterstützung zu aktivieren. Also kommt doch NTFS gleich mit dem aktuellen Mac OS mit, ist aber wohl aus lizenztechnischen oder Support-Gründen deaktivert.

Um die Schreibfunktion zu aktivieren bedarf es einiger Änderungen im Mountscript. Mac Attack bietet diese gleich fertig als Download an:

http://macattack.speedesign.de/2009/09/20/ntfs-unter-snow-leopard-ohne-zusatzprogramme/

NTFS-3G

NTFS-3G ist der NTFS Treiber für das Fuse: Filesystem in Userspace – Framework. Dieses Framework ermöglicht es verschiedene Filesysteme einfach in ein System einzubinden und es wurde unter Linux ins Leben gerufen. So kann man neben NTFS diverse Filesystem Treiber herunter laden und einbinden. Bei vorheringen NTFS-3G Versionen musste man erst Fuse und dann einzeln den NTFS Treiber installieren. Die aktuelle Version nimmt einem diese Arbeit ab und installiert das aktuelle Fuse gleich mit. Im Gegensatz zur ersten Lösung läuft diese und die folgenden nicht nur unter Snow Leopard.

http://www.ntfs-3g.org/

http://macntfs-3g.blogspot.com/

http://code.google.com/p/macfuse/

NTFS-3G kommerziell

Von Tuxera gibt es auch eine kommerzielle Variante von NTFS-3G. Diese soll laut Internetseite ab September auch für den Mac erhältlich werden. Was dann genau die Vorteile/Unterschiede sind kann ich euch jetzt auch nicht sagen. Vermutlich schon mal kommerzieller Support.

http://www.tuxera.com/products/tuxera-ntfs-for-mac/

Paragon NTFS für Mac

Paragon mit NTFS für Mac ein kommerzielles Komplettpaket an. Das Paket kostet 29,95 €.

http://www.paragon-software.com/de/home/ntfs-mac/

Alle Pakete sollen brauchbar und einwandfrei laufen. Selbst laufen habe ich nur NTFS 3G, weil dieses nie Probleme bereitet hat und ich seit dem Umzug von Windows auf Mac auch kein NTFS Datenträger mehr genutzt habe am Mac. In einigen Foren habe ich allerdings über eventuelle Probleme bei professionellen oder intensiven Anwendungen gelesen. Aber dazu kann ich so schlecht etwas sagen, und ich würde sowieso keine NTFS Festplatte als Produktivplatte am Mac einsetzen. Dazu eignen sich die Mac eigenen Filesysteme sicher besser.

Prinzipiell hatte ich NTFS schon von Anfang an auf meinem Mac laufen. Auf die Idee gebracht doch noch drüber zu Bloggen hat mich ein entsprechender Beitrag bei OS X Freeware. Danke dafür.

The FAT file system (FAT FS) enables the inserting developer from Zeeis to easily and integrates MS-DOS or the window compatible FAT file system using the nature mutually and all main operating system all equipment rapidly. It tested the use in several true time operating system and the canned food not to have RTOS. This high performance source code was the minimum memory consumption in RAM and ROM optimal, was cheap and is high speed.
In 2008, the Zeeis FAT file system product shipped the completely successful 25,000,000 equipment, is the leading filing system product for mobile, the electrical appliances and the storage application.