Welcome to part 1 of my blog series, “Birth of a cloud” (and you thought these came from the cabbage patch!). In these posts, I’ll be taking you from the inception and planning of the data center cloud to an important mature component of the green data center.

For some time now, we’ve operated internal use and customer virtual machines using the excellent Xen hypervisor. As good as the standalone virtualization has been, it’s time to step it up a notch and take advantage of all the capabilities of a cloud, so we’re using the new Xen Cloud Platform as a good foundation.

For this project, we’re using Rackable Systems servers with 2x AMD Opteron 275 dual core CPUs, 8Gb memory, and 2GbE ports. These aren’t state of the art by any means, but they’ve been doing yeoman duty as standalone hosts and they’re rock solid hardware (we only use Linux virtual machines so there’s no need for hardware virtualization support).

We’re starting off with 6 cloud hosts but we’ve laid out the network for the full 16 cloud hosts that the Xen Cloud Platform we’re using is capable of (figuring 6 core Opterons, that’s 96 cores). The network uses a single switch, split in two networks, one for data traffic (all port 1s) and one for storage traffic (all port 2s).

Storage for the cloud is a common NFS server. This allows us to take advantage of XCPs workload balancer and the ability to shift live virtual machines between cloud hosts seamlessly.

The cloud is also integrated into the data center DC power distribution. Cloud hosts are split between separate DC power feeds which, with. the addition of XCPs work load balancer, will insure maximum reliability.

This is a basic overview of the hardware infrastructure to support a virtual machine cloud. In part 2, I’ll talk about establishing the cloud and bringing it up under manual control. In part 3, I’ll talk about using the work load balancer for automatic control of the cloud and finally, in part 4, I’ll talk about advanced projects that provide the ultimate in green results, such as the effort to allow the cloud to power down unneeded machines when demand is low. Stay tuned!

Vern, SwiftWater Telecom

We are migrating our Asterisk test environment from VMware to Xen.  We always used the free VMware (VMware Server) which worked fine other than one huge limitation: There was no way for our virtual machines to access telephony cards (eg: Sangoma or Digium T1 pci cards).  Xen provides that feature and more by allowing your virtual machines to access pretty much any device you can access from within a regular linux installation.

Here are the steps I took to add Xen to my CentOS test server and fire up my first asterisk guest complete with its own digium pri card.  For my first asterisk guest i created a server with all the latest and greatest asterisk software and appropriately named it bleeder.

1. Install Xen – This step turns your current server into the Dom0, or “the special privileged domain that is used to administer normal guest domains”

yum install xen kernel-xen
modify /boot/grub/grub.conf to boot the xen kernel by default
reboot

2. Download a 1G partition image that contains an ext3 filesystem with the linux kernel and a minimal set of CentOS packages and extend it to 4G.  Stacklet.com is a great resource for these.  I found the CentOS image i was looking for here.  This comes with the partition image and two example config files.  I copied the partition image to a directory i could remember where i had sufficient space and extended it using the following commands.  The first command (dd) extends the actual file.  This is analogous to extending a partition on a disk.

[root@fsdev images]# dd bs=1k seek=4096k count=1 if=/dev/zero of=centos.5.3.x86.img

Note that this command produces a “sparse” file. That means that though the file can grow up to a total size of 4GB, it only takes up the space on disk that it is actually using.  Here’s the proof:

[root@fsdev images]# ls -lh
total 2.1G
-rw-r--r-- 1 k3leland k3leland  4.1G Nov 15 19:33 centos.5-3.x86.img
[root@fsdev images]# du -h centos.5-3.x86.img
1.1G  centos.5-3.x86.img

The rest of the commands are to resize the actual ext3 filesystem.  Find the first available loopback device and assign our partition image to it:

[root@fsdev images]# losetup -f
/dev/loop0
[root@fsdev images]# losetup /dev/loop0 centos.5-3.x86.img

Now Resize!

[root@fsdev images]# resize2fs /dev/loop0
resize2fs 1.39 (29-May-2006)
Please run 'e2fsck -f /dev/loop0' first.

[root@fsdev images]# e2fsck -f /dev/loop0
...output omitted...
[root@fsdev images]# resize2fs /dev/loop0 4G
resize2fs 1.39 (29-May-2006)
Resizing the filesystem on /dev/loop0 to 1048576 (4k) blocks.
The filesystem on /dev/loop0 is now 1048576 blocks long.

Lastly unassign our file to the loopback device:

[root@fsdev images]# losetup -d /dev/loop0

Here is an extra command that may come in handy when dealing with partition images. This allows you to mount the image and explore the contents:

[root@fsdev images]# mount -o loop -t ext3 centos.5-3.x86.img /mnt
[root@fsdev images]# ls /mnt/
bin   dev  home  lost+found  mnt  poweroff  root  selinux  sys  usr
boot  etc  lib   media       opt  proc      sbin  srv      tmp  var

3. Copy the pygrub config to /etc/xen/bleeder and make some slight modifcations to it.  I gave it its very own ip address and indicated i wanted it bridged at the ethernet level by adding this line:

vif = [ 'ip=xx.xx.xx.xx, bridge=xenbr0' ]

I updated the disk directive to reflect the new location that i moved the image to:

disk = ['file:/var/asterisk/xen/images/centos.5-3.x86.img,sda1,w']

And lastly I assigned the virtual machine the digium T1 card using the pci directive.  Note: I found the pci identifier in the output of lspci.

pci=['0a:01.0']

4. Unbind the pri card from my Dom0 so that i will be able to boot my bleeder guest by loading the pciback kernel module with the hide option:

modprobe pciback 'hide=(0a:01.0)'

If this worked you will see some messages in dmesg:

pciback 0000:0a:01.0: seizing device
PCI: Enabling device 0000:0a:01.0 (0310 -> 0313)
ACPI: PCI Interrupt 0000:0a:01.0[A] -> GSI 20 (level, low) -> IRQ 21
ACPI: PCI interrupt for device 0000:0a:01.0 disabled

If this command doesn’t work it will not give any indication other than you will not see those messages in dmesg. This command will not work if you currently have a driver loaded that is binded to that device. That means you may have to:

service zaptel/dahdi stop

and try again:

rmmod pciback
modprobe pciback 'hide=(0a:01.0)'

5. Now we are ready to boot our guest for the first time using the xm command.  The -c option indicates we want a console on the new vm, and the name of the vm, bleeder, corresponds to the name of the config file in /etc/xen.  If everything works you will eventually get a login prompt (root/password by default)

[root@fsdev ~]# xm create -c bleeder
Using config file "/etc/xen/bleeder".

One error that prevented booting for the first time was:

Error: pci: improper device assignment specified: pci: 0000:0f:00.0 must be co-assigned to the same guest with 0000:0f:06.0, but it is not owned by pciback.

I resolved this by moving my pci card to a different slot in the system. I believe this is a limitation in Xen but if you have the flexibility to move your card it beats waiting for a bug fix;)  Here is the output of lspci and lspci -t before and after the move:

[k3leland@Dom0 ~]$ lspci | grep Digium
0f:06.0 Ethernet controller: Digium, Inc. Wildcard TE120P single-span T1/E1/J1 card (rev 11)
[k3leland@Dom0 ~]$ lspci -t
-[0000:00]-+-00.0
           +-01.0-[0000:01]--
           +-1c.0-[0000:09-0a]--+-00.0-[0000:0a]--
           |                    \-00.1
           +-1c.4-[0000:0d]----00.0
           +-1c.5-[0000:0e]----00.0
           +-1d.0
           +-1d.1
           +-1d.2
           +-1d.3
           +-1d.7
           +-1e.0-[0000:0f]--+-00.0
           |                 \-06.0
           +-1f.0
           +-1f.1
           +-1f.2
           \-1f.3

[k3leland@Dom0 ~]$ lspci | grep Digium
0a:01.0 Ethernet controller: Digium, Inc. Wildcard TE120P single-span T1/E1/J1 card (rev 11)
[k3leland@fsdev ~]$ lspci -t
-[0000:00]-+-00.0
           +-01.0-[0000:01]--
           +-1c.0-[0000:09-0a]--+-00.0-[0000:0a]----01.0
           |                    \-00.1
           +-1c.4-[0000:0d]----00.0
           +-1c.5-[0000:0e]----00.0
           +-1d.0
           +-1d.1
           +-1d.2
           +-1d.3
           +-1d.7
           +-1e.0-[0000:0f]----00.0
           +-1f.0
           +-1f.1
           \-1f.3

6. Now we can proceed to configure and install software on our new virtual guest as it was a physical server. The output of lspci in the guest should produce a single line with our digium T1 card. The system we got from Stacklet.com provides the bare minimum so the first step is to bring it up to date and install the prerequisite tools:

yum update
yum install vim-enhanced gcc gcc-c++ subversion make kernel-xen-devel libtermcap-devel

Checkout Asterisk and Friends from svn and make and make install asterisk, dahdi-linux, dahdi-tools, libpri.

Deviations from boring ‘configure;make;make install’ routine included a couple of things:

When configuring asterisk i chose to disable xmldoc:

./configure –disable-xmldoc

After making and installing dahdi-linux be sure to modify /etc/dahdi/* to your needs, or use dahdi_genconf.

TODO in Future:

• test performance of virtualized asterisk
• install asterisk and dahdi-linux on vm from binaries using asterisk xen pacakges at packages.asterisk.org.
• install thirdlane, trixbox, and asterisknow a xen guests

Recommended book and primary reference for this project:

Virt-df works on Xen so you can display free disk space in a Xen VM.

# virt-df -h 2>&1 | grep -v '/dev/kqemu'
Domain-0 seems to have no disk devices
Filesystem                                Size       Used  Available  Use%
RHEL39FV64:/dev/hda1                     98.7M      13.5M      80.1M 18.8%
RHEL39FV64:/dev/hda2                      6.8G       1.8G       4.7G 31.0%
RHEL39FV64:/dev/hdb1                     30.4M       1.0M      27.8M  8.5%
RHEL39FV32:/dev/hda1                     98.7M      14.5M      79.1M 19.9%
RHEL39FV32:/dev/hda2                      6.8G       1.5G       4.9G 27.8%
RHEL39FV32:/dev/hdb1                     30.4M       1.0M      27.8M  8.5%
RHEL53PV64:/dev/VolGroup00/LogVol00       6.6G       2.1G       4.2G 37.3%
RHEL53PV64:/dev/hda1                     98.7M      13.1M      80.5M 18.5%
RHEL48PV64:/dev/VolGroup00/LogVol00       6.7G       2.1G       4.3G 36.6%
RHEL48PV64:/dev/hda1                     98.7M       9.2M      84.4M 14.5%
RHEL52PV32:/dev/VolGroup00/LogVol00       6.6G       2.5G       3.7G 43.5%
RHEL52PV32:/dev/hda1                     98.7M      12.7M      80.9M 18.0%
RHEL53PV32:/dev/VolGroup00/LogVol00       6.6G       1.9G       4.4G 34.1%
RHEL53PV32:/dev/hda1                     98.7M      12.8M      80.8M 18.2%
RHEL52FV32:/dev/VolGroup00/LogVol00       6.7G       2.6G       3.8G 43.5%
RHEL52FV32:/dev/hda1                     98.7M      14.5M      79.1M 19.8%
RHEL52FV32:/dev/hdb1                     30.3M       1.4M      27.4M  9.7%
RHEL48PV32:/dev/VolGroup00/LogVol00       6.7G       1.8G       4.6G 31.7%
RHEL48PV32:/dev/hda1                     98.7M       8.6M      85.0M 13.9%
RHEL47FV32:/dev/VolGroup00/LogVol00       6.7G       1.9G       4.5G 32.6%
RHEL47FV32:/dev/hda1                     98.7M      13.9M      79.7M 19.2%
RHEL47PV32:/dev/VolGroup00/LogVol00       6.7G       1.8G       4.6G 31.3%
RHEL47PV32:/dev/hda1                     98.7M       8.6M      85.0M 13.9%

It’s a bit noisy giving a warning about opening /dev/kqemu for each guest. I’ve grepped out the warnings above. The “Domain-0 seems to have no disk devices” warning is bug 538041.

Ebbene, dopo essere riuscito ad ottenere una macchina para-virtualizzata con XEN con il supporto per le Huge Pages come descritto nel mio post precedente, era arrivato il momento di verificare se effettivamente c’erano problemi con questa accoppiata.

Confesso che nella fretta come al solito ho voluto bruciare le tappe ed ho subito cercato di configurare le Huge pages e di provare a far partire il database Oracle (sempre l’istanza 11gR2). La fretta è stata una cattiva consigliera come il solito, perché ho litigato un po’ con i parametri di configurazione della memoria di Oracle per far stare tutto nella RAM che ho a disposizione. Seguirò nella descrizione lo stesso ordine con cui ho eseguito le operazioni.

Configurazione del sistema operativo

Una volta “abilitato” il kernel all’utilizzo delle Huge Pages, bisogna configurare quanta parte di memoria debba essere dedicata  a queste pagine giganti, infatti andando a vedere il file /proc/meminfo si osserva:

[root@oel53test11gR201 ~]# cat /proc/meminfo |grep Huge
HugePages_Total:      0
HugePages_Free:       0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

per farlo in modo definitivo (si può fare anche al volo, ma siccome le pagine devo essere contigue potrebbe essere necessario un reboot affinché il sistema trovi sufficente memoria contigua libera) si modifica il file /etc/sysctl.con aggiungendo le righe:

vm.nr_hugepages = 420
vm.hugetlb_shm_group = 505

La prima riga corrisponde al numero di pagine, che moltiplicato per la dimensione della pagina (nel mio caso 2048K) la dimensione in byte della memoria che si vuole gestire in questo modo (in questo esempio 420*2048k=840M). La seconda riga è importante se si intende far utilizzare le Huge Pages ad oracle: indica il gruppo del sistema operativo che  può utilizzare le huge pages con le chiamate di sistema shmat/smget (Vedi vm/hugetlbpage.txt) Questo secondo la documentazione, ma vedremo dopo che a me questo non torna.

Secondo alcune note del Metalink Oracle, ad esempio la 560501.1, occorre modificare anche il file /etc/security/limits.conf aggiungendo le due righe:

oracle soft memlock n
oracle hard memlock n

Dove n è il numero di KByte che si assegnerà alla SGA (immagino sia il caso di mettere un numero un po’ più grande)

Modifcato i file si può fare un reboot e  dopo di ciò si ottiene:

[root@oel53test11gR201 ~]# cat /proc/meminfo |grep Huge
HugePages_Total:   420
HugePages_Free:    420
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Configurazione di Oracle

Stando alla nota Oracle Metalink 749851.1 l’impostazione “Automatic Memory Management (AMM)” di Oracle 11g non è compatibile con l’uso delle HugePages, quindi occorre settare a 0 i parametri MEMORY_TARGET e MEMORY_MAX_TARGET .  Ho quindi impostato i vari parametri SGA_TARGET, SGA_MAX_SIZE PGA_AGGREGATE_TARGET, con qualche difficoltà. Non ho verificato se l’incompatibilità è certificata anche per la R2 di 11g, in ogni caso questa incompatibilità spiega il perché nei manuali ufficiali non viene incoraggiato l’utilizzo delle Huge Pages.

Dopo aver aggiustato in più fasi tutti i parametri Oracle sono arrivato finalmente ad un risultato. Ad un certo punto sono arrivato al livello che sembrava tutto a posto ma da /proc/meminfo risultavano ancora tutte le pagine libere, ad indicare che Oracle non le usava: era perché il numero di pagine che avevo riservato ancora non bastavano per tutta la SGA, infatti aumentandolo da 405 a 420 ho risolto.

… risolto poco, perché il risultato del tentativo di avvio dell’istanza Oracle è stato un reboot della macchina virtuale con la visualizzazione sulla console di XEN di un bel “kernel panic”.

A questo punto ho fatto quello che avrei dovuto fare subito, ho provato le hugepages con un programmino di test, precisamente quello descritto in questo bel articolo.

Riporto quindi il messaggio in console:

Unable to handle kernel paging request at ffff88005cc122f0 RIP:
[<ffffffff802cd3ec>] hugetlb_no_page+0×1e3/0×27e
PGD 13cd067 PUD 15cf067 PMD 16b6067 PTE 801000005cc12065
Oops: 0003 [1] SMP
last sysfs file: /block/dm-0/range
CPU 0
Modules linked in: autofs4(U) hidp(U) nfs(U) lockd(U) fscache(U) nfs_acl(U) rfcomm(U) l2cap(U) bluetooth(U) sunrpc(U) dm_multipath(U) scsi_dh(U) scsi_mod(U) ipv6(U) xfrm_nalgo(U) crypto_api(U) parport_pc(U) lp(U) parport(U) pcspkr(U) xennet(U) dm_raid45(U) dm_message(U) dm_region_hash(U) dm_mem_cache(U) dm_snapshot(U) dm_zero(U) dm_mirror(U) dm_log(U) dm_mod(U) xenblk(U) ext3(U) jbd(U) uhci_hcd(U) ohci_hcd(U) ehci_hcd(U)
Pid: 3435, comm: oracle Tainted: G      2.6.18-2.6.18-164xenhugepages #1
RIP: e030:[<ffffffff802cd3ec>]  [<ffffffff802cd3ec>] hugetlb_no_page+0×1e3/0×27e
RSP: e02b:ffff880063017d18  EFLAGS: 00010246
RAX: ffff88005cc122f0 RBX: 0000000048c0d0e7 RCX: 0000000048c0d027
RDX: 0000000000000000 RSI: 0000000048c0d0e7 RDI: 000000008bdffe98
RBP: 0000000000000000 R08: ffffffff804fbfc3 R09: 0000000000000908
R10: 0000000000000000 R11: 0000000000000000 R12: ffff880002fe5000
R13: ffff8800a12e4160 R14: 000000008bdffe98 R15: ffff88005fb5ce58
FS:  00002abe89f78e60(0000) GS:ffffffff805ce000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000
Process oracle (pid: 3435, threadinfo ffff880063016000, task ffff880063ac87a0)
Stack:  00000001804f62c0  ffff88005cc122f0  ffff880009a95b80  0000000000000000
ffff88005cc122f0  ffff880009a95b80  000000008bdffe98  0000000000000001
ffff88005fb5ce58  ffffffff802cd4dc
Call Trace:
[<ffffffff802cd4dc>] hugetlb_fault+0×55/0xba
[<ffffffff80208b0c>] __handle_mm_fault+0×5c/0×1354
[<ffffffff80241249>] vma_prio_tree_insert+0×20/0×38
[<ffffffff8026766a>] do_page_fault+0xf7b/0×12e0
[<ffffffff8020e6fe>] do_mmap_pgoff+0×658/0×7c3
[<ffffffff80264901>] _spin_lock_irqsave+0×9/0×14
[<ffffffff80231992>] __up_write+0×27/0xf2
[<ffffffff8026082b>] error_exit+0×0/0×6e

Code: 48 89 18 83 7c 24 04 00 74 23 41 f6 47 28 08 75 1c 48 8b 4c
RIP  [<ffffffff802cd3ec>] hugetlb_no_page+0×1e3/0×27e
RSP <ffff880063017d18>
CR2: ffff88005cc122f0
<0>Kernel panic – not syncing: Fatal exception

Va detto, che l’errore non avviene nel momento in cui il programma alloca le pagine (con shmget) ma quando cerca di scriverci dentro. Ho anche provato ad eseguire il programmino di test dopo aver tolto dal file /etc/sysctl.conf la riga vm.hugetlb_shm_group = 505, in questo modo stando alla documentazione solo l’utente root dovrebbe essere in grado di utilizzare le Huge Pages, ma nel mio caso non è cambiato nulla, la macchina si riavviata con lo stesso errore.

Dopo giorni (ma in realtà erano mesi che volevo farlo) sono riuscito a ricompilare il kernel su Oracle Enterprise Linux 5.3. Si tratta di una installazione su una macchina virtuale XEN che ho fatto qualche settimana fa per installare e testare Oracle 11gR2. Una delle cose che ho notato gia tempo fa è che sia sulla macchina “ospitante” (Dom0 il linguaggio XEN) che sulle macchine para-virtualizzate linux non è configurato il supporto per le HugePages, mentre su una macchina fully-virtualised tale supporto è presente (la macchina fully virtualised mi era necessaria per installare un versione linux 32 bit, usata a suo tempo per testare 11gR1 quando la versione 64 bit non era ancora uscita).

Ho sempre pensato che se nel kernel compilato con il supporto XEN non c’era il supporto per le Huge Pages ci doveva essere un motivo ben preciso, qualche vincolo strutturale; purtroppo però a tutt’oggi non ho trovato una spiegazione di  cio’, non una nota dove si dica che le macchine virtuali non possono usare Huge Pages.

Un po’ di tempo fa ho scoperto, grazie al blog di Sergio Leunissen, che Oracle ha messo a disposizione un server YUM pubblico. Si tratta di un server da cui è possibile scaricare e installare in modo quasi automatico i pacchetti rpm aggiornati, utilizzando una utility chiamata appunto YUM. Chiaramente in questi aggiornamenti non sono inclusi le patch di sicurezza che vengono fornite solo con il supporto ULN. Da qui ho scaricato il pacchetto RPM dei sorgenti del kernel. Gia questa fase è stata per me problematica, perché con YUM non sono stato capace di farlo, ho scaricato l’rpm con wget e l’ho installato con rpm.

Una volta installati i sorgenti del kernel sono passato alla difficoltà successiva: come compilare il kernel? I pacchetti rpm con in sorgenti vengono installati con una struttura particolare, che non conoscevo molto, francamente io quando smanettavo di più con linux ero una amante della distrubuzione Slackware e della sua organizzazione a pacchetti tar.gz. In ogni caso ho scoperto l’utility rpmbuild, sulla quale però non è che la documentazione abbondi.

Fra mille ricerche il link più utile che ho trovato è stato senz’altro questo. In tutti però, compresi questo e questo manca una informazione chiara su come funziona la configurazione. Infatti io continuavo a modificare a mano il file .config aggiungendo le righe:

CONFIG_HUGETLBFS=y
CONFIG_HUGETLB_PAGE=y

Ma sistematicamente osservavo che il make me le “spianava”, cancellava le righe togliendo di fatto il supporto alle Huge Pages. Da sottolineare poi un’altra cosa che non capivo: facendo make menuconfig (e un paio di volte mi sono fatto anche il make config con l’interminabile sfilza di domande) non compariva nessuna voce relativa alle stesse huge pages. Solo grazie ad un errore nel file .config ho poi scoperto che a  “resettare” il mio file .config era l’istruzione:

scripts/kconfig/conf -s arch/x86_64/Kconfig

lanciata da make. Ulteriori indagini mi hanno portato al file fs/Kconfig dove ho trovato:

config HUGETLBFS
bool “HugeTLB file system support”
depends X86 || IA64 || PPC64 || SPARC64 || SUPERH || (S390 && 64BIT) \
|| BROKEN
depends !XEN
help
hugetlbfs is a filesystem backing for HugeTLB pages, based on
ramfs. For architectures that support it, say Y here and read
<file:Documentation/vm/hugetlbpage.txt> for details.

If unsure, say N.

Che in sostanza stabilisce che se è selezionato il supporto per XEN viene disabilitato quello per le Huge Pages. Ho verificato, con “make menuconfig” ma anche deselezionando la voce “XEN -> Privileged Guest (domain 0)” sotto “File Systems -> Pseudo filesystems” non compare la voce “HugeTLB file system support”. (Non sono solo io ad aver problemi su come scrivere “filesystem” se come una unica parola  o come due divise, visto che sul questo menu’ compare scritto in entrambi i modi). Invece modificando il file fs/Kconfig così:

config HUGETLBFS
bool “HugeTLB file system support”
depends X86 || IA64 || PPC64 || SPARC64 || SUPERH || (S390 && 64BIT) \
|| BROKEN
#depends !XEN
default y
help
hugetlbfs is a filesystem backing for HugeTLB pages, based on
ramfs. For architectures that support it, say Y here and read
<file:Documentation/vm/hugetlbpage.txt> for details.

If unsure, say N.

Magicamente la voce “HugeTLB file system support” compare sotto “HugeTLB file system support”

Quindi sono riuscito a ricompilare il kernel con il supporto a XEN e alle Huge Pages, vedremo cosa accadrà.

Questi in sintesi i passi che ho seguito:

1. scaricato il pacchetto  kernel-2.6.18-164.el5.src.rpm da public-yum.oracle.com
2. aggiunto l’utente mockbuild con gruppo mockbuild (altrimenti non funziona l’installazione dell’rpm).
3. installato con rpm -i kernel-2.6.18-164.el5.src.rpm; in questo modo finisce tutto sotto /usr/src/redhat, con una struttura particolare
4. Secondo il consiglio visto da più parti, ad esempio anche qui, ho copiato tutto il contenuto di /usr/src/redhat in un’altro posto (/var/tmp/rpmbuild) e creando il file  ~/.rpmmacros con la riga %_topdir /var/tmp/rpmbuild.
5. la radice per la compilazione dei sorgenti è diventata nel mio caso /var/tmp/rpmbuild/BUILD/kernel-2.6.18/linux-2.6.18.x86_64/, ho modificato il file Makefile impostando la variabile EXTRAVERSION = -xenhuge2.6.18-164  (come saggiamente indicato qui)
6. modificato il file fs/Kconfig come descritto sopra
7. lanciato il comando “make rpm ” La compilazione non è durata giorni, mi sembra circa un’ora.
8. a questo punto sotto /var/tmp/rpmbuild/RPMS ho trovato il mio rpm che ho installato con rpm -i, piccola curiosità: sotto /boot mi sono trovato un file vmlinuz da 52 MB contro i 2 MB circa degli altri kernel.
9. Ho creato l’immagine del ramdisk con mkinitrd, sempre come indicato dalla guida di HowToForge e modificato il file /boot/grub/menu.lst

Ho riavviato la macchina virtuale e selezionato dal menu di grup il nuovo kernel e incredibilmente è partita.  Curiosando sotto /var/tmp/rpmbuild/BUILD/kernel-2.6.18xenhuge2.6.18-164 ho visto un file vmlinuz da circa 2 MB, accanto a un file vmlinux da circa 52 MB (dimensione uguale al byte al file vmlinuz-2.6.18-xenhuge2.6.18-164 che rpm mi ha installato sotto /boot) allora ho provato a sostituire il file da 52 MB sotto boot con il file vmlinuz (opportunamente rinominato) ed ho riavviato. Anche così funziona. Evidentemente il file più piccolo è il kernel compresso ma al boot viene gestito in modo trasparente (indagherò su questo).

Conclusione

I motivi che mi hanno spinto a investire il mio tempo in queste ricerche ed esperimenti sono imperscrutabili. Un’operazione simile va contro il principio che cerco di adottare solitamente, ovvero seguire strade e metodi certificati. Solamente una morbosa curiosità, il fascino del lato oscuro di Linux giustificano cio’ che ho fatto. Sono però soddisfatto, perché potrò adesso testare Oracle su Linux con le Huge Pages.

Perkara install ulang memang gampang, tapi setting yang kita buat sebelum kita backup akan hilang, sedih banget khan, makanya sebelum diapa-apain dan masih berjalan normal sebaiknya konfigurasi yang kita buat di backup dulu baik lisensi, backup router maupun backup script yang kita buat, caranya?

- Backup Konfigurasi router.

[nama admin@nama router] /system backup save name=<buat nama file backup>

- Export  konfigurasi yang kita buat,  dapat dilihat disini.

- Backup License ID Key.

Export License ID menggunakan console winbox masuk ke menu System > License > Export Key.

Lalu bagaimana jika router yg kita tiba2 hang (error) atau kita tak sengaja memasukkkan pake yang ternyata tidak sesuai dengan spesifikasi hardware yang kita miliki, banyak kasus yang terjadi, kita menambahkan atau mengupgrade router kita dengan memasukkan XEN, ini menyebabkan router kita error dan harus di install ulang.

Berikut cara termudah yang harus dipersiapkan.

1. Komputer/PC + CD-ROM
2. DOM (Disk on Module) yang akan kita install ulang dan sudah tercolok ke IDE atau SATA pada mainboard.
3. Download ISO file disitus mikrotik.com atw mikrotik.co.id, ambil dan sesuaikan dengan komputer PC atau x86.
4. Kemudian burning ke cd kosong dan masukkan hasil burningan ke cd-rom komputer yg akan kita install dan atur pada bios cd-rom pada boot yang pertama.
5. Setelah masuk pada masuk instalasi pilih paket apa aja yg kita install, tekan “i” untuk proses instalasi dan DOM akan di hapus dan di install ulang. 
6. Setelah instalasi selesai dan tidak terjadi error masuk lagi ke menu System > License dan import file lisensi yang kita backup tadi, kemudia kita juga dapat mengembalikan konfigurasi yang telah kita backup tadi.

Kalau yang tidak di backup terpaksa di konfigurasi ulang router MikroTik anda.

Tautan situs : www.mikrotik.com, mikrotik.co.id

Today I woke up to several alerts from Linode informing me that one of my VPS nodes was exceeding the Disk I/O threshold that I had set. Curiously this VPS is used as a HTTP web proxy and whilst it gets about 300-400 visitors per day (mainly china) this morning I was seeing over 800 visitors in Google Analytics.

Attempting to ssh to the server failed with timeouts although the PHP web application was still responding to requests over HTTP fine. I suspect sshd was failing to reverse-lookup my IP address in any reasonable amount of time, or perhaps IP Tables – (Note to self: Look in to why that happened). Thankfully Linode provide out of band / console access via SSH and AJax so all was not lost.

Looking at the Network rrdgraph it shows that the server was approaching 7Mbit/s of HTTP traffic and almost 50GB had been consumed today alone. Whilst the server seemed to handle the load without problem (minus ssh access) consuming 50GB+ per day would quickly max out my monthly data transfer allowance with Linode – this wasn’t acceptable. I modified the firewall to accept HTTP/HTTPS traffic from my IP only in order to investigate and the load suddenly stopped and SSH was alive again.

Initially I had suspected that some sort of automated bot was using ehproxy.info to do automated scans and attacks but a closer inspection of the traffic showed an even number of distributed IPs (all from China – as Google Analytics confirms) all clicking various porn sites. I guess everyone in China was feeling horny this afternoon!

Further analysis of the access.log shows that the server (Linode XenU VPS with 720MB of ram) was handling 62 hits sec (2428863/39600) and lighttpd was dealing with the load no problem. Pretty good considering this is a pure PHP application utilising php-cgi.

For the record, the top five IP addresses were:

Hits : IP address

Una serie di interessanti post recenti di Dizwell sulla virtualizzazione hanno risvegliato in me l’interesse per queste tecnologie. In particolare egli ha iniziato con un bel post in cui faceva una sorta di recensione di sistemi di virtualizzazione, in seguito egli si è concentrato su quello apparentemente più interessante XenServer di Citrix.  Apro qui una parentesi, il sito di citrix su XenServer è http://www.citrix.com/xenserver/ ma io lo trovo frequentemente giu, una cosa preoccupante direi. Chiusa la parentesi.

XenServer sembra un prodotto interessante, non molto diverso come base da Oracle VM, ovvero una distribuzione Linux minimale; la differenza la fa il “contorno”, a partire dagli strumenti di amministrazione per arrivare alle caratteristiche “enterprise”, che nel caso di XenServer sono quelle che si comprano, mentre la base è gratuita.

L’ultimo post di Dizwel, è un po’ “amaro”, infatti egli ha incontrato dei limiti di XenServer che gli impediscono di  utilizzarlo in “produzione”, produzione tra virgolette perché il suo progetto è quello di consolidare gli ambienti di test. I suoi racconti mi hanno incuriosito e spinto a capire meglio come funziona Xen, a tale proposito voglio segnalare che fra i manuali di Red Hat ve ne è uno sulla virtualizzazione molto interessante, qui. Volevo segnalare questo manuale perché vi ho trovato cose che ho cercato anche tempo fa, come un metodo per generare nuovi mac-address o il sistema per accedere ai file all’interno delle immagini dei dischi.

Vi sono due dubbi che mi rimangono:

1) cosa sono i template di cui parla Dizwell; in effetti non mi è chiaro il meccanismo di creazione di una nuova VM, io utilizzo l’interfaccia grafica di virtual machine manager ma non ho ancora capito i meccanismi che stanno dietro. L’ultima VM che ho creata l’ho fatta con OEL 5.3 senza bisogno di “template”.

2)sulle vm che ho creato, di tipo “para-virtualised” il kernel xen non attivo il supporto per le HugePages, mi chiedo se questo sia dovuto al fatto che in questa configurazione non è possibile averle oppure se semplicemente è possibile ricompilare il kernel ed attivare tale supporto.

Sono convinto che conoscere a fondo i meccanismi di funzionamento di Xen possono aiutare a gestirlo e utilizzarlo al meglio, quindi farò di tutto per colmare le mie lacune.

There have been lots discussions lately about what’s happening around Citrix XenServer. Perhaps too many. For what it is worth, I was one of the people discussing this on the net (Twitter, Blogs etc) with some other folks. I originally drafted a blog post when Citrix bought XenSource but it never made it (officially because I was busy, unofficially because I couldn’t figure out “why”).

I think that what it is happening is pretty clear at this point. The market landscape is being consolidated with Oracle acquiring VirtualIron as well as the “Sun Xen thing” within the overall grand plan of the acquisition (of the remaining) of Sun. All these solutions have hardly, in the past few years, managed to make a difference in the industry and their names were floating around more with the hope that VMware could feel more pressure and competition, and hence lower the prices. In the meanwhile, VMware increased their price which speaks for itself.

This is leaving (apparently) the x86 virtualization market with 3 relevant viable alternatives that are VMware, Microsoft and Citrix. I have always said this is going to be a two-horse race and I still stand behind this statement. The first horse is VMware and the second horse is what I call Microtrix ™. There have been a nice Twitter discussion a few days ago on why Citrix bought XenSource and the future of it etc. This was my tweet in the discussion which, in a way, summarizes my thinking:

Source

http://slurl.com/secondlife/Xeniversity/208/231/81

Este post faz referência sobre virtualização computacional, modelo de abstração de recursos computacionais, afim de prover redução de recursos necessários para consolidar diferentes arquiteturas, sob um único hardware básico (ou avançado, indo do gosto do freguês).

Pretendo cobrir a parte sobre virtualização de sistemas operacionais (virtualização de plataformas), portanto, não falarei detalhadamentesobre as outras abordagens de virtualização. Apesar de ser um assunto relativamente novo, é um assunto, ao mesmo tempo, relativamente muito velho. Entendo por virtualização a grande invenção da memória virtual (que abstrai um segundo dispositivo de armazenamento, como memória principal, e estende sobre esse o espaço de endereçamento de instruções), as interfaces virtuais de rede (onde, por exemplo, dependendo das capacidades do dispositivo, pode-se criar diversas interfaces de rede virtuais, com diversos pontos de acesso em IP, por exemplo, sob um único dispositivo físico), as interfaces virtuais de armazenamento (que podem ser, desde abstrações via rede, até um storage inteiro virtual), VPNs (que nada mais são que redes inteiras virtuais), virtualização de desktops (como terminais burros, ou por exemplo, serviços de terminais remotos), virtualização e clustering de sistemas de gestão de bancos de dados, e por aí vai!

Falando sobre Hypervisor

Para entrar no tema que eu quero sobre virtualização, não dá pra começar sem falar de uma entidade chamada de Hypervisor. Este é o sujeito responsável por prover uma interface lógica entre o que vai prover um ambiente de virtualização (chamado comumente de hospedeiro) e o que vai ser virtualizado (chamado comumente de convidado). Técnicamente, é uma porção de código que é executado pelo software hospedeiro, e age como um controlador de hardware e monitor do sistema operacional do software convidado. Geralmente, nesses moldes, o sistema operacional convidado, executa sob um nível abaixo do nível do Hypervisor.

A primeira figura exibe quais as camadas definidas na arquitetura de virtualização a qual estou me referindo.

Figura 1 – Arquitetura básica da virtualização provida pelo Hypervisor.

Há dois principais modos de execução do Hypervisor. O modo nativo (portanto, para virtualization), e o modo abrigado (portanto, full virtualization).

O primeiro modo de execução, chamado abrigado (com a palavrinha mágica – hosted), faz com que toda a plataforma de hardware disponível ao convidado seja totalmente virtualizada. Portanto, são implementados e aplicados gatilhos e tratadores nas chamadas de sistema e instruções de máquina (instruções nativas – sistema operacional convidado não sofre nenhuma modificação) enviadas pelo sistema operacional convidado, e convertidas para as instruções de máquina e chamadas de sistema do sistema operacional hospedeiro, portanto, executadas nativamente. O resultado de tais ações é novamente convertido para o formato de retorno do sistema operacional convertido, e tudo caminha nas mil maravilhas.

A figura 2 mostra um modelo esquemático de aplicação do modelo 1 do Hypervisor.

Figura 2 – Arquitetura básica do modelo hospedado.

Definem-se como vantagens dessa abordagem, o fato sistema operacional convidado não precisa ser modificado; para este, é uma máquina física como qualquer outra. Não há custos adicionais, nem modificação no software original, reservando as características originais do sistema operacional convidado. A grande desvantagem dessa abordagem é o desempenho comprometido (em relação ao desempenho original, o que não significa que fique lento – leia-se sem condições mínimas de emprego em um ambiente corporativo de produção). A queda do desempenho dá-se pela interpretação de todas as instruções disparadas pelo hardware virtual disponível para a máquina convidada. Além disso, claro, ainda há o escalonamento de processos do sistema operacional, através de um núcleo premptivo.

O segundo modo de execução, chamado nativo (com a palavrinha mágica – bare-metal), cria um meio termo nas coisas. Através de um núcleo de chamadas de sistema (vulgo Kernel, no mundo *nix) modificado, torna o sistema operacional convidado, alguém que sabe que está sendo virtualizado, mas que consegue, através desse núcleo modificado, alcançar nativamente o hardware disponível na arquitetura do sistema hospedeiro. Mágico, não?

Este novo núcleo, é então parte do Hypervisor (que agora é parte do sistema operacional convidado, e parte do sistema operacional hospedeiro), fazendo com que seja mais simples chegar ao hardware, e fazer algumas traduções das chamadas mais críticas do sistema.

Como vantagens da técnica, está principalmente o desempenho, tornando as coisas mais fáceis, fazendo com que não haja tradução de instruções, nem emulação de hardware. Por exemplo, se você comprou sua nova Intel® PRO/1000 GT Desktop Adapter, ele vai ser Intel no sistema hospedeiro, e Intel no sistema convidado. Na primeira abordagem, ela seria Intel no sistema hoespedeiro, e Network Virtual Driver, no sistema convidado. A principal desvantagem é a modificação do núcleo do sistema convidado, aumentando a probabilidade do número de falhas (por sair da árvore principal de desenvolvimento do software, e ter dedos de terceiros), incompatibilidade de padrões, aumento de custo de manutenção, e blá blá blá.

A figura 3 define um modelo esquemático de como aconteceria a implantação do segundo caso.

Figura 3 – Implantação do modelo nativo do Hypervisor.

Em relação ao modelo de virtualização de plataformas, com a presença de um Hypervisor, o assunto termina por aqui. Não há muito mais novidades na área.

Continuando com o assunto de virtualização, é válido dizer que o hardware x86 é particularmente difícil de virtualizar. Com essa motivação, tanto a Intel, quanto a AMD, criaram tecnologias de virtualização diretamente embutidas no processador. A Intel a chamou de Intel-VT (VanderPool), e a AMD, AMD-V (Pacifica). Tais extensões são responsáveis por adicionar capacidades que são difíceis ou ineficientes de virtualizar via software, facilitando o trabalho do pessoal que desenvolve na área.

Falando sobre implementações

Há diversos softwares comerciais e livres que implementam este modelo de monitor de máquinas virtuais, e com a idéia geral de virutalização de plataformas, dá pra nomear algumas categorias, de forma simplificada, e alguns softwares virtualizadores, que a implementam.

No primeiro modelo, nativo, encaixam-se nomes como VMware ESX Server, IBM System z Hypervisor, Microsoft Hyper-V, Xen, Citrix XenServer, Oracle VM Server, Parallels Server, Sun’s Logical Domains Hypervisor, e finalmente, o Kernel-based Virtual Machine (KVM), disponível nas distribuições empresariais de Linux. Todos esses softwares permitem a virtualização do sistema operacional hospedeiro, bem como o sistema operacional convidado, exigindo a modificação de seu núcleo.

Já no segundo modelo, hospedado, encaixam-se nomes como VMware Server, VMware Workstation, VMware Fusion, o projeto livre QEMU, Microsoft Virtual PC e Microsoft Virtual Server, Sun’s VirtualBox, Parallels Workstation e Parallels Desktop. Estes software virtualizam qualquer sistema opercional comum, sem a necessidade de sua modificação.

Virtualização e instâncias de sistemas operacionais

Especificamente, o Sun Solaris e o BSD, implementam um outro modelo de virtualização de sistema operacional.

O Sun Solaris (a partir de 2005) implementa Zones, definido como um abstração virtual do sistema operaciona, onde aplicações executam. Tais aplicações são protegidas e separadas umas das outras. Através de uma única instância principal do sistema operacional, pode-se gerencial diversas Zones (como instâncias do mesmo sistema).

O uso do gerenciamento de recursos da arquitetura em questão, faz com que uma Zone seja chamada de Container, criando o conceito de Solaris Containers. A maioria das pessoas utilizam  os dois termos, sem fazer distinção entre eles.

Existem dois tipos de Zones. As Zones nativas, criam instâncias do mesmo sistema operacional (Solaris), enquanto as Branded Zones, disponibilizam uma insfra estrutura virtual de hardware, para que seja possível criar instâncias de outros sistemas operacionais, como o Linux, por exemplo.

Da mesma maneira, existem os Jails (que vieram antes dos Containers, em 1990, criados a partir do FreeBSD). Jail implementa o mesmo sentido de abstração do sistema operacional, criando instâncias desse, fazendo com que haja muito mais flexibilidade do uso do sistema operacional, em um ambiente de produção.

Acredito que o uso de instâncias de sistema operacional, tenha tido como principal motivação, a flexibilidade de modelos de segurança que o *nix não conseguia prover, mas isso é um outro assunto.

Para terminar o post, volto a dizer que virtualização pode ser vista de diversas formas, e existem diversos modelos e ferramentas para implementar. Vão desde abstrações de hardware de entrada e saída, até abstrações do sistema operacional. O que vale realmente realmente, é estudar e medir as necessidades e as disponibilidades, para que seja feita a escolha certa.

O modelo de cloud computing se encaixa em virtualizações. É um tema recente, poderoso, mas com a escolha errada, pode se tornar extremamente perigoso.

Até a próxima!

Referências:

http://en.wikipedia.org/wiki/Virtualization

http://en.wikipedia.org/wiki/Hypervisor

http://www.sun.com/servers/coolthreads/ldoms/index.jsp

http://en.wikipedia.org/wiki/Paravirtualization

http://www.ibm.com/developerworks/br/library/l-hypervisor/index.html

http://blogs.technet.com/lpalma/archive/2008/10/01/paravirtualiza-o-emula-o-bare-metal.aspx

http://www.intel.com/products/desktop/adapters/pro1000gt/pro1000gt-overview.htm

Setup a private cloud using open source eucalyptus on Ubuntu 9.04 jaunty OS. I used ‘xen 3.0′ instead of default package ‘kvm’ for Ubuntu 9.04.
This Private cloud setup uses ‘STATIC’ mode and list of IPs and machine address are provided to the Eucalyptus installation to be used, because I was using the existing subnet and pick the available IPs from the subnet.

Contents

• Cloud – setup private cloud
• Technical background
• Front end setup
• Back end setup
• Cloud integration
• ec2  or euca2ools setup
• Create guest machine
• Creating EMIs
• Run instance
• Troubleshoot
• Useful Commands

Cloud – setup private cloud

My interest in cloud computing started when I worked on a project in Impetus Infotech. Project was to deploy application on Amazon cloud and setup the clusters for MySql, Apache and other technologies to load balance. While working on Amazon cloud I got some insights of the cloud computing and various underlying technologies.

Here I am going to illustrate all the steps that I have followed to setup cloud. I will be describing the cloud packages, how to setup front end and back end setup, cloud implementation, ec2  or euca2ools setup, creating xen-supported guest machine, creating EMIs for eucalyptus, run/stop instances and some xen/eucalyptus/ec2 useful command.

Technical background

When I start on cloud setup, I was sure I will go for XEN rather than KVM. Then came the Ubuntu jaunty as this version was built for the virtualization and designed with built-in support. I used eucalyptus, an opensource tool that provide Amazon like features and is EC2 tools compliance. So I starts researching the contents on all these technologies. The more I study, the more I got confident to use these technologies for cloud setup.

Though, Ubuntu prefers KVM for the visualization support and has added to the default packages. But limitation of KVM to run only virtualization extensions (Intel VT or AMD-V) processors only(see http://www.linux-kvm.org/page/Main_Page), makes my decision to use XEN for virtualization.

Xen, is the underlaying technology used by eucalyptus. Xen hypervisor allows several guest operating systems to be executed on the same computer hardware concurrently. Xen partitions a single physical machine into multiple virtual machines, to provide server consolidation and utility computing. Existing applications and binaries run unmodified. The hypervisor controls the MMU, CPU scheduling, and interrupt controller, presenting a virtual machine to guests.

The first guest operating system, called in Xen terminology “domain 0″ (dom0), is booted automatically when the hypervisor boots and given special management privileges and direct access to all physical hardware by default. The system administrator can log into dom0 in order to manage any further guest operating systems, called “domain U” (domU) in Xen terminology.

Eucalyptus has 3 major packages.
Cluster Controller:  provides support for the virtual network overlay
Cloud Controller: includes the front-end services and the Walrus storage system.
Node Controller: that interacts with XEN to manage individual VMs.

The basic architecture how client, cloud controller, cluster controller and node controllers communicate with each other in a eucalyptus setup.

—-> eucalyptus-nc

|

|

Client —> eucalyptus-cloud —-> eucalyptus-cc ——>  eucalyptus-nc

|

|

—->  eucalyptus-nc

Packages provided by Ubuntu are: eucalyptus-cloud, eucalyptus-cc and eucalyptus-nc.

In my private cloud setup, the front end runs both eucalyptus-cloud and eucalyptus-cc and 3 machines act as node controller and run eucalyptus-nc.

Front end setup

I have setup a Dell machine as front end. Front end runs both eucalyptus-cloud and eucalyptus-cc. So I install both packages on this machine.

Following are the steps for installing the front end:

Step 1: Prepare a raw ubuntu OS system.

Install Ubuntu 9.04 on a machine. Select ssh server to be installed on the machine by default, so We can connect to machine remotely.

Step 2: Install postfix.

Postfix is a mail server. Eucalyptus send mails to users that are added by admin on eucalyptus web interface. So we install postfix that eucalyptus will use as mail server.

Steps to install postfix and to setup configurations and testing successful Postfix installation
command# apt-get install postfix

Set configurations
command# sudo dpkg-reconfigure postfix

Set following parameters with given values.


Test postfix installation
command# EHLO localhost


Check your mail (if received your mail is working)

Step 3 Install Cloud and Cluster controller

Install the packages provided by Ubuntu, all the dependencies will be installed automatically by package manager.

command# apt-get install eucalyptus-cloud eucalyptus-cc

Step 4 Install dhcp server

Install the DHCP server which is used by eucalyptus for assigning the IPs to instances. DHCP server should not be in running state and set the DHCP server not to run when system is booted up again.

command# apt-get install dhcp3-server
command# /etc/init.d/dhcdbd stop
command# update-rc.d -f dhcdbd remove

Set the DHCP server path in the eucalyptus config file ‘/etc/eucalyptus/eucalyptus.conf‘
VNET_DHCPDAEMON=”/usr/sbin/dhcpd3″

Step 5 Eucalyptus configuration

Set the static mode for eucalyptus so that eucalyptus uses IP’s from the pool of IP’s provided in the config file and use the machine addresses provided along with IPs.

Configuration file for eucalyptus is ‘/etc/eucalyptus/eucalyptus.conf‘
open /etc/eucalyptus/eucalyptus.conf and set the following parameters

VNET_INTERFACE=”peth0″
VNET_BRIDGE=”eth0″

Locate ‘kvm‘ and replace to ‘xen‘
locate VNET_MODE=”SYSTEM” and disable it
locate VNET_MODE=”STATIC” and enable it

Add machine and IPs for instances to be used and change the netowork setting to current network settings


Step 6 Other configurations

Edit : /usr/share/eucalyptus/gen_libvirt_xml
Locate all ‘sda‘ to ‘xvda‘

Edit : /etc/xen/xend-config.sxp



Step 7 Rebbot the system to take effect

Step 8 Cloud configuarions

Once eucalyptus front end has been installed, we can use the eucalyptus web interface and setup cluster and set other parameters.

Eucalyptus cloud web URL: https://<front end ip>:8443/

Under configuration tab add cluster name
Set host to ‘localhost‘

Create kernel and ramdisk images and set in eucalyptus site. (Procedure to create images will be described later in the page)

Restart the eucalyptus daemon to take effects

Back end setup

On the systems where we want to run instances, we will install node controller and XEN 3.0. Since I used the XEN as virtualization package, I install the node controller package eucalyptus-nc manually. I installed required dependencies using the package manager apt-get, but the eucalyptus-nc was installed by donwloading and manually installing bypassing dependencies list.

Step 1 Hardware compatibility

eucalyptus-nc requires ‘libvirt-bin‘ package. Libvirt is a C toolkit to interact with the virtualization capabilities of recent versions of Linux (and other OSes). More info can be found at: http://libvirt.org/

To install ‘libvirt-bin‘ package, certain flags should be supported by CPU.

command# egrep ‘(vmx|svm)’ /proc/cpuinfo
[if no print], libvirt-bin cant be installed

Stop the apparmor for Ubuntu System
command# /etc/init.d/apparmor stop
command# update-rc.d -f apparmor remove

Step 2 Install XEN package

I am using the XEN virtualization package instead of Ubuntu’s default KVM, I install the eucalyptus-nc dependencies manually.
command# apt-get install iproute  iptables module-init-tools python2.5 python2.6
command# apt-get  install xen-utils
command# apt-get  install ubuntu-xen-server

Step 3 Install libvirt-bin package

eucalyptus-nc requires ‘libvirt-bin‘ package. Libvirt is a C toolkit to interact with the virtualization capabilities of recent versions of Linux (and other OSes). More info can be found at: http://libvirt.org/

command# apt-get install adduser bridge-utils dnsmasq-base iptables libsasl2-2 libxen3 logrotate
command# apt-get install netcat-openbsd libavahi-client3 libavahi-common3 libc6 libdbus-1-3
command# apt-get install libgcrypt11 libgnutls26 libhal1 libpolkit-dbus2 libpolkit2 libreadline5 libselinux1
command# apt-get install  libtasn1-3 libvirt0    libxml2   zlib1g policykit

Download the libvirt-bin package from URL: http://packages.ubuntu.com/jaunty/libvirt-bin

I used i386 package, http://mirrors.kernel.org/ubuntu/pool/main/libv/libvirt/libvirt-bin_0.6.1-0ubuntu5_i386.deb

Install lib-virt
command# dpkg -i libvirt-bin_0.6.1-0ubuntu5_i386.deb
command# sudo adduser $USER libvirtd

Step 4 Install eucalyptus node package

command# apt-get install apache2  eucalyptus-common eucalyptus-gl libapache2-mod-axis2c
command# apt-get install libaxis2c0 librampart0 dhcp3-server vlan aoetools
command# apt-get install libc6 libcurl3-gnutls  libvirt0  zlib1g

Download eucalyptus-nc package from Ubuntu package repository.

eucalyptus-nc package URL: http://packages.ubuntu.com/jaunty/eucalyptus-nc

Download the i386 package from URL:
http://mirrors.kernel.org/ubuntu/pool/universe/e/eucalyptus/eucalyptus-nc_1.5~bzr266-0ubuntu2_i386.deb

command# dpkg -i eucalyptus<version>

Step 5 Grub loader for Xen Kernel

Setup grub menus and load xen images to boot into XEN kernel.
Download the modules and image from URLs:
wget http://ftp.debian.org/debian/pool/main/l/linux-2.6/linux-modules-2.6.26-2-xen-686_2.6.26-15_i386.deb
wget http://ftp.debian.org/debian/pool/main/l/linux-2.6/linux-image-2.6.26-2-xen-686_2.6.26-15_i386.deb

command# sudo dpkg -i linux-modules-2.6.26-2-xen-686_2.6.26-15_i386.deb
command# sudo dpkg -i linux-image-2.6.26-2-xen-686_2.6.26-15_i386.deb

Step 6 Eucalyptus Configuerations

Stop the eucalyptus daemon and set the configuration depending upon your installation.
command# /etc/init.d/eucalytpus-nc stop

Edit : /etc/sysctl.conf
(uncomment net.ipv4.ip_forward=1)

Edit : /etc/eucalyptus/eucalyptus.conf
(set)



Edit : /usr/share/eucalyptus/gen_libvirt_xml
locate ‘sda’ s and replace to ‘xvda‘ s

for static ips
Edit : /etc/xen-tools/xen-tools.conf



command# /etc/init.d/eucalyptus-nc start

Cloud integration

Now, we have front controller and node controllers ready. We have to setup now the communication between cluster controller and node controllers. We will add the nodes list to the cloud, so cloud know about the available node.

command# sudo euca_conf -addnode <node ip> <node ip> <node ip>

This will add nodes to cloud config file and replicates the permission and certificate files to the nodes to be able to have password-less communication between nodes and cloud.

To check the cloud installation, and node communication, you can run ‘euca-describe-availability-zones verbose‘ command and check if all the nodes are listed in output.

ec2 or euca2ools setup

Euca2ools setup

Install dependencies
command# apt-get install curl libopenssl-ruby ruby
Download the appropriate tarball from http://open.eucalyptus.com/downloads

Steps

command# tar zxvf euca2ools-1.0-*.tar.gz
command# cd euca2ools-1.0-* (Replace this with the actual directory name)
command# sudo -s
command# echo deb file://${PWD} ./ >> /etc/apt/sources.list
command# apt-get update

command# apt-get install euca2ools
WARNING: The following packages cannot be authenticated!
…
Install these packages without verification [y/N]? y

ec2 tools Setup

Eucalyptus supports ec2-ami-tools-1.3-26357 and ec2-api-tools-1.3-30349 versions of ec2 tools.

Install dependencies
Install jdk1.6.0_12 java version.

Steps
Download ec2-ami-tools-1.3-26357 and ec2-api-tools-1.3-30349.
unzip these archives
Copy the ‘lib’ and ‘bin’ folders under ~/.euca folder.

Export parameters



Set URL and credentials

Download ‘euca2-admin-x509.zip‘ file from cloud site at https://<front end ip>:8443/.
Unzip this archive under ~/.euca folder and export the eucarc file.

command# source ~/.eucarc

Create guest machine

I create my image using vmbuilder.
More info on vmbuilder can be found on: https://help.ubuntu.com/8.04/serverguide/C/ubuntu-vm-builder.html

Steps

Select a directory where all files will be stored. I used ‘/usr/akash/cloud‘

Create a ‘part‘ file under directory.
The content of ‘part‘ is following:
root 400
/mnt/ephemeral 0 /dev/sda2
swap 1 /dev/sda3

Create ‘firstboot‘ file, which is run first time, instance boots up.
apt-get -y install openssh-server

Create the image.
command# vmbuilder xen ubuntu  –firstboot ./firstboot –mirror=http://192.168.152.87:9999/ubuntu –addpkg ssh –rootsize=4096 –arch i386 –verbose –debug
This command creates the image file that i used for creating emi. You can found the image under ‘ubuntu-xen‘ folder.
This image  can be tested, if its compatible to xen. Use xm create command.

command# xm create /usr/akash/cloud/ubuntu-xen/xen.cfg

Check the status of emi using xm list command.
command# xm list
If you see your emi ID in the output, your image can be run with XEN.

Creating EMIs

Now, cloud is setup, we should create images that can be run using eucalyptus. We will create emis for kernel, ramdisk and instance.

kernel emi

Steps
command# mkdir kernel
command# ec2-bundle-image -i vmlinuz-2.6.28-13-server -d ./kernel –kernel true
command# ec2-upload-bundle -b eucaKernel -m ./kernel/vmlinuz-2.6.28-13-server.manifest.xml
command# EKI=`ec2-register eucaKernel/vmlinuz-2.6.28-13-server.manifest.xml | awk ‘{print $2}’`
command# echo $EKI

ramdisk emi

Steps
command# mkdir ramdisk
command# ec2-bundle-image -i initrd.img-2.6.28-13-server -d ./ramdisk –ramdisk true
command# ec2-upload-bundle -b eucaRamdisk -m ./ramdisk/initrd.img-2.6.28-13-server.manifest.xml
command# ERI=`ec2-register eucaRamdisk/initrd.img-2.6.28-13-server.manifest.xml | awk ‘{print $2}’`
command# echo $ERI

instance emi

Steps
command# mkdir image
command# ec2-bundle-image -i root.img -d ./image
command# ec2-upload-bundle -b plabImage -m ./image/root.img.manifest.xml
command# EMI=`ec2-register plabImage/root.img.manifest.xml | awk ‘{print $2}’`
command# echo $EMI

Run instance

Add key pair
command# ec2-add-keypair mykey > ~/.euca/mykey.priv
command# chmod 0600 ~/.euca/mykey.priv
command# ec2-run-instances emi-73FD0EAD -k mykey –kernel eki-8D701398 –ramdisk eri-E47F14E8 –instance-type m1.xlarge

Troubleshoot

Once I started installing eucalyptus, I encounter many issues and face problems because of no/less help available. To resolve the issues I have to google or search http://open.eucalyptus.com forums or post my issues in forums to get the solution. I will try to list some of the issues thats I face, it might be of some help if you got similar issues.

Eucalyptus Logs

When I got any issue, my first area of interest was the logs generated by eucalyptus and its dependencies. Eucalyptus maintain all the logs under pre-defined directory ‘/var/log/eucalyptus‘.

euca_test_nc.log
On node controllers, if there is any issue with starting eucalytus-nc command, check euca_test_nc.log file for any errors. euca_test_nc.log file contains all the errors eucalytus-nc encounter while starting.

nc.log
nc.log file contains all the logs generated in cluster-node communication.
While starting a instance, initially I wasnt aware of buggy log handling of eucalyptus and it takes a lot time to understand this. While starting instance, and keep on adding the logs for non-existing network while instance is currently in booting up. Because of this issue, I got following in the nc.log file

[EUCAINFO  ] doDescribeResource() invoked
[EUCAINFO  ] doDescribeResource() invoked
[EUCAERROR ] libvirt: Domain not found (code=42)

which indicates, domain can not be started, but after some time this error is resolved automatically when network is started for the instance. So we can omit this error message in nc.log file.

Cloud resources

Some times when you install the eucalyptus and cluster, cloud and nodes are running successfully on systems. Still you can not run instances, this may be issue with the node resource registration with the cluster. Or all the resources has been consumed by the instances. You can check the resource status for cluster using ‘ec2-describe-availability-zones verbose’ command.

command# ec2-describe-availability-zones verbose

Output of above command could be like this.
AVAILABILITYZONE        iCloud    UP localhost
AVAILABILITYZONE        |- vm types     free / max   cpu   ram  disk
AVAILABILITYZONE        |- m1.small     0002 / 0002   1    128    15
AVAILABILITYZONE        |- c1.medium    0002 / 0002   1    256    15
AVAILABILITYZONE        |- m1.large     0001 / 0001   2    512    17
AVAILABILITYZONE        |- m1.xlarge    0001 / 0001   2   1024    20
AVAILABILITYZONE        |- c1.xlarge    0000 / 0000   4   2048    20
AVAILABILITYZONE        |- 192.168.145.71       certs[cc=true,nc=true] @ Wed Sep 30 02:12:15 EDT 2009

From the output, we can clearly visualize the resource management, free/used resources and number of instance we can run. Output also lists the nodes registered/available currently to cluster.

Instance issues

Some times, I got stuck with instance related issues. When I start instance using ec2 tools, instance will die and nc.log or any other log file doesnt provide useful info. So, best way to investigate issue and save time to run the instance using xm command provided by xen.

Common steps are,
- copy the image file used to create emi to any of the node.
- create xen conf file, check following xen config file content



kernel = '/boot/vmlinuz-2.6.28-13-server'
ramdisk = '/boot/initrd.img-2.6.28-13-server'

memory = 128
root = '/dev/xvda1 ro'

disk = [
'file:/usr/akash/cloud/ubuntu-xen/root.img,xvda1,w',
'file:/usr/akash/cloud/ubuntu-xen/swap.img,xvda2,w'
]

name = 'Ubuntu'
dhcp = 'dhcp'
vif = ['']

on_poweroff = 'destroy'
on_reboot = 'restart'
on_crash = 'restart'

extra = 'xencons=tty console=tty1 console=hvc0'

- run the instance using xm command.
command# xm create xen.cfg

- while xm create is starting instance, use console command to check the console output.
command# xm console Ubuntu
Ubuntu – is the name option provided in xen.cfg file

Useful commands

Xen commands

command# xm list
list all the available Xen instances you’re allowed to control

command# xm create xen.cfg
run the instance using xen.cfg file options

command# xm status
current status of your Xen guest

command# xm console <domain id>
allow you to access your Xen guest’s console

command# xm restore <domain file>
Restore a domain from a saved state

command# xm save <domain id>
Save a domain state to restore later

command# xm pause <domain id>
Pause execution of a domain

command##  xm shutdown <domain id>
Shutdown a domain

command# xm destroy <domain id>
Terminate a domain immediately

command# xm mem-max <domain id>
Set the maximum amount reservation for a domain

command# xm mem-set <domain id>
Set the current memory usage for a domain

EC2 command

Delete image
command# ec2-deregister emi-55150E20

command# euca-delete-bundle -a $EC2_ACCESS_KEY -s $EC2_SECRET_KEY –url $S3_URL -b eucaubuntu -p image –clear

Check cloud status, returns all the available nodes and resource available
command# euca-describe-availability-zones verbose

EUCALYPTUS Commands

Add new node
command# euca_conf -addnode 192.168.145.71

Delete node
command# euca_conf -delnode 192.168.145.71

Sync keys between nodes and cluster controller
command# $EUCALYPTUS/usr/sbin/euca_sync_key -c$EUCALYPTUS/etc/eucalyptus/eucalyptus.conf

References

1. Ubuntu eucalyptus setup
2. lib-virt
3. Eucalyptus
4. Xen
5. KVM

Instalując pod Xenem dystrubucję Debian Lenny można napotkać na kilka problemów. Oto one wraz z przepisem jak je rozwiązać:

• Zawieszanie się Lennego podczas bootowania. Problemem jest konsola, a właściwie błędne zdefiniowanie. Rozwiązaniem jest dodatnie następującego zaklęcia do pliku konfiguracyjnego wirtualkę xenową, następującej linii:
  extra = 'console=hvc0 xencons=tty'
• Jeżeli ktoś zainstaluje pakiet openssh-server, to podczas próby połączenia dostanie komunikat typu:
  
  

  Server refused to allocate pty stdin: is not a tty

  Rozwiązaniem jest dopisanie w instancji Lennego, do pliku /etc/fstab następującej kombinacji znaków:
  none /dev/pts devpts rw 0 0

Are you running Xen / “xm create” and you get this error?

Try this (RHEL/CentOS): service haldaemon start

- and have a nice day!

Недавно начал пробовать систему виртуализации Xen. После быстрого чтения мануалов, смог установить виртуальную машину с Debian, что говорит о достаточно простой конфигурации (для начальных нужд). Итак, в качестве хостовой машины (dom0) будет использоваться CentOS 5.2, запущенный на компьютере с поддержкой аппаратной виртуализации (AMD-V или Intel-VT). Поддержка аппаратной виртуализации нужна, если вы собираетесь запускать Windows. Первое что необходимо, установить ядро с поддержкой Xen. В стандартных репозитариях CentOS есть такое ядро, kernel-xen версии 2.6.18-164 с Xen 3.0

yum install kernel-xen

После установки, в /boot появится ядро vmlinuz-2.6.18-164.el5xen. Теперь надо загрузиться с ним. Можео либо вручную его выбрать при загрузке grub, либо отредактировать файл /boot/grub/menu.lst и параметр default сделать равным 0 (по-умолчанию, kernel-xen автоматчиески прописывает себя первым ядром в меню grub). После загрузки с ядром Xen’а, работу его можно проверить наличием интерфейсов xenbr и vif в выводе ifconfig:

[root@gigi ~]# ifconfig

peth1     Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF

inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link

UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1

RX packets:1484845567 errors:0 dropped:848 overruns:0 frame:0

TX packets:21318806 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:107304750948 (99.9 GiB)  TX bytes:1495739489 (1.3 GiB)

Memory:dc080000-dc0a0000

vifxenv0  Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF

inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link

UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1

RX packets:17643569 errors:0 dropped:0 overruns:0 frame:0

TX packets:1467766901 errors:0 dropped:390908 overruns:0 carrier:0

collisions:0 txqueuelen:32

RX bytes:863953497 (823.9 MiB)  TX bytes:100008573528 (93.1 GiB)

vif0.1    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF

inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link

UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1

RX packets:3588998 errors:0 dropped:0 overruns:0 frame:0

TX packets:1483233162 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:0

RX bytes:370499902 (353.3 MiB)  TX bytes:107193417651 (99.8 GiB)

xenbr1    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF

UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1

RX packets:3930114 errors:0 dropped:0 overruns:0 frame:0

TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:0

RX bytes:2361888673 (2.1 GiB)  TX bytes:0 (0.0 b)

А также, посмотреть вывод команды xm list, он должен выглядеть примерно так:

[root@gigi ~]# xm list Name                                      ID Mem(MiB) VCPUs State   Time(s)Domain-0                                   0      512     4 r-----  45273.0

Запись Domain-0 обозначает, что запущена машина с dom0 (наша хостовая) и значит, все работает Из известных проблем, следует упомянуть, не используйте сетевые карты Realtek на чипсетах 8169 и ему подобном, т.к. при использовании такой сетевой карты, не сохраняются сетевые параметры (не применяются при создании моста xenbr), а также MAC адрес становится FF:FF:FF:FF:FF:FF В следующей заметке (очень скоро) расскажу про создание Linux и Windows виртуальных машин. Ссылки: Официальный сайт Официальная документация Документация на русском

Seit einiger Zeit hatte ich das Problem die Zeit auf einer Xen-virtualisierten Maschine zu verändern. Alle Versuche scheiterten, bis ich die Zeit auf dem Wirtsystem dom0 änderte und siehe da, die Gastmaschinen hatte auch alle die korrekte Zeit.

Eine Suche bei Google brachte mich dann auch zu einem Xen FAQ.
In dem FAQ wird beschrieben das unter Xen standardmäßig die Zeit der Gastmaschinen an die Wirtmaschine dom0 gekoppelt ist.