We are in the process of deploying some new infrastructure to store the 150+GB of new content (media only, not including text) uploaded to WordPress.com daily.

WordPress.com data in GB

After some searching and testing, we have decided to use the open source software MogileFS developed in part by our friends at Six Apart. Our initial deployment is going to be 180TB of storage in a single data center and we plan to expand this to include multiple data centers in early 2010. In order to get that amount of storage affordably, the options are limited. We thought about building Backblaze devices, but decided that the ongoing management of these in our hosting environment would be prohibitively complicated. We eventually settled on Dell’s MD PowerVault series. Our configuration consists of:

• 4 x Dell R710 ( 32GB RAM/2 x Intel E5540/2 x 146GB SAS RAID 1)
• 4 x Dell MD3000 (15 x 1TB 7200 RPM HDD each)
• 8 x Dell MD1000 (15 x 1TB 7200 RPM HDD each)

Each Dell R710 is connected to a MD3000 and then 2 MD1000s are connected to each MD3000. The end result is 4 self-contained units, each providing 45TB of storage for a total of 180TB.

Illustration by Joe Rodriguez

Our proof of concept was deployed on a single Dell 2950 connected to a MD1000 and things worked relatively flawlessly. We could use all of our existing tools to monitor, manage, and configure the devices when needed. Little did I know the MD3000s were so much of a pain Since we are using MogileFS which handles the distribution of files across various hosts and devices, we wanted these devices setup in what I thought was a relatively simple JBOD configuration. Each drive would be exported as a device to the OS, then we would mount 45 devices per machine and MogileFS would take care of the rest. Didn’t exactly work that way.

When the hardware was initially deployed to us, they were configured in a high availability (HA) setup, with each controller on the MD3000 connected to a controller on the R710. This way, if a controller fails, in theory the storage is still accessible. The problem with this type of setup is that in order to make it work flawlessly, you need to use the Dell multi-path proxy and mpt drivers, not the ones provided by the Linux kernel. Dell’s provided stuff doesn’t work on Debian. Initially, without multipath configured, some confusing stuff happens — we had 90 devices detected by the OS (/dev/sdb through /dev/sdcn), but every other device was un-reachable. After some trial and error with various multipath configurations, and some help I ended up with this:

apt-get install multipath-tools

Our multipath.conf:

defaults {
        getuid_callout "/lib/udev/scsi_id -g -u -s /block/%n"
		user_friendly_names on
}
devices {
        device {
                vendor DELL*
                product MD3000*
                path_grouping_policy failover
                getuid_callout "/lib/udev/scsi_id -g -u --device=/dev/%n"
                features "1 queue_if_no_path"
                path_checker rdac
                prio_callout "/sbin/mpath_prio_rdac /dev/%n"
                hardware_handler "1 rdac"
                failback immediate
        }
}
blacklist {
       device {
               vendor DELL.*
               product Universal.*
       }
       device {
               vendor DELL.*
               product Virtual.*
       }
}

multipath -F
multipath -v2
/etc/init.d/multipath-tools start


This gave me a bunch of device names in /dev/mapper/* which I could access, partition, format, and mount. A few things to note:

• user_friendly_names doesn’t seem to work. The devices were all still labeled by their WWID even with that option enabled
• The status of the paths as shown by multipath -ll seemed to change over time (from active to ghost). Not sure why.
• Even with all of this set up and working, I still was seeing the occasional I/O error and path failure reported in the logs

After a few hours of “fun” with this, I decided that it wasn’t worth the hassle or complexity and since we have redundant storage devices anyway, we would just configure the devices in “single path” mode and mount them directly and forego multipath. Not so fast…according to Dell engineers, “single path mode” is not supported. Easy enough, lets un-plug one of the controllers, creating our own “single path mode” and everything should work, right? Sort of.

If you just go and unplug the controller while everything is running, nothing works. The OS needs to re-scan the devices in order to address them properly. The easiest way for this to happen is to reboot (sure this isn’t Windows?). After a reboot, the OS properly saw 45 devices (/dev/sdb – /dev/sdau) which is what I would have expected. The only problem was that every other device was inaccessible! It turns out, that the MD3000 tries to balance the devices across the 2 controllers, and 1/2 of the drives had been assigned a preferred path of controller 1 which was unplugged. After some additional MD3000 configuration, we were able to force all of the devices to prefer controller 0 and everything was accessible once again.

Only other thing worth noting here is that the MD3000 exports an addition device that you may not recognize:

scsi 1:0:0:31: Direct-Access DELL Universal Xport 0735 PQ: 0 ANSI: 5

For us this was LUN 31 and the number doesn’t seem user configurable, but I suppose other hardware may assign a different LUN. This is a management device for the MD3000 and not a device that you can or should partition, format, or mount. We just made sure to skip it in our setup scripts.

I suppose if we were running Red Hat Enterprise Linux, CentOS, SUSE, or Windows, this would have all worked a bit more smoothly, but I don’t want to run any of those. We have over 1000 Debian servers deployed and I have no plans on switching just because of Dell. I really wish Dell would make their stuff less distro-specific — it would make things easier for everyone.

Is anyone else successfully running this type of hardware configuration on Debian using multipath? Have you tested a failure? Do you have random I/O errors in your logs? Would love to hear stories and tips.

I have some more posts to write about our adventures in Dell MD land. The next one will be about getting Dell’s SMcli working on Debian, and then after that a post with some details of our MogileFS implementation.

* Thanks to the fine folks at Layered Tech for helping us tweak the MD3000 configuration throughout this process.

Entre administradores de entornos de almacenamiento en fibra es conocido el problema que hay con la compatibilidad entre dispositivos de proveedores diferentes, por ejemplo, al añadir una cabina de discos  a una SAN y que no funcione con el failover (multipath) ya existente en el servidor porque no están en la matriz de interoperatividad de fabricantes. Dejaré en este post unas notas relativas a estos problemas.

Primeramente definiré Interoperatividad:

La Interoperatividad es la condición mediante la cual sistemas heterogéneos pueden intercambiar procesos o datos ( procedente de la Wikipedia ) o la capacidad de los productos de diferentes fabricantes puedan operar conjuntamente ( procedente de BNET).

En entornos de almacenamiento en fibra la Interoperatividad es bastante compleja, cumpliéndose que si toda una red SAN no está en la lista de compatibilidad de hardware de los vendedores no operará de la forma esperada. Hace algunos años, el acceder a información de interoperatividad era un trabajo frustante debido a que esta información no era pública por parte de la mayoría de fabricantes y por lástima en muchos casos usada de forma interesada comercialmente. En la actualidad se ha realizado un gran esfuerzo por la mayoría de ellos, creándose una tendencia cada vez más recia en comunicar esta información, al mismo tiempo que los grandes proveedores de soluciones de almacenamiento se ajustan a estándares para universalizar sus productos y flexibilizar una mayor integración en cualquier entorno heterogéneo cada vez más común.

En almacenamiento hay diferentes organismos para mejorar estos grandes problemas de incompatibilidades entre diferentes proveedores, de forma que establecen estándares y protocolos para una mejor comunicación e integración de dispositivos de almacenamiento. Entre todas ellas es destacable es el trabajajo que realiza SNIA (Storage Networking Industry Association) cuyo proposito es encabezar en la industria el desarrollo y promoción de estándares, tecnologías y servicios educacionales para apoderar la gestión de información. Todos los grandes fabricantes de almacenamiento son miembros de la SNIA y de forma paulatina va creciendo un marco de interoperatividad entre todos ellos. Aún así, debido al proceso lento de salida y aceptación de estos estándares se realizan de forma independiente muchas alianzas tecnológicas entre diferentes fabricantes que es su mayoría pretenden impulsar propios productos de avanzada tecnología consiguiendo en gran cantidad de casos ser pioneros y mejorar su cartera de productos y especificaciones y por consiguiente competir comercialmente, forzando que el resto de competidores tecnológicos acepten e integren estas nuevas especificaciones aún no estandarizadas, por esta necesidad de compatibilidad en entornos heterogéneos y para que no penalice su imagen comercial.

A continuación se aportará  información junto con enlaces de  los proveedores de almacenamiento para el conocimiento de la Interoperabilidad de sus productos con las de otros fabricantes.

Algunas Tablas de Interoperatividad y HCL ( Listas de compatibilidad Hardware )

Fabricantes

Emulex

Emulex es unos de los líderes en interoperatividad en SAN, su reputación “It Just Works” se basa en un compromiso con los estándares, abierto al soporte de software y hardware, y al testeo extensivo de interoperatividad.

A continuación se listan sus Tablas de Interoperatividad con los siguientes fabricantes :

Información y enlaces procedente de Emulex.

QLogic

QLogic presenta un magnífico trabajo disponible en su web de interoperatividad al igual que de su historia altamente recomendable disponible en su web, que recopila sus productos, soluciones y servicios que proporcionan, al igual que su certificaciones con distintos proveedores de soluciones SAN. Un documento nada habitual en esta tecnología.

A continuación se listan las Tablas de Interoperatividad con los siguientes fabricantes :

3PAR | AC&NC | Apple | ATTO Technology | BlueArc | Compellent | Dell | Dell EqualLogic | DNF | Dot Hill | EMC | Enhance Technology | Fujitsu Computer Products of America | Fujitsu Computer Systems | Hewlett-Packard | Hitachi Data Systems | Hitachi Global StorageTechnologies | HP LeftHand SAN (aka LeftHand Networks) | IBM | Infortrend | Intransa | iQstor | iStor | LSI | Matrox | NEC | NetApp | Nexsan | Overland | Pillar | Promise | Qualstar | Quantum | RELDATA | Seagra | SEPATON | Spectra Logic | StoneFly | StoreVault | Sun Microsystems | Symantec | Tandberg Data | Texas Memory Systems | Thales nCipher | Winchester Systems | Xiotech | Xyratex

Información y enlaces procedentes de QLogic.

Brocade – McData

Brocade pone a disposición un documento con la compatibilidad/interoperatividad de sus productos en la sección de compatibilidad enlazable aquí llamado Brocade Data Center Ready.

Sun Microsystems

Proporciona un listado de soluciones de almacenamiento certificadas con Microsoft en el siguiente documento:

http://www.sun.com/software/windows/storage_cert.pdf

En su sección de almacenamiento y servidores individualmente es completada con tablas de soportabilidad/certificación/interoperatividad de cada dispositivo, y una sección genérica a la Interoperatividad de todos sus productos.

IBM

Presenta una sección de productos para SAN:

http://www-03.ibm.com/systems/uk/storage/san/index.html

Cada dispositivo va acompañado de un documento de Interoperatividad llamado Interoperability Matrix.

EMC

Proporcionan un documento con las matrices de interoperatividad en la sección de Interoperatividad de su portal. Sus  clientes  pueden utilizar una herramienta de navegación para la interoperatividad en Powerlink.

HP

Sección de almacenamiento en su portal:

http://welcome.hp.com/country/es/es/leb/storage.html

NETAPP

Sección de Interoperatividad en su portal:

http://www.netapp.com/us/technology/interop.html

VMWARE

Dispone uns sección para consultas de compatibilidad desde su web con distintos proveedores:

http://www.vmware.com/resources/compatibility/search.php?action=base&deviceCategory=san

Proporciona un documento con las matrices de compatibilidad completa Storage/SAN Compatibility Guide.

….  sólo son algunos proveedores de fibra.

En ocasiones te puedes encontrar con el problema de que necesitas hacer algo relacionado con discos, caminos, …. y tienes un volumen lógico que no te permite hacerlo. Me explico:

Si necesitas eliminar caminos con “multipath -F” para hacer algún cambio en la configuración, te puede pasar que al ejecutarlo te diga algo así como: “disco: map in use” donde “disco” se corresponde con el punto de montaje de un volumen lógico. Si tienes un cluster, y ejecutas “service clvmd status” y te dice que tienes el volumen lógico activo, y no encuentras la manera de que deje de estar activo (parando servicios, …) lo puedes hacer a través de la instrucción:

“vgchange -a n mi_grupo_de_volumenes”

Fuente: http://www.centos.org/docs/5/html/Cluster_Logical_Volume_Manager/VG_activate.html

Esto te desactivará el grupo de volúmenes y podrás seguir con tu camino.

Saludos.

A few days ago there was in error on one of our Cisco MDS 9120 fiber switches. The current environment at this datacenter consists of two Cisco MDS 9120 SAN switches with servers redundantly connected between the two. These switches are used to connect the servers to our fiber channel (FC) storage systems. In this case, the servers are generally mapped to an EMC, NetApp, and two RamSans. Below outlines a basic example of what can be expected from multipathing and the Linux environment when there is loss of connectivity to one leg of the fiber network.

Below is an excerpt of the log coming from a dual fiber connected server. In this case, the max active paths would be 2. If the server had quad fiber connectivity it could have a max of 4 paths in our environment. As seen from the log, multipathd reported the loss of a path to each LUN an logged it “remaining active paths: 1″. Once the errors were corrected in the failed switch, multipathd reported the restoration “remaining active paths: 2“.

[root@testsrv01 ~]# cat /var/log/messages | grep multipathd
May  6 10:30:55 testsrv01 multipathd: TempLUN1: remaining active paths: 1
May  6 10:30:55 testsrv01 multipathd: 8:224: readsector0 checker reports path is up
May  6 10:30:55 testsrv01 multipathd: 8:224: reinstated
May  6 10:30:55 testsrv01 multipathd: RanSanLUN2: remaining active paths: 1
May  6 10:30:56 testsrv01 multipathd: 65:128: readsector0 checker reports path is up
May  6 10:30:56 testsrv01 multipathd: 65:128: reinstated
May  6 10:30:57 testsrv01 multipathd: TempLUN1: remaining active paths: 2
May  6 10:30:58 testsrv01 multipathd: 65:224: readsector0 checker reports path is up
May  6 10:30:58 testsrv01 multipathd: 65:224: reinstated
May  6 10:30:58 testsrv01 multipathd: RanSanLUN2: remaining active paths: 2

To be safe, the ‘multipath’ command was used via the Command Line Interface (CLI) on the server(s) to verify that the storage systems were visable to the OS.

NAME
       multipath - Device mapper target autoconfig
SYNOPSIS
       multipath [options] [device]

       -l     show the current multipath topology from information fetched in sysfs and the device mapper
       -ll    show the current multipath topology from all available information (sysfs, the device mapper ...)

Truncated results are below that shows connectivity to the RamSans and EMC. This server is not mapped to the NetApp.

[root@testsrv01 ~]# multipath -ll
ramsan_DramCache (25050c24001094501)
[size=9 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [prio=4][active]
 \_ 2:0:3:0  sdah 66:16  [active][ready]
 \_ 2:0:4:0  sdai 66:32  [active][ready]
 \_ 1:0:3:0  sdal 66:80  [active][ready]
 \_ 1:0:4:0  sdan 66:112 [active][ready]

ramsan_FlashSSD (25050c24021094501)
[size=1853 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [prio=4][active]
 \_ 2:0:3:1  sdaj 66:48  [active][ready]
 \_ 2:0:4:1  sdak 66:64  [active][ready]
 \_ 1:0:3:1  sdam 66:96  [active][ready]
 \_ 1:0:4:1  sdao 66:128 [active][ready]

EMC_LUN1 (461a963004637699a05bs3464f76)
[size=100 GB][features="1 queue_if_no_path"][hwhandler="0"]
\_ round-robin 0 [prio=2][active]
 \_ 2:0:2:8  sdab 65:176 [active][ready]
 \_ 1:0:2:8  sdl  8:176  [active][ready]

Notes: Make sure to verify that all servers have multipathing setup and working correctly, all of the Linux servers never missed a beat! The database (Oracle) and Websphere servers kept on going. The same can not be said about the Exchange environment. Due to a configuration problem there, multipathing was not functioning and it lost connectivity to its storage LUNs.  The VMware cluster survived fine as well.

Graphic design is being the promotional key in every spare of business and individual life. Business organizations seek graphic tools to have publicity by dint of bill board, catalogs, magazines, web pages, store directories, banners, flyers, posters, ads, galleries logo and a lot. Individuals hunt such service for meet different specious motive; for example one can easily get bulk photographs printed using graphic design using Photoshop in a very cost effective manner instead of going to a studio with negative film. Different plug-ins, filters, gradients and other options in graphic design makes any object alive and beautiful.

Clipping path, Color Mask/Multiple Clipping Path and masking are the most common and popular techniques used by the graphic professionals. Pen tool, Magic wand tool, Lasso tool re used initially to erase a part from the image background to compromise those tasks. Experts generally use the Pen tool to expertise themselves with more accuracy and credibility to the customers to place an image in a new background.

What is clipping path?

Simply clip means to cut off any particle from an item. Path cuts out an image. Clipping path simply means erasing selected outlined points from the background. As for example a tailor cuts- off outlined part of a cloth with a scissor.  The object becomes moveable to any other inspired background while clipped.

Procedure:

After running adobe Photoshop,   you will open an image from the File menu. Then you will click on the pen tool in the tool box and see the nib icon on the top toolbar below the menu. Than you will increase the zoom by pressing ctrl add +. Then it will be easy to clip. Pressing Alt you will break the curve in an image. After completion of the path you will make selection. Than copying the selected image you will paste it your inspired background.

Benefits of Pen tool in Clipping Path

Pen tool will help to give a wonderful background in lieu of damaged, old and scratched one in an image to promote your further intentions. It will provide superior, perfect and alive image as well as quality service of yours. The manual pen tool will ensure the selection of the area without changing or modifying a single pixel in an image.  After clipping path by it you will get a bitmap image within a vector path which will be saved as EPS format. EPS format is a file that contains both the vector and bitmap data in the same image. So, this image can be used not only in Photoshop directly but also in Quark Press, Adobe PageMaker, Adobe Illustrator, Macromedia FreeHand and CorelDraw as a vector object for farther design. It is user friendly for the PDF also.

Image Masking:

Masking is created to make an adjustment and attachment of a new layer with the background layer to hide the unwanted pixel with the black and white color exhibition in it.

Procedure of masking in adobe Photoshop:

After copying a parted object from a picture by a lasso tool or pen tool you will paste it in the back ground layer in the Photoshop. Than selecting the image on the layer palate, you will click on the add layer mask in the layer palate. Than you will click on the gradient tool in the tool bar and constantly will see whether the gradient is black and white color in the option bar. If the gradient remains with multicolor on the option bar than, you will double click on the gradient with the mouse in the option bar and apply black and white color. You can also make it changing foreground and background color in the tool bar. Now clicking on gradient toolbar you will drag on the image from the right side. Thus masking will be effective. Color Mask/Multiple Clipping Path: Color Mask will Give your image a fresh new look by applying Photoshop path to multiple objects in a photograph. It will add color and texture to the same image to give totally different look.

Ariful Islam

Marketing Manager

Photoshop Clipping Path (PCP)

Who does not want to have an image with a wonderful back ground? Once, people would go to historical places, sea beaches, hill tracts, river backs, and other places expending money labor and time to have a beautiful background to their images. Now we have a few times to tackle our monotonous and memorize ourselves with the pictures. Now we get any kind of amazing scenery as our image background through search engines. What ever, to have a distinct back ground we are to erase our core object from the existing, old, unexpected and damaged background. We get these tasks done by many tools in Adobe Photoshop like Quick Mask, Magnetic Lasso, Magic Wand and the Pen Tool. Every tool will compromise us to erase our unexpired background quickly and the average way but the manually pen tool. It will erase our core objects in a wonderful way we like the most and expire. So when we will give our images in a new back ground individually than must use pen tool to clip. Again when we will give our bulk images to have catalog, posters, brochures, websites, or other forms as company publicity perspective than we also ask to the graphic professionals and studio owners about the using of Pentool. By virtue of using Pentool, our image will have a bitmap image within a vector path which will be saved as the EPS format. EPS format is a file that contains both the vector and bitmap data in the same image. So, this image can be used easily not only in Photoshop directly but also in Quark Press, Adobe PageMaker, Adobe Illustrator, Macromedia FreeHand and CorelDraw for farther design. It will be user friendly for the PDF also. The printing will also be with the super quality. Though Pentool takes more time to erase any object, yet it will compromise us with the best, super and the spotless images as we always expire.

Ariful Islam

Marketing Manager

Proactive Infotrade Ltd.

Setting Up storage:

For performance benefits while using ASM we would be using iSCSI devices instead of virtual devices on NFS filesystem. We would also use Multipath software along with multiple SCSI Channels or fibre channels between the hosts and storage. This is to avoid single point of fault in the I/O channel and will also enhance performance.

For more info in Multipath: http://en.wikipedia.org/wiki/Multipath_I/O

Hardware setup for the scope of this document:

-sh-3.00$ more /etc/redhat-release

Red Hat Enterprise Linux ES release 4 (Nahant Update 5)

-sh-3.00$ more /proc/meminfo

MemTotal: 8164980 kB

-sh-3.00$ more /proc/cpuinfo |grep processor

processor : 0

processor : 1

processor : 2

processor : 3

Filer: NetApps FS 3050

Install iSCSI software on all the host used for RAC setup:

Download the latest package of iSCSI daemon and utility program from redhat and install it on all the hosts.

sudo rpm –Uvh iscsi-initiator-utils-4.0.3.0-6.x86_64.rpm

Install and configure Linux multipath software on all the hosts:

We are using Linux dm-multipath support for our setup:

Download the latest rpm from redhat and install it on all the hosts.

sudo rpm –Uvh device-mapper-multipath-0.4.5-27.RHEL4.x86_64.rpm

Load the dm_multipath kernel module:

modprobe dm_multipath

Edit the configuration file /etc/multipath.conf as follow:

Comment out the following line:

devnode_blacklist { devnode “*” }

Create a device-specific section at the end of the file for the storage system as follow:

devices {

device {

vendor “NETAPP”

product “LUN”

path_grouping_policy multibus

features “1 queue_if_no_path”

path_checker readsector0

failback immediate

}

}

# This is an example configuration file for device mapper multipath.

# For a complete list of the default configuration values, see

# /usr/share/doc/device-mapper-multipath-0.4.5/multipath.conf.defaults

# For a list of configuration options with descriptions, see

# /usr/share/doc/device-mapper-multipath-0.4.5/multipath.conf.annotated

# Blacklist all devices by default. Remove this to enable multipathing

# on the default devices.

#devnode_blacklist {

# devnode “*”

#}

## Use user friendly names, instead of using WWIDs as names.

defaults {

user_friendly_names yes

}

## By default, devices with vendor = “IBM” and product = “S/390.*” are

## blacklisted. To enable mulitpathing on these devies, uncomment the

## following lines.

#devices {

# device {

# vendor “IBM”

# product “S/390 DASD ECKD”

# path_grouping_policy multibus

# getuid_callout “/sbin/dasdview -j -f /dev/%n”

# path_checker directio

# }

#}

##

## This is a template multipath-tools configuration file

## Uncomment the lines relevent to your environment

##

#defaults {

# udev_dir /dev

# polling_interval 10

# selector “round-robin 0″

# path_grouping_policy multibus

# getuid_callout “/sbin/scsi_id -g -u -s /block/%n”

# prio_callout /bin/true

# path_checker readsector0

# rr_min_io 100

# rr_weight priorities

# failback immediate

# no_path_retry fail

# user_friendly_name yes

#}

##

## The wwid line in the following blacklist section is shown as an example

## of how to blacklist devices by wwid. The 3 devnode lines are the

## compiled in default blacklist. If you want to blacklist entire types

## of devices, such as all scsi devices, you should use a devnode line.

## However, if you want to blacklist specific devices, you should use

## a wwid line. Since there is no guarantee that a specific device will

## not change names on reboot (from /dev/sda to /dev/sdb for example)

## devnode lines are not recommended for blacklisting specific devices.

##

#devnode_blacklist {

# wwid 26353900f02796769

# devnode “^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*”

# devnode “^hd[a-z]“

#}

#multipaths {

# multipath {

# wwid 3600508b4000156d700012000000b0000

# alias yellow

# path_grouping_policy multibus

# path_checker readsector0

# path_selector “round-robin 0″

# failback manual

# rr_weight priorities

# no_path_retry 5

# }

# multipath {

# wwid 1DEC_____321816758474

# alias red

# }

#}

#devices {

# device {

# vendor “COMPAQ “

# product “HSV110 (C)COMPAQ”

# path_grouping_policy multibus

# getuid_callout “/sbin/scsi_id -g -u -s /block/%n”

# path_checker readsector0

# path_selector “round-robin 0″

# hardware_handler “0″

# failback 15

# rr_weight priorities

# no_path_retry queue

# }

# device {

# vendor “COMPAQ “

# product “MSA1000 “

# path_grouping_policy multibus

# }

#}

devices {

device {

vendor “NETAPP”

product “LUN”

path_grouping_policy multibus

features “1 queue_if_no_path”

path_checker readsector0

failback immediate

}

Start the multipath daemon by entering the following command:

sudo /etc/init.d/multipathd start

Add the multipath service to the boot sequence by entering the following commands:

chkconfig –add multipathd

chkconfig –list multipathd

chkconfig multipathd on

Install iSCSI Initiator Support Kit on all the RAC Nodes:

To have a better view on iSCSI devices from hosts, you should download and install iSCSI support Kit from NetApp as follow:

Download iSCSI Linux Initiator Support Kit from http://now.netapp.com/NOW/download/software/kit_iscsi/Linux/ to a working directory on the hosts

Untar the file like

tar -xvf netapp_linux_tools_1_5.tar.gz

When I downloaded the file from this website, I found that the file has a .Z extension but when I tried to unzip the file it gave me error. The I just untar the file and it worked.

cd to the subdirectory netapp_linux_tools_1_5 and run install

./install

Modify iSCSI configurations on all RAC servers:

/sbin/iscsi-iname command will give to the iSCSI initiator name, which can be used to update the file /etc/initiatorname.iscsi. I have added the node name to the initiator name for node reference.

Edit the configuration file /etc/initiatorname.iscsi on all host to contain single line:

[oracle@oradbdev02 ~]$ sudo more /etc/initiatorname.iscsi

## DO NOT EDIT OR REMOVE THIS FILE!

## If you remove this file, the iSCSI daemon will not start.

## If you change the InitiatorName, existing access control lists

## may reject this initiator. The InitiatorName must be unique

## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.

Edit the configuration file /etc/iscsi.conf on all hosts as follow:

If you are using dm-multipath support, remove the comment of the ConnFailTimeout line in the Session Timeout Settings section and set the value to

ConnFailTimeout=5.

Add the following line for any one iSCSI-enabled interface on each storage system that you will use for iSCSI LUNs:

DiscoveryAddress=storage_system_IPaddress

# ============================================================================

# iSCSI Configuration File Sample – see iscsi.conf(5)

# ============================================================================

#

# All of the configuration parameters described in this file are applied

# globally to all targets, unless they are overridden by a local setting. The

# three types of local categories that can override the global settings are:

#

# Target Name (i.e., TargetName)

# Network (i.e., Subnet or Address)

# SCSI Routing Instance (i.e., DiscoveryAddress)

#

# All parameters that are localized to one of the categories above must be

# indented by at least one white space or a tab character. If the parameter is

# not indented, it will be interpreted as a global parameter (see examples for

# each parameter).

#

# The parameters are global if they appear before any DiscoveryAddress

# or TargetName or Subnet entries. The global entries need to

# start at column 1. See the iscsi.conf man page’s Note section for more

# details

#

# If more that one entry exists for any given parameter (either global or

# local), the last entry has precedence.

#

# If a parameter setting under the network category conflicts with a different

# setting of the same parameter under the discovery address or target name

# category (for the same target), the network setting will have precedence.

#

# If a parameter is not specified in the iscsi.conf file, the default setting is

# used. The default values for all parameters can be found in the readme file.

#

# In the sample settings shown below, the following definitions apply:

# = any alpha-numeric text string

# = any numeric text string

#

#
= valid portal address of the form a.b.c.d[:e]

#

# ============================================================================

# PARAMETERS

# ============================================================================

#

# ———————–

# AUTHENTICATION SETTINGS

# ———————–

# To globally configure a CHAP username and password for initiator

# authentication by the target(s), uncomment the following lines:

#

#OutgoingUsername=

#OutgoingPassword=

#

# The maximum length for both the password and username is 256 characters.

# An example username and password would be:

#

#OutgoingUsername=alice

#OutgoingPassword=nty57nbe

#

# To globally configure a CHAP username and password for target(s)

# authentication by the initiator, uncomment the following lines:

#

#IncomingUsername=

#IncomingPassword=

#

# The maximum length for both the password and username is 256 characters.

# An example username and password would be:

#

#IncomingUsername=bill

#IncomingPassword=ghot67

#

# The global authentication settings can be overridden on a per discovery

# address basis. An example of a unique username and password for all targets

# found at address 192.168.10.94 would be:

#

#DiscoveryAddress=192.168.10.94

# OutgoingUsername=fred

# OutgoingPassword=uhyt6h

# and/or

#

#DiscoveryAddress=192.168.10.94

# IncomingUsername=mary

# IncomingPassword=kdhjkd9l

#

# —————

# DIGEST SETTINGS

# —————

# To globally enable CRC32C digest checking for the header part of

# iSCSI PDUs, uncomment the following line:

#

#HeaderDigest=always

#

# To globally disable digest checking for the header data part of

# iSCSI PDUs, uncomment the following line:

#

#HeaderDigest=never

#

# To globally allow the targets to control the setting of the digest checking,

# with the initiator requesting a preference of enabling the checking, uncomment

# the following line:

#

#HeaderDigest=prefer-on

#

# To globally allow the targets to control the setting of the digest checking,

# with the initiator requesting a preference of disabling the checking,

# uncomment the following line:

#

#HeaderDigest=prefer-off

#

# The global digest settings can be overridden on a per target name basis. An

# example of enabling header digest checking just for target iqn.1987-

# 05.com.cisco:00.0d1d898e8d66.t0 would be:

#

#TargetName=iqn.1987-05.com.cisco:00.0d1d898e8d66.t0

# HeaderDigest=always

#

# It should also be noted that if the initiator and the target have incompatible

# settings (e.g., target set for “always” and initiator set for “never”), the

# login will fail.

#

# ———————-

# ENABLE/DISABLE TARGETS

# ———————-

# To globally enable/disable group of targets use the following option.

#

# TargetNames mentioned after the below entry will be enabled by default.

#Enabled=yes

#

# TargetNames mentioned after the below entry will be disabled by default.

#Enabled=no

#

# To specifically enable/disable a target, use the following entry

# under Targetname.

#

#TargetName=iqn.1987-05.com.cisco:00.0d1d898e8d66.t0

# Enabled=yes

#

# —————————

# CONNECTION TIMEOUT SETTINGS

# —————————

# To globally specify the time to wait for a login PDU to be received from

# the target in response to a login request sent by the initiator before failing

# the connection attempt, uncomment the following line:

#

#LoginTimeout=

#

# where is in seconds. A setting of “0″ will result in commands never

# being timed out.

#

# IdleTimeout and ActiveTimeout determine when a ping has to be sent to check

# the status of the connection when there is no traffic on the connection from

# the target.

#

# To globally specify the time to wait on a connection with no tasks active in

# the session, uncomment the following line:

#

#IdleTimeout=

#

# where is in seconds. A setting of “0″ will result in a ping never

# being sent.

#

# To globally specify the time to wait on a connection with tasks active in the

# session, uncomment the following line:

#

#ActiveTimeout=

#

# where is in seconds. A setting of “0″ will result in a ping never

# being sent.

#

# To globally specify the time to wait for a ping response after a ping has been

# sent to a target before failing the existing connection and initiating a new

# one, uncomment the following line:

#

#PingTimeout=

#

# where is in seconds. A setting of “0″ will result in the ping command

# never timing out.

#

# The global connection timeout settings can be overridden on a per target name,

# discovery address or IP address basis. An example of setting the

# “LoginTimeout” value to 12 seconds for just target iqn.1987-

# 05.com.cisco:00.0d1d898e8d66.t0 would be:

#

#TargetName=iqn.1987-05.com.cisco:00.0d1d898e8d66.t0

# LoginTimeout=12

#

# An example of setting the “PingTimeout” value to 8 seconds for just all

# targets found at address 192.168.10.94 would be:

#

#DiscoveryAddress=192.168.10.94

# PingTimeout=8

#

# An example of setting the “IdleTimeout” value to 3 seconds for just all

# targets found on subnet 192.168.10.94 would be:

#

#Subnet=192.168.10.0/24

# IdleTimeout=3

#

# ————————

# SESSION TIMEOUT SETTINGS

# ————————

# To globally specify the length of time to wait for session re-establishment

# before failing SCSI commands back to the application, uncomment the

# following line:

#

ConnFailTimeout=5

#

# where is in seconds. A setting of “0″ will result in commands never

# being failed back due to connection failure.

#

# The global session timeout settings can be overridden on a per target name

# basis. An example of setting the “ConnFailTimeout” value to 5 seconds for

# just target iqn.1987-05.com.cisco:00.0d1d898e8d66.t0 would be:

#

#TargetName=iqn.1987-05.com.cisco:00.0d1d898e8d66.t0

# ConnFailTimeout=5

#

# If a third party multipathing application is being used,

# then the “ConnFailTimeout” should be set to smaller value

# such as 15. This value is just a guideline so the actual value will be

# dependent on the users operating environment.

#

# ——————————-

# ERROR HANDLING TIMEOUT SETTINGS

# ——————————-

# To globally specify the length of time to wait for an abort command to

# complete before declaring the abort command has failed, uncomment the

# following line:

#

#AbortTimeout=

#

# where is in seconds. A setting of “0″ will result in commands never

# being timed out.

#

# To globally specify the length of time to wait for a reset command to complete

# before declaring that the reset command has failed, uncomment the following

# line:

#

#ResetTimeout=

#

# where is in seconds. A setting of “0″ will result in commands never

# being timed out.

#

# The global error handling timeout settings can be overridden on a per target

# name or per IP address basis. An example of setting the “AbortTimeout” value

# to 10 seconds for just target iqn.1987-05.com.cisco:00.0d1d898e8d66.t0 would

# be:

#

#TargetName=iqn.1987-05.com.cisco:00.0d1d898e8d66.t0

# AbortTimeout=10

#

# An example of setting the “ResetTimeout” value to 6 seconds for just all

# targets found on portal 192.168.10.94 would be:

#

#Subnet=192.168.10.0/24

# ResetTimeout=6

#

# —————————–

# CONTINUOUS DISCOVERY SETTINGS

# —————————–

# To globally specify that all discovery sessions be kept open, uncomment the

# following line:

#

#Continuous=yes

#

# To globally specify that all discovery sessions be closed once discovery is

# completed, uncomment the following line:

#

#Continuous=no

#

# The global continuous discovery setting can be overridden on a per target

# basis. An example of setting “Continuous” to “no” for just target iqn.1987-

# 05.com.cisco:00.0d1d898e8d66.t0 would be:

#

#TargetName=iqn.1987-05.com.cisco:00.0d1d898e8d66.t0

# Continuous=no

#

# ———————————

# ASYNC EVENT NOTIFICATION SETTINGS

# ———————————

# To globally specify that the initiator wants to receive vendor specific async

# events from the target(s), uncomment the following line:

#

#SendAsyncText=yes

#

# To globally specify that the initiator does not want to receive vendor

# specific async events from the target(s), uncomment the following line:

#

#SendAsyncText=no

#

# The SendAsyncText key can be specified for a particular Discovery Address.

#

# The global async event notification setting can be overridden on a per target

# basis. An example of setting “SendAsyncText” to “no” for just target iqn.1987-

# 05.com.cisco:00.0d1d898e8d66.t0 would be:

#

#TargetName=iqn.1987-05.com.cisco:00.0d1d898e8d66.t0

# SendAysncText=no

#

# ————————————

# iSCSI OPERATIONAL PARAMETER SETTINGS

# ————————————

# To globally enable R2T flow control (i.e., the initiator must wait for an R2T

# command before sending any data), uncomment the following line:

#

#InitialR2T=yes

#

# To globally disable R2T flow control (i.e., the initiator has an implied

# initial R2T of “FirstBurstLength” at offset 0), uncomment the following line:

#

#InitialR2T=no

#

# To globally enable immediate data (i.e., the initiator sends unsolicited data

# with the iSCSI command packet), uncomment the following line:

#

#ImmediateData=yes

#

# To globally disable immediate data (i.e., the initiator does not send

# unsolicited data with the iSCSI command PDU), uncomment the following line:

#

#ImmediateData=no

#

# To globally specify the maximum number of data bytes the initiator can receive

# in an iSCSI PDU from a target, uncomment the following line:

#

#MaxRecvDataSegmentLength=

#

# where is the number of bytes in the range of 512 to (2^24-1)

#

# To globally specify the maximum number of unsolicited data bytes the initiator

# can send in an iSCSI PDU to a target, uncomment the following line:

#

#FirstBurstLength=

#

# where is the number of bytes in the range of 512 to (2^24-1)

#

# To globally specify the maximum SCSI payload that the initiator will negotiate

# with the target for, uncomment the following line:

#

#MaxBurstLength=

#

# where is the number of bytes in the range of 512 to (2^24-1)

#

# To globally specifiy the maximum number of bytes that can be sent over a TCP

# connection by the initiator before receiving an acknowledgement from the

# target, uncomment the following line:

#

#TCPWindowSize=

#

# where is the number of bytes in the range of 512 to (2^24-1)

#

# The global iSCSI operational parameter setting can be overridden on a per

# target basis. An example of setting multiple parameters for just target

# iqn.1987-05.com.cisco:00.0d1d898e8d66.t0 would be:

#

#TargetName=iqn.1987-05.com.cisco:00.0d1d898e8d66.t0

# InitialR2T=no

# ImmediateData=no

# MaxRecvDataSegmentLength=128 * 1024

# FirstBurstLength=262144

# MaxBurstLength=(16 * 1024 * 1024) – 1024

# TCPWindowSize=262144

#

# The global “TCPWindowSize” setting can also be overridden on a per portal

# basis. An example of setting the “TCPWindowSize” for just subnet 10.77.13.0/16

# would be:

#

#Subnet=10.77.13.0/16

# TCPWindowSize=262144

#

# —————-

# Network Category

# —————-

# To localize parameters to targets on a particular network (i.e., to

# override the global settings), you need to use either the “Subnet” or

# “Address” settings. The format for the “Subnet” setting is a.b.c.d/e. An

# example of these settings would be:

#

#Subnet=10.4.100.0/24

#

# The format for “Address” is a.b.c.d. An example of these settings would be:

#

#Address=10.4.100.0

#

# The following parameters can be specified using the network category:

#

# 1) Connection Timeout Settings

# 2) Error Handling Timeout Settings

# 3) TCP Settings

#

# ————————–

# Discovery Address Category

# ————————–

# To localize parameters to targets found on a particular discovery address

# (i.e., to override the global settings), you need to use the

# “DiscoveryAddress” setting. The format for the “DiscoveryAddress” setting is

# a.b.c.d, a.b.c.d:e (where e is a TCP port number), or an instance name. An

# example of these settings would be:

#

#DiscoveryAddress=10.4.100.0

# or

#DiscoveryAddress=10.4.100.1:3260

# or

#DiscoveryAddress=scisrouter1

#

# The following parameters can be specified using the discovery address

# category:

#

# 1) Authentication Settings

# 2) ConnectionTimeout Settings

# 3) Continuous Discovery settings

# 4) AsyncEvent Notification Settings

#

# ——————–

# Target Name Category

# ——————–

# To localize parameters to targets identified by a particular target name

# (i.e., to override the global settings), you need to use the

# “TargetName” setting. The format for the “TargetName” setting is

# either the ‘iqn’ or ‘eui’ format. An example of these settings would be:

#

#TargetName=iqn.1987-05.com.cisco:00.0d1d898e8d66.t0

#

# The following parameters can be specified using the target name category:

#

# 1) CRC Settings

# 2) iSCSI Operational Parameter settings

# 3) Connection Timeout Settings

# 4) Session Timeout Settings

# 5) Error Handling Timeout Settings

# 6) TCP Settings

# 7) Enable/Disable targets

#

Configure the storage system for iSCSI:

Perform the following tasks on the storage system console:

I would recommend taking help from your storage team to make these changes instead of trying it yourself. These steps don’t fall in the scope of an Oracle DBA.

You can ask your storage admin to create luns of the size you require for your oracle database. Also mention that you would like these luns to be visible from all your RAC nodes.

Enter the following command to check whether iSCSI service is running:

iscsi status

iSCSI service is not running

If it is not running, enter the following command to start it and check it again:

iscsi start

iscsi status

iSCSI service is running

Create iSCSI initiator group with the following command:

igroup create -i -t linux

Example:

igroup create -i -t linux ractest

Add iSCSI initiators to the iSCSI initiator group with the following command:

igroup add

where initiator name is defined in the host’s configuration file /etc/initiatorname.iscsi.

Example:

igroup add ractest iqn.1987-05.com.cisco:01.45b2c1c2e7:oradbdev02

Use the following command to see the initiators and initiator groups

igroup show

Example:

igroup show

igroup1 (iSCSI) (ostype: linux):

iqn.1987-05.com.cisco:01.45b2c1c2e7:oradbdev02 (not logged in)

Create volumes that contain LUNs with the following commands:

vol create size {k|m|g|t}

vol options nosnap on # disable snapshots

snap reserve 0 # disable snapshot reservation

snap sched 0 0 0 # disable the automatic snapshot schedule

For example, the following commands create an 2 GB volume /vol/vol1 on the aggregate aggr1:

vol create ractest aggr1 size 2g

vol options vol1 nosnap on

snap reserve vol1 0

snap sched vol1 0 0 0

Create LUNs with the following command:

lun create -s size -t linux lun_path

where lun_path is the LUN’s path name that includes the volume and qtree.

For example, the following commands create a 500 GB LUN /vol/vol1/ractest.lun in the volume /vol/vol1:

lun create -s 2g -t linux /vol/vol1/ractest.lun

You can use the following command to see all LUNs (From OS):

[oracle@oradbdev02 ~]$ sudo sanlun lun show

Password:

filer: lun-pathname device filename adapter protocol lun size lun state

silo: /vol/vol1/ractest.lun /dev/sda host0 iSCSI 2g (2147483648) GOOD

Map the LUN to an igroup with the following command:

lun map lun_path initiator_group

Example:

lun map /vol/vol1/ractest.lun ractest

Adamlar sonunda yaptı! (Onlardan birisi de benim)

Xpeech VoIP Gateway Türkçe arayüz ve IVR sistemi bulunduruyor. Bunların yanı sıra sektörde en çok tercih edilen Quintum, Audiocodes ve Linksys marka gatewayler ile yarışacak ve geçecek özelliklere sahip.(benim fikrim değil, gerçekten öyle.) Quintum gibi multipath ama ucuz, Audiocodes gibi kaliteli nedeni aynı DSP, Linksys’in yapamadıklarını yapıyor ve 1′den 32′ye kadar analog ve pri port seçenekleri bulunmakta. Neticede ürünün özellikleri şöyle;

SIP Destekliyor
Multipath (Internette sorun oluştuğunda üzerindeki FXO-PSTN portlarına yönlendiriyor)
DSP:Audiocodes
Aynı anda Phonebook ve SIP Proxy destekliyor
FXO-PSTN alternatifleri: zira PSTN’li gatewayler operatorlere hitap ederken, FXO gatewayler dahili üzerinde konumlandırılabilir.
Phonebook Manager özelliği: Phonebook kullanılan fazla şubeli yapılarda kontrol mekanizmasını kolaylaştırıcak bir yapı
Bütünleşik Router Fonksiyonları(ah birde modem olsaymış)

Daha fazla ayrıntı için Web’i vereyim ama mutlaka deneyin (kendi malım diye demiyorum)

İthalatcı firma : www.bircom.com.tr
Firma: www.xpeech.com