Not sure where I got this from, but its great troubleshooting info for SCSI conflicts.

To identify if SCSI reservation conflicts are preventing a LUN from being accessed:
Verify that the LUN is detected by the ESX host at boot time.

For ESX 3.x, run the command:

# cd /vmfs/devices/disks
# ls vmh*

The output appears similar to:

vmhba0:0:0:0
vmhba0:0:43:0
vmhba0:0:52:0

For ESX 4.X, run the command:

# esxcfg-scsidevs -c

The output appears similar to:

Device UID Device Type Console Device Size Plugin Display Name
eui.00000e1100b81bfc Direct-Access /dev/sda 70007MB NMP Local FUJITSU Disk (eui.00000e1100b81bfc)
mpx.vmhba0:C0:T0:L0 CD-ROM /dev/sr0 0MB NMP Local HL-DT-ST CD-ROM (mpx.vmhba0:C0:T0:L0)
mpx.vmhba3:C0:T0:L0 Direct-Access /dev/cciss/c0d0 34727MB NMP Local VMware Disk (mpx.vmhba3:C0:T0:L0)

If the LUN is not listed, rescan the storage. For more information, see Performing a rescan of the storage (1003988).

Check dmesg. Look for lines that may provide some information if the LUN is having an issue:

Vendor: EMC Model: SYMMETRIX Rev: 5771
Type: Direct-Access ANSI SCSI revision: 03
scsi0 (0,0,52) : RESERVATION CONFLICT
scsi0 (0,0,52) : RESERVATION CONFLICT
scsi0 (0,0,52) : RESERVATION CONFLICT
scsi0 (0,0,52) : RESERVATION CONFLICT
scsi0 (0,0,52) : RESERVATION CONFLICT
scsi0 (0,0,52) : RESERVATION CONFLICT
sdh : READ CAPACITY failed.
status = c, message = 00, host = 0, driver = 00
scsi0 (0,0,52) : RESERVATION CONFLICT
scsi0 (0,0,52) : RESERVATION CONFLICT
scsi0 (0,0,52) : RESERVATION CONFLICT
VMWARE: Device that would have been attached as scsi disk sdh at scsi0,
channel 0, id 0, lun 52
Has not been attached because this path could not complete a READ
command eventhough a TUR worked.
result = 0×18 key = 0×0, asc = 0×0, ascq = 0×0
VMWARE: Device that would have been attached as scsi disk sdh at scsi0,
channel 0, id 0, lun 52
Has not been attached because it is a duplicate path or on a passive
path
scan_scsis starting finish
scan_scsis done with finish

In this example, LUN 52 is inaccessible in the ESX host cluster. Since it is listed in the output of step one, it was accessible at some point (a host reserved the LUN and never released it, possibly due to a SAN switch reboot in the middle of the reservation operation).
To resolve this:
For ESX 3.0, examine all hosts for any pending reservations with the command:

# tail -1 /proc/vmware/scsi/vmhba[0-9]/[0-9]:*

The output appears similar to:

==> /proc/vmware/scsi/vmhba0/0:0 /proc/vmware/scsi/vmhba0/0:43 /proc/vmware/scsi/vmhba0/0:44 /proc/vmware/scsi/vmhba0/0:45 /proc/vmware/scsi/vmhba0/0:46 /proc/vmware/scsi/vmhba1/0:50 /proc/vmware/scsi/vmhba1/0:51 /proc/vmware/scsi/vmhba1/0:52 /proc/vmware/scsi/vmhba1/0:53 /proc/vmware/scsi/vmhba1/0:54 /proc/vmware/scsi/vmhba2/0:0 <==
Active: 0 Queued: 0 Reserved: N Pending Reserves: 0

The ESX host that has Pending Reserves with a value that is larger than 0 is holding the lock.

For ESX 3.5 and 4.0, examine all hosts (also applies to hosts for any pending reservations) with the command:

# esxcfg-info | egrep -B5 “s Reserved|Pending”

The output appears similar to:

Note: Not all of the output is shown.

|—-Console Device…………………./dev/sdd
|—-DevfsPath…………………….. /vmfs/devices/disks
/vml.02000000006001c230d8abfe000ff76c51486715db504552432035
|—-SCSI Level……………………..6
|—-Queue Depth…………………….128
|—-Is Pseudo………………………false
|—-Is Reserved…………………….false
|—-Pending Reservations…………….0
–
|—-Console Device…………………./dev/sda
|—-DevfsPath……………………../vmfs/devices/disks
/vml.02000000006001c230d8abfe000ff76c198ddbc13e504552432035
|—-SCSI Level……………………..6
|—-Queue Depth…………………….128
|—-Is Pseudo………………………false
|—-Is Reserved…………………….false
|—-Pending Reservations……………. 1

The ESX host that has Pending Reserves with a value that is larger than 0 is holding the lock.

Perform a LUN reset to clear the lock.

For ESX 3.x, run the command:

# vmkfstools –lock lunreset /vmfs/devices/disks/vmhba1\:0\:52\:0

For ESX 4.x, run the command:

# vmkfstools –lock lunreset /vmfs/devices/disks/vml.02000000006001c230d8abfe000ff76c198ddbc13e504552432035

Verify that the LUN no longer has any Pending Reserves.

For ESX 3.x, run the command:

# tail -1 /proc/vmware/scsi/vmhba1/0\:52

The output appears similar to:

Active: 0 Queued: 0 Reserved: N Pending Reserves: 0

For ESX 3.5, run the command:

# esxcfg-info -s | egrep -B5 “s Reserved|Pending”

For ESX 4.x, run the command:

# esxcfg-info -s | egrep -B16 “s Reserved|Pending”

Note: If the Pending Reserves is not 0 or Is Reserved is not false, the LUN reset did not succeed.

Do a refresh of the storage window on each ESX host.

If the datastore is not mounted, perform a rescan.

WD S25: SAS σκληρός δίσκος 10.000 RPM, από τη Western Digital! – ( HWBOX ) O/C on first boot

Πριν λίγα χρόνια, μόνο και μόνο που θα ακούγαμε ότι η Western Digital αποφάσισε να μπει στην αγορά των SAS και SCSI, θα μας φαινόταν αληθινά παράξενο και παράδοξο, αφού η εταιρεία είχε γίνει πλέον από διάσημη στα μάτια μας με τους WD Raptor και WD Velociraptor σκληρούς δίσκους των 10.000 RPM και δεν φαινόταν η “ανάγκη” να μπει σε μια αγορά όπως οι enterprises. Τώρα όμως, τα πράγματα αλλάζουν και με τους SSD να υπάρχουν στην αγορά και τις μεγάλες εταιρείες ακόμα να εμπιστεύονται τους SAS HDD για τις δουλειές και τους servers τους, ο WD S25 φαίνεται να είναι μια από τις ιδανικές λύσεις.[next]

 

Temiz çalışır durumda 3 yıl garantili SCSI Ultra 10K Diskler

9GB Ultra3 68PIN 30$

18GB Ultra3 68PIN 50$

36GB Ultra3 68PIN 75$

72GB Ultra3 68PIN 150$

İleltişim : yvzhosts@gmail.com

Tel : 0-532-5015586

Se trata del error hardware más frecuente en cualquier tipo de servidor/ordenador o dispositivo dedicado de almacenamiento:

Fallos de disco duro con errores de escritura

Lo aconsejable en estos casos sería hacer un backup del disco que presenta problemas y sustituirlo. Con la frecuencia que estos errores se presentan, provocadas por factores de entorno (humedad, temperatura, vibraciones, fallos electricos o mecánicos, … ) o por fin de vida útil, la prevención es la herramienta más eficaz de forma que:

Se disponga de un sistema que controle dichas variables de entorno (Utilizar herramientas de monitorización basadas en SMART) y El sistema tenga tolerancia a fallos de disco ( por ejemplo RAID1,5) de forma que cuando presenta un fallo, el disco es aislado y solicita a su administrador que lo sustituya, previniendo las pérdidas de datos y su acceso , ya sea por recostrucción automática en disco auxiliar (spare) y/o recreación de la información (algoritmos de paridad).

No siempre se administran sistemas con un diseño previsor a estos tipos de errores, pueden ser por deficiencias provocadas por los presupuestos de la empresa ( el disco duro históricamente era un soporte de almacenamiento bastante caro) o por algo muy “typical spanish”:

El aprendizaje a base de palos porque no hay previsión.

Por ejemplo, errores hardware en el almacenamiento que provoca caida de algunos de los servicios básicos de una empresa, incluso con pérdidas  datos valiosos e irrecuperables.  Esto generará entonces un replanteamiento de la seguridad en el almacenamiento. Una auditoría y una inversión acorde con el valor y cantidad de datos eliminaría este riesgo, no olvidemos que se suele cumplir la máxima de:

Se tiene lo que se paga

Aún así, sean técnicos o responsables de sistemas de una empresa, convertidos en unos Mc Gyver, tratan de cubrir muchas deficiencias de las instalaciones informáticas, al recaer en ellos la responsabilidad de mantener unas instalaciones inseguras y con bajo presupuesto … Me refiero a esos Googleadores natos, con el destornillador siempre a mano,

que tienen disponibilidad absoluta cuando el jefe lo llama a las 12 de la noche porque no puede conectarse a la VPN de la empresa desde su casa.

La migración de entornos a Linux es una opción muy típica para estas instalaciones, debido a la compatibilidad con el hardware, facilidad de control y supervisión y por lo tanto seguridad, con paquetes completos para estas labores de administración sin ningún coste (para los más arerriesgados), o disponer de un soporte de todo el software de una distribución Enterprise a bajo precio.

En la actualidad, debido al aumento de capacidad y la bajada de precios del disco duro, hasta un usuario particular puede permitirse pagarse el hardware para un sistema raid tolerante a fallos de disco en el PC de su casa, …, tema pendiente para otro post.

Volviendo al tema que tocamos ahora, que hacer cuando hay que reparar unos sectores defectuosos, por ejemplo de un disco sin garantía o contrato de soporte o simplemente testear cuales son esos sectores para valorar el riesgo. Se pueden usar herramientas específicas para esta causa, que tienen los fabricantes de discos y que están basado en la tecnología SMART. Poniendo un ejemplo, en la creación de un raid 5 para uso doméstico, me encontré que la herramienta de software  para raid mdadm se negaba a integrar en el nuevo volumen uno de los discos de tera y medio comprados específicamente para esta función. La finalización del paso de creación del raid  (mdadm –create) terminaba con un sistema degradado. Después de varias búsquedas por internet, por sintomatología determino que es error de escritura para un sector de la LBA específico, por lo que me dispongo abrir un caso a Soporte de Seagate. Soporte me solicitó usar una herramienta propia para diagnóstico y reparación de sectores defectuosos, pero el consejero Google me llevó por otros derroteros, e intenté repararlo con herramientas propias de la distribución. Por equivalencia, como estas herramientas de diagnóstico se basan en SMART instalé smartmontools, paquete de aplicaciones que incluye Debian y demás distros en linux para diagnóstico de discos con SMART implementado.   Lo que a continuación expondré son unos pasos prácticos para en primer momento diagnóstico, y si es posible reparación.

Para diagnosticar y reparar, en el campo que sea lo primero es formarse y el conocimiento no aparece de forma espontánea, viene después de mucho trabajo y experiencia, aunque una cultura de base acelera el proceso de aprendizaje de forma exponencial. Dejo varios links para este propósito:

Básico

- Procedente de la Wikipedia: Una magnífica introducción al disco duro, componentes físicos, divisiones físicas y lógicas, formas de direccionamientos, tecnologías usadas,  … , Un trabajo inmejorable de síntesis en un artículo y en castellano!

- Procedente de Wikipedia: Definición de tecnología SMART,  parámetros que controla para diagnóstico de error y en castellano.

- Procedente de Monografías: Aunque es información básica, en ocasiones obsoleta y orientado a entornos de Microsoft. Se definen conceptos básicos como tiempos de acceso o buffer de disco interesantes.

- Procedente de Wikipedia: Definición, historia, especificaciones,… sobre discos SATA (Serial-ATA).

- Google: El oráculo de Delfos, hasta ahora …  responde preguntas y sobre todo crea nuevas para seguirte respondiendo.

Herramientas referidas en este post

smartmontools – Paquete que contiene dos utilidades smartctl y smartd, para controlar y monitorizar sistemas de almacenamiento que usan SMART.

sg3-tools – Paquete para enviar comandos scsi a dispositivos scsi.

Seatools for DOS: Herramienta de Seagate para diagnóstico y reparación de discos.

Procedimiento de diagnóstico de sectores defectuosos

En este ejemplo se tratará de un entorno GNU/Linux, distribución Debian, al usar sus repositorios se usará la utilidad de paquetes nativas (apt-get) de la distribución, de forma análoga se usará yum en las variantes de Red Hat, Yast en Novell/Suse, …

En un sistema Debian, podremos tirar de repositorio para instalar el paquete smartmontools:

Comprobar que está en repositorio:

nas-ferras:~# apt-cache search smartmontools
smart-notifier – graphical hard disk health status notifier
smartmontools – control and monitor storage systems using S.M.A.R.T.

Instalar paquete smartmontools:

nas-ferras:~# apt-get install smartmontools
Leyendo lista de paquetes… Hecho
Creando árbol de dependencias
Leyendo la información de estado… Hecho
Se instalarán los siguientes paquetes NUEVOS:
smartmontools
0 actualizados, 1 se instalarán, 0 para eliminar y 1 no actualizados.
Se necesita descargar 0B/332kB de archivos.
Se utilizarán 848kB de espacio de disco adicional después de esta operación.
Seleccionando el paquete smartmontools previamente no seleccionado.
(Leyendo la base de datos …
91982 ficheros y directorios instalados actualmente.)
Desempaquetando smartmontools (de …/smartmontools_5.38-2+lenny1_amd64.deb) …
Procesando disparadores para man-db …
Configurando smartmontools (5.38-2+lenny1) …
Not starting S.M.A.R.T. daemon smartd, disabled via /etc/default/smartmontools (warning).

Comprobar la versión instalada:

nas-ferras:~# dpkg -l smartmontools
Desired=Unknown/Install/Remove/Purge/Hold
| Estado=No/Instalado/Config-files/Desempaquetado/Fallo-config/Medio-inst/espera-disparo/pendiente-disparo
|/ Err?=(ninguno)/Retenido/Requiere-reinst/X=ambos problemas (Estado,Err: mayúsc.=malo)
||/ Nombre                            Versión                          Descripción
+++-=================================-=================================-==================================================================================
ii  smartmontools                     5.38-2+lenny1                     control and monitor storage systems using S.M.A.R.T.

DIAGNÓSTICO

Ejecución de test sobre disco sospechoso:

Se puede usar la versión short o long:

nas-ferras:~# smartctl -t long /dev/sda
smartctl version 5.38 [x86_64-unknown-linux-gnu] Copyright (C) 2002-8 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===
Sending command: “Execute SMART Extended self-test routine immediately in off-line mode”.
Drive command “Execute SMART Extended self-test routine immediately in off-line mode” successful.
Testing has begun.
Please wait 255 minutes for test to complete.
Test will complete after Wed Oct 14 01:13:52 2009

Use smartctl -X to abort test.

Una revisión del log mostrará los resultados de los diferentes tests ejecutados sobre el disco de la siguiente forma:

nas-ferras:~# smartctl -l selftest /dev/sda
smartctl version 5.38 [x86_64-unknown-linux-gnu] Copyright (C) 2002-8 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed: read failure       90%        99         2597324513
# 2  Extended offline    Completed: read failure       90%        62         2597324513
# 3  Short offline       Completed: read failure       90%        61         2597324513
# 4  Short offline       Completed: read failure       90%        61         2597324513
# 5  Short offline       Completed: read failure       90%        61         2597324513
# 6  Short offline       Completed: read failure       90%        60         2597324513
# 7  Extended offline    Completed: read failure       90%        60         2597324513
# 8  Extended offline    Completed: read failure       90%        58         2597324513
# 9  Extended offline    Completed: read failure       90%        58         2597324513
#10  Extended offline    Completed: read failure       90%        58         2597324513

Que apunta en todos sus test ejecutados a fallos de lectura en el sector 2597324513.

Ya tenemos diagnóstico,  un sector está defectuoso, siendo el causante de que mdadm no lo quiera integrar como elemento en el raid5, etiquetándolo como disco en fallo.

REPARACIÓN

La reparación será una reasignación del sector en fallo, para un disco SCSI/Fibra sería posible realizarlo con las herramientas del paquete sg3-utils siguiendo los siguientes pasos: Instalación #apt-get install sg3-utils, confirmación del sector defectuoso #sg_verify –lba=2597324513 /dev/sda y reasignar el sector en el firmware del disco con # sg_reassign –grown /dev/sda #sg_reassign –address=2597324513 /dev/sda. Más detallado se encuentra éste procedimiento basado en un caso práctico en éste link.

Para nuestro caso, se trata de un disco SATA, en principio no podemos disfrutar de las sg3-utils para los tipo SCSI, pero una pregunta a Google responde que este paquete en versiones actuales ya tiene soporte para este tipo de discos y trataremos de probar suerte.

nas-ferras:~# sg_verify –lba=2597324513 /dev/sda
verify (10):  Descriptor format, current;  Sense key: Medium Error
Additional sense: Unrecovered read error – auto reallocate failed
Descriptor type: Information
0×0000ffff9ad006e1
medium or hardware error, reported lba=0xffff9ad006e1
nas-ferras:~# sg_reassign –address=2597324513 /dev/sda
REASSIGN BLOCKS not supported
nas-ferras:~# sg_verify –lba=2597324513 /dev/sda
verify (10):  Descriptor format, current;  Sense key: Medium Error
Additional sense: Unrecovered read error – auto reallocate failed
Descriptor type: Information
0×0000ffff9ad006e1
medium or hardware error, reported lba=0xffff9ad006e1

En un principio no responde nada parecido a dispositivo no soportado por la herramienta y la información que proporciona es coherente, pareciendo que interactúa con dispositivos SATA. Pero partiendo  de que se trata de una herramienta que a bajo nivel trabaja con dd, tampoco parece raro que detecte errores de lectura para cualquier dispositivo ya que no tendría porque usar los driver sg ( SCSI generic) para dispositivos SCSI y fibra en esta comprobación.

Nos quitará las dudas cuando forzaremos la reasignación del sector(reallocate):

nas-ferras:~# sg_reassign –grown /dev/sda
READ DEFECT DATA (10) not supported

nas-ferras:~# sg_reassign –address=2597324513 /dev/sda
REASSIGN BLOCKS not supported

En un principio parece que esta tools no está preparada para la reasignación de sectores en dispositivos  SATA, o el firmware del disco no es compatible con las sg3-tools. De todas formas no encuentro referencias a que interactúe con dispositivos SATA en ninguna parte del man de sg3-tools o sg_reassign.

Persistiendo en la forma de realizar una reasignación del sector defectuoso desde linux, se formatea el disco en ext3 y se sigue la guía Bad block HOWTO for smartmontools para el caso de una partición extended:

Se formatea en ext3

nas-ferras:~#mkfs.ext3 /dev/sda2

Se recogen los atributos de la partición /dev/sda:

nas-ferras:~# smartctl -A /dev/sda
smartctl version 5.38 [x86_64-unknown-linux-gnu] Copyright (C) 2002-8 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate     0×000f   103   097   006    Pre-fail  Always       -       88063229
3 Spin_Up_Time            0×0003   100   100   000    Pre-fail  Always       -       0
4 Start_Stop_Count        0×0032   100   100   020    Old_age   Always       -       45
5 Reallocated_Sector_Ct   0×0033   100   100   036    Pre-fail  Always       -       2
7 Seek_Error_Rate         0×000f   100   253   030    Pre-fail  Always       -       675601
9 Power_On_Hours          0×0032   100   100   000    Old_age   Always       -       103
10 Spin_Retry_Count        0×0013   100   100   097    Pre-fail  Always       -       0
12 Power_Cycle_Count       0×0032   100   100   020    Old_age   Always       -       45
184 Unknown_Attribute       0×0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0×0032   060   060   000    Old_age   Always       -       40
188 Unknown_Attribute       0×0032   100   093   000    Old_age   Always       -       59
189 High_Fly_Writes         0×003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0×0022   061   055   045    Old_age   Always       -       39 (Lifetime Min/Max 25/40)
194 Temperature_Celsius     0×0022   039   045   000    Old_age   Always       -       39 (0 21 0 0)
195 Hardware_ECC_Recovered  0×001a   039   030   000    Old_age   Always       -       88063229
197 Current_Pending_Sector  0×0012   100   100   000    Old_age   Always       -       1
198 Offline_Uncorrectable   0×0010   100   100   000    Old_age   Offline      -       1
199 UDMA_CRC_Error_Count    0×003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0×0000   100   253   000    Old_age   Offline      -       59180354371687
241 Unknown_Attribute       0×0000   100   253   000    Old_age   Offline      -       3587000442
242 Unknown_Attribute       0×0000   100   253   000    Old_age   Offline      -       3852047273

Se busca comienzo de la partición sda2:

nas-ferras:~# fdisk -lu /dev/sda

Disco /dev/sda: 1500.3 GB, 1500301910016 bytes
255 heads, 63 sectors/track, 182401 cylinders, 2930277168 sectores en total
Units = sectores of 1 * 512 = 512 bytes
Disk identifier: 0×0009ec06

Disposit. Inicio    Comienzo      Fin      Bloques  Id  Sistema
/dev/sda1   *          63    29302559    14651248+  83  Linux
/dev/sda2        29302560 2930272064  1450484752+  83  Linux

Como sda2 empieza en el sector 293302560 y el sector en error es 2597324513, para obtener la posición del sector en error relativo a la partición sda2 hay que hacer la resta 2597324513 – 293302560 = 2304021953.

Para saber el tamaño de bloque usado por el sistema de ficheros en sda2 :

nas-ferras:~# tune2fs -l /dev/sda2 | grep Block
Block count:              362621188
Block size:               4096
Blocks per group:         32768

Para finalizar las cuentas nos queda por saber que bloque del sistema de ficheros contiene esta LBA. Que sigue la siguiente ecuación

  b = (int)((L-S)*512/B)
Donde:
b = Número de bloque en el sistema de ficheros
B = Tamaño de bloque en bytes del sistema de fichero
L = LBA del sector defectuoso
S = Sector de comienzo de la partición que muestra la herramienta fdisk -lu
y (int) que denota la parte entera.

En nuestro ejemplo quedaría así:

b=((int)(2597324513 - 293302560)*512/4096)= (int) 288002744,125=288002744

Comprobamos que el bloque no está en uso:

nas-ferras:~# debugfs
debugfs 1.41.3 (12-Oct-2008)
debugfs:  open /dev/sda2

debugfs:  testb 288002744
Block 288002744 not in use

Y por último realizar un dd para forzar un reallocate del sector defectuoso como se describe en la guía:

nas-ferras:~# dd if=/dev/zero of=/dev/sda2 bs=4096 count=1 seek=288002744
1+0 records in
1+0 records out
4096 bytes (4,1 kB) copied, 0,000607736 s, 6,7 MB/s

Ejecuto test dos veces:

nas-ferras:~# smartctl -t short /dev/sda
smartctl version 5.38 [x86_64-unknown-linux-gnu] Copyright (C) 2002-8 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===
Sending command: “Execute SMART Short self-test routine immediately in off-line mode”.
Drive command “Execute SMART Short self-test routine immediately in off-line mode” successful.
Testing has begun.
Please wait 1 minutes for test to complete.
Test will complete after Wed Oct 14 01:14:43 2009

Sorpresa !! No se ha producido una nueva reasignación de sectores, se puede comprobar mirando del atributo Reallocated_Sector_Ct en la salida del comando smartctl -A /dev/sda:

nas-ferras:~# smartctl -A /dev/sda
smartctl version 5.38 [x86_64-unknown-linux-gnu] Copyright (C) 2002-8 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate     0×000f   103   097   006    Pre-fail  Always       -       89275771
3 Spin_Up_Time            0×0003   100   100   000    Pre-fail  Always       -       0
4 Start_Stop_Count        0×0032   100   100   020    Old_age   Always       -       46
5 Reallocated_Sector_Ct   0×0033   100   100   036    Pre-fail  Always       -       2
7 Seek_Error_Rate         0×000f   100   253   030    Pre-fail  Always       -       679040
9 Power_On_Hours          0×0032   100   100   000    Old_age   Always       -       104
10 Spin_Retry_Count        0×0013   100   100   097    Pre-fail  Always       -       0
12 Power_Cycle_Count       0×0032   100   100   020    Old_age   Always       -       46
184 Unknown_Attribute       0×0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0×0032   060   060   000    Old_age   Always       -       40
188 Unknown_Attribute       0×0032   098   093   000    Old_age   Always       -       61
189 High_Fly_Writes         0×003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0×0022   066   055   045    Old_age   Always       -       34 (Lifetime Min/Max 26/34)
194 Temperature_Celsius     0×0022   034   045   000    Old_age   Always       -       34 (0 21 0 0)
195 Hardware_ECC_Recovered  0×001a   038   030   000    Old_age   Always       -       89275771
197 Current_Pending_Sector  0×0012   100   100   000    Old_age   Always       -       1
198 Offline_Uncorrectable   0×0010   100   100   000    Old_age   Offline      -       1
199 UDMA_CRC_Error_Count    0×003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0×0000   100   253   000    Old_age   Offline      -       143872814481511
241 Unknown_Attribute       0×0000   100   253   000    Old_age   Offline      -       3587013738
242 Unknown_Attribute       0×0000   100   253   000    Old_age   Offline      -       3874184123

Y lógicamente los resultados de los test ejecutados después del dd, siguen dando fallo en mismo sector:

nas-ferras:~# smartctl -l selftest /dev/sda
smartctl version 5.38 [x86_64-unknown-linux-gnu] Copyright (C) 2002-8 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed: read failure       90%       103         2597324513
# 2  Short offline       Completed: read failure       90%       103         2597324513
# 3  Extended offline    Completed: read failure       90%        99         2597324513
# 4  Extended offline    Completed: read failure       90%        62         2597324513
# 5  Short offline       Completed: read failure       90%        61         2597324513
# 6  Short offline       Completed: read failure       90%        61         2597324513
# 7  Short offline       Completed: read failure       90%        61         2597324513
# 8  Short offline       Completed: read failure       90%        60         2597324513
# 9  Extended offline    Completed: read failure       90%        60         2597324513
#10  Extended offline    Completed: read failure       90%        58         2597324513
#11  Extended offline    Completed: read failure       90%        58         2597324513
#12  Extended offline    Completed: read failure       90%        58         2597324513

Tampoco este HOWTO nos lleva a la reparación del sector defectuoso.

Herramienta SEATOOLS for DOS

Queda usar la herramienta propia de Seagate llamada SEATOOLS, que se trata de un LiveCD basado en FreeDOS que posee una interfaz gráfica que reconoce los discos SATA y proporciona herramientas de diagnóstico, reparación y configuración. Esta herramienta se puede descargar desde el siguiente link.

Los resultados de ejecución de esta herramienta han sido eficaces, detecta en su Basic Test Short  el sector defectuoso, y en su Basic Test Long en su finalización solicita la reparación del mismo. Con la reparación ha pasado los test del mdadm  e integrado en el raid5 sin problemas. Sólo queda tenerlo un tiempo en observación para comprobar que es sólamente un error puntual en el sector o el disco es inestable con tendencia a errores.

Conclusiones:

• Herramientas basadas en SMART como smartctl, es eficaz en el diagnóstico de sectores defectuosos.
• La herramientas de diagnóstico propias de los fabricantes de discos, pueden ser el único medio para la reparación de sectores defectuosos. Además, automatizan el proceso de análisis y reparación de forma cómoda. Cuyo único inconveniente en la herramienta usada es que al ser LiveCD se necesita reiniciar el ordenador y dejarlo fuera de servicio varias horas mientras se ejecuta el análisis de disco. Una integración en el sistema operativo de esta herramienta sería lo ideal si no se quieren parar los servicios de la máquina durante tanto tiempo. Otra opción, la que usé, es apagar el ordenador, extraer el disco y ponerlo en otro PC para que ejecute el diagnóstico, cuya parada fue de pocos minutos (apagar, extraer disco y encender).
• De forma preventiva es altamente aconsejable usar otra herramienta como smartd perteneciente al mismo paquete de smartmontools, que realiza un continua monitorización del estado de los discos, quedando pendiente debido a su extensión para otro post.

A while back, I discussed speculation from EMC around Emulex’s proposed cloud-block storage appliance, E3s (Enterprise Elastic Storage).  With my current focus on Cloud Storage, I thought it would be good to delve a bit deeper into some of the aspects of why block-based cloud computing could prove tricky and why without an appliance it may be impossible.

Block Storage Legacy

Today, block-based I/O still uses the SCSI protocol to communicate between a host/server and a disk storage device.  SCSI has been around since 1981 when devices were physically connected to the server itself using a controller board and old-style ribbon cables.  Obviously we’ve progressed somewhat since then and seen the virtualisation of the physical SCSI interface into Fibre Channel and IP SCSI implementations (and in the future FCoE).  Both FC and iSCSI have removed the need for a dedicated SCSI controller card and replaced it with Host Bus Adaptors (HBAs) and Network Interface Cards (NICs).  Irrespective of this, the underlying communication protocol remains the same and the concept of the “Initiator” (host/server) and “Target” (storage device) persist today.  The Initiator starts (or initiates) an I/O request; the target services that request, reading or writing data.

Writing In Blocks

We need to focus for a moment on the concept of block-based I/O versus file-based I/O.  Block-based I/O has no concept of the format of the data being written to the block device (let’s call it a LUN).  This is in contrast to file-based I/O where the storage device understands the data format and manages the content accordingly, ensuring data access is serialised correctly and that files are held in a logical structure (a file system).  Unfortunately block-based I/O is just “dumb” storage and the host itself is responsible for overlaying a file system onto block-based devices.   These JBBDs (“Just a  Block-Based Device”) can be used singly or combined in complex ways to create the file system the host sees.  This combination can be achieved using native Logical Volume Managers (LVMs) on the operating system, or add-ons such as Veritas Volume Manager.  Consequently, an individual LUN could contain an entire file system or only a small component of one.  Either way, the host operating system depends on one feature of the storage device to ensure data integrity and that’s Write Order Consistency.

Write Order Consistency

Preserving the order in which data is written to disk is a fundamental requirement for modern Journalling file systems like NTFS.  Retaining write consistency ensures the file system can be recovered in the event of a server failure or failure of the link between server and disk.  Ordered writes are also essential where data is replicated from one storage array to another usually at a remote location.  In the event that the primary array is lost, the file system can be recovered in a consistent fashion on the remote device, even if it isn’t 100% up to date.

Remote replication can occur either synchronously or asynchronously.  In synchronous mode, I/Os are acknowledged from the remote array before the I/O is confirmed as completed to the host.  This ensures both the primary and remote copies are write-order consistent because the host doesn’t receive acknowledgement of I/O completion until both the local and remote copies are written.  Write-order consistency is implicitly guaranteed.  However, the penalty for this guarantee is the increase in latency (or host I/O response time) which results and increases with the distance between the two devices.  Asynchronous replication is slightly different.  Write I/Os to the primary array are acknowledged immediately, then queued up for writing to the remote device.  The delay in writing data to the remote location is dependent on the bandwidth available and the latency (effectively the distance) between the devices. In any event, as long as write-order is maintained, the remote copy can be recovered even if it doesn’t represent the absolute latest copy of data.  One last consideration.  We touched earlier on how LVMs can combine single LUNs to create complex file systems.  When asychronous replication is involved, write I/O for a single entity like a file system need to be treated together for write-order consistency.  Therefore Consistency Groups allow multiple LUNs to be grouped together for write ordering, ensuring that all I/O for the file system is ordered correctly.  This requirement doesn’t exist for synchronous replication as the consistency is guaranteed by ensuring the write I/O has completed on both the source and target array before acknowledgement to the host.

The Need for an Appliance

Knowing how write-ordering affects consistency is important in understanding how a block-based device would be replicated into “the cloud”.  Due to latency issues, it is unlikely that synchronous replication would be offered as a method of replicating data from a host server into Amazon S3 or EMC Atmos.  Instead, the most likely process will be asynchronous processing and that means installation of a dedicated appliance.  The question is, where in the data path should it sit?

The Splitter

There’s no requirement for a host/server to be connected into a storage array in order to utilise cloud storage.  Instead, at some point in the data path between host and local disk, a copy of the write I/O needs to be taken.  This could occur at the file system level using a host agent or in a SAN environment could happen in the fabric or from the array itself.  Wherever data is taken from, some form of “I/O splitter” is needed to capture write I/O as it is being transferred to disk.    This technology already exists today in products like EMC’s RecoverPoint and Brocade’s Data Mobility Manager.

So here are our requirements for a block-based storage protocol:

• Write Order Consistency
• Consistency groups
• An I/O splitter or replicator

A Theoretical Implementation

Here’s how I think block-based storage could be implemented.  I’ll use the Atmos and Amazon S3 protocols to demonstrate the process.  Firstly, data will be stored in blocks.  Both S3 and Atmos store data as objects and so each object will need to represent a block.  The file system structure can be used to store individual LUNs, with a directory representing a LUN.  For example LUN 12, block 12343 could have the object name:

\S3\Array1\LUN12\12343

It’s worth noting that there’s a distinct difference between the way Atmos and S3 implement updates to objects.  S3 replaces the entire object, whereas Atmos allows part of an object to be updated.  So, Atmos could store an entire LUN as an object whereas S3 can’t, unless the entire LUN is replaced on each write.  Clearly that’s impractical but does indicate that each API implementation will have certain benefits and disadvantages.

So, how big are these blocks going to be?  Ideally, they’d be as small as a typical hard drive block at 512 bytes, however blocks of this size will seriously hamper throughput if write consistency is to be maintained; imagine 5ms latency into the cloud; that’s only 200 IOPS and consequently a throughput of 100KB/s. What’s more practical is a block size matching the O/S  file system and/or database, say 8KB.  Even at this size, with 5ms latency and a single thread of I/O, that’s only 1.6MB/s throughput.

Obviously this level of throughput is not going to be acceptable and there’s a real sticking point here.  The cloud isn’t intelligent.  It will write data as it’s received.  There’s no locking control and the delivery mechanism could be unpredictable.  If writes are issued in parallel, there’s no way to guarantee the I/Os are written to the cloud in the right order.  So perhaps a different approach is required.  Data writes to the target LUN need to be written in a log format, with the name of the object comprising both the block number and a sequence number.  This could be something simple as follows:

\Atmos\Array1\LUN12\SequenceNumber-BlockNumber

e.g.   \Atmos\Array1\LUN12\123445343434366-12343

As the LUN is written to, the sequence number (unique to the LUN or consistency group) is incremented for each write.  The I/Os can then be written in parallel as the sequence numbers track what has and has not been received.  At this point there are two choices – retain all the block updates (unlikely due to the growth in storage usage) or post process the writes, deleting all the written blocks where another later copy exists and where there are no gaps in the sequence numbers.  If there is a gap, then the LUN writes are only guaranteed back to the point where the sequence number gap occurred.  Restoring the LUN for access means processing the LUN block data before it can be read again by the host.

Summary

OK, so this post presents an idea of some of the issues involved in writing block-level data into the cloud.  Data needs to retain integrity and consistency, but performance and throughput are an issue.  Cloud storage has no intelligence, so writing and managing data needs to be handled somewhere, probably using an appliance.  The appliance guarantees the data integrity which can’t be achieved with the cloud alone.  Each Cloud Storage API implementation will have similar features, so using generic CRUD (Create/Read/Update/Delete) commands on objects representing blocks means any service could be used to store data.  It also enables data to be replicated between services so vendor lock-in can be avoided.

I’d be interested in receiving feedback to see if anyone else has thought how block-based cloud storage could be achieved.

Our client is looking for a Storage Diagnostic Test Engineer.  This is a full time position in the San Francisco Bay Area.

This individual would be a principal diagnostic and test engineer for FCoE (Fibre Channel over Ethernet) development. This engineer will be responsible for Protocol conformance testing and test plan development and execution of a FCoE SAN Fabric designed around a converged NIC adaptor or CNA.

Description:

Major functions and duties:

• Debug issues with Fibre Channel , block devices, fabric switches and HBAs
• Develop test suites and code instrumentation to evaluate HBA performance and protocol conformance
• Interact with team in multiple locations
• Unit test and replicate SAN Fabric issues
• Design and communicate solutions and plans effectively

Detailed knowledge of:

• SCSI Protocol for block based devces
• Fibre channel protocol and SANs

Interactive Skills:

• Interact with the team, vendors and customers to replicate and solve issues rapidly and efficiently
• A pragmatic approach to prioritization of tasks
• Communicate clearly and succinctly solutions and analysis

Requirements:

• Minimum 10 yrs development experience.
  Fibre Channel expertise in SANs including debugging protocol problems a plus
• Masters Degree in Electrical Engineering and CS

If you are interested, please send your resume to kathy@mindsource.com.

We thought it would be a good idea to spend a few weeks highlighting some of the different scanner manufacturers and what kind of equipment and service they offer and the benefits of purchasing their product.  This week we are going to cover Fujitsu scanners. 

Fujitsu offers a very wide range of scanner models from the ScanSnap S300 which is rated at 8 pages per minute (ppm) all the way up to the 5900C, which is rated at 100ppm.  For a more complete list of the Fujitsu models available, click here.  The majority of Fujitsu’s mid-range scanners come with an on-site 3 month Next Business Day response warranty.  They also come with a nice package of software including:  Kofax VRS, Adobe Acrobat Standard, and ScandAll21.  Also, most of the scanners are now Dual Interface: Ultra SCSI & USB connectivity, see blog about USB vs. SCSI scanners.

A couple of the newer models that Fujitsu has released, the 6670 and 6770, have 15,000 page daily duty cycles.  That means that they can scan 15,000 pages in one day and not shorten the life span of the scanner.  That is about twice the daily duty cycle of the models they replaced, the 5650C and the 5750C.  That high of daily duty cycle is quite good for a scanner in its price range.  For the lower volume and price range, another two new models are fi-6130 and fi-6140. These little workhorses scan 40ppm and 60ppm and an optional Flatbed can be purchased separately for both models.

Other benefits of the Fujitsu scanners are the service offerings. Fujitsu is one of the few remaining scanner manufacturers that maintains their own warranty and post warranty contracts. This permits for high availability of quality phone support prior to an on-site repair call as well as high availability of parts when and if an on-site service call is needed. Fujitsu service offerings range from Basic coverage (any hardware failure that isn’t consumables, see blog about the difference between parts and consumables) to the more elaborate ScanCare package. ScanCare is a more comprehensive contract which can include several Preventative Maintenance (PM) visits per year and consumables kit(s) depending on the model. First time buyers can sometimes experience sticker shock over the cost of some of these contracts but once you calculate the guaranteed PM visits, the consumables and the end user training,  you should be able to quickly gauge the value. Plus to replace a major part on scanners can often run into the thousands of dollars.  Contracts are much cheaper than replacement scanners and if it’s a mission critical scanner can you really afford not to be covered?

Andrea Latham, CDIA+
ImageSource, Inc.
Phone 360.943.9273
www.imagesourceinc.com

der esxi server hat von haus aus probleme ide laufwerke bzw den ide controller zu erkennen, sodass eine installation mit dem hinweis Unable to find a supported device to write abgebrochen wird. mithilfe der anpassung einer kleinen datei ist es jedoch möglich esxi zu installieren. sobald man bei der installation an dem punkt angelangt ist, der einem sagt das die hdd nicht erkannt wurde muss man mithilfe von ALT+F1 in die erste shell wechseln und sich als root  ohne pw anmelden. mithilfe des befehls

lspci

bzw

fdisk -l

kann man feststellen ob die hdd vom kernel erkannt wurde. sollte dies der fall sein kann man mithilfe von

vi /usr/lib/vmware/installer/Core/TargetFilter.py

esxi dazu überzeugen die platte doch zu erkennen. man muss lediglich nach dem abschnitt def IDEFilter(lun) suchen und folgenden eintrag

return interface.GetInterfaceType() == ScsiInterface.SCSI_IFACE_TYPE_IDE

durch folgenden ersetzen:

return interface.GetInterfaceType() == ScsiInterface.SCSI_IFACE_TYPE_ISCSI

nun muss man nur noch vi beenden und install in die shell tippern, dann kann der fröhliche installations spaß weitergehen!

Açılımı Internet Small Computer System Interface olan iSCSI, bilgisayar depo belleklerini uzaktan ekleme/yönetme mimarisi olarak tanımlanabilir. iSCSI, *SCSI komutlarını IP ağları üzerinden yayınlayarak ağlar üzerindeki veri transferini kolaylaştırır ve depo belleklerinin uzaktan yönetilmesini sağlar. Aynı zamanda iSCSI, yerel ağlar üzerinden veri iletiminde de kullanılabilir.

Esasen, basitçe iSCSI iki bilgisayarın IP ağları üzerinden *SCSI komutları ile görüşmesini sağlar.

iSCSI, SCSI aletlerinin iletişimini sağlayabilmesine rağmen, sistem yöneticileri iSCSI’yi genellikle bellek depoları üzerindeki disk alanlarına erişmek için sunucu bilgisayar olarak kullanıyorlar.

*SCSI : İstemci-sunucu(client – server) mimarisi

Over the course of my time selling imaging hardware, i.e. scanners, many customers ask me which is better…SCSI or USB 2.0?  Actually some don’t even ask, they insist that SCSI (small computer system interface) is still the fastest option.  Unfortunately for them, almost all scanner manufacturers have moved to using a USB 2.0 interface as option on their scanners and some are USB 2.0 exclusively. 

When the first USB (Universal Serial bus) it was slower than SCSI and at times Flakey. Anyone who has ever installed a SCSI device knows that there was definitely room for improvement in this technology as well. Between double checking SCSI ID’s and triple checking the Termination you could spend quite a bit of time sorting out a SCSI install if it didn’t go right the first time. But when USB 2.0 came out it was stable and fast. In fact there is little to no degradation in speed with the scanners.  In fact, Kofax has stopped manufacturing SCSI cards all together.  VRS used to be a limitation of USB in a production environment because it required a SCSI adrenaline board interface, but now the latest versions of VRS are more versatile and work with USB 2.0 just as well if not better than SCSI.

USB 2.0 is working its way to becoming the standard in imaging technology because it less expensive than SCSI and so much easier to work with.  You also have one less point of failure with the USB 2.0 vs. the SCSI card.  So for those people with the common misconception that SCSI is faster and better, I’m here to tell you that USB 2.0 is just as good when it comes to speed and less expensive for your pocketbook. It also opens up the door to using laptops.

ImageSource, Inc. offers the all of the best names in imaging hardware – Fujitsu, Bowe Bell & Howell, Panasonic, Canon– each one with their own strengths, but all equipped with the newest and latest connectivity of USB 2.0

Andrea Latham, CDIA+

Inside Sales

ImageSource, Inc.

What a couple of days!  We’re certainly keeping the guys a HP support in employment.  This is a tale of two device, two support experiences and load of woes…

Server

Yesterday one of our three main servers decided to reboot of it own accord.  I wash just starting to look into the problem when it did it again.  A blue screen flashed up with the words “Fatal Hardware Error” and it restarted.  It’d be nice to know what hardware is fatal!

I run the ultimate boot cd, http://www.ultimatebootcd.com/, Memtest86+ runs without error as does the CPU stress test.  Restart the server and boot Windows long enough to check the system logs.  Last error before reboot was an error on the SCSI bus.  The SCSI controller is embedded on the motherboard – oh hum!  I raise a case with HP.

HP are adament that since I can boot into Windows there’s not a problem with the SCSI controller.  Could it be one of the hard drives?  The RAID controller doesn’t think so.  On HP’s advice we install the HP system management tools and the latest firmware for the various server components.  The HP system management tools insist all hardware is working perfectly, no problems.

Today the server restarted atleast 6 times between 7am and 12pm.  The most we’re getting out of HP’s logs is an angry “Automatic Server Recovery” that is upset with the server keep restarting.  The ASR is not as angry as I am!  HP get us to patch the server again with some more monitoring agents to try and diagnose what is causing all the problems.  What do you know, now we want it to reboot itself the bloody thing has been as solid as a rock all afternoon.  Grrrr.

HP have been very fast to respond and couldn’t be  more helpful.  Top class server support.  The server is in warranty until January 2010.

Printer

We have a mamouth HP Colour Laser Jet 5550dtn A3 printer (actually we have two).  Yesterday it stopped accepting print jobs.  Nick investigated and found a strange message on the LCD – “RFU not loaded, reintall via parallel”.  WTF?

We found an old laptop with a parallel port, a compatible (HP specialist) printer cable and a copy of the firmware from HP’s website.  We tried to upload the firmware back to the printer, as per instructions.

Printer says “firmware loaded, cycling power”.  Laptop says “receiving device not ready, 0 files copied”.  Well, the printer claims it recieved the firmware but the power cycling is taking it’s time.  So we leave it over night.  This morning the LCD still says “firmware loaded, cycling power”.  Mmmm somethings not right here, says I and turns the printer off at the mains.  When I turn it back on the motors come to life, but the rest of the printer is dead.

I think we a b0rked the firmware!

Raise a case with HP who unhelpfully phone me twice to say the printer is out of warranty.  Both times they agreed to forward the case on to the sales department so I can get the replacement parts.  Still no word from HP.  After being so helpful with the server it looks like I’ve stumbled upon HP support of old.

Luckily the firmware is on a compact flash card and searching the internet shows a retail price of about $230.  So hopefully one of my standard suppliers can fix me up with one of these.  If I can’t repair it, it’s £3500 for a new printer!

Ouch!

Error “Windows is unable to Install to the selected location.” 0×80300001

Dell 670 with SCSI Raid controller. Loaded drivers for Controller but didn’t replace the Windows install dvd this error is because the disk was not put back in.

Stupid Microsoft, please change this error to mean something or give a prompt for install disk so we stop wasting time.