多通道儲存出現鬼盤

本來應該只有連線到主儲存的通道才會在/dev下生成裝置檔案，但是有一定機率出現：備儲存的通道被linux識別到了，但是無法使用，於是在oracle伺服器上報通道io錯誤。

這個錯誤不會對生產造成實質影響，如果要幹掉鬼盤，需要儲存那邊進行排查，並且客戶端伺服器需要重啟。

messages日誌報錯：

Oct 31 14:10:58 p2ccdbbj01 kernel: end_request: I/O error, dev sdap, sector 0
Oct 31 14:10:58 p2ccdbbj01 kernel: end_request: I/O error, dev sdap, sector 8
Oct 31 14:10:58 p2ccdbbj01 kernel: end_request: I/O error, dev sdap, sector 0
Oct 31 14:10:58 p2ccdbbj01 kernel: end_request: I/O error, dev sdaq, sector 0
Oct 31 14:10:58 p2ccdbbj01 kernel: end_request: I/O error, dev sdaq, sector 0
Oct 31 14:10:58 p2ccdbbj01 kernel: end_request: I/O error, dev sdaq, sector 2097024
Oct 31 14:10:58 p2ccdbbj01 kernel: end_request: I/O error, dev sdaq, sector 2097136
Oct 31 14:10:58 p2ccdbbj01 kernel: end_request: I/O error, dev sdaq, sector 0
Oct 31 14:10:58 p2ccdbbj01 kernel: end_request: I/O error, dev sdaq, sector 8
Oct 31 14:10:58 p2ccdbbj01 kernel: end_request: I/O error, dev sdaq, sector 0
Oct 31 14:10:58 p2ccdbbj01 kernel: end_request: I/O error, dev sdcf, sector 0
Oct 31 14:10:58 p2ccdbbj01 kernel: end_request: I/O error, dev sdcg, sector 0
Oct 31 14:10:58 p2ccdbbj01 kernel: end_request: I/O error, dev sdap, sector 0
Oct 31 14:10:58 p2ccdbbj01 kernel: end_request: I/O error, dev sdaq, sector 0
Oct 31 14:10:58 p2ccdbbj01 kernel: end_request: I/O error, dev sdcf, sector 0
Oct 31 14:10:58 p2ccdbbj01 kernel: end_request: I/O error, dev sdcg, sector 0
Oct 31 14:10:58 p2ccdbbj01 kernel: end_request: I/O error, dev sdap, sector 0
Oct 31 14:10:58 p2ccdbbj01 kernel: end_request: I/O error, dev sdaq, sector 0
Oct 31 14:10:59 p2ccdbbj01 kernel: end_request: I/O error, dev sdcf, sector 0
Oct 31 14:10:59 p2ccdbbj01 kernel: end_request: I/O error, dev sdcg, sector 0

/proc/partitions可以看到sdcf等

  69    16     153600 sdcd
  69    17     153570 sdcd1
  69    32     153600 sdce
  69    33     153570 sdce1
  69    48    1048576 sdcf
  69    64    1048576 sdcg
253     0  208666624 dm-0
253     1  208666624 dm-1
253     2  418381824 dm-2

但是用fdisk -l  | grep sdcf看不到報錯的裝置

最後在/dev/disks/by-path下發現，出問題的裝置的確是fc裝置，只是後面的地址和其它通道出入很大

lrwxrwxrwx 1 root root 11 Oct 30 18:20 pci-0000:42:00.0-fc-0x5006016446e037ac:0x0012000000000000-part1 -> ../../sdbj1
lrwxrwxrwx 1 root root 10 Oct 30 18:20 pci-0000:42:00.0-fc-0x5006016446e037ac:0x0013000000000000 -> ../../sdbk
lrwxrwxrwx 1 root root 11 Oct 30 18:20 pci-0000:42:00.0-fc-0x5006016446e037ac:0x0013000000000000-part1 -> ../../sdbk1
lrwxrwxrwx 1 root root 10 Oct 30 18:20 pci-0000:42:00.0-fc-0x5006016446e03829:0x0000000000000000 -> ../../sdcf
lrwxrwxrwx 1 root root 10 Oct 30 18:20 pci-0000:42:00.0-fc-0x5006016b46e037ac:0x0000000000000000 -> ../../sdbl
lrwxrwxrwx 1 root root 11 Oct 30 18:21 pci-0000:42:00.0-fc-0x5006016b46e037ac:0x0000000000000000-part1 -> ../../sdbl1
lrwxrwxrwx 1 root root 10 Oct 30 18:20 pci-0000:42:00.0-fc-0x5006016b46e037ac:0x0001000000000000 -> ../../sdbm
lrwxrwxrwx 1 root root 11 Oct 30 18:20 pci-0000:42:00.0-fc-0x5006016b46e037ac:0x0001000000000000-part1 -> ../../sdbm1

lrwxrwxrwx 1 root root 10 Oct 30 18:20 pci-0000:42:00.0-fc-0x5006016b46e03829:0x0000000000000000 -> ../../sdcg