Sample Header Ad - 728x90

zpool status hangs forever

2 votes
1 answer
3700 views
Openindiana oi_151a7. zpool with 15x 1TB raidz3 + 2x 8 disk 2TB raidz2. Many disk operations such as zpool status, creating a new iscsi volume (using sbdadm) etc hangs. Using iostat (-xne) I can determine that two block devices have errors, one is with high probability a cdrom, floppy or such, and the other is a 2TB disk (of one of the raidz2 vdevs the zpool consist of). iostat -xne 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 1344 22 1366 c10t32d0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 52 385 437 c9t5000C50057CEF65Fd0 The problem with "c9t5000C50057CEF65Fd0" is that it's not listed in 'format'. 'format' further lists all 16 2TB disks, while iostat lists 17. One of the 2TB disks was replaced 2-3 months ago so Im thinking it's possible that iostat still think it's present although its not. Let me point out that I know I have 16 2TB disks physically in the host, and 16 (as 2x8 disk raidz2) in vdevs. This is a node for backup, so while a reboot is definitely possible it would be very convenient to be able to investigate it further. Iostat and format in it's entirety: extended device statistics ---- errors --- r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b s/w h/w trn tot device 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 69 0 69 c5t0d0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0 c4t0d0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0 c4t1d0 5.0 0.0 80.1 0.0 0.0 0.0 0.0 3.4 0 2 0 0 0 0 c4t2d0 7.0 4.0 112.1 96.1 0.0 0.0 0.0 2.7 0 3 0 0 0 0 c4t3d0 13.0 5.0 208.1 184.1 0.0 0.1 0.0 3.0 0 5 0 0 0 0 c4t4d0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0 c7t50000393D8CAD353d0 143.6 121.1 531.8 681.2 0.0 2.2 0.0 8.3 0 63 0 0 0 0 c7t50000393D8CAD313d0 144.1 132.6 516.6 682.2 0.0 2.4 0.0 8.6 0 69 0 0 0 0 c7t50000393D8CAD243d0 140.1 121.6 614.6 681.9 0.0 3.0 0.0 11.5 0 74 0 0 0 0 c7t50000393D8CAD217d0 142.6 136.6 568.6 678.4 0.0 2.4 0.0 8.5 0 67 0 0 0 0 c7t50000393D8CAD2A7d0 143.1 149.1 539.1 680.9 0.0 2.3 0.0 7.9 0 69 0 0 0 0 c7t50000393D8CAD32Bd0 150.1 118.1 571.1 683.2 0.0 2.7 0.0 10.0 0 71 0 0 0 0 c7t50000393D8CAD29Bd0 143.1 111.6 502.6 679.4 0.0 2.2 0.0 8.7 0 62 0 0 0 0 c7t50000393D8CAD1FBd0 146.1 156.1 539.1 679.2 0.0 2.4 0.0 7.8 0 69 0 0 0 0 c 7t50000393D8CAD24Bd0 146.1 136.6 533.8 680.7 0.0 2.4 0.0 8.5 0 67 0 0 0 0 c7t50000393D8CAD25Bd0 143.1 129.6 529.1 679.9 0.0 2.1 0.0 7.8 0 60 0 0 0 0 c7t50000393D8CABD2Fd0 149.1 123.6 511.6 682.9 0.0 2.5 0.0 9.3 0 73 0 0 0 0 c7t50000393D8CAD35Fd0 142.6 140.6 540.6 680.4 0.0 2.4 0.0 8.3 0 69 0 0 0 0 c7t50000393D8CAD3BFd0 143.6 112.6 555.4 683.4 0.0 2.8 0.0 10.8 0 70 0 0 0 0 c7t50000393D8CAD34Fd0 140.1 143.6 514.1 680.9 0.0 2.5 0.0 8.9 0 71 0 0 0 0 c7t50000393D8CAD33Fd0 138.6 137.6 624.9 682.4 0.0 2.3 0.0 8.4 0 70 0 0 0 0 c7t50000393D8CAD36Fd0 123.1 227.1 494.3 1304.1 0.0 1.3 0.0 3.8 0 35 0 0 0 0 c9t5000C50056114317d0 54.0 122.6 336.2 426.5 0.0 0.3 0.0 1.5 0 8 0 0 0 0 c9t5000C500561161F3d0 51.0 104.1 319.7 427.3 0.0 0.5 0.0 3.2 0 12 0 0 0 0 c9t5000C5005611763Fd0 46.5 101.1 145.3 428.5 0.0 0.3 0.0 2.1 0 9 0 0 0 0 c9t5000C50055FFF9EBd0 38.5 128.6 198.6 427.0 0.0 0.3 0.0 1.6 0 9 0 0 0 0 c9t5000C50055FC6FD3d0 42.5 98.1 168.4 430.0 0.0 0.3 0.0 2.0 0 10 0 0 0 0 c9t5000C50055FFC403d0 53.5 143.6 336.5 423.3 0.0 0.3 0.0 1.5 0 9 0 0 0 0 c9t5000C5005611431Fd0 52.0 143.6 330.2 419.5 0.0 0.3 0.0 1.4 0 8 0 0 0 0 c9t5000C50056000BCBd0 36.0 148.6 270.4 426.5 0.0 0.2 0.0 1.2 0 7 0 0 0 0 c9t5000C50055FC7BF7d0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0 c4t5d0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 1344 22 1366 c10t32d0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 1 10 11 c10t33d0 108.6 257.2 642.4 1297.8 0.0 1.0 0.0 2.7 0 29 0 0 0 0 c9t5000C50057CE4F2Bd0 116.1 238.7 620.1 1295.3 0.0 1.1 0.0 3.2 0 34 0 0 0 0 c9t5000C50057CF51D7d0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 52 385 437 c9t5000C50057CEF65Fd0 126.1 261.2 600.4 1294.3 0.0 1.0 0.0 2.5 0 32 0 0 0 0 c9t5000C50057CE4ED7d0 119.6 238.2 653.7 1289.8 0.0 1.0 0.0 2.9 0 32 0 0 0 0 c9t5000C50057CE50A7d0 112.1 220.6 484.6 1300.8 0.0 1.0 0.0 2.9 0 30 0 0 0 0 c9t5000C50057CEF10Bd0 109.1 257.2 572.6 1294.6 0.0 1.1 0.0 3.0 0 31 0 0 0 0 c9t5000C50057CE37DFd0 121.1 242.2 600.6 1293.8 0.0 1.1 0.0 3.0 0 33 0 0 0 0 c9t5000C50057CE4A6Bd0 0. c4t0d0 /pci@0,0/pci1002,5a1e@d/pci103c,3245@0/sd@0,0 1. c4t1d0 /pci@0,0/pci1002,5a1e@d/pci103c,3245@0/sd@1,0 2. c4t2d0 /pci@0,0/pci1002,5a1e@d/pci103c,3245@0/sd@2,0 3. c4t3d0 /pci@0,0/pci1002,5a1e@d/pci103c,3245@0/sd@3,0 4. c4t4d0 /pci@0,0/pci1002,5a1e@d/pci103c,3245@0/sd@4,0 5. c4t5d0 /pci@0,0/pci1002,5a1e@d/pci103c,3245@0/sd@5,0 6. c7t50000393D8CABD2Fd0 /pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cabd2f,0 7. c7t50000393D8CAD1FBd0 /pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad1fb,0 8. c7t50000393D8CAD2A7d0 /pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad2a7,0 9. c7t50000393D8CAD3BFd0 /pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad3bf,0 10. c7t50000393D8CAD24Bd0 /pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad24b,0 11. c7t50000393D8CAD25Bd0 /pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad25b,0 12. c7t50000393D8CAD29Bd0 /pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad29b,0 13. c7t50000393D8CAD32Bd0 /pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad32b,0 14. c7t50000393D8CAD33Fd0 /pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad33f,0 15. c7t50000393D8CAD34Fd0 /pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad34f,0 16. c7t50000393D8CAD35Fd0 /pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad35f,0 17. c7t50000393D8CAD36Fd0 /pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad36f,0 18. c7t50000393D8CAD217d0 /pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad217,0 19. c7t50000393D8CAD243d0 /pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad243,0 20. c7t50000393D8CAD313d0 /pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad313,0 21. c7t50000393D8CAD353d0 /pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad353,0 22. c9t5000C50055FC6FD3d0 /scsi_vhci/disk@g5000c50055fc6fd3 23. c9t5000C50055FC7BF7d0 /scsi_vhci/disk@g5000c50055fc7bf7 24. c9t5000C50055FFC403d0 /scsi_vhci/disk@g5000c50055ffc403 25. c9t5000C50055FFF9EBd0 /scsi_vhci/disk@g5000c50055fff9eb 26. c9t5000C50057CE4A6Bd0 /scsi_vhci/disk@g5000c50057ce4a6b 27. c9t5000C50057CE4ED7d0 /scsi_vhci/disk@g5000c50057ce4ed7 28. c9t5000C50057CE4F2Bd0 /scsi_vhci/disk@g5000c50057ce4f2b 29. c9t5000C50057CE37DFd0 /scsi_vhci/disk@g5000c50057ce37df 30. c9t5000C50057CE50A7d0 /scsi_vhci/disk@g5000c50057ce50a7 31. c9t5000C50057CEF10Bd0 /scsi_vhci/disk@g5000c50057cef10b 32. c9t5000C50057CF51D7d0 /scsi_vhci/disk@g5000c50057cf51d7 33. c9t5000C50056000BCBd0 /scsi_vhci/disk@g5000c50056000bcb 34. c9t5000C500561161F3d0 /scsi_vhci/disk@g5000c500561161f3 35. c9t5000C5005611763Fd0 /scsi_vhci/disk@g5000c5005611763f 36. c9t5000C5005611431Fd0 /scsi_vhci/disk@g5000c5005611431f 37. c9t5000C50056114317d0 /scsi_vhci/disk@g5000c50056114317 38. c10t32d0 /iscsi/disk@0000iqn.1984-05.com.dell%3Apowervault.md3200i.6782bcb000311a0a000000004df9adc70001,0 39. c10t33d0 /iscsi/disk@0000iqn.1984-05.com.dell%3Apowervault.md3200i.6782bcb000311a0a000000004df9adc70001,31
Asked by 3molo (189 rep)
Jun 4, 2014, 10:57 AM
Last activity: Apr 6, 2019, 01:30 AM