Last weekend, I found the origin of the SVM bug using the mdb tool. Good reading !!
After restarting the server, I wanted to mount a filesystem (metaset object), but the following command was not responding...
# metaset
^Cmetaset: Interrupt
Very strange... I tested another SVM command.
# metastat
^Cmetastat: Interrupt
Hmm "bizarre"... All SVM commands were blocked. Is there a blocked SVM process ?
# pgrep -fl meta
12140 /usr/sbin/metasync -r
Yes of course !? But this process is only used to resync a meta element. Is there a problem with a device ? My first action : I checked the system logs looking for disk errors. No errors
were reported in the system logs.
What does the metasync process do ? I check the activity of this process
# truss -aefl -p 12140
12140/1: psargs: /usr/sbin/metasync -r
^C
Nothing... A more thorough analysis is essential to understand this problem. I use mdb to check a stack of this thread.
# mdb -k
Loading modules: [ unix genunix dtrace specfs ufs sd mpt pcisch sgsbbc ssd fcp fctl md ip hook neti qlc sgenv
sctp arp usba lofs zfs cpc fcip random crypto logindmux ptm nfs ipc ]
>
> 0t12140::pid2proc
60053be24f0
> 60053be24f0::walk thread |::findstack -v
stack pointer for thread 30160427920: 2a100e92d21
[ 000002a100e92d21 sema_p+0x138() ]
000002a100e92dd1 biowait+0x6c(600520ca500, 0, 18b7c00, 300b17ae000, 8, 600520ca500)
000002a100e92e81 default_physio+0x388(12f650c, 200, 0, 600520ca540, 12e6c38,
600520ca538)
000002a100e92fb1 scsi_uscsi_handle_cmd+0x1b8(20000000b0, 1, 60053d18238, 12f650c,
600520ca500, 60053fed7c0)
000002a100e930a1 sd_send_scsi_cmd+0x114(20000000b0, 1964c00,
60053fed7c0, 1, 3009f2ca6c0, 2a100e93a30)
000002a100e93161 sd_send_scsi_RDWR+0x2bc(600500ccf80,
10000, 14, 2a100e93a1c, 2a100e93a70, 1)
000002a100e93281 sd_use_efi+0x90(3009f2ca6c0, 1, 0, 6a945a3b, 130a3fc, 3041df72000)
000002a100e93351 sd_validate_geometry+0xc8(3009f2ca6c0, 1, 60, 1, 7, ea000050)
000002a100e93411 sd_ready_and_valid+0x2d4(3009f2ca6c0, ea, ea000050, 600500ccf80, 30, 0)
000002a100e93521 sdopen+0x248(8, 3009f2ca6c0, 3, 196c6d8, 3009f2ca7a0, 0)
000002a100e935d1 PxOpenNativeDev+0x148(600515ab510, 3, 701ae328, 4, 60050003e00, 3011bb3a7a0)
000002a100e936e1 PxSolUpdateSize+0x78(600515ab510, 60053e523d0, 60053e525f0, 20, ffffffff, 3)
000002a100e937b1 PowerPlatformBottomDispatch+0x150(600515ab510, 60053e523d0, 0, 0, 60050555558,
20000000b0)
000002a100e939d1 PowerBottomDispatch+0xa0(600515ab510, 60053e523d0, 0, 0, 600547cd74c,
600515ab250)
000002a100e93aa1 PowerBottomDispatchPirp+0x88(600515ab250, 60053e523d0, 9, 60053e523d0, 600515a5718,
600515ab510)
000002a100e93b71 PowerDispatch+0x3a4(600515a9470, 60053e523d0, 60053e52630, 0, 600547cd718,
600515ab250)
000002a100e93c81 GpxDispatch+0x68(600515a9378, 60053e523d0, 600547cd728, 60053e525b0, 600515a57e8,
0)
000002a100e93d61 PowerDispatch+0x264(600515a9378, 60053e523d0, 60053e52610, 0, 0, 600515ab250)
000002a100e93e71 GpxDispatch+0x68(600515a9280, 60053e523d0, ffffffffffffffff, 7bf12e987bf22408,
600515a5780, 0)
000002a100e93f51 PowerDispatch+0x264(600515a9280, 60053e523d0, 60053e525f0, 0, 2a100e94531,
600515ab250)
000002a100e94061 GpxDispatch+0x68(600515a9188, 60053e523d0, 20f, 0, 60050555558, 0)
000002a100e94141 PowerDispatch+0x264(600515a9188, 60053e523d0, 60053e525d0, 0, 600547cd74c,
600515ab250)
000002a100e94251 GpxDispatch+0x68(600515a9090, 60053e523d0, 9, 60053e523d0, 600515a5718, 0)
000002a100e94331 PowerDispatch+0x264(600515a9090, 60053e523d0, 60053e525b0, 0, 600547cd718,
600515ab250)
000002a100e94441 PxUpdateSize+0x80(600515a9090, 0, 600547cd728, 60053e525b0, 600547cd728,
60053e523d0)
000002a100e94531 power_open+0x5c8(2a100e94f48, 40000003, 2, 60050003e00, 600515ab250,
600515ab510)
000002a100e94691 spec_open+0x4f8(2a100e950b8, 224, 60050003e00, a01, 60053b3f2d0, 0)
000002a100e94751 fop_open+0x78(2a100e950b8, 2, 60050003e00, 40000003, 6005483d900,
6005483d900)
000002a100e94801 dev_lopen+0x34(2a100e95170, 3, 4, 60050003e00, ffffffff, ffffffffffffffff)
000002a100e948c1 md_layered_open+0x120(13, 2a100e95258, 1, 300ad44b084, 20000000b0,
60050003e00)
000002a100e94981 stripe_open_all _devs+0x188(1a, 1, 0, 0, 0, 65)
000002a100e94a61 stripe_open+0xa0(65, 3, 4, 60050135f68, 600500ede00, 1)
000002a100e94b11 md_layered_open+0xb8(0, 2a100e954b0, 1, 60050135f68, 65, 60050003e00)
000002a100e94bd1 mirror_open_all _devs+0x240(1, 60050135770, 2a100e958c0, 300ad585070, 64, 1)
000002a100e94cc1 mirror_internal_open+0xf0(600501357a0, 3, 4, 0, 2a100e958c0, 0)
000002a100e94d71 mirror_resync_unit+0x74(60053d0d018, 60053d0d000, 60050135770, 2a100e958c0, 0,
1)
000002a100e94e31 mirror_admin_ioctl+0x1f8(e, ffbffb00, 102003, 2a100e958c0, 54, 102003)
000002a100e94f41 md_admin_ioctl+0x130(19c4c00, 19, ffbffb00, 102003, 2a100e958c0, 5615)
000002a100e95011 mdioctl+0xf4(550003ffff, 5615, ffbffb00, 102003, 60053b97100, 1f)
000002a100e950e1 fop_ioctl+0x20(600548dd840, 5615, ffbffb00, 102003, 60053b97100, 1288f38)
000002a100e95191 ioctl+0x184(0, 600501e7778, ffbffb00, fffffff8, 0, 5615)
000002a100e952e1 syscall_trap32+0xcc(0, 5615, ffbffb00, fffffffffffffff8, 0, ffbffb51)
Really interesting, the kthead is suspended by a scsi operation (The function biowait suspends processes pending completion of block I/O operations). Which is the device used ?
> 600520ca500::print -t buf_t
{
int b_flags = 0x2200061
struct buf *b_forw = 0
struct buf *b_back = 0
struct buf *av_forw = 0
struct buf *av_back = 0
o_dev_t b_dev = 0
size_t b_bcount = 0x200
union b_un = {
caddr_t b_addr = 0x303666e1480
struct fs *b_fs = 0x303666e1480
struct cg *b_cg = 0x303666e1480
struct dinode *b_dino = 0x303666e1480
daddr32_t *b_daddr = 0x303666e1480
}
lldaddr_t _b_blkno = {
longlong_t _f = 0
struct _p = {
int32_t _u = 0
int32_t _l = 0
}
}
char b_obs1 = '\0'
size_t b_resid = 0
clock_t b_start = 0
struct proc *b_proc = 0
struct page *b_pages = 0
clock_t b_obs2 = 0
size_t b_bufsize = 0
int (*)() b_iodone = 0
struct vnode *b_vp = 0
struct buf *b_chain = 0
int b_obs3 = 0
int b_error = 0
void *b_private = 0x300edf4b000
dev_t b_edev = 0x20000000b0
ksema_t b_sem = {
void *[2] _opaque = [ 0, 0 ]
}
ksema_t b_io = {
void *[2] _opaque = [ 0x30160427920, 0 ]
}
struct buf *b_list = 0
struct page **b_shadow = 0x60053df97c0
void *b_dip = 0x6005013d6d0
struct vnode *b_file = 0
offset_t b_offset = 0xffffffffffffffff
}
The macro devt enables us to obtain his minor and major numbers
> 0x20000000b0::devt
MAJOR MINOR
32 176
With the numbers, it's easy to find the name of device (There is a lot of differents methods to obtain the device). For exemple :
# cd /devices
# ls -Rl > /tmp/bruno
# view /tmp/bruno
[...]
/ssm@0,0/pci@18,600000/pci@1/scsi@2
[...]
brw-r----- 1 root sys 32, 176 Mar 23 18:46 sd@9,0:a
[...]
# grep "/ssm@0,0/pci@18,600000/pci@1/scsi@2/sd@9" /etc/path_to_inst
"/ssm@0,0/pci@18,600000/pci@1/scsi@2/sd@9,0" 22 "sd"
# ls -l /dev/dsk | grep "/ssm@0,0/pci@18,600000/pci@1/scsi@2/sd@9"
lrwxrwxrwx 1 root root 57 Aug 22 2007 c2t9d0s0 ->
../../devices/ssm@0,0/pci@18,600000/pci@1/scsi@2/sd@9,0:a
lrwxrwxrwx 1 root root 57 Aug 22 2007 c2t9d0s1 ->
../../devices/ssm@0,0/pci@18,600000/pci@1/scsi@2/sd@9,0:b
lrwxrwxrwx 1 root root 57 Aug 22 2007 c2t9d0s2 ->
../../devices/ssm@0,0/pci@18,600000/pci@1/scsi@2/sd@9,0:c
lrwxrwxrwx 1 root root 57 Aug 22 2007 c2t9d0s3 ->
../../devices/ssm@0,0/pci@18,600000/pci@1/scsi@2/sd@9,0:d
lrwxrwxrwx 1 root root 57 Aug 22 2007 c2t9d0s4 ->
../../devices/ssm@0,0/pci@18,600000/pci@1/scsi@2/sd@9,0:e
lrwxrwxrwx 1 root root 57 Aug 22 2007 c2t9d0s5 ->
../../devices/ssm@0,0/pci@18,600000/pci@1/scsi@2/sd@9,0:f
lrwxrwxrwx 1 root root 57 Aug 22 2007 c2t9d0s6 ->
../../devices/ssm@0,0/pci@18,600000/pci@1/scsi@2/sd@9,0:g
lrwxrwxrwx 1 root root 57 Aug 22 2007 c2t9d0s7 ->
../../devices/ssm@0,0/pci@18,600000/pci@1/scsi@2/sd@9,0:h
The unix device is c2t9d0. Impossible to obtain information on this device (format, prtvtoc, ... do not respond). If your production system is correctly configured, Explorer runs every week.
This is the moment to use it.
# cd /opt/SUNWexplo/output/explorer.84549349.server-2013.03.16.23.05/disks
# grep c2t9d0 diskinfo
c2t9d0 SEAGATE ST314655LSUN146G 0491 0719S16ZB2 primary
With the type of device determined, all we have to do is open a ticket with support and change it.
In conclusion, the breakdown of the device not having been brutal, a ioctl has remained suspended. As a result all SVM commands remained suspended. Using mdb, it was easy to diagnose the problem
and solve it quickly.
