Hi Robert,
I did started a ncftpget and managed to get 6 pdflush processes running in
state D, hopefully this will give us a chance to debug it.
I've attached the entire Alt+SysReq+T output here because I have no idea how
to read it.
Thanks,
Jason
-------Original Message-----
From: Robert Hancock
Sent: Saturday 06 May 2006 10:20
To: Jason Schoonover
Subject: Re: High load average on disk I/O on 2.6.17-rc3
Jason Schoonover wrote:
> Hi Robert,
>
> There are, this is the relevant output of the process list:
>
> ...
> 4659 pts/6 Ss 0:00 -bash
> 4671 pts/5 R+ 0:12 cp -a test-dir/ new-test
> 4676 ? D 0:00 [pdflush]
> 4679 ? D 0:00 [pdflush]
> 4687 pts/4 D+ 0:01 hdparm -t /dev/sda
> 4688 ? D 0:00 [pdflush]
> 4690 ? D 0:00 [pdflush]
> 4692 ? D 0:00 [pdflush]
> ...
>
> This was when I was copying a directory and then doing a performance test
> with hdparm in a separate shell. The hdparm process was in [D+] state and
> basically waited until the cp was finished. During the whole thing there
> were up to 5 pdflush processes in [D] state.
>
> The 5 minute load average hit 8.90 during this test.
>
> Does that help?
Well, it obviously explains why the load average is high, those D state
processes all count in the load average. It may be sort of a cosmetic
issue, since they're not actually using any CPU, but it's still a bit
unusual. For one thing, not sure why there are that many of them?
You could try enabling the SysRq triggers (if they're not already in
your kernel/distro) and doing Alt-SysRq-T which will dump the kernel
stack of all processes, that should show where exactly in the kernel
those pdflush processes are blocked..
015c0ca> sys_select+0x9a/0x166 <b0104a5b> do_IRQ+0x22/0x2b
<b010267b> syscall_call+0x7/0xb
snmpd S 3C14DC00 0 3771 1 3778 3764 (NOTLB)
f7975b58 f78d8300 0000000f 3c14dc00 00000000 000186a0 d5831281 000f68dc
dfbc4688 dfbc4580 b21a0540 b2014340 7b773b00 000f9663 00000001 b01273fc
b21f0500 f78aef00 00200286 b011e999 f7975b68 f7975b68 00200286 00000000
Call Trace:
<b01273fc> add_wait_queue+0x12/0x30 <b011e999> lock_timer_base+0x15/0x2f
<b026fd89> schedule_timeout+0x6c/0x8b <b011ecff> process_timeout+0x0/0x9
<b015baca> do_select+0x3b6/0x41e <f88df929> ext3_mark_iloc_dirty+0x2a7/0x345 [ext3]
<b015b65c> __pollwait+0x0/0xb8 <b011357b> default_wake_function+0x0/0x15
<b011357b> default_wake_function+0x0/0x15 <b011357b> default_wake_function+0x0/0x15
<b011357b> default_wake_function+0x0/0x15 <b011357b> default_wake_function+0x0/0x15
<b011357b> default_wake_function+0x0/0x15 <b0113571> try_to_wake_up+0x327/0x331
<b01609e5> inode_init_once+0x4c/0x14c <b0149394> cache_alloc_refill+0x39e/0x45d
<b0149062> cache_alloc_refill+0x6c/0x45d <b01117b5> __wake_up_common+0x2a/0x4f
<b017605f> proc_alloc_inode+0x41/0x66 <b0160911> alloc_inode+0xf5/0x17d
<b016e6c0> inotify_d_instantiate+0x3b/0x5e <b015fd63> d_instantiate+0x45/0x78
<b0160557> d_rehash+0x4f/0x5f <b0178a39> proc_lookup+0x77/0xaf
<b0137669> get_page_from_freelist+0x7e/0x31a <b0137895> get_page_from_freelist+0x2aa/0x31a
<b015bd1a> core_sys_select+0x1e8/0x2a2 <b01ad8db> vsnprintf+0x423/0x461
<b0137c3a> free_hot_cold_page+0x91/0xcd <b0137c90> __pagevec_free+0x1a/0x24
<b01393de> release_pages+0x147/0x150 <b0142040> page_remove_rmap+0x27/0x2a
<b021303e> lock_sock+0x85/0x8d <b0213d18> sk_free+0xa8/0xe2
<b0210e8e> sock_destroy_inode+0x13/0x16 <b015f456> dput+0x1b/0x11e
<b015c0ca> sys_select+0x9a/0x166 <b011b272> sys_gettimeofday+0x22/0x51
<b010267b> syscall_call+0x7/0xb
smbd S 000F4240 0 3776 3764 (NOTLB)
df977fb0 00000000 00000000 000f4240 00000000 00028b0a 00000012 00000002
dff18178 dff18070 dfb78580 b200c340 1685e740 000f41ff 00000000 00000000
00000001 00000007 df977fbc dff18070 00000001 df977fbc b0282b65 00000007
Call Trace:
<b011fa32> sys_pause+0x14/0x1a <b010267b> syscall_call+0x7/0xb
sshd S 004C4B40 0 3778 1 3875 3820 3771 (NOTLB)
f7f77b60 b23c4374 00000000 004c4b40 00000000 00000000 f8545f80 000f68dd
dfb81138 dfb81030 b02b0480 b200c340 ca71c780 000f95ea 00000000 00000000
dfa76ac8 00200246 b01273fc fffffffc df989a80 00200246 b023aee4 00000000
Call Trace:
<b01273fc> add_wait_queue+0x12/0x30 <b023aee4> tcp_poll+0x24/0x134
<b026fd30> schedule_timeout+0x13/0x8b <b02112c0> sock_poll+0x13/0x17
<b015baca> do_select+0x3b6/0x41e <f88df929> ext3_mark_iloc_dirty+0x2a7/0x345 [ext3]
<b015b65c> __pollwait+0x0/0xb8 <b011357b> default_wake_function+0x0/0x15
<b011357b> default_wake_function+0x0/0x15 <b014f430> __getblk+0x1d/0x211
<f8874995> do_get_write_access+0x44a/0x45c [jbd] <b01114b5> __activate_task+0x47/0x56
<b0113571> try_to_wake_up+0x327/0x331 <f88e698a> __ext3_journal_stop+0x19/0x37 [ext3]
<b01271f4> autoremove_wake_function+0x18/0x3a <b01117b5> __wake_up_common+0x2a/0x4f
<b0149062> cache_alloc_refill+0x6c/0x45d <b01ab175> kobject_put+0x16/0x19
<b01ab69b> kobject_release+0x0/0xa <b0215457> __alloc_skb+0x4a/0xee
<b0213988> sock_alloc_send_skb+0x5a/0x188 <b0217138> memcpy_fromiovec+0x29/0x4e
<b0214593> sock_def_readable+0x11/0x5e <b02659e1> unix_stream_sendmsg+0x21b/0x2e6
<b015bd1a> core_sys_select+0x1e8/0x2a2 <b0211077> do_sock_write+0xa4/0xad
<b0137669> get_page_from_freelist+0x7e/0x31a <b0137953> __alloc_pages+0x4e/0x261
<b0142611> __page_set_anon_rmap+0x2b/0x2f <b013d845> do_wp_page+0x244/0x286
<b013d869> do_wp_page+0x268/0x286 <b0215117> skb_dequeue+0x3b/0x41
<b0160690> destroy_inode+0x48/0x4c <b015f456> dput+0x1b/0x11e
<b015c0ca> sys_select+0x9a/0x166 <b014a58a> filp_close+0x4e/0x57
<b010267b> syscall_call+0x7/0xb
rpc.statd S 006ACFC0 0 3820 1 3828 3778 (NOTLB)
df9a9b60 b01114b5 b0328460 006acfc0 00000000 00000000 00000100 00000000
dfb7a138 dfb7a030 b21a0540 b2014340 3b0f8a80 000f41ff 00000001 f78a4c80
00000246 00000246 b01273fc 00000304 df917b80 00000246 b023aee4 00000000
Call Trace:
<b01114b5> __activate_task+0x47/0x56 <b01273fc> add_wait_queue+0x12/0x30
<b023aee4> tcp_poll+0x24/0x134 <b026fd30> schedule_timeout+0x13/0x8b
<b02112c0> sock_poll+0x13/0x17 <b015baca> do_select+0x3b6/0x41e
<b015b65c> __pollwait+0x0/0xb8 <b011357b> default_wake_function+0x0/0x15
<b011357b> default_wake_function+0x0/0x15 <b011357b> default_wake_function+0x0/0x15
<b015b65c> __pollwait+0x0/0xb8 <b011357b> default_wake_function+0x0/0x15
<b021a02c> net_rx_action+0x98/0x146 <b011bdf9> __do_softirq+0x57/0xc0
<b011bfc0> local_bh_enable+0x5b/0x66 <b021aff9> dev_queue_xmit+0x1e5/0x1eb
<b02377ea> ip_output+0x1c0/0x1f6 <b02369b7> ip_push_pending_frames+0x351/0x3dd
<b02170ea> memcpy_toiovec+0x29/0x4e <b0217824> skb_copy_datagram_iovec+0x49/0x1a6
<b0215932> kfree_skbmem+0x65/0x69 <b024e64d> udp_recvmsg+0x18a/0x1d3
<b021318b> sock_common_recvmsg+0x36/0x4b <b0212a1e> sock_recvmsg+0xf1/0x10d
<b01271dc> autoremove_wake_function+0x0/0x3a <b015bd1a> core_sys_select+0x1e8/0x2a2
<b0210e04> move_addr_to_user+0x3a/0x52 <b0212d12> sys_recvfrom+0xfc/0x134
<b021303e> lock_sock+0x85/0x8d <b0211aa5> sys_bind+0x73/0xa0
<b0213d18> sk_free+0xa8/0xe2 <b0210e8e> sock_destroy_inode+0x13/0x16
<b015f456> dput+0x1b/0x11e <b015c0ca> sys_select+0x9a/0x166
<b014a58a> filp_close+0x4e/0x57 <b010267b> syscall_call+0x7/0xb
ntpd S 39E048C0 0 3828 1 3836 3820 (NOTLB)
df8d7b60 0583c2ce 00000127 39e048c0 00000000 00000000 00000000 b011bdf9
b21ddb98 b21dda90 b02b0480 b200c340 48b33bc0 000f966c 00000000 dff27540
00000246 b021773c dff27540 df98a118 df8d7bd4 df98a100 dff27540 00000000
Call Trace:
<b011bdf9> __do_softirq+0x57/0xc0 <b021773c> datagram_poll+0x21/0xc0
<b026fd30> schedule_timeout+0x13/0x8b <b02112c0> sock_poll+0x13/0x17
<b015baca> do_select+0x3b6/0x41e <f88df929> ext3_mark_iloc_dirty+0x2a7/0x345 [ext3]
<b015b65c> __pollwait+0x0/0xb8 <b011357b> default_wake_function+0x0/0x15
<b011357b> default_wake_function+0x0/0x15 <b011357b> default_wake_function+0x0/0x15
<b011357b> default_wake_function+0x0/0x15 <b021aff9> dev_queue_xmit+0x1e5/0x1eb
<b02377ea> ip_output+0x1c0/0x1f6 <b0113571> try_to_wake_up+0x327/0x331
<b0214593> sock_def_readable+0x11/0x5e <b01117b5> __wake_up_common+0x2a/0x4f
<b0111803> __wake_up+0x29/0x3c <b02145b5> sock_def_readable+0x33/0x5e
<b0243570> tcp_rcv_established+0x40e/0x69a <f886501c> ipt_hook+0x1c/0x20 [iptable_filter]
<b024963e> tcp_v4_do_rcv+0x23/0x2ba <b0232a0a> ip_local_deliver_finish+0x0/0x193
<b0232a0a> ip_local_deliver_finish+0x0/0x193 <b024a72a> tcp_v4_rcv+0x889/0x8dd
<b0232bf8> ip_local_deliver+0x5b/0x1fd <b0232cf6> ip_local_deliver+0x159/0x1fd
<b0213117> sk_reset_timer+0x12/0x1e <b02447b1> tcp_send_delayed_ack+0xb7/0xbd
<b015bd1a> core_sys_select+0x1e8/0x2a2 <f886501c> ipt_hook+0x1c/0x20 [iptable_filter]
<b011f676> __sigqueue_free+0x2a/0x2f <b010710a> convert_fxsr_to_user+0xdf/0x12e
<b0107274> save_i387+0xf0/0x105 <b0101e68> setup_sigcontext+0xde/0x11c
<b0102490> do_notify_resume+0x4ab/0x58b <b010729f> convert_fxsr_from_user+0x16/0xd6
<b015c0ca> sys_select+0x9a/0x166 <b0101c66> restore_sigcontext+0x102/0x14c
<b0101fc0> sys_sigreturn+0x98/0xbd <b010267b> syscall_call+0x7/0xb
atd S 000F4240 0 3836 1 3843 3828 (NOTLB)
df8b5f1c df8b5ee8 30b8a000 000f4240 00000000 00000000 445cddcf f7e23f5c
b21c8178 b21c8070 b21a0540 b2014340 a2ec1c80 000f93e0 00000001 b2014d94
b2014d8c b2014d8c 1268d3c8 00000282 b012996a 00000000 e1b033c8 00000000
Call Trace:
<b012996a> hrtimer_start+0xc5/0xd0 <b0270109> do_nanosleep+0x3b/0x63
<b01299b8> hrtimer_nanosleep+0x43/0xfe <b0120244> do_sigaction+0x92/0x149
<b0129596> hrtimer_wakeup+0x0/0x1c <b0129ab0> sys_nanosleep+0x3d/0x51
<b010267b> syscall_call+0x7/0xb
cron S 03A2C940 0 3843 1 3859 3836 (NOTLB)
f7e23f1c f7e23ee8 f8475800 03a2c940 00000000 00000000 445ce895 f7fb5cc4
dff6e648 dff6e540 b02b0480 b200c340 23033000 000f9663 00000000 b200cd94
b200cd8c b200cd8c 264ea2e8 00000282 b012996a 00000000 2e074ae8 00000000
Call Trace:
<b012996a> hrtimer_start+0xc5/0xd0 <b0270109> do_nanosleep+0x3b/0x63
<b01299b8> hrtimer_nanosleep+0x43/0xfe <b0120244> do_sigaction+0x92/0x149
<b0129596> hrtimer_wakeup+0x0/0x1c <b0129ab0> sys_nanosleep+0x3d/0x51
<b010267b> syscall_call+0x7/0xb
login S 00895440 0 3859 1 5750 3860 3843 (NOTLB)
df9abf68 f7f5f580 df9aa000 00895440 00000000 00000000 b18f9d4c 7ee02065
dfb81b58 dfb81a50 b02b0480 b200c340 40787740 000f95d1 00000000 f7880740
00000140 c647b080 b026f8ed df9abfb4 00000286 00000000 00895440 00000000
Call Trace:
<b026f8ed> schedule+0xa0d/0xaaf <b011a7cd> do_wait+0x8ab/0x94d
<b011357b> default_wake_function+0x0/0x15 <b011a896> sys_wait4+0x27/0x2a
<b010267b> syscall_call+0x7/0xb
getty S 000F4240 0 3860 1 3861 3859 (NOTLB)
df975edc 00000246 00000000 000f4240 00000000 00000000 00000020 b01fb342
dfb7bb98 dfb7ba90 dfb81a50 b200c340 85ce1000 000f41ff 00000000 00000000
000000ff b1fdc520 b02b5ba0 b02b5ba0 b01393de df975ec8 00000001 00000202
Call Trace:
<b01fb342> do_con_write+0x141f/0x1450 <b01393de> release_pages+0x147/0x150
<b01179cd> release_console_sem+0x157/0x191 <b026fd30> schedule_timeout+0x13/0x8b
<b01fb3ba> con_write+0x1d/0x23 <b01273fc> add_wait_queue+0x12/0x30
<b01f2383> read_chan+0x321/0x53e <b011357b> default_wake_function+0x0/0x15
<b01ee431> tty_read+0x5d/0x9f <b014bc1d> vfs_read+0xa3/0x13a
<b014c5cc> sys_read+0x3b/0x64 <b010267b> syscall_call+0x7/0xb
getty S 000F4240 0 3861 1 3862 3860 (NOTLB)
df9a7edc 00000246 00000000 000f4240 00000000 00000000 00000020 b01fb342
dfbc7138 dfbc7030 b02b0480 b200c340 861a5b40 000f41ff 00000000 00000000
000000ff b1fdc520 b02b5ba0 b02b5ba0 b01393de df9a7ec8 00000001 00000000
Call Trace:
<b01fb342> do_con_write+0x141f/0x1450 <b01393de> release_pages+0x147/0x150
<b026fd30> schedule_timeout+0x13/0x8b <b01fb3ba> con_write+0x1d/0x23
<b01273fc> add_wait_queue+0x12/0x30 <b01f2383> read_chan+0x321/0x53e
<b011357b> default_wake_function+0x0/0x15 <b01ee431> tty_read+0x5d/0x9f
<b014bc1d> vfs_read+0xa3/0x13a <b014c5cc> sys_read+0x3b/0x64
<b010267b> syscall_call+0x7/0xb
getty S 000F4240 0 3862 1 3863 3861 (NOTLB)
df99bedc 00000246 00000000 000f4240 00000000 00000000 861a5b40 000f41ff
dfbba138 dfbba030 dfbc7030 b200c340 861a5b40 000f41ff 00000000 00000000
000000ff b1fdc520 b02b5ba0 b02b5ba0 b200c380 df99bec8 00000001 00000206
Call Trace:
<b01179cd> release_console_sem+0x157/0x191 <b026fd30> schedule_timeout+0x13/0x8b
<b01fb3ba> con_write+0x1d/0x23 <b01273fc> add_wait_queue+0x12/0x30
<b01f2383> read_chan+0x321/0x53e <b011357b> default_wake_function+0x0/0x15
<b01ee431> tty_read+0x5d/0x9f <b014bc1d> vfs_read+0xa3/0x13a
<b014c5cc> sys_read+0x3b/0x64 <b010267b> syscall_call+0x7/0xb
getty S 002DC6C0 0 3863 1 3864 3862 (NOTLB)
f7ed3edc 00000246 00000000 002dc6c0 00000000 00000000 00000020 b01fb342
dffafb98 dffafa90 b21a0540 b2014340 85fbd6c0 000f41ff 00000001 00000000
000000ff b1fb66e0 b02b5ba0 b02b5ba0 b01393de f7ed3ec8 00000001 00000000
Call Trace:
<b01fb342> do_con_write+0x141f/0x1450 <b01393de> release_pages+0x147/0x150
<b026fd30> schedule_timeout+0x13/0x8b <b01fb3ba> con_write+0x1d/0x23
<b01273fc> add_wait_queue+0x12/0x30 <b01f2383> read_chan+0x321/0x53e
<b011357b> default_wake_function+0x0/0x15 <b01ee431> tty_read+0x5d/0x9f
<b014bc1d> vfs_read+0xa3/0x13a <b014c5cc> sys_read+0x3b/0x64
<b010267b> syscall_call+0x7/0xb
getty S 000F4240 0 3864 1 5964 3863 (NOTLB)
df9ededc 00000246 00000000 000f4240 00000000 00000000 00000020 b01fb342
dfb75138 dfb75030 b21a0540 b2014340 8638dfc0 000f41ff 00000001 00000000
000000ff b1fb66e0 b02b5ba0 b02b5ba0 b01393de df9edec8 00000001 00000000
Call Trace:
<b01fb342> do_con_write+0x141f/0x1450 <b01393de> release_pages+0x147/0x150
<b026fd30> schedule_timeout+0x13/0x8b <b01fb3ba> con_write+0x1d/0x23
<b01273fc> add_wait_queue+0x12/0x30 <b01f2383> read_chan+0x321/0x53e
<b011357b> default_wake_function+0x0/0x15 <b01ee431> tty_read+0x5d/0x9f
<b014bc1d> vfs_read+0xa3/0x13a <b014c5cc> sys_read+0x3b/0x64
<b010267b> syscall_call+0x7/0xb
sshd S 08B3C880 0 3875 3778 3877 3887 (NOTLB)
df99fb60 008a184e 0000012b 08b3c880 00000000 00000000 97a171c0 000f4a7c
b21dfb58 b21dfa50 dffbfa50 b200c340 3f0f6200 000f4a7d 00000000 00000000
df99fb54 b0111803 00000000 00000000 00000003 00000046 f797100c 00000286
Call Trace:
<b0111803> __wake_up+0x29/0x3c <b026fd30> schedule_timeout+0x13/0x8b
<b01ee160> tty_poll+0x4a/0x54 <b015baca> do_select+0x3b6/0x41e
<b021d346> neigh_lookup+0x9c/0xa3 <b015b65c> __pollwait+0x0/0xb8
<b011357b> default_wake_function+0x0/0x15 <b011357b> default_wake_function+0x0/0x15
<b011357b> default_wake_function+0x0/0x15 <b011357b> default_wake_function+0x0/0x15
<b0226a72> qdisc_restart+0x14/0x17f <b021aff9> dev_queue_xmit+0x1e5/0x1eb
<b02377ea> ip_output+0x1c0/0x1f6 <b023717e> ip_queue_xmit+0x34d/0x38f
<b01114b5> __activate_task+0x47/0x56 <b011e999> lock_timer_base+0x15/0x2f
<b011f2c3> __mod_timer+0x90/0x98 <b0213117> sk_reset_timer+0x12/0x1e
<b0243dea> update_send_head+0x7a/0x80 <b02468c8> __tcp_push_pending_frames+0x68f/0x74b
<b0215457> __alloc_skb+0x4a/0xee <b02133db> release_sock+0xf/0x97
<b023d9ae> tcp_sendmsg+0x8ab/0x985 <b015bd1a> core_sys_select+0x1e8/0x2a2
<b0211077> do_sock_write+0xa4/0xad <b0211625> sock_aio_write+0x56/0x63
<b014b99a> do_sync_write+0xc0/0xf3 <b01271dc> autoremove_wake_function+0x0/0x3a
<b015c0ca> sys_select+0x9a/0x166 <b014c630> sys_write+0x3b/0x64
<b010267b> syscall_call+0x7/0xb
bash S 02CD29C0 0 3877 3875 (NOTLB)
df8afedc b01ecee7 00000000 02cd29c0 00000000 00000000 df8b780c df8b7800
dffbfb58 dffbfa50 b02b0480 b200c340 3f0f6200 000f4a7d 00000000 00000010
00000000 df8aff14 dffbfa50 df8affbc b01209c2 f7814ac0 f7814c18 00000000
Call Trace:
<b01ecee7> tty_ldisc_deref+0x52/0x61 <b01209c2> dequeue_signal+0x36/0xa4
<b026fd30> schedule_timeout+0x13/0x8b <b01273fc> add_wait_queue+0x12/0x30
<b01f2383> read_chan+0x321/0x53e <b011357b> default_wake_function+0x0/0x15
<b01ee431> tty_read+0x5d/0x9f <b014bc1d> vfs_read+0xa3/0x13a
<b014c5cc> sys_read+0x3b/0x64 <b010267b> syscall_call+0x7/0xb
sshd S 1BA81400 0 3887 3778 3889 5760 3875 (NOTLB)
df8b1b60 df8b1b58 df8b1b58 1ba81400 00000000 00000000 474e1feb 000f4a7e
dffbb648 dffbb540 b02b0480 b200c340 4758da80 000f4a7e 00000000 00000001
df8b1b54 b0111803 00000000 00000000 00000003 00000046 f780480c 00000000
Call Trace:
<b0111803> __wake_up+0x29/0x3c <b026fd30> schedule_timeout+0x13/0x8b
<b01ee160> tty_poll+0x4a/0x54 <b015baca> do_select+0x3b6/0x41e
<b024a28e> tcp_v4_rcv+0x3ed/0x8dd <b015b65c> __pollwait+0x0/0xb8
<b011357b> default_wake_function+0x0/0x15 <b011357b> default_wake_function+0x0/0x15
<b011357b> default_wake_function+0x0/0x15 <b011357b> default_wake_function+0x0/0x15
<b0226a72> qdisc_restart+0x14/0x17f <b021aff9> dev_queue_xmit+0x1e5/0x1eb
<b02377ea> ip_output+0x1c0/0x1f6 <b023717e> ip_queue_xmit+0x34d/0x38f
<b021aff9> dev_queue_xmit+0x1e5/0x1eb <b02377ea> ip_output+0x1c0/0x1f6
<b023717e> ip_queue_xmit+0x34d/0x38f <b01055a9> timer_interrupt+0x72/0x7a
<b011e999> lock_timer_base+0x15/0x2f <b011f2c3> __mod_timer+0x90/0x98
<b0213117> sk_reset_timer+0x12/0x1e <b0243dea> update_send_head+0x7a/0x80
<b02468c8> __tcp_push_pending_frames+0x68f/0x74b <b0104a5b> do_IRQ+0x22/0x2b
<b0215457> __alloc_skb+0x4a/0xee <b02133db> release_sock+0xf/0x97
<b023d9ae> tcp_sendmsg+0x8ab/0x985 <b015bd1a> core_sys_select+0x1e8/0x2a2
<b0211077> do_sock_write+0xa4/0xad <b0211625> sock_aio_write+0x56/0x63
<b014b99a> do_sync_write+0xc0/0xf3 <b01271dc> autoremove_wake_function+0x0/0x3a
<b015c0ca> sys_select+0x9a/0x166 <b014c630> sys_write+0x3b/0x64
<b010267b> syscall_call+0x7/0xb
bash S 02CD29C0 0 3889 3887 (NOTLB)
df9a3edc b01ecee7 00000000 02cd29c0 00000000 00000000 4758da80 000f4a7e
dff18b98 dff18a90 dffbb540 b200c340 4758da80 000f4a7e 00000000 00000000
00000016 0202c205 00000002 1b02000a b200c380 f792d280 f7f8c634 f7f87280
Call Trace:
<b01ecee7> tty_ldisc_deref+0x52/0x61 <b026fd30> schedule_timeout+0x13/0x8b
<b01273fc> add_wait_queue+0x12/0x30 <b01f2383> read_chan+0x321/0x53e
<b011357b> default_wake_function+0x0/0x15 <b01ee431> tty_read+0x5d/0x9f
<b014bc1d> vfs_read+0xa3/0x13a <b014c5cc> sys_read+0x3b/0x64
<b010267b> syscall_call+0x7/0xb
bash S 01312D00 0 5750 3859 (NOTLB)
b85cfedc 00000246 00000000 01312d00 00000000 000186a0 00000020 b01fb342
dfe73138 dfe73030 b02b0480 b200c340 05584d80 000f966b 00000000 00000000
000000ff 00000297 b01ba027 000003d4 00000297 b201e000 df808800 00000000
Call Trace:
<b01fb342> do_con_write+0x141f/0x1450 <b01ba027> vgacon_set_cursor_size+0x34/0xd1
<b026fd30> schedule_timeout+0x13/0x8b <b01179cd> release_console_sem+0x157/0x191
<b01273fc> add_wait_queue+0x12/0x30 <b01f2383> read_chan+0x321/0x53e
<b011357b> default_wake_function+0x0/0x15 <b01ee431> tty_read+0x5d/0x9f
<b014bc1d> vfs_read+0xa3/0x13a <b014c5cc> sys_read+0x3b/0x64
<b010267b> syscall_call+0x7/0xb
sshd S 0F518240 0 5760 3778 5762 5787 3887 (NOTLB)
effd1b60 058431d4 00000127 0f518240 00000000 00000000 00000000 b011bdf9
dfbba648 dfbba540 b02b0480 b200c340 61511d00 000f966c 00000000 00000001
effd1b54 b0111803 00000000 00000000 00000003 00000046 f7d6200c 00000000
Call Trace:
<b011bdf9> __do_softirq+0x57/0xc0 <b0111803> __wake_up+0x29/0x3c
<b026fd30> schedule_timeout+0x13/0x8b <b01ee160> tty_poll+0x4a/0x54
<b015baca> do_select+0x3b6/0x41e <b015b65c> __pollwait+0x0/0xb8
<b011357b> default_wake_function+0x0/0x15 <b011357b> default_wake_function+0x0/0x15
<b011357b> default_wake_function+0x0/0x15 <b011357b> default_wake_function+0x0/0x15
<b0226a72> qdisc_restart+0x14/0x17f <b021aff9> dev_queue_xmit+0x1e5/0x1eb
<b02377ea> ip_output+0x1c0/0x1f6 <b023717e> ip_queue_xmit+0x34d/0x38f
<b021b373> netif_receive_skb+0x22c/0x24a <f89008a0> tg3_poll+0x5e8/0x643 [tg3]
<b011e999> lock_timer_base+0x15/0x2f <b011f2c3> __mod_timer+0x90/0x98
<b0213117> sk_reset_timer+0x12/0x1e <b0243dea> update_send_head+0x7a/0x80
<b02468c8> __tcp_push_pending_frames+0x68f/0x74b <b0104a5b> do_IRQ+0x22/0x2b
<b0215457> __alloc_skb+0x4a/0xee <b02133db> release_sock+0xf/0x97
<b023d9ae> tcp_sendmsg+0x8ab/0x985 <b015bd1a> core_sys_select+0x1e8/0x2a2
<b0211077> do_sock_write+0xa4/0xad <b0211625> sock_aio_write+0x56/0x63
<b014b99a> do_sync_write+0xc0/0xf3 <b01271dc> autoremove_wake_function+0x0/0x3a
<b015c0ca> sys_select+0x9a/0x166 <b014c630> sys_write+0x3b/0x64
<b010267b> syscall_call+0x7/0xb
bash S 068E7780 0 5762 5760 6115 (NOTLB)
d3b45f54 b013d869 b1d3cc80 068e7780 00000000 00000000 dfeb9740 b1d3cc80
dffbf138 dffbf030 dfbc7a50 b200c340 68f58540 000f9654 00000000 00000000
bfdb9080 b134470c 69e64065 f7d6f574 0000001d 000003d8 f789ee00 aff805a4
Call Trace:
<b013d869> do_wp_page+0x268/0x286 <b011a7cd> do_wait+0x8ab/0x94d
<b011357b> default_wake_function+0x0/0x15 <b011a896> sys_wait4+0x27/0x2a
<b011a8ac> sys_waitpid+0x13/0x17 <b010267b> syscall_call+0x7/0xb
sshd S 08C30AC0 0 5787 3778 5789 5760 (NOTLB)
b5dbfb60 f792d2e0 000005ea 08c30ac0 00000000 00000000 a4ab0bc0 000f966b
dfbc2648 dfbc2540 b02b0480 b200c340 a5ccf680 000f966b 00000000 00000001
b5dbfb54 b0111803 00000000 00000000 00000003 00000046 dcd5400c 00000000
Call Trace:
<b0111803> __wake_up+0x29/0x3c <b026fd30> schedule_timeout+0x13/0x8b
<b01ee160> tty_poll+0x4a/0x54 <b015baca> do_select+0x3b6/0x41e
<b015b65c> __pollwait+0x0/0xb8 <b011357b> default_wake_function+0x0/0x15
<b011357b> default_wake_function+0x0/0x15 <b011357b> default_wake_function+0x0/0x15
<b011357b> default_wake_function+0x0/0x15 <b0226a72> qdisc_restart+0x14/0x17f
<b021aff9> dev_queue_xmit+0x1e5/0x1eb <b02377ea> ip_output+0x1c0/0x1f6
<b023717e> ip_queue_xmit+0x34d/0x38f <b011bdf9> __do_softirq+0x57/0xc0
<b010314c> apic_timer_interrupt+0x1c/0x24 <b0244573> tcp_transmit_skb+0x5dc/0x600
<b02468c8> __tcp_push_pending_frames+0x68f/0x74b <b0215457> __alloc_skb+0x4a/0xee
<b02133db> release_sock+0xf/0x97 <b023d9ae> tcp_sendmsg+0x8ab/0x985
<b015bd1a> core_sys_select+0x1e8/0x2a2 <b0211077> do_sock_write+0xa4/0xad
<b0211625> sock_aio_write+0x56/0x63 <b014b99a> do_sync_write+0xc0/0xf3
<b01271dc> autoremove_wake_function+0x0/0x3a <b015c0ca> sys_select+0x9a/0x166
<b014c630> sys_write+0x3b/0x64 <b010267b> syscall_call+0x7/0xb
bash S 055D4A80 0 5789 5787 (NOTLB)
de097edc b01ecee7 00000000 055d4a80 00000000 0001b207 a5ccf680 000f966b
dfbc2b58 dfbc2a50 dfbc2540 b200c340 a5ccf680 000f966b 00000000 00000000
00000000 de097f14 dfbc2a50 de097fbc b200c380 f7f3f580 f7f3f6d8 f7f3f6c4
Call Trace:
<b01ecee7> tty_ldisc_deref+0x52/0x61 <b01216d0> get_signal_to_deliver+0x1c0/0x372
<b026fd30> schedule_timeout+0x13/0x8b <b01273fc> add_wait_queue+0x12/0x30
<b01f2383> read_chan+0x321/0x53e <b011357b> default_wake_function+0x0/0x15
<b01ee431> tty_read+0x5d/0x9f <b014bc1d> vfs_read+0xa3/0x13a
<b014c5cc> sys_read+0x3b/0x64 <b010267b> syscall_call+0x7/0xb
pdflush D 37E8BE80 0 5842 11 6085 3699 (L-TLB)
c7803eec b1bfa900 b1bfa920 37e8be80 00000000 00000000 fc4c7fab 000f9668
dfe67688 dfe67580 b21a0540 b2014340 805ef140 000f966c 00000001 00000000
00000286 b011e999 00000286 b011e999 c7803efc c7803efc 00000286 00000000
Call Trace:
<b011e999> lock_timer_base+0x15/0x2f <b011e999> lock_timer_base+0x15/0x2f
<b026fd89> schedule_timeout+0x6c/0x8b <b011ecff> process_timeout+0x0/0x9
<b026ebf5> io_schedule_timeout+0x29/0x33 <b01387eb> pdflush+0x0/0x1b5
<b01a0a1d> blk_congestion_wait+0x55/0x69 <b01271dc> autoremove_wake_function+0x0/0x3a
<b0137f01> background_writeout+0x7d/0x8b <b01388ee> pdflush+0x103/0x1b5
<b0137e84> background_writeout+0x0/0x8b <b01271a1> kthread+0xa3/0xd0
<b01270fe> kthread+0x0/0xd0 <b0100bc5> kernel_thread_helper+0x5/0xb
syslogd S 029F6300 0 5964 1 6051 3864 (NOTLB)
c8d39b60 f88df961 c2e1aed8 029f6300 00000000 00000000 f7c7c260 00001000
dffbd178 dffbd070 b02b0480 b200c340 baf35680 000f9666 00000000 00000000
b013595f dffe2340 00011200 b733848c 00000246 b01273fc df9d4680 00000000
Call Trace:
<f88df961> ext3_mark_iloc_dirty+0x2df/0x345 [ext3] <b013595f> mempool_alloc+0x21/0xbf
<b01273fc> add_wait_queue+0x12/0x30 <b026fd30> schedule_timeout+0x13/0x8b
<b02112c0> sock_poll+0x13/0x17 <b015baca> do_select+0x3b6/0x41e
<b015b65c> __pollwait+0x0/0xb8 <b011357b> default_wake_function+0x0/0x15
<f887355d> __journal_file_buffer+0x111/0x1e7 [jbd] <f8874995> do_get_write_access+0x44a/0x45c [jbd]
<f887355d> __journal_file_buffer+0x111/0x1e7 [jbd] <f8874df9> journal_dirty_metadata+0x173/0x19e [jbd]
<b013595f> mempool_alloc+0x21/0xbf <b013595f> mempool_alloc+0x21/0xbf
<b01a543b> as_set_request+0x1c/0x6d <b01a5a45> as_update_iohist+0x38/0x2a5
<b01a63ee> as_add_request+0xc0/0xef <b019ea34> elv_insert+0x9b/0x138
<b01a27e6> __make_request+0x319/0x350 <b01342a3> generic_file_buffered_write+0x482/0x56f
<b01a0268> generic_make_request+0x168/0x17a <b013595f> mempool_alloc+0x21/0xbf
<b01a0ad6> submit_bio+0xa5/0xaa <b015bd1a> core_sys_select+0x1e8/0x2a2
<b0133749> wait_on_page_writeback_range+0xb3/0xf7 <b011f676> __sigqueue_free+0x2a/0x2f
<b0120981> __dequeue_signal+0x14b/0x156 <b01053b7> do_gettimeofday+0x1b/0x9d
<b01053b7> do_gettimeofday+0x1b/0x9d <b011b6ba> getnstimeofday+0xf/0x25
<b01acb36> rb_insert_color+0xa6/0xc8 <b01294f4> enqueue_hrtimer+0x58/0x7f
<b012996a> hrtimer_start+0xc5/0xd0 <b011abdd> do_setitimer+0x16c/0x47d
<b015c0ca> sys_select+0x9a/0x166 <b0101fc0> sys_sigreturn+0x98/0xbd
<b010267b> syscall_call+0x7/0xb
klogd R running 0 6051 1 5964 (NOTLB)
pdflush D 0E205540 0 6085 11 6130 5842 (L-TLB)
b4d01eec f792d280 00000204 0e205540 00000000 00000000 ad7d58c0 000f966b
dffbbb58 dffbba50 b21a0540 b2014340 7e1b1bc0 000f966c 00000001 00000000
00000286 b011e999 00000286 b011e999 b4d01efc b4d01efc 00000286 00000000
Call Trace:
<b011e999> lock_timer_base+0x15/0x2f <b011e999> lock_timer_base+0x15/0x2f
<b026fd89> schedule_timeout+0x6c/0x8b <b011ecff> process_timeout+0x0/0x9
<b026ebf5> io_schedule_timeout+0x29/0x33 <b01387eb> pdflush+0x0/0x1b5
<b01a0a1d> blk_congestion_wait+0x55/0x69 <b01271dc> autoremove_wake_function+0x0/0x3a
<b0137f01> background_writeout+0x7d/0x8b <b01388ee> pdflush+0x103/0x1b5
<b0137e84> background_writeout+0x0/0x8b <b01271a1> kthread+0xa3/0xd0
<b01270fe> kthread+0x0/0xd0 <b0100bc5> kernel_thread_helper+0x5/0xb
ncftpget D 568FC100 0 6115 5762 (NOTLB)
cb351a88 87280008 b013595f 568fc100 00000008 00051615 f2a64c80 b21cc340
dfbc7b58 dfbc7a50 b02b0480 b200c340 77405900 000f966c 00000000 00000282
b011e999 f2a64cd8 00000000 dfec8b84 00000000 dfec8b84 b01a216b 00000000
Call Trace:
<b013595f> mempool_alloc+0x21/0xbf <b011e999> lock_timer_base+0x15/0x2f
<b01a216b> get_request+0x55/0x283 <b026f9b5> io_schedule+0x26/0x30
<b01a2432> get_request_wait+0x99/0xd3 <b01271dc> autoremove_wake_function+0x0/0x3a
<b01a2786> __make_request+0x2b9/0x350 <b01a0268> generic_make_request+0x168/0x17a
<b013595f> mempool_alloc+0x21/0xbf <b01a0ad6> submit_bio+0xa5/0xaa
<b015032c> bio_alloc+0x13/0x22 <b014dad0> submit_bh+0xe6/0x107
<b014f9d9> __block_write_full_page+0x20e/0x301 <f88e17e8> ext3_get_block+0x0/0xad [ext3]
<f88e0633> ext3_ordered_writepage+0xcb/0x137 [ext3] <f88e17e8> ext3_get_block+0x0/0xad [ext3]
<f88def99> bget_one+0x0/0xb [ext3] <b016a1d3> mpage_writepages+0x193/0x2e9
<f88e0568> ext3_ordered_writepage+0x0/0x137 [ext3] <b013813f> do_writepages+0x30/0x39
<b0168988> __writeback_single_inode+0x166/0x2e2 <b01aa9d3> __next_cpu+0x11/0x20
<b0136421> read_page_state_offset+0x33/0x41 <b0168f5e> sync_sb_inodes+0x185/0x23a
<b01691c6> writeback_inodes+0x6e/0xbb <b0138246> balance_dirty_pages_ratelimited_nr+0xcb/0x152
<b013429e> generic_file_buffered_write+0x47d/0x56f <f88e698a> __ext3_journal_stop+0x19/0x37 [ext3]
<f88dfde0> ext3_dirty_inode+0x5e/0x64 [ext3] <b0168b4c> __mark_inode_dirty+0x28/0x14c
<b01354da> __generic_file_aio_write_nolock+0x3c8/0x405 <b0211c91> sock_aio_read+0x56/0x63
<b013573c> generic_file_aio_write+0x61/0xb3 <f88dde72> ext3_file_write+0x26/0x92 [ext3]
<b014b99a> do_sync_write+0xc0/0xf3 <b016201a> notify_change+0x2d4/0x2e5
<b01271dc> autoremove_wake_function+0x0/0x3a <b014bdb0> vfs_write+0xa3/0x13a
<b014c630> sys_write+0x3b/0x64 <b010267b> syscall_call+0x7/0xb
pdflush D 00F42400 0 6130 11 6146 6085 (L-TLB)
f57ddeec b1964a00 b1fdf940 00f42400 00000000 00000000 b1c75ca0 b1f29320
dfb75648 dfb75540 b21a0540 b2014340 7b01a6c0 000f966c 00000001 00000000
00000286 b011e999 00000286 b011e999 f57ddefc f57ddefc 00000286 00000000
Call Trace:
<b011e999> lock_timer_base+0x15/0x2f <b011e999> lock_timer_base+0x15/0x2f
<b026fd89> schedule_timeout+0x6c/0x8b <b011ecff> process_timeout+0x0/0x9
<b026ebf5> io_schedule_timeout+0x29/0x33 <b01387eb> pdflush+0x0/0x1b5
<b01a0a1d> blk_congestion_wait+0x55/0x69 <b01271dc> autoremove_wake_function+0x0/0x3a
<b0137f01> background_writeout+0x7d/0x8b <b01388ee> pdflush+0x103/0x1b5
<b0137e84> background_writeout+0x0/0x8b <b01271a1> kthread+0xa3/0xd0
<b01270fe> kthread+0x0/0xd0 <b0100bc5> kernel_thread_helper+0x5/0xb
pdflush D 004C4B40 0 6146 11 6156 6130 (L-TLB)
dec91eec 00000001 b02f20e0 004c4b40 00000000 00000000 00000000 b02f20e0
dfb80178 dfb80070 b21a0540 b2014340 7b8afb00 000f966c 00000001 00000000
00000286 b011e999 00000286 b011e999 dec91efc dec91efc 00000286 00000000
Call Trace:
<b011e999> lock_timer_base+0x15/0x2f <b011e999> lock_timer_base+0x15/0x2f
<b026fd89> schedule_timeout+0x6c/0x8b <b011ecff> process_timeout+0x0/0x9
<b026ebf5> io_schedule_timeout+0x29/0x33 <b01387eb> pdflush+0x0/0x1b5
<b01a0a1d> blk_congestion_wait+0x55/0x69 <b01271dc> autoremove_wake_function+0x0/0x3a
<b0137f01> background_writeout+0x7d/0x8b <b01388ee> pdflush+0x103/0x1b5
<b0137e84> background_writeout+0x0/0x8b <b01271a1> kthread+0xa3/0xd0
<b01270fe> kthread+0x0/0xd0 <b0100bc5> kernel_thread_helper+0x5/0xb
pdflush D 04B571C0 0 6156 11 6168 6146 (L-TLB)
d959feec b1b7c500 b1f0e020 04b571c0 00000000 00000000 14c754c0 000f9669
dff15b98 dff15a90 b21a0540 b2014340 7b10e900 000f966c 00000001 00000000
00000286 b011e999 00000286 b011e999 d959fefc d959fefc 00000286 00000000
Call Trace:
<b011e999> lock_timer_base+0x15/0x2f <b011e999> lock_timer_base+0x15/0x2f
<b026fd89> schedule_timeout+0x6c/0x8b <b011ecff> process_timeout+0x0/0x9
<b026ebf5> io_schedule_timeout+0x29/0x33 <b01387eb> pdflush+0x0/0x1b5
<b01a0a1d> blk_congestion_wait+0x55/0x69 <b01271dc> autoremove_wake_function+0x0/0x3a
<b0137f01> background_writeout+0x7d/0x8b <b01388ee> pdflush+0x103/0x1b5
<b0137e84> background_writeout+0x0/0x8b <b01271a1> kthread+0xa3/0xd0
<b01270fe> kthread+0x0/0xd0 <b0100bc5> kernel_thread_helper+0x5/0xb
pdflush D 000F4240 0 6168 11 6156 (L-TLB)
caa13eec 0000000a b23ec9cc 000f4240 00000000 00000000 b01117b5 ebb31f74
b21dd178 b21dd070 b21a0540 b2014340 7ee17900 000f966c 00000001 00000000
00000286 b011e999 00000286 b011e999 caa13efc caa13efc 00000286 00000000
Call Trace:
<b01117b5> __wake_up_common+0x2a/0x4f <b011e999> lock_timer_base+0x15/0x2f
<b011e999> lock_timer_base+0x15/0x2f <b026fd89> schedule_timeout+0x6c/0x8b
<b011ecff> process_timeout+0x0/0x9 <b026ebf5> io_schedule_timeout+0x29/0x33
<b01387eb> pdflush+0x0/0x1b5 <b01a0a1d> blk_congestion_wait+0x55/0x69
<b01271dc> autoremove_wake_function+0x0/0x3a <b0137f01> background_writeout+0x7d/0x8b
<b01388ee> pdflush+0x103/0x1b5 <b0137e84> background_writeout+0x0/0x8b
<b01271a1> kthread+0xa3/0xd0 <b01270fe> kthread+0x0/0xd0
<b0100bc5> kernel_thread_helper+0x5/0xb
[Index of Archives]
[Kernel Newbies]
[Netfilter]
[Bugtraq]
[Photo]
[Stuff]
[Gimp]
[Yosemite News]
[MIPS Linux]
[ARM Linux]
[Linux Security]
[Linux RAID]
[Video 4 Linux]
[Linux for the blind]
[Linux Resources]
