Re: X-freeze after clflush changes [Was: 2.6.23-rc6-mm1]

[Valdis.Kletnieks@vt.edu]

On Fri, 21 Sep 2007 21:43:20 +0200, Jiri Slaby said:
> On 09/21/2007 09:38 PM, Jiri Slaby wrote:
> > It is rather the other user who adds the page to some other list while bein
g at
> > deferred_pages list. Could you try my debug patch
> > (http://lkml.org/lkml/2007/9/19/141)?
> 
> or the whitespace non-damaged version:
> http://www.fi.muni.cz/~xslaby/sklad/pageattr_debug

Gaak. Is that thing *supposed* to spew zillions of lines of output?

Some of the hits we get (I'm wondering if anything after the first makes
any sense, or if we're just slowly watching the corruption spread - the
thing ended up near 23K lines long before I gave up and hit the poweroff
button because there was no end in sight):

(If there's something specific you want me to find in the output,
like "the first time we see XYZ", yell...)

[  103.701000] POISONS (ffff81000117dc88): ffff810006d14000, ffff8100034225c0
[  103.701000]
[  103.701000] Call Trace:
[  103.701000]  [<ffffffff803580aa>] __list_add+0xd7/0x138
[  103.701000]  [<ffffffff80358117>] list_add+0xc/0x11
[  103.701000]  [<ffffffff80270865>] free_hot_cold_page+0xe8/0x16d
[  103.701000]  [<ffffffff8027093e>] free_hot_page+0xb/0xd
[  103.701000]  [<ffffffff80270958>] __free_pages+0x18/0x21
[  103.701000]  [<ffffffff80270990>] free_pages+0x2f/0x34
[  103.701000]  [<ffffffff8028922d>] kmem_freepages+0xc5/0xce
[  103.701000]  [<ffffffff8028957f>] slab_destroy+0x3c/0x53
[  103.701000]  [<ffffffff80289663>] free_block+0xcd/0x110
[  103.701000]  [<ffffffff8028973a>] drain_array+0x94/0xc9
[  103.701000]  [<ffffffff8028a8c3>] cache_reap+0x0/0x105
[  103.701000]  [<ffffffff8028a948>] cache_reap+0x85/0x105
[  103.701000]  [<ffffffff80243d5d>] run_workqueue+0x8e/0x125
[  103.701000]  [<ffffffff8024478d>] worker_thread+0x0/0xe7
[  103.701000]  [<ffffffff80244869>] worker_thread+0xdc/0xe7
[  103.701000]  [<ffffffff80247f13>] autoremove_wake_function+0x0/0x38
[  103.701000]  [<ffffffff80247ddd>] kthread+0x49/0x78
[  103.701000]  [<ffffffff8020cfc8>] child_rip+0xa/0x12
[  103.701000]  [<ffffffff80247d94>] kthread+0x0/0x78
[  103.701000]  [<ffffffff8020cfbe>] child_rip+0x0/0x12
[  103.701000]

[  103.701000] POISONS (ffff81000117eac0): ffff810006d55000, ffff8100034225c0
[  103.701000]
[  103.701000] Call Trace:
[  103.701000]  [<ffffffff803580aa>] __list_add+0xd7/0x138
[  103.701000]  [<ffffffff80358117>] list_add+0xc/0x11
[  103.701000]  [<ffffffff80270865>] free_hot_cold_page+0xe8/0x16d
[  103.701000]  [<ffffffff8027093e>] free_hot_page+0xb/0xd
[  103.701000]  [<ffffffff80270958>] __free_pages+0x18/0x21
[  103.701000]  [<ffffffff80270990>] free_pages+0x2f/0x34
[  103.701000]  [<ffffffff8028922d>] kmem_freepages+0xc5/0xce
[  103.701000]  [<ffffffff8028957f>] slab_destroy+0x3c/0x53
[  103.701000]  [<ffffffff80289663>] free_block+0xcd/0x110
[  103.701000]  [<ffffffff8028973a>] drain_array+0x94/0xc9
[  103.701000]  [<ffffffff8028a8c3>] cache_reap+0x0/0x105
[  103.701000]  [<ffffffff8028a948>] cache_reap+0x85/0x105
[  103.701000]  [<ffffffff80243d5d>] run_workqueue+0x8e/0x125
[  103.701000]  [<ffffffff8024478d>] worker_thread+0x0/0xe7
[  103.701000]  [<ffffffff80244869>] worker_thread+0xdc/0xe7
[  103.701000]  [<ffffffff80247f13>] autoremove_wake_function+0x0/0x38
[  103.701000]  [<ffffffff80247ddd>] kthread+0x49/0x78
[  103.701000]  [<ffffffff8020cfc8>] child_rip+0xa/0x12
[  103.701000]  [<ffffffff80247d94>] kthread+0x0/0x78
[  103.701000]  [<ffffffff8020cfbe>] child_rip+0x0/0x12
[  103.701000]
(That trace repeats 16 times, then we see:)
[  106.284000] POISONS (ffff810004432810): ffff810005291378, ffff81000524e618
[  106.284000] 
[  106.284000] Call Trace:
[  106.284000]  [<ffffffff80517086>] __down_write_nested+0x3d/0xa1
[  106.284000]  [<ffffffff803580aa>] __list_add+0xd7/0x138
[  106.284000]  [<ffffffff802765fe>] vma_prio_tree_add+0xc9/0xe0
[  106.284000]  [<ffffffff802324f0>] copy_process+0xc63/0x1515
[  106.284000]  [<ffffffff80232eff>] do_fork+0x75/0x20b
[  106.284000]  [<ffffffff80353d54>] __up_write+0xf0/0x100
[  106.284000]  [<ffffffff8020c17e>] system_call+0x7e/0x83
[  106.284000]  [<ffffffff8020a64f>] sys_clone+0x23/0x25
[  106.284000]  [<ffffffff8020c497>] ptregscall_common+0x67/0xb0
[  106.284000] 
..
[  106.284000] POISONS (ffff810004432768): ffff81000524e618, ffff81000524e618
[  106.284000]
[  106.284000] Call Trace:
[  106.284000]  [<ffffffff803580aa>] __list_add+0xd7/0x138
[  106.284000]  [<ffffffff802765fe>] vma_prio_tree_add+0xc9/0xe0
[  106.284000]  [<ffffffff802324f0>] copy_process+0xc63/0x1515
[  106.284000]  [<ffffffff80232eff>] do_fork+0x75/0x20b
[  106.284000]  [<ffffffff80353d54>] __up_write+0xf0/0x100
[  106.284000]  [<ffffffff8020c17e>] system_call+0x7e/0x83
[  106.284000]  [<ffffffff8020a64f>] sys_clone+0x23/0x25
[  106.284000]  [<ffffffff8020c497>] ptregscall_common+0x67/0xb0
[  106.284000]  
...
[  106.285000] POISONS (ffff810003637b30): ffff810003637c18, 0000000000000246
[  106.285000]
[  106.285000] Call Trace:
[  106.285000]  [<ffffffff803580aa>] __list_add+0xd7/0x138
[  106.285000]  [<ffffffff80358117>] list_add+0xc/0x11
[  106.285000]  [<ffffffff80248110>] add_wait_queue+0x2c/0x40
[  106.285000]  [<ffffffff8029ac28>] __pollwait+0xd6/0xdf
[  106.285000]  [<ffffffff802b80a3>] inotify_poll+0x29/0x5c
[  106.285000]  [<ffffffff8029a498>] do_select+0x2fa/0x50d
[  106.285000]  [<ffffffff8029ab52>] __pollwait+0x0/0xdf
[  106.285000]  [<ffffffff8022d480>] default_wake_function+0x0/0xf
[  106.285000]  [<ffffffff8051471d>] __down_trylock+0x4d/0x5a
[  106.285000]  [<ffffffff80517241>] __down_failed_trylock+0x35/0x3a
[  106.285000]  [<ffffffff8022c1b3>] __update_rq_clock+0x1a/0xe5
[  106.285000]  [<ffffffff80270e41>] __alloc_pages+0x5c/0x2b5
[  106.285000]  [<ffffffff8029a89e>] core_sys_select+0x1f3/0x2a2
[  106.285000]  [<ffffffff802453c3>] alloc_pid+0x2f8/0x34f
[  106.285000]  [<ffffffff80353c5b>] __up_read+0x7a/0x83
[  106.285000]  [<ffffffff8024b25b>] up_read+0x9/0xb
[  106.285000]  [<ffffffff80519a92>] do_page_fault+0x405/0x7ac
[  106.285000]  [<ffffffff8029acf0>] sys_select+0xbf/0x17b
[  106.285000]  [<ffffffff8020c17e>] system_call+0x7e/0x83
[  106.285000] POISONS (ffff810003637ba0): ffffffff8060ff48, ffffffff8051471d
[  106.285000]
[  106.285000] Call Trace:
[  106.285000]  [<ffffffff803580aa>] __list_add+0xd7/0x138
[  106.285000]  [<ffffffff80358117>] list_add+0xc/0x11
[  106.285000]  [<ffffffff80248110>] add_wait_queue+0x2c/0x40
[  106.285000]  [<ffffffff8029ac28>] __pollwait+0xd6/0xdf
[  106.285000]  [<ffffffff80477b83>] datagram_poll+0x23/0xd5
[  106.285000]  [<ffffffff8046ea63>] sock_poll+0x18/0x1a
[  106.285000]  [<ffffffff8029a498>] do_select+0x2fa/0x50d
[  106.285000]  [<ffffffff8029ab52>] __pollwait+0x0/0xdf
[  106.285000]  [<ffffffff8022d480>] default_wake_function+0x0/0xf
[  106.285000]  [<ffffffff8022d480>] default_wake_function+0x0/0xf
[  106.285000]  [<ffffffff8022d480>] default_wake_function+0x0/0xf
[  106.285000]  [<ffffffff8051471d>] __down_trylock+0x4d/0x5a
[  106.285000]  [<ffffffff80517241>] __down_failed_trylock+0x35/0x3a
[  106.285000]  [<ffffffff8022c1b3>] __update_rq_clock+0x1a/0xe5
[  106.285000]  [<ffffffff80270e41>] __alloc_pages+0x5c/0x2b5
[  106.285000]  [<ffffffff8029a89e>] core_sys_select+0x1f3/0x2a2
[  106.285000]  [<ffffffff802453c3>] alloc_pid+0x2f8/0x34f
[  106.285000]  [<ffffffff80353c5b>] __up_read+0x7a/0x83
[  106.285000]  [<ffffffff8024b25b>] up_read+0x9/0xb
[  106.285000]  [<ffffffff80519a92>] do_page_fault+0x405/0x7ac
[  106.285000]  [<ffffffff8029acf0>] sys_select+0xbf/0x17b
[  106.285000]  [<ffffffff8020c17e>] system_call+0x7e/0x83
[  106.285000] POISONS (ffff810003637ba0): ffffffff8060ff48, ffffffff8051471d
[  106.285000]
[  106.285000] Call Trace:
[  106.285000]  [<ffffffff803580aa>] __list_add+0xd7/0x138
[  106.285000]  [<ffffffff80358117>] list_add+0xc/0x11
[  106.285000]  [<ffffffff80248110>] add_wait_queue+0x2c/0x40
[  106.285000]  [<ffffffff8029ac28>] __pollwait+0xd6/0xdf
[  106.285000]  [<ffffffff80477b83>] datagram_poll+0x23/0xd5
[  106.285000]  [<ffffffff8046ea63>] sock_poll+0x18/0x1a
[  106.285000]  [<ffffffff8029a498>] do_select+0x2fa/0x50d
[  106.285000]  [<ffffffff8029ab52>] __pollwait+0x0/0xdf
[  106.285000]  [<ffffffff8022d480>] default_wake_function+0x0/0xf
[  106.285000]  [<ffffffff8022d480>] default_wake_function+0x0/0xf
[  106.285000]  [<ffffffff8022d480>] default_wake_function+0x0/0xf
[  106.285000]  [<ffffffff8051471d>] __down_trylock+0x4d/0x5a
[  106.285000]  [<ffffffff80517241>] __down_failed_trylock+0x35/0x3a
[  106.285000]  [<ffffffff8022c1b3>] __update_rq_clock+0x1a/0xe5
[  106.285000]  [<ffffffff80270e41>] __alloc_pages+0x5c/0x2b5
[  106.285000]  [<ffffffff8029a89e>] core_sys_select+0x1f3/0x2a2
[  106.285000]  [<ffffffff802453c3>] alloc_pid+0x2f8/0x34f
[  106.285000]  [<ffffffff80353c5b>] __up_read+0x7a/0x83
[  106.285000]  [<ffffffff8024b25b>] up_read+0x9/0xb
[  106.285000]  [<ffffffff80519a92>] do_page_fault+0x405/0x7ac
[  106.285000]  [<ffffffff8029acf0>] sys_select+0xbf/0x17b
[  106.285000]  [<ffffffff8020c17e>] system_call+0x7e/0x83
[  106.285000] 
...
[  109.705000] POISONS (ffff8100047638c8): 0000000000200200, 0000000000000000
[  109.705000]
[  109.705000] Call Trace:
[  109.705000]  <IRQ>  [<ffffffff803580aa>] __list_add+0xd7/0x138
[  109.705000]  [<ffffffff8023c44c>] internal_add_timer+0xbb/0xbd
[  109.705000]  [<ffffffff8023d0a2>] __mod_timer+0xb8/0xd6
[  109.705000]  [<ffffffff803ec908>] ata_qc_issue+0x4bd/0x517
[  109.705000]  [<ffffffff8023d306>] mod_timer+0x3d/0x42
[  109.705000]  [<ffffffff8034a3d2>] as_antic_waitnext+0x33/0x41
[  109.705000]  [<ffffffff8034b10f>] as_dispatch_request+0x1e0/0x3e6
[  109.705000]  [<ffffffff8034206b>] elv_next_request+0x1a1/0x1b3
[  109.705000]  [<ffffffff803e1032>] scsi_request_fn+0x74/0x345
[  109.826000]  [<ffffffff803463cd>] blk_run_queue+0x42/0x75
[  109.826000]  [<ffffffff803df887>] scsi_run_queue+0x1d1/0x1e0
[  109.826000]  [<ffffffff803dfaa4>] scsi_next_command+0x36/0x46
[  109.826000]  [<ffffffff803dfcf4>] scsi_end_request+0xc2/0xd5
[  109.826000]  [<ffffffff803dff6d>] scsi_io_completion+0x105/0x34f
[  109.826000]  [<ffffffff803eca57>] ata_hsm_qc_complete+0xf5/0xfe
[  109.826000]  [<ffffffff803e7737>] sd_rw_intr+0x184/0x1b2
[  109.826000]  [<ffffffff803e07dc>] scsi_device_unbusy+0x89/0x91
[  109.826000]  [<ffffffff803dbd05>] scsi_finish_command+0x56/0x5f
[  109.826000]  [<ffffffff803e093e>] scsi_softirq_done+0xd0/0xd8
[  109.826000]  [<ffffffff803f1438>] ata_interrupt+0x20c/0x224
[  109.826000]  [<ffffffff80346784>] blk_done_softirq+0x63/0x72
[  109.826000]  [<ffffffff802395ad>] __do_softirq+0x57/0xc7
[  109.826000]  [<ffffffff8020d32c>] call_softirq+0x1c/0x28
[  109.826000]  [<ffffffff8020e916>] do_softirq+0x34/0x87
[  109.826000]  [<ffffffff80239443>] irq_exit+0x3f/0x90
[  109.826000]  [<ffffffff8020eaad>] do_IRQ+0x144/0x167
[  109.826000]  [<ffffffff8020c691>] ret_from_intr+0x0/0xa
[  109.826000]  <EOI>  [<ffffffff880135ef>] :processor:acpi_idle_enter_bm+0x2cd/0x346
[  109.826000]  [<ffffffff880135e5>] :processor:acpi_idle_enter_bm+0x2c3/0x346
[  109.826000]  [<ffffffff8043caca>] menu_select+0x6b/0x94
[  109.826000]  [<ffffffff8043bbb9>] cpuidle_idle_call+0x71/0x9e
[  109.826000]  [<ffffffff8043bb48>] cpuidle_idle_call+0x0/0x9e
[  109.826000]  [<ffffffff8020aefc>] cpu_idle+0xca/0x110
[  109.826000]  [<ffffffff80514fb8>] rest_init+0x7c/0x7e
[  109.826000]  [<ffffffff806d5ac6>] start_kernel+0x2c9/0x2d4
[  109.826000]  [<ffffffff806d5111>] _sinittext+0x111/0x118
[  109.826000]
...
[  111.313000] POISONS (ffff810003637b68): 00007fffd149e2a0, ffff810003637be8
[  111.313000]
[  111.313000] Call Trace:
[  111.313000]  [<ffffffff803580aa>] __list_add+0xd7/0x138
[  111.313000]  [<ffffffff80358117>] list_add+0xc/0x11
[  111.313000]  [<ffffffff80248110>] add_wait_queue+0x2c/0x40
[  111.313000]  [<ffffffff8029ac28>] __pollwait+0xd6/0xdf
[  111.313000]  [<ffffffff80477b83>] datagram_poll+0x23/0xd5
[  111.313000]  [<ffffffff8046ea63>] sock_poll+0x18/0x1a
[  111.313000]  [<ffffffff8029a498>] do_select+0x2fa/0x50d
[  111.313000]  [<ffffffff8029ab52>] __pollwait+0x0/0xdf
[  111.313000]  [<ffffffff8022d480>] default_wake_function+0x0/0xf
[  111.313000]  [<ffffffff8022d480>] default_wake_function+0x0/0xf
[  111.313000]  [<ffffffff80519a92>] do_page_fault+0x405/0x7ac
[  111.313000]  [<ffffffff80323c61>] inode_has_perm+0x6b/0x7a
[  111.313000]  [<ffffffff8032840f>] selinux_inode_permission+0x98/0x9d
[  111.313000]  [<ffffffff80322dbf>] avc_has_perm+0x4c/0x5e
[  111.313000]  [<ffffffff8029a89e>] core_sys_select+0x1f3/0x2a2
[  111.313000]  [<ffffffff80353c5b>] __up_read+0x7a/0x83
[  111.313000]  [<ffffffff80248153>] remove_wait_queue+0x2f/0x38
[  111.313000]  [<ffffffff8029acf0>] sys_select+0xbf/0x17b
[  111.313000]  [<ffffffff8022d480>] default_wake_function+0x0/0xf
[  111.313000]  [<ffffffff8020c17e>] system_call+0x7e/0x83
[  111.313000]

Attachment: pgpr6QOLb761J.pgp
Description: PGP signature