Strange OOM problem (swap empty)

[Disconnect <dc.disconnect@gmail.com>]

Last night one of our testbed fileservers went OOM for unknown
reasons.  (The log is attached.)  Its a moderately complex setup - 2
dual-core Opteron chips with 8G of (matched) ram under Xen.  The file
array (which was doing heavy writes from this node using rsync) is an
external multi-connected scsi using clvmd and ocfs2.  (The other node
had no problems, but it was not doing anything other than idling with
2 xen guests, also idle.)

Local filesystems and swap are on an internal scsi raid rather than
the shared device.

The confusing part is that the dump shows swap almost completely
unused. So shouldn't memory pressure have forced something out to
swap?

I can provide any other info that would be helpful (and I'm subscribed
to the list).

Jul 10 18:23:50 dvsc3 kernel: Free swap  = 5961616kB
Jul 10 18:23:50 dvsc3 kernel: Total swap = 6032368kB

Attachment: OOM.log
Description: Binary data

host                   : dvsc3
release                : 2.6.18-4-xen-amd64
version                : #1 SMP Fri May 4 02:40:51 UTC 2007
machine                : x86_64
nr_cpus                : 4
nr_nodes               : 1
sockets_per_node       : 2
cores_per_socket       : 2
threads_per_core       : 1
cpu_mhz                : 2193
hw_caps                : 178bfbff:e3d3fbff:00000000:00000010:00000001:00000000:00000003
total_memory           : 8191
free_memory            : 6536
xen_major              : 3
xen_minor              : 0
xen_extra              : .3-1
xen_caps               : xen-3.0-x86_64
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : Tue Oct 17 22:09:52 2006 +0100 
cc_compiler            : gcc version 4.1.2 20061028 (prerelease) (Debian 4.1.1-19)
cc_compile_by          : ultrotter
cc_compile_domain      : debian.org
cc_compile_date        : Fri Nov  3 00:21:27 CET 2006
xend_config_format     : 2

Attachment: cpuinfo
Description: Binary data

Attachment: free
Description: Binary data

Name                                      ID Mem(MiB) VCPUs State   Time(s)
Domain-0                                   0      512     4 r-----     24.7
dv-xeng3.internal                          3      512     1 -b----      2.2
dv-xeng5.internal                          4      512     1 -b----      2.1