Re: NFSD stops working

["Dr.Peer-Joachim Koch" <pkoch@bgc-jena.mpg.de>]

Hi,

here is some output from /var/log/messages

and also this one comes sometimes very often:
Apr 11 15:07:34 sizilia kernel: RPC svc_write_space: some sleeping on 
00000100eeb86c80
Apr 11 15:07:51 sizilia last message repeated 1829 times
                                              ^^^^

sizilia:~ # grep "Apr 11 16:34" /var/log/messages
Apr 11 16:34:04 sizilia kernel: Unable to handle kernel NULL pointer 
dereference at 000000000000003c RIP:
Apr 11 16:34:04 sizilia kernel: <ffffffff80231b56>{__down_read+6}
Apr 11 16:34:04 sizilia kernel: PML4 ebfb3067 PGD f08b0067 PMD 0
Apr 11 16:34:04 sizilia kernel: Oops: 0002 [1] SMP
Apr 11 16:34:04 sizilia kernel: CPU 0
Apr 11 16:34:04 sizilia kernel: Pid: 32304, comm: nfsd Tainted: P   U 
(2.6.5-7.191-smp SLES9_SP2_BRANCH-200506281458560000)
Apr 11 16:34:04 sizilia kernel: RIP: 0010:[<ffffffff80231b56>] 
<ffffffff80231b56>{__down_read+6}
Apr 11 16:34:04 sizilia kernel: RSP: 0018:00000100e5b13688  EFLAGS: 00010006
Apr 11 16:34:04 sizilia kernel: RAX: 00000100f2a84d20 RBX: 
fffffffffffffff4 RCX: 00000100d530d000
Apr 11 16:34:04 sizilia kernel: RDX: 00000100f7fef288 RSI: 
00000000ffffffff RDI: 0000000000000038
Apr 11 16:34:04 sizilia kernel: RBP: 00000100d4b93000 R08: 
0000010037ffe810 R09: 0000000000000000
Apr 11 16:34:04 sizilia kernel: R10: 00000100f7fef298 R11: 
00000100f7fef2a8 R12: 00000100f54a8c00
Apr 11 16:34:04 sizilia kernel: R13: 0000000000001000 R14: 
0000000000001000 R15: 0000000000000001
Apr 11 16:34:04 sizilia kernel: FS:  0000002a9588e6e0(0000) 
GS:ffffffff80562e80(0000) knlGS:00000000556d5860
Apr 11 16:34:04 sizilia kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 
000000008005003b
Apr 11 16:34:04 sizilia kernel: CR2: 000000000000003c CR3: 
0000000000101000 CR4: 00000000000006e0
Apr 11 16:34:04 sizilia kernel: Process nfsd (pid: 32304, threadinfo 
00000100e5b12000, task 00000100f2a84d20)
Apr 11 16:34:04 sizilia kernel: Stack: 0000000000001000 0000000000001000 
0000000000000001 0000000000000246
Apr 11 16:34:04 sizilia kernel:        fffffffffffffff4 ffffffff80193097 
00000004ecccad58 00000100f6265c00
Apr 11 16:34:04 sizilia kernel:        00000100d530d900 0000000000000001
Apr 11 16:34:04 sizilia kernel: Call 
Trace:<ffffffff80193097>{bio_map_user+247} 
<ffffffffa034cb61>{:cvfs:linuxif_map_bio+49}
Apr 11 16:34:04 sizilia kernel: 
<ffffffffa0387cdd>{:cvfs:cv_buf_strat+525} 
<ffffffffa03b6a77>{:cvfs:cvZoneAlloc+679}
Apr 11 16:34:04 sizilia kernel: 
<ffffffffa036f796>{:cvfs:CvDoIo+1190} 
<ffffffffa037036f>{:cvfs:cvStrategy+2575}
Apr 11 16:34:04 sizilia kernel: 
<ffffffffa03881ac>{:cvfs:AlignedIO+380} <ffffffffa037273c>{:cvfs:rwcvp+5516}
Apr 11 16:34:04 sizilia kernel: 
<ffffffffa03bc01b>{:cvfs:MdOsAlloc+139} 
<ffffffffa0352251>{:cvfs:cv_uio_allocate_using_uio+161}
Apr 11 16:34:04 sizilia kernel: 
<ffffffffa03737f1>{:cvfs:read_cvp+273} 
<ffffffffa038f332>{:cvfs:cvfs_read+210}
Apr 11 16:34:04 sizilia kernel: 
<ffffffffa04fffd0>{:nfsd:nfsd_acceptable+0} 
<ffffffff8018c74a>{do_readv_writev+490}
Apr 11 16:34:04 sizilia kernel: 
<ffffffffa034c730>{:cvfs:cvfs_read_wrapper+0} 
<ffffffffa034d9df>{:cvfs:linuxif_is_procname+31}
Apr 11 16:34:04 sizilia kernel: 
<ffffffffa038f407>{:cvfs:cvfs_open+55} 
<ffffffffa05026d2>{:nfsd:nfsd_read+818}
Apr 11 16:34:04 sizilia kernel: 
<ffffffffa0509d60>{:nfsd:nfsd3_proc_read+240} 
<ffffffffa04fd130>{:nfsd:nfsd_dispatch+256}
Apr 11 16:34:04 sizilia kernel: 
<ffffffff803522d2>{svc_process+1010} 
<ffffffff80135c50>{default_wake_function+0}
Apr 11 16:34:04 sizilia kernel:        <ffffffffa04fd460>{:nfsd:nfsd+0} 
<ffffffffa04fd6a5>{:nfsd:nfsd+581}
Apr 11 16:34:04 sizilia kernel: 
<ffffffff8013aa56>{schedule_tail+70} <ffffffff801112b7>{child_rip+8}
Apr 11 16:34:04 sizilia kernel:        <ffffffffa04fd460>{:nfsd:nfsd+0} 
<ffffffffa04fd460>{:nfsd:nfsd+0}
Apr 11 16:34:04 sizilia kernel:        <ffffffff801112af>{child_rip+0}
Apr 11 16:34:04 sizilia kernel:
Apr 11 16:34:04 sizilia kernel: Code: f0 fe 4f 04 0f 88 d2 00 00 00 8b 
07 85 c0 78 15 48 8d 4f 08
Apr 11 16:34:04 sizilia kernel: RIP <ffffffff80231b56>{__down_read+6} 
RSP <00000100e5b13688>
Apr 11 16:34:04 sizilia kernel: CR2: 000000000000003c
Apr 11 16:34:04 sizilia kernel:  <1>Unable to handle kernel NULL pointer 
dereference at 000000000000003c RIP:
Apr 11 16:34:04 sizilia kernel: <ffffffff80231b56>{__down_read+6}
Apr 11 16:34:04 sizilia kernel: PML4 ebfb3067 PGD f08b0067 PMD 0
Apr 11 16:34:04 sizilia kernel: Oops: 0002 [2] SMP
Apr 11 16:34:04 sizilia kernel: CPU 0
Apr 11 16:34:04 sizilia kernel: Pid: 32305, comm: nfsd Tainted: P   U 
(2.6.5-7.191-smp SLES9_SP2_BRANCH-200506281458560000)
Apr 11 16:34:04 sizilia kernel: RIP: 0010:[<ffffffff80231b56>] 
<ffffffff80231b56>{__down_read+6}
Apr 11 16:34:04 sizilia kernel: RSP: 0018:00000100e8525688  EFLAGS: 00010006
Apr 11 16:34:04 sizilia kernel: RAX: 00000100eafdb390 RBX: 
fffffffffffffff4 RCX: 0000000000000000
Apr 11 16:34:04 sizilia kernel: RDX: 0000010037ffe800 RSI: 
00000000000000d0 RDI: 0000000000000038
Apr 11 16:34:04 sizilia kernel: RBP: 00000100d4ed1000 R08: 
0000000000000000 R09: 00000100f7ec1910
Apr 11 16:34:04 sizilia kernel: R10: 00000100f7f27880 R11: 
0000000000000002 R12: 00000100f54a8c00
Apr 11 16:34:04 sizilia kernel: R13: 0000000000001000 R14: 
0000000000001000 R15: 0000000000000001
Apr 11 16:34:04 sizilia kernel: FS:  0000002a9588e6e0(0000) 
GS:ffffffff80562e80(0000) knlGS:00000000556d5860
Apr 11 16:34:04 sizilia kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 
000000008005003b
Apr 11 16:34:04 sizilia kernel: CR2: 000000000000003c CR3: 
0000000000101000 CR4: 00000000000006e0
Apr 11 16:34:04 sizilia kernel: Process nfsd (pid: 32305, threadinfo 
00000100e8524000, task 00000100eafdb390)
Apr 11 16:34:04 sizilia kernel: Stack: 0000000000000000 fffffffffffffff4 
00000100d4ed1000 0000000000000246
Apr 11 16:34:04 sizilia kernel:        fffffffffffffff4 ffffffff80193097 
00000004ecccad58 00000100f6265c00
Apr 11 16:34:04 sizilia kernel:        00000100d530d8c0 0000000000000001
Apr 11 16:34:04 sizilia kernel: Call 
Trace:<ffffffff80193097>{bio_map_user+247} 
<ffffffffa034cb61>{:cvfs:linuxif_map_bio+49}
Apr 11 16:34:04 sizilia kernel: 
<ffffffffa0387cdd>{:cvfs:cv_buf_strat+525} 
<ffffffffa03b6a77>{:cvfs:cvZoneAlloc+679}
Apr 11 16:34:04 sizilia kernel: 
<ffffffffa036f796>{:cvfs:CvDoIo+1190} 
<ffffffffa037036f>{:cvfs:cvStrategy+2575}
Apr 11 16:34:04 sizilia kernel: 
<ffffffffa03881ac>{:cvfs:AlignedIO+380} <ffffffffa037273c>{:cvfs:rwcvp+5516}
Apr 11 16:34:04 sizilia kernel: 
<ffffffffa03bc01b>{:cvfs:MdOsAlloc+139} 
<ffffffffa0352251>{:cvfs:cv_uio_allocate_using_uio+161}
Apr 11 16:34:04 sizilia kernel: 
<ffffffffa03737f1>{:cvfs:read_cvp+273} 
<ffffffffa038f332>{:cvfs:cvfs_read+210}
Apr 11 16:34:04 sizilia kernel: 
<ffffffffa04fffd0>{:nfsd:nfsd_acceptable+0} 
<ffffffff8018c74a>{do_readv_writev+490}
Apr 11 16:34:04 sizilia kernel: 
<ffffffffa034c730>{:cvfs:cvfs_read_wrapper+0} 
<ffffffffa034d9df>{:cvfs:linuxif_is_procname+31}
Apr 11 16:34:04 sizilia kernel: 
<ffffffffa038f407>{:cvfs:cvfs_open+55} 
<ffffffff8018db13>{open_private_file+195}
Apr 11 16:34:04 sizilia kernel: 
<ffffffffa0501dbe>{:nfsd:nfsd_open+302} 
<ffffffffa05026d2>{:nfsd:nfsd_read+818}
Apr 11 16:34:04 sizilia kernel: 
<ffffffffa0509d60>{:nfsd:nfsd3_proc_read+240} 
<ffffffffa04fd130>{:nfsd:nfsd_dispatch+256}
Apr 11 16:34:04 sizilia kernel: 
<ffffffff803522d2>{svc_process+1010} 
<ffffffff80135c50>{default_wake_function+0}
Apr 11 16:34:04 sizilia kernel:        <ffffffffa04fd460>{:nfsd:nfsd+0} 
<ffffffffa04fd6a5>{:nfsd:nfsd+581}
Apr 11 16:34:04 sizilia kernel:        <ffffffff801112b7>{child_rip+8} 
<ffffffffa04fd460>{:nfsd:nfsd+0}
Apr 11 16:34:04 sizilia kernel:        <ffffffffa04fd460>{:nfsd:nfsd+0} 
<ffffffff801112af>{child_rip+0}
Apr 11 16:34:04 sizilia kernel:
Apr 11 16:34:04 sizilia kernel:
Apr 11 16:34:04 sizilia kernel: Code: f0 fe 4f 04 0f 88 d2 00 00 00 8b 
07 85 c0 78 15 48 8d 4f 08
Apr 11 16:34:04 sizilia kernel: RIP <ffffffff80231b56>{__down_read+6} 
RSP <00000100e8525688>
Apr 11 16:34:04 sizilia kernel: CR2: 000000000000003c
Apr 11 16:34:04 sizilia kernel:  <1>Unable to handle kernel NULL pointer 
dereference at 000000000000003c RIP:
Apr 11 16:34:04 sizilia kernel: klogd 1.4.1, ---------- state change 
----------
Apr 11 16:34:04 sizilia kernel: <ffffffff80231b56>{__down_read+6}
Apr 11 16:34:04 sizilia kernel: PML4 f542a067 PGD f55ef067 PMD 0
Apr 11 16:34:04 sizilia kernel: Oops: 0002 [3] SMP
Apr 11 16:34:04 sizilia kernel: CPU 1
Apr 11 16:34:04 sizilia kernel: Pid: 32306, comm: nfsd Tainted: P   U 
(2.6.5-7.191-smp SLES9_SP2_BRANCH-200506281458560000)
Apr 11 16:34:04 sizilia kernel: RIP: 0010:[<ffffffff80231b56>] 
<ffffffff80231b56>{__down_read+6}
Apr 11 16:34:04 sizilia kernel: RSP: 0018:00000100e56e7688  EFLAGS: 00010006
Apr 11 16:34:04 sizilia kernel: RAX: 00000100eafdaa80 RBX: 
fffffffffffffff4 RCX: 0000000000000000
Apr 11 16:34:04 sizilia kernel: RDX: 00000100f7fc9400 RSI: 
00000000000000d0 RDI: 0000000000000038
Apr 11 16:34:04 sizilia kernel: RBP: 00000100d86bc000 R08: 
0000000000000000 R09: 00000100f7ec1950
Apr 11 16:34:04 sizilia kernel: R10: 00000100f7ea7298 R11: 
00000100f7ea72a8 R12: 00000100f54a8c00
Apr 11 16:34:04 sizilia kernel: R13: 0000000000001000 R14: 
0000000000001000 R15: 0000000000000001
Apr 11 16:34:04 sizilia kernel: FS:  0000002a9588e6e0(0000) 
GS:ffffffff80562f00(0000) knlGS:0000000056d95bb0
Apr 11 16:34:04 sizilia kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 
000000008005003b
Apr 11 16:34:04 sizilia kernel: CR2: 000000000000003c CR3: 
0000000037e06000 CR4: 00000000000006e0
Apr 11 16:34:04 sizilia kernel: Process nfsd (pid: 32306, threadinfo 
00000100e56e6000, task 00000100eafdaa80)
Apr 11 16:34:04 sizilia kernel: Stack: 0000000000000000 fffffffffffffff4 
00000100d86bc000 0000000000000246
Apr 11 16:34:04 sizilia kernel:        fffffffffffffff4 ffffffff80193097 
00000004ecccad58 00000100f6265c00
Apr 11 16:34:04 sizilia kernel:        00000100d530dd00 0000000000000001
Apr 11 16:34:04 sizilia kernel: Call 
Trace:<ffffffff80193097>{bio_map_user+247} 
<ffffffffa034cb61>{:cvfs:linuxif_map_bio+49}
Apr 11 16:34:04 sizilia kernel: 
<ffffffffa0387cdd>{:cvfs:cv_buf_strat+525} 
<ffffffffa03b6a77>{:cvfs:cvZoneAlloc+679}
Apr 11 16:34:04 sizilia kernel: 
<ffffffffa036f796>{:cvfs:CvDoIo+1190} 
<ffffffffa037036f>{:cvfs:cvStrategy+2575}
Apr 11 16:34:04 sizilia kernel: 
<ffffffffa03881ac>{:cvfs:AlignedIO+380} <ffffffffa037273c>{:cvfs:rwcvp+5516}
Apr 11 16:34:04 sizilia kernel: 
<ffffffffa03bc01b>{:cvfs:MdOsAlloc+139} 
<ffffffffa0352251>{:cvfs:cv_uio_allocate_using_uio+161}
Apr 11 16:34:04 sizilia kernel: 
<ffffffffa03737f1>{:cvfs:read_cvp+273} 
<ffffffffa038f332>{:cvfs:cvfs_read+210}
Apr 11 16:34:04 sizilia kernel: 
<ffffffffa04fffd0>{:nfsd:nfsd_acceptable+0} 
<ffffffff8018c74a>{do_readv_writev+490}
Apr 11 16:34:04 sizilia kernel: 
<ffffffffa034c730>{:cvfs:cvfs_read_wrapper+0} 
<ffffffffa034d9df>{:cvfs:linuxif_is_procname+31}
Apr 11 16:34:05 sizilia kernel: 
<ffffffffa038f407>{:cvfs:cvfs_open+55} 
<ffffffff8018db13>{open_private_file+195}
Apr 11 16:34:05 sizilia kernel: 
<ffffffffa0501dbe>{:nfsd:nfsd_open+302} 
<ffffffffa05026d2>{:nfsd:nfsd_read+818}
Apr 11 16:34:05 sizilia kernel: 
<ffffffffa0509d60>{:nfsd:nfsd3_proc_read+240} 
<ffffffffa04fd130>{:nfsd:nfsd_dispatch+256}
Apr 11 16:34:05 sizilia kernel: 
<ffffffff803522d2>{svc_process+1010} 
<ffffffff80135c50>{default_wake_function+0}
Apr 11 16:34:05 sizilia kernel:        <ffffffffa04fd460>{:nfsd:nfsd+0} 
<ffffffffa04fd6a5>{:nfsd:nfsd+581}
Apr 11 16:34:05 sizilia kernel:        <ffffffff801112b7>{child_rip+8} 
<ffffffffa04fd460>{:nfsd:nfsd+0}
Apr 11 16:34:05 sizilia kernel:        <ffffffffa04fd460>{:nfsd:nfsd+0} 
<ffffffff801112af>{child_rip+0}
Apr 11 16:34:05 sizilia kernel:
Apr 11 16:34:05 sizilia kernel:
Apr 11 16:34:05 sizilia kernel: Code: f0 fe 4f 04 0f 88 d2 00 00 00 8b 
07 85 c0 78 15 48 8d 4f 08
Apr 11 16:34:05 sizilia kernel: RIP <ffffffff80231b56>{__down_read+6} 
RSP <00000100e56e7688>
Apr 11 16:34:05 sizilia kernel: CR2: 000000000000003c
Apr 11 16:34:05 sizilia kernel:  <1>Unable to handle kernel NULL pointer 
dereference at 000000000000003c RIP:
Apr 11 16:34:05 sizilia kernel: <ffffffff80231b56>{__down_read+6}
Apr 11 16:34:05 sizilia kernel: PML4 f16c2067 PGD e320f067 PMD 0
Apr 11 16:34:05 sizilia kernel: Oops: 0002 [4] SMP
Apr 11 16:34:05 sizilia kernel: CPU 0
Apr 11 16:34:05 sizilia kernel: Pid: 32307, comm: nfsd Tainted: P   U 
(2.6.5-7.191-smp SLES9_SP2_BRANCH-200506281458560000)
Apr 11 16:34:05 sizilia kernel: RIP: 0010:[<ffffffff80231b56>] 
<ffffffff80231b56>{__down_read+6}
Apr 11 16:34:05 sizilia kernel: RSP: 0018:00000100e758b688  EFLAGS: 00010006
Apr 11 16:34:05 sizilia kernel: RAX: 00000100e917d5e0 RBX: 
fffffffffffffff4 RCX: 0000000000000000
Apr 11 16:34:05 sizilia kernel: RDX: 0000010037ffe800 RSI: 
00000000000000d0 RDI: 0000000000000038
Apr 11 16:34:05 sizilia kernel: RBP: 00000100d51c0000 R08: 
0000000000000000 R09: 00000100f7ec1c70
Apr 11 16:34:05 sizilia kernel: R10: 00000100f7f27880 R11: 
0000000000000002 R12: 00000100f54a8c00
Apr 11 16:34:05 sizilia kernel: R13: 0000000000001000 R14: 
0000000000001000 R15: 0000000000000001
Apr 11 16:34:05 sizilia kernel: FS:  0000002a9588e6e0(0000) 
GS:ffffffff80562e80(0000) knlGS:00000000556d5860
Apr 11 16:34:05 sizilia kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 
000000008005003b
Apr 11 16:34:05 sizilia kernel: CR2: 000000000000003c CR3: 
0000000000101000 CR4: 00000000000006e0
Apr 11 16:34:05 sizilia kernel: Process nfsd (pid: 32307, threadinfo 
00000100e758a000, task 00000100e917d5e0)
Apr 11 16:34:05 sizilia kernel: Stack: 0000000000000000 fffffffffffffff4 
00000100d51c0000 0000000000000246
Apr 11 16:34:05 sizilia kernel:        fffffffffffffff4 ffffffff80193097 
00000004ecccad58 00000100f6265c00
Apr 11 16:34:05 sizilia kernel:        00000100d530d880 0000000000000001
Apr 11 16:34:05 sizilia kernel: Call 
Trace:<ffffffff80193097>{bio_map_user+247} 
<ffffffffa034cb61>{:cvfs:linuxif_map_bio+49}
Apr 11 16:34:05 sizilia kernel: 
<ffffffffa0387cdd>{:cvfs:cv_buf_strat+525} 
<ffffffffa03b6a77>{:cvfs:cvZoneAlloc+679}
Apr 11 16:34:05 sizilia kernel: 
<ffffffffa036f796>{:cvfs:CvDoIo+1190} 
<ffffffffa037036f>{:cvfs:cvStrategy+2575}
Apr 11 16:34:05 sizilia kernel: 
<ffffffffa03881ac>{:cvfs:AlignedIO+380} <ffffffffa037273c>{:cvfs:rwcvp+5516}
Apr 11 16:34:05 sizilia kernel: 
<ffffffffa03bc01b>{:cvfs:MdOsAlloc+139} 
<ffffffffa0352251>{:cvfs:cv_uio_allocate_using_uio+161}
Apr 11 16:34:05 sizilia kernel: 
<ffffffffa03737f1>{:cvfs:read_cvp+273} 
<ffffffffa038f332>{:cvfs:cvfs_read+210}
Apr 11 16:34:05 sizilia kernel: 
<ffffffffa04fffd0>{:nfsd:nfsd_acceptable+0} 
<ffffffff8018c74a>{do_readv_writev+490}
Apr 11 16:34:05 sizilia kernel: 
<ffffffffa034c730>{:cvfs:cvfs_read_wrapper+0} 
<ffffffffa034d9df>{:cvfs:linuxif_is_procname+31}
Apr 11 16:34:05 sizilia kernel: 
<ffffffffa038f407>{:cvfs:cvfs_open+55} 
<ffffffff8018db13>{open_private_file+195}
Apr 11 16:34:05 sizilia kernel: 
<ffffffffa0501dbe>{:nfsd:nfsd_open+302} 
<ffffffffa05026d2>{:nfsd:nfsd_read+818}
Apr 11 16:34:05 sizilia kernel: 
<ffffffffa0509d60>{:nfsd:nfsd3_proc_read+240} 
<ffffffffa04fd130>{:nfsd:nfsd_dispatch+256}
Apr 11 16:34:05 sizilia kernel: 
<ffffffff803522d2>{svc_process+1010} 
<ffffffff80135c50>{default_wake_function+0}
Apr 11 16:34:05 sizilia kernel:        <ffffffffa04fd460>{:nfsd:nfsd+0} 
<ffffffffa04fd6a5>{:nfsd:nfsd+581}
Apr 11 16:34:05 sizilia kernel:        <ffffffff801112b7>{child_rip+8} 
<ffffffffa04fd460>{:nfsd:nfsd+0}
Apr 11 16:34:05 sizilia kernel:        <ffffffffa04fd460>{:nfsd:nfsd+0} 
<ffffffff801112af>{child_rip+0}
Apr 11 16:34:05 sizilia kernel:
Apr 11 16:34:05 sizilia kernel:
Apr 11 16:34:05 sizilia kernel: Code: f0 fe 4f 04 0f 88 d2 00 00 00 8b 
07 85 c0 78 15 48 8d 4f 08
Apr 11 16:34:05 sizilia kernel: RIP <ffffffff80231b56>{__down_read+6} 
RSP <00000100e758b688>
Apr 11 16:34:05 sizilia kernel: CR2: 000000000000003c

--
Mit freundlichem Gruss
    Peer-Joachim Koch
_________________________________________________________
Max-Planck-Institut fuer Biogeochemie
Dr. Peer-Joachim Koch
Hans-Knöll Str.10            Telefon: ++49 3641 57-6705
D-07745 Jena                 Telefax: ++49 3641 57-7705

begin:vcard
fn:Peer-Joachim Koch
n:Koch;Peer-Joachim
org;quoted-printable:MPI f=C3=BCr Biogeochemie;DVA
adr:;;Hans-Knoell-Str. 10;Jena;;07745;Germany
email;internet:pkoch@bgc-jena.mpg.de
title:Dr. 
tel;work:+49 3641 576705
tel;fax:+49 3641 577705
x-mozilla-html:FALSE
version:2.1
end:vcard

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature