Hey Roland (and any iommu/ppc/dma experts out there):
I'm debugging a data corruption issue that happens on PPC64 systems
running rdma on kernels where the iommu page size is 4KB yet the host
page size is 64KB. This "feature" was added to the PPC64 code recently,
and is in kernel.org from 2.6.23. So if the kernel is built with a 4KB
page size, no problems. If the kernel is prior to 2.6.23 then 64KB page
configs work too. Its just a problem when the iommu page size != host
page size.
It appears that my problem boils down to a single host page of memory
that is mapped for dma, and the dma address returned by dma_map_sg() is
_not_ 64KB aligned. Here is an example:
app registers va 0x000000002d9a3000 len 12288
ib_umem_get() creates and maps a umem and chunk that looks like (dumping
state from a registered user memory region):
umem len 12288 off 12288 pgsz 65536 shift 16
chunk 0: nmap 1 nents 1
sglist[0] page 0xc000000000930b08 off 0 len 65536 dma_addr 000000005bff4000 dma_len 65536
So the kernel maps 1 full page for this MR. But note that the dma
address is 000000005bff4000 which is 4KB aligned, not 64KB aligned. I
think this is causing grief to the RDMA HW.
My first question is: Is there an assumption or requirement in linux
that dma_addressess should have the same alignment as the host address
they are mapped to? IE the rdma core is mapping the entire 64KB page,
but the mapping doesn't begin on a 64KB page boundary.
If this mapping is considered valid, then perhaps the rdma hw is at
fault here. But I'm wondering if this is an PPC/iommu bug.
BTW: Here is what the Memory Region looks like to the HW:
TPT entry: stag idx 0x2e800 key 0xff state VAL type NSMR pdid 0x2
perms RW rem_inv_dis 0 addr_type VATO
bind_enable 1 pg_size 65536 qpid 0x0 pbl_addr 0x003c67c0
len 12288 va 000000002d9a3000 bind_cnt 0
PBL: 000000005bff4000
Any thoughts?
Steve.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[Index of Archives]
[Kernel Newbies]
[Netfilter]
[Bugtraq]
[Photo]
[Stuff]
[Gimp]
[Yosemite News]
[MIPS Linux]
[ARM Linux]
[Linux Security]
[Linux RAID]
[Video 4 Linux]
[Linux for the blind]
[Linux Resources]
