Re: Dropping NETIF_F_SG since no checksum feature.

["Michael S. Tsirkin" <mst@mellanox.co.il>]

Quoting r. David Miller <davem@davemloft.net>:
> Subject: Re: Dropping NETIF_F_SG since no checksum feature.
> 
> From: "Michael S. Tsirkin" <mst@mellanox.co.il>
> Date: Wed, 11 Oct 2006 23:23:39 +0200
> 
> > With my patch, there is a huge performance gain by increasing MTU to 64K.
> > And it seems the only way to do this is by S/G.
> 
> Numbers?
> 

I created two subnets on top of the same pair infiniband HCAs:

root@sw069 ~]# ifconfig ib0
ib0       Link encap:UNSPEC  HWaddr
00-00-04-04-FE-80-00-00-00-00-00-00-00-00-00-00
          inet addr:12.4.3.69  Bcast:12.255.255.255  Mask:255.0.0.0
          inet6 addr: fe80::202:c902:20:ee45/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:2044  Metric:1
          RX packets:1382531 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2725206 errors:0 dropped:5 overruns:0 carrier:0
          collisions:0 txqueuelen:128
          RX bytes:71892772 (68.5 MiB)  TX bytes:5290011992 (4.9 GiB)

[root@sw069 ~]# ifconfig ibc0
ibc0      Link encap:UNSPEC  HWaddr
00-03-04-06-FE-80-00-00-00-00-00-00-00-00-00-00
          inet addr:11.4.3.69  Bcast:11.255.255.255  Mask:255.0.0.0
          inet6 addr: fe80::202:c902:20:ee45/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:65484  Metric:1
          RX packets:115647 errors:0 dropped:0 overruns:0 frame:0
          TX packets:253403 errors:0 dropped:4 overruns:0 carrier:0
          collisions:0 txqueuelen:128
          RX bytes:6014720 (5.7 MiB)  TX bytes:16589589008 (15.4 GiB)

The other side was configured with 12.4.3.68 for MTU 65484
and 11.4.3.68 for MTU 2044. 

And then I just run netperf:
[root@sw069 ~]#
[root@sw069 ~]# /mswg/work/mst/netperf-2.4.2/src/netperf -f M -H 12.4.3.68 -c -C
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 12.4.3.68 (12.4.3.68)
port 0 AF_INET
Recv   Send    Send                          Utilization       Service Demand
Socket Socket  Message  Elapsed              Send     Recv     Send    Recv
Size   Size    Size     Time     Throughput  local    remote   local   remote
bytes  bytes   bytes    secs.    MBytes  /s  % S      % S      us/KB   us/KB

 87380  16384  16384    10.00       286.45   40.20    25.28    5.482   3.448

[root@sw069 ~]# /mswg/work/mst/netperf-2.4.2/src/netperf -f M -H 11.4.3.68
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 11.4.3.68 (11.4.3.68)
port 0 AF_INET
Recv   Send    Send
Socket Socket  Message  Elapsed
Size   Size    Size     Time     Throughput
bytes  bytes   bytes    secs.    MBytes/sec

 87380  16384  16384    10.01     782.55

This is all very preliminary - but I hope you get the idea -
increasing MTU is very helpful for infiniband, and infiniband adapters
handle large S/G lists without problems, but the verbs
do not include support for IP checksums, so these must be done in software.

So what we would like, is for the infiniband network device to say
"I don't support checksums, I only support S/G" and then for
network layer to do the checksumming for us piggybacking on data copy
at least for cases where it does perform the copy.

Does this makes sense now?

-- 
MST
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/