Problems getting 2 NICs to work.

Hi.

I've got several HP Proliant servers running mostly Fedora versions of redhat. I'm using kernel 2.4.22-1 right now on the ones that are problematic.

These MBs on the HPs have dual gigabit ethernet controllers in them. /proc/pci reports them correctly as "Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 2)".

I've got two public class C networks with two separate ISPs. When I initialize eth0 to either of my public IP ranges with their respective gateways everything works just fine. So now I'm trying to add an IP from the other ISP's range so that I've got multiple pathways to these public servers.

Whether I add a virtual copy of eth0 with the other ISPs IP/gateway combo or if I plug eth1 into the switch and give it one, networking stops working. I should mention that both ISPs come in to the same router and are in the smae switch plane so all wires from the servers are on the same set of switches.

Here's the route table when things are ok:
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
204.117.218.0 * 255.255.255.0 U 0 0 0 eth0
169.254.0.0 * 255.255.0.0 U 0 0 0 eth0
127.0.0.0 * 255.0.0.0 U 0 0 0 lo
default 204.117.218.254 0.0.0.0 UG 0 0 0 eth0

What the hell is that 169.254.0.0 statement? I've searched hi and lo and can't figure it out. I'm making a blind guess that it's an IPv6 thing? I didn't put it there and removing it doesn't survive a reboot.

Anyhow, at this point all networking is fine. I can ping and trace out and vice versa. Now, when I initialize eth1 here's the table:

Destination Gateway Genmask Flags Metric Ref Use Iface
12.168.88.0 * 255.255.255.0 U 0 0 0 eth1
204.117.218.0 * 255.255.255.0 U 0 0 0 eth0
169.254.0.0 * 255.255.0.0 U 0 0 0 eth1
127.0.0.0 * 255.0.0.0 U 0 0 0 lo
default 204.117.218.254 0.0.0.0 UG 0 0 0 eth0

As soon as this is activated, networking stops functioning. I can't ping in or out. Am I missing some key step? I know that it works becasue I'm running a firewall on another machine that's running a 2.4.20 kernel and it was autoconfigured and it's functioning properly.

Can anyone give me a clue?

Thanks,
Kevin