Don't forget the color depth. If you go to 8- or 16-bit color rather than the 24-bit color, you cut your memory requirements by 33-66%. You can then get the 1152x864 size, but not as many colors. Here is the video RAM needed for an 1152x864 display:
24-bit: 24MB 16-bit: 15MB 8-bit: 7.5MB
If all you can give is 8MB, you're limited to an 8-bit 1152x864 display (256 colors). The formula is:
video RAM needed (in MB) = (width x height x number-of-bits)/1048576
Your formula is calculating the amount of memory in Mega _bits_ not Mega _Bytes_. You have to divide the number by 8. So you get
video RAM needed (in MB) = (width x height x number-of-bits) / 8388608
-- mads
