Does the ext3 fs have a 2gig limit to begin with?
No, it doesn't. My home fileserver has ext3 filesystems and has no trouble handling DVD ISO files, some of which exceed 4GB.
Least whenever I try and record an avi longer than 2 gig it craps out.
Most likely the process that is writing the avi file does not have large file support.
Paul.