Swap memory is not useful for the textures, so forget the 4GB virtual. We are talking (preferably, but not exclusively, dedicated video) ram here.
If you have a video card that uses shared memory (that means onboard and does not have it's own), or your AGP aperture is set really low, or if you have a lot of memory hogs running before you start rfactor, yes you will run out of memory. Not because you don't have enough, but because it is poorly organized.
Press CTRL-ALT-DELETE on desktop, and select the performance tab. Look at Physical Memory, and the check available. I have 2GB (2095528K) and 1,5 GB is available (1539692K) as I type this.
Press ctrl-F immediately when you get to the track. You should get a green bar. You can see use and surplus, surplus being coloured. When it turns yellow, rfactor has run out of memory for the textures, and has resized them. When it turns red, it has resized them again.
It's only after we've lost everything, that we are free to do anything.