I was also thinking that newer drivers would be slower so i kept my old 44.03 or 41.09 for a long time but i just downloaded and installed the latest official 52.16 and i feel that game effects are better (wet reflection, smog effect in heavy weather for example).
So just have a try, quality and speed are there.