I reckon most people will be done with the temples after two full days, but that's cos im a temple lover rather than enthusiast. Spot on re: pool. However I still think the town is definitely worth an afternoons wander (I really liked it).
If you take three days over the temples you'd probably end up with a free afternoon anyway. You can only get 1, 3 or 5 day passes anyway iirc.
Some fantastic restaurants in both Siem Reap and Phnom Penh.