Using fast VIS and RAD is fine for test compiles, but a full compile is mandatory when creating a version for release. Your map will run slower, especially with a fast VIS compile, as VIS doesn't do proper visibility calculations on the map (hence the speed).
There are ways to speed up VIS. The most easiest is the use of the func_detail entity. Converting complex brush-based geometry to a func_detail will greatly assist VIS when it's performing its calculations.
Increased compile times can be caused by many things: large, open maps, lightmap grids, and, the most common of all, unoptimised maps.
This is a great guide on how optimising Source maps: it gives a brief insight into how BSP (the Source map format) works and renders. It's nessecary to grasp even just a basic idea of this, as it'll give you a clearer indication as to what's causing problems down the track.