A 4-plus hour long compile for a map the size of Dust 2 that's no more complex than any of the other official maps? What do you have, a 500 MHz Celeron and 128 MB of RAM? That's just not right. But yeah, fast VIS could be a problem if you haven't blocked off the different areas from view of each other very well. You never said anything about the r_speeds though, so I don't actually know if the map itself is much of a problem.
One thing you might want to check though, is how many programs you have running in the background. If you've got a lot, that may be the problem, because even running a simple hollowed cube map on my computer, if I have Hammer, GCFScape, a couple web pages, Chatzilla and Azureus running, then Half-Life will slow down like it's 1999.