And for some strange reason, the ladder does work if the visual ladder's there, but only when crouching. I don't get that.
Ok, moving it out substantially fixed it. I'm going to try moving it back towards the visual ladder unit by unit, see if I can find out where the edge of the 'not working' zone is.
EDIT: Apparently the minimum distance between the func_ladder and the func_door_rotating visual ladder is approximately 19 units.
Admittedly, I did end up dead on the floor at the other end of the room, but at least it's fixed.