Black holes are, perhaps, improperly described as giant. Since they're basically just a point in space, it is the sheer gravitational force that these points can exert on outer objects. This force is the out-of-proportions thing that is huge. (think about the relation of size-to-gravitational influence: how extraordinarily large is a star, yet it is weaker than that theoretical point)
Soup Miner, are you suggesting that the black hole point is the most dense thing in the Universe? Or it is something totally different from neutron stars?