Why not use Skype, Ventrilo or TeamSpeak?
The biggest reason is it sucks resources away from SL, which uses pretty much everything it can get as it is now (especially with the official client. I'm on Nicholaz 17d, and since it has more memory holes fixed, it would be a little better).
Of course, if you just wanna hold a private conversation, why not have it either in SL via text (which the voicechat program will begin degrading the experience of), or in your program of choice (hoping they have or can easily obtain it)?
Now, the last part brings up another interesting aspect: compatibility. None of these programs work with the others. Skype is most popular, it appears, as it remains free as long as you go PC-to-PC, so it's the more logical choice. However, if my memory serves me right, it throws advertising up in the window, and some people really don't like that. What SL adding voice does is it removes this issue completely.
Personally, I hope the Lindens automatically make the entire grid voice-enabled, especially since the voice chat usage is a Preferences option in the first place. Don't wanna hear it? The switch is just a couple keystrokes and mouse-clicks away (Ctrl-P, Voice Chat tab, Enable Voice Chat checkbox). You won't even be able to tell who's using it, since you don't see the little white dots. Might even be nice to have a Voice button on the toolbar for even easier access to that switch. Tune in when you want, tune out when you don't wanna be bothered.