How to make a dialog appear less busy?

The right-hand configuration side of my current connection dialog looks reasonable enough at the moment:

However, with the addition of several (very handy!) options, it is starting to look too busy:

What are some techniques I can apply to make it less so?