Grouped path layout approach

I am trying to come up with the most intuitive approach for presenting a user with a dialog where they may setup one to n groups and then proceed to add paths to those groups.

Currently I use a FlowLayoutPanel and add ListBox elements corresponding to the number of groups the user chooses.

I am stuck on the best approach for either using events such as MouseClick and MouseDown to add and remove paths. I can use a right click to invoke a FolderBrowserDialog and add a new path but then deleting a path or a group for that matter becomes less intuitive.

Does combining all the functionality into mouse events make this worse for a user, if so should I have explicit buttons that Add/Remove for both groups and paths within groups? A ListBox can highlight the selected item correlating a delete button but the ability to delete the ListBox itself may not be so obvious?

Or I could just be using a naive approach altogether (UI design is not my forte)? Anyone have any suggestions?