Adding a new VIP group is easy. Right click on your VIP list and select "Add new group". This will open a new window and ask you to type a name for the new group.
To make the groups show up on the VIP list, you also need to select "Show groups", also by right clicking on the VIP list.
Once you have added a new group, you can assign players to the group, by right clicking on their name and selecting "Edit (playername)". Then in the following window tick the groups that the player belongs to.
Removing group is a little more tricky. You must make sure that "Show groups" is enabled and that at least one player is assigned to that group. If that's the case, right click on the group name in the VIP list and select "Remove group (group name)".
Note: you cannot remove the three predefined groups: Enemies, Friends and Trading Partners. Also, you can only have 5 of your own groups. If you already have 5, you will have to remove one before you can add new.