Personally I'd say nuke the little b@st@rds. While they don't particularly bother me, I'd hate to hear about some unsuspecting visitor with an allergy go into shock 3 pitches up because they don't have an epi pen with them. While they may not be aggressive now, they will be soon as things warm up more and they are building their condos. Besides, these are not honeybees, doing mankind a service by pollinating our zucchinis, these are wasps.
By the way, from climbing at the Gunks I have found that if they are buzzing you, it is possible to kill them in mid-air with a sharp blow from an open hand; a guidebook works even better. Just don't miss, 'cause that just gets 'em angry.