I was the second or third party to climb a route, when it was not a free ascent, and loose block, crack fill with dirt, unsafe pillar are daily climbing for me.
Sure, if you have the 2nd or 3rd ascent at a "backcountry" crag, I an see breaking hand holds and have to contend with dirt filled cracks. What I was referring to is North Conway and its neighborhood you frequent. You do know there are climbers around who take the time and trouble to clean overgrown older routes so they and us can climb them again, right?
Here's one for you, one of the safety related things one learn to do on a route (suspected to have loose blocks and weak holds) is to test the hold before fully weighing it. A lot of routes on Canon qualify. And if the hold feels weak or loose, find a way around it, or use it very gently. Dropping broken rock onto or near the belayer sounds to me like one of the most unsafe things to do, dear grand maestro of safety. Even the beginners plying the carriage road down at The Gunks know this tidbit!!
P.S. Like John, I have had only 1 or 2 hand holds break on me ever, and not because I have not had my share of choss. And the broken part was small, like an arrowhead. I have broken footholds too, on a wide variety of routes, all small flaky or pebble stuff. But I once bagged a 2-pound chunk of loose rock directly above one of the belay anchors on The Standard Route at WH. Luckily, I had a day pack with me so we took it to take top before disposing of it.