It does get into a gray area, particularly when dealing with more chossy rock, but to lay a boundary, I think I would have to go with intent. If you are thinking "I want to make a hold here" or get rid of this hold there to make the route harder, then it has gone into chipping. If you are doing something that is noticeable to others and pissing off the majority of other climbers, then it is pretty obvious you have gone over the line.
I am not one who rabidly believes any intentional altering of the rock is automatically cause for a public hanging, but our proscribing against it serves us well. The problem is twofold to start with. First are the hacks like Timmy Fairfield who make a mess out of the rock, leaving it obvious that it has been altered, thereby destroying the essence for others of why most of us climb outdoors. Secondly, it is a very slippery slope which can lead to all kinds of stupidity. Even if someone was artful, we could end up with a bunch of routes like gym climbs, all comfortable holds, even difficulty with individuality beaten out of them. A little humility is in order if you think you are wiser and more creative than Mother Nature