For certain players, such as John Grabow (who is a type A middle reliever), they will not be able to find another team willing to give up a 1st round draft pick in order to sign them.  In Buster Olney's blog (subscription required), he has a quote from an AL GM who says, "If they offered [Grabow] arbitration, there's no chance another team would sign him. You're not going to give up a No. 1 pick for a middle reliever."  This is a fairly recent phenomenon, as teams have just started figuring out the value of Rule 4 picks.

We saw this happen last year when players like Orlando Hudson, Juan Cruz, and Orlando Cabrera were offered arbitration which drove their prices and bargaining position way down in the free agency market.  For players like Grabow, clubs won't be willing to give up a first round draft pick to sign them.  This essentially forces the player to either go to arbitration (which typically won't give the player the raise they would get on the FA market) or try to negotiate a new contract with the same club from a weak bargaining position.  Players can get a clause on their next contract that forbids teams from offering them arbitration (as Cabrera did), but, again, this drives down their value.

So for players in a certain strata (lower-end type A and lower-end type B) to stay in MLB, they are essentially forced to re-sign with their current team.  In effect, compensation picks allow teams to perpetually renew contracts for certain players.

For those of you who say that compensation picks help lower-end teams like the Pirates, I would say that their supposed goal to compensate lower-end teams for losing players to free agency has failed.  Good players going to free agency typically come from good teams, so better clubs get more compensation picks.  Additionally, as the Yankees showed last year, if you have more spending power, you can get more type A free agents while giving up less.  They signed CC Sabathia, AJ Burnett, and Mark Teixeira, but only had one first round pick to forfeit.

Hopefully, the MLBPA will manage to eliminate compensation picks during their next negotiations with ownership.