Jim Carrey might have been willing to gain 50 pounds to play Curly Howard, but his film has lost an entire body. Sean Penn, who was scheduled to play Larry Fine in the Three Stooges, has just dropped out of the project.

The Farrelly brothers film would have joined the two-time Oscar winner with Carrey and Benicio Del Toro. (The Che star was picked to star as Moe Howard.) Unfortunately, Sean has reportedly decided to put family before work. (His rocky relationship with Robin Wright Penn is currently back on.) That choice means giving up a role in Cartel as well.

It’s not known at this point what Sean’s leave will mean for the long-awaited comedy. Stooges was slated to begin production sometime in August. If the studio opts to wait for his return, the film could be delayed for another year.