Will American Idol producers keep it in the family when they replace outgoing judge Simon Cowell? Rumor has it AI is considering doing just that by moving host Ryan Seacrest into Cowell’s soon to be vacated seat on the judging panel.

American Idol judge Simon Cowell told host Ryan Seacrest he is “very sad” to be leaving the show. Not sad enough, however, to stick around for another season. “I think you gotta know when it’s time to go.” Cowell told Seacrest on his KIIS-FM morning show Tuesday.