The band got help from Bert, Big Bird Oscar the Grouch, Cookie Monster and more.
Two more celebs were unmasked on Wednesday night.
Let us know your favorite musician appearance on 'Sesame Street' by voting in our poll.
"I've played a lot of instruments, but I've never played a cookie before!" the pop star laughs.
Bert and his old pal Ernie dropped by 'Elle' to play the popular 'Song Association' game.
He performed on 'Sing Along With Mitch' and was a teenage idol in Japan before spending 46 years on the iconic kids TV program.
The new sneak peek also features Samuel L. Jackson & Ava DuVernay.