Letterman Bandleader Paul Shaffer

Paul Shaffer is finally stepping out in front. Before landing the gig as David Letterman's bandleader and sidekick on Late Night (now called Late Show) in 1982, the native Canadian navigated a long, winding path to fame: from playing music in Toronto strip clubs to joining the original Saturday Night Live band and cast in the '70s — and along the way co-composing the Weather Girls' hit "It's Raining Men" and starring in a short-lived sitcom. His memoir, We'll Be Here the Rest of Our Lives, is chock-full of tales from the 59-year-old's show-business days. Shaffer spoke with TIME about being a struggling musician, the perils of late-night TV and why he can't discuss Letterman's personal life — even as Letterman himself can't seem to stop.

After being on Letterman's show for 27 years, what inspired you to finally sit down and write a memoir?
I actually had tried it once, about 10 years ago. I thought I could do it myself, but I ended up having to give the money back. I didn't finish it. Later, I ran into David Ritz, who is my co-author, at a Bon Jovi rehearsal. He had written the Ray Charles biography. So I knew who he was, and he asked if I ever wanted to do a book.

When you were playing clubs in Toronto, did you feel like you had made it?
Even though I was working in this strip club, [I felt like] I'd made it to some degree. But I was kind of depressed. It felt like I was going to be doing that for the rest of my life. And basically I am. I think that the title [of the book] is a little philosophical, in that I am basically doing that kind of work. I'm doing a five-night-a-week lounge gig. I just do it for David Letterman.

I think a lot of people probably don't realize you were part of a sitcom called A Year at the Top.
The sitcom looked good. It was produced by Don Kirshner, who was a very hot rock 'n' roll producer, and Norman Lear, who was at the top of his game producing about 12 major sitcoms at once, including All in the Family, The Jeffersons, Good Times, One Day at a Time. We were a show about guys who had sold their souls to the devil to become rock stars. It was a hell of an experience, though: being in Hollywood, coming out with a new sitcom, being on the same lot where Soul Train taped. Luckily, when the show didn't last, I got my old job on Saturday Night Live back.

You were in the original Blues Brothers but weren't able to be in the first movie. What did that feel like?
I had lovingly put together that band with John Belushi and Dan Aykroyd and really felt part of it. Then that situation came up where I was co-producing a record for Gilda Radner at the same time and didn't get it finished on time, and had to make a choice between John and Gilda. And I chose to be loyal to Gilda. But it was devastating not being in the movie. John was very hurt by it. We entered into what I like to call a show-business feud. Just like Dean Martin and Jerry Lewis. But just like Dean and Jerry, we reunited on the Jerry Lewis telethon.

Can you think of any really challenging moments when you had to improvise in the middle of a show?
Certainly the very first show on Letterman. I couldn't believe that show business was going to be this disorganized. Billy Murray was the main guest. He came by early in the day to talk to the writers about what he might do. And he had this vague idea that he might sing "Let's Get Physical." And then he said he had to go feed his dog and left, and he never came back! There was no rehearsal at all. I didn't even know what key he was going to sing in. He comes back, just in time for the show, and I just started the music and he went for it. It was totally improvised. I think that's true reality television.

You've recorded and performed with hundreds of artists. Could you pick your ideal band from all the people in show business whom you've worked with?
Well, that's kind of cool, can I pick from dead people too? For rock drums, Keith Moon. Bass guitar, Will Lee from my band. There is no one better. I love Jeff Beck on guitar. And Ronnie Laws on tenor saxophone.

And yourself?
I might just produce it. And we'd do Duane Eddy covers all night.

Letterman has been in the news quite a bit lately. Do you think that will help your book sales?
I have no idea. I really don't.

What did you think of his confession last week?
You know, I just can't talk about it. There is a legal proceeding going on. I've been advised that I can't comment on that stuff.

Is Late Show something you want to stick with? What are your upcoming ambitions?
Certainly I'm going to stick with it for as long as we continue to do it. After that I think I can just lie down. I want to learn to sight-read music. And to play the bass pedals on the organ. Those are my only ambitions.

Is there anyone in the music business you have never played with whom you'd want to?
Elvis and Sinatra and Sam Cooke. Otherwise, I did pretty well.