The architect David Adjaye talks with TIME about moving past signature architecture and collaborating with Brad Pitt