I hate genre rules... Everybody mentions rear sets, but if you look at vintage cafe pictures you won't see many. The big thing was to make your bike fast without much budget.
The truth is that the style has evolved. Just like a modern chopper doesn't look like one from back in the day, the modern interpretation of the cafe bike has changed. There are so many influences, especially with access to the internet: street fighters, flat/street tracker, even chopper styles have changed what might be considered cool.
My advice is look at a lot of pictures, and eventually you'll get a feel for what you want to build.