I want to create a 'premium' ad network for a niche market, where I want to invite sites which are suitable for the ads.. is there a software or service which helps setting up an ad network?
For software you can use OpenX as an ad server.
As for creating a premium network, you have a chicken and egg problem. Sites won't sign up for exclusive ads unless you have good advertisers and can offer good rates. Advertisers won't come until you have good sites.
I think what you are trying to build is something like The DECK so if you haven't you might want to check out their service.
