I want to create a 'premium' ad network for a niche market, where I want to invite sites which are suitable for the ads.. is there a software or service which helps setting up an ad network?


For software you can use OpenX as an ad server.

As for creating a premium network, you have a chicken and egg problem. Sites won't sign up for exclusive ads unless you have good advertisers and can offer good rates. Advertisers won't come until you have good sites.

I think what you are trying to build is something like The DECK so if you haven't you might want to check out their service.