I want to create a 'premium' ad network for a niche market, where I want to invite sites which are suitable for the ads.. is there a software or service which helps setting up an ad network?
Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.
For software you can use OpenX as an ad server.
As for creating a premium network, you have a chicken and egg problem. Sites won't sign up for exclusive ads unless you have good advertisers and can offer good rates. Advertisers won't come until you have good sites.
I think what you are trying to build is something like The DECK so if you haven't you might want to check out their service.