Is there a way to search only books available in nearby libraries in Google Books?
No, Google does not display that information. However, the Worldcat webapp http://www.worldcat.org/ will do what you want. In their own words:
WorldCat.org lets you search the collections of libraries in your community and thousands more around the world. WorldCat grows every day thanks to the efforts of librarians and other information professionals.
You can see if your local libraries are indexed by searching for your library or town here.
No. Google books stores one copy of every book (that they have gotten to). They do not publish information about which library they got that book for nor which library's have that book. Your best bet is to just call up local libraries if you come across a book you like. If they don't have it they may be able to tell you who does or even get it on a library loan.