How to improve the user experience of an illustrated map?

Are there any recommended suggestions to improve the interaction cost for an illustrated map? More precisely, any time a user clicks on a "point of interest" and then goes back to the map, all items load again. Also, from usability testing and watching analytic recordings, it seems as if users aren't clicking all the way through and instead just "hovering" over illustrations on the map. Any suggestions on how we can get users more engaged with the map content? Are there any other sites that have set a good precedent for this?

Link to map here: http://thebambergergroup.com/map/murray-hill/nyc