A shorter alternative route (and more properly called the Mosman Bay Walk) is to start at Cremorne Point wharf and follow the shoreline around to Musgrave St. This takes around 1 hour and there are lots of places to have a picnic, or you could stop ot the characterful Mosman Rowers for a seafood lunch.
You can then either catch the South Mosman ferry or continue on to Taronga.
Why? Great views and lost of interesting places to see on the way.