Getting out of the Central District is a good idea for those who want to see something other than high skyscrapers. A nice option might be catching a bus from Hyde Park and getting to Christison Park and its surroundings.
This is not a popular spot for tourists, so you won't feel like a stranger in the city. In addition, and what is even better, from here you will probably get the best views of the city, with the perfect skyline, featuring the Habour Bridge, the Sydney Opera and the Television Tower.
A good start point is Christison Park and that will allow you to take a walk around the Dover Heights and Rose Bay area, which is really recommended. From there, you might be able to walk along the ocean cliffs that surround the bay, so watch out if you suffer from vertigo!