Just turn off Sydney Road in Balgowlah to Woodlands Street and follow to the end, then turn right at the end of the road; there is parking and access to Grotto Point Walk.
Take a stroll down the bush walk, take your time and you can stop at the Aboriginal sandstone animal engravings which are hundreds of years old.
You can access the rocks from small tracks which run next to the lighthouse. This is a very quiet spot.
You can cast out towards Middle or North Head and be in the right area to catch some of the pelagic species that the Harbour is famous for.
This is the only access the fish have to Middle Harbour - feeding on the way in and as the tide runs out, they pass you again.
Techniques for fishing this area is with a small bobby cork or with as little weight as the conditions will allow. Also because you are fishing in such a large area of water, burley is very important if you want to have a good session.
Using a burley of stale bread or chicken pellets soaked in some tuna oil should do the job.
Leave a good few hours to get to this spot, and there is a bit of rock climbing after you reach the lighthouse. But the views are fantastic.
This link will give you a map but getting there is relatively easy.