Brian Smith worked with garden guru Paul Sorensen in his Leura nursery for 15 years but during a visit back to the site last month, he was disappointed at the state of the old arboretum.
He pointed out many trees which needed dead wood cut back, suckers that should be removed and stubs protruding where branches had been cut but should have been pruned flush to the trunk.
He said the area had been left untended for so long before any restoration work had started, it would be almost impossible to return it to its best.
"To try and bring it back to what it was - to me it will never happen. They are just trying to work with the material they've got. If they can preserve the remaining trees, they've got a bit of a chance."
The Herbert St site has been divided into four sections: one where a number of townhouses have been built, one section of open space, one where the original Sorensen home remains and the fourth which contains the arboretum and where a new building has been constructed and approval given for a retail nursery.
But Mr Smith, who runs his own nursery in Moorland near Taree, said he couldn't see that venture succeeding.
"It isn't going to work because there's nowhere to grow your stock. It's all too shady.
"It would take someone with a bit of skill not only to run the nursery but to maintain the arboretum and there would be very, very few people around who could do it."
Paul Sorensen planned his arboretum to give customers an idea of what mature specimens would look like. Under each large tree were smaller plants of the same species for sale and interspersed among the trees were shrubs which provided propagation material.
The plant displays, plus the stone walls at the nursery, provided inspiration to many amateur gardeners in the Mountains.
Kerry Brown, a horticulturalist and former manager of Everglades, Sorensen's most famous Mountains garden, said the new building would be an ideal site for a garden heritage centre to boost the tourist economy.
"Over six decades, Sorensen and his nursery provided designs and plants for hundreds of gardens up and down the Mountains," she said.
She described Mountains gardens as a "unique tourism asset".
"Where else in the world do you find streetscapes looking across mature canopies of cool climate exotics and natives to a world heritage wilderness?
"A Blue Mountains garden heritage centre could guide visitors and interpret our horticultural riches in the same way that the National Parks and Wildlife Centre at Blackheath guides people about our wilderness," Ms Brown said.
A spokesman for the site's developer, Jaymay Developments, said preliminary steps were being taken to seek out potential tenants for the new building. It may be used as a nursery, cafe, art gallery or some other purpose, he said.
