The Hume coal project proposes to extract up to 3.5 million tonnes of coal each year for 19 years and the associated Berrima Rail Project involves the development of a new rail spur and connecting loop.
The DEP has referred the projects to the Independent Planning Commission for a public hearing.
"The department has undertaken a rigorous examination of the proposals, which included seeking independent expert advice on a range of issues, including groundwater, mine design, economics and noise," it said.
"The department received a total of 12,666 submissions on the project, including more than 5000 submissions from the local area, of which 97% were objections."
Wingecarribee Shire Council objected to the project, while key government agencies including the Department of Industry - Lands and Water, the Environment Protection Authority and WaterNSW all raised concerns about impacts on groundwater and surface water.
The Resources Regulator also said the proposed mining method was untested and that the mine could be unsafe for workers.
"The department considers there is currently considerable scientific uncertainty about the level of environmental damage to both groundwater and surface water resources," it said.
"The predicted drawdown impacts on the ground water aquifer would be the most significant for any mining project that has ever been assessed in NSW."
Hume Coal project director Greig Duncan said while the company was disappointed by the findings of the state's planning department, it was important to understand the determining authority for State Significant Developments in NSW, such as the Hume Coal project, was the IPC.
"The NSW Department of Planning and Environment is yet another stakeholder involved in the state's planning process and as such is entitled to put forward its opinion on the project proposal," he said.
"Despite the department's findings, the company strongly believes it has put forward a strong proposal, which is low impact and ensures the long-term sustainability of the operation while generating hundreds of employment opportunities and millions of economic benefits for the Southern Highlands."
NSW Greens resources spokesman Jeremy Buckingham welcomed the DPE's recommendation but said NSW needed a coal strategy to deal with new coal mine applications and phasing out coal in a strategic way.
"This is a strong rejection of the proposed Berrima coal mine echoing the concerns that the local community have held for many years," Buckingham said.
"It's a pity it has taken almost eight years and the eve of the NSW election for the government to finally knock this bad mine on the head. The local Liberal MP Pru Goward failed to stop this mine progressing through the planning system despite widespread concern, even when she was planning minister.
"The tide has well and truly turned on coal in an age of climate change.
"NSW now needs a comprehensive coal strategy that should forbid any new coal mines and implement a strategy to phase out coal mining over the next decades with transition plans for the regions involved."
The "untested" mining approach the Resources Regulator refers to is what Hume Coal calls the "first workings" mining system.
It is a system developed by Hume's engineering team in which underground main roadways or "mains" will be developed. That is followed by mining panels at an angle off the mains, together forming the trunk and limbs of the mine.
Continuous miners will be used for mining at typical widths in standard drivages of between 5.2m to 5.5m.
Over-width drivages, typically up to 6m, may be developed from time-to-time for specific purposes such as niches for equipment, and will be geotechnically assessed on a case-by-case basis.
Following development of the gate roads, a narrow head continuous miner and continuous haulage system will be used to develop a series of parallel drives - also referred to as plunges - off the outside edges of the gate roads.
The plunges will each be 4m wide and angled at 70 degrees to the gate roads.
Pillars will vary in size depending on the depth of cover.
