A new test could be more effective at detecting dangerous prostate cancer than current screening methods, a study suggests.