Paramedics were called to Bells Beach, 100km southwest of Melbourne, just before 2.20 this afternoon.
Three people, all in their 20s, were injured in the collapse.
Ambulance Victoria said a 28-year-old man suffered serious upper-body injuries.
Victoria Police said the man died as he was being airlifted to a Melbourne hospital.
A man and a woman sustained non-life-threatening injuries and were taken to hospital.
Victoria Police believe a cliff face upwards of 30 metres high gave way and collapsed on the group.
Six people were sitting under the cliff face when rocks fell and struck three of them.
No one else was injured.