Put potatoes in a large pot and cover with cold water. Cover and bring to a boil over high heat. Reduce heat to medium and cook until potatoes are tender when pierced, about 15 minutes. Drain; return to pot.
Meanwhile, in a large frying pan, cook hazelnuts over medium heat, stirring often, until fragrant, about 5 minutes. Add butter and cook, stirring frequently, until butter is golden brown and flecked with brown bits and hazelnuts are dark brown, about 5 minutes. Pour hot hazelnut mixture into a bowl and set aside.
Mash hot potatoes. Add milk, yogurt, salt, and pepper, mashing to blend. Transfer to a serving bowl and spoon about half the warm hazelnut mixture over the top; serve the rest on the side.