Heat the oven to 350°F/180°C.
Heat the olive oil and butter in a wide skillet over medium heat. Add the onion and cook until soft and pale golden, about 6 minutes. Add the mushrooms and cook until their liquid has evaporated, about 5 minutes. Stir in the garlic and cook just until fragrant, about 30 seconds.
Add the cream, milk, salt, pepper, and nutmeg. Bring to a gentle simmer and cook until lightly thickened, 5 to 7 minutes. Remove from heat.
Arrange the potato slices in overlapping layers in a baking dish. Pour the sauce evenly over the potatoes and scatter the Gouda on top. Cover and bake for 15 minutes.
Uncover and continue baking until the potatoes are tender and the top is deeply golden, about 25 minutes more. Let rest briefly before serving.