In a shallow bowl, whisk together the egg, milk, and orange zest (or vanilla). Dip each brioche slice into the mixture, ensuring both sides are well coated.
Melt the butter in a skillet over medium heat. Add the soaked brioche and cook until golden brown and crisp, about 2-3 minutes per side. Spoon a little melted butter over the top as it cooks for extra richness.
In a clean, dry skillet over medium heat, spread the sugar in an even layer. Allow it to melt undisturbed until it turns a deep amber color and small bubbles form.
Reduce the heat to low, then carefully whisk in the heavy cream. Stir until smooth and glossy.
Plate the warm French toast, add a generous scoop of your favorite ice cream, and drizzle the hot caramel sauce over the top.