Combine the first 4 ingredients in a bowl. Spoon into a 9-inch round cake pan coated with cooking spray.
Place 2/3 cup sugar and next 4 ingredients (2/3 cup sugar through egg) in a large bowl; beat with a mixer at medium speed until weel blended (for about 5 minutes). Lightly spoon flour into dry measuring cups; level with a knife. Combine flour, baking powder, baking soda, and salt, stirring well with a whisk. Add the flour mixture to sugar mixture alternately with buttermilk, beginning and ending with flour mixture; mix after each addition.
Spoon batter over peach mixture in pan. Bake at 350 for 45 minutes or until a wooden pick inserted in the center comes out clean. Cool 10 minutes in pan on a wire rack. Place a plate upside down on top of cake, and invert onto plate. Serve warm with frozen yogurt and caramel syrup.