Preheat oven to 350°F. Grease two 9-inch pans and line with parchment. Whisk dry ingredients together in a bowl and set aside.
Cream butter with both sugars on medium speed for 4 to 5 minutes until light and fluffy. Add eggs one at a time, then sour cream and vanilla.
Add flour mixture in three additions alternating with buttermilk in two additions, mixing on low only until just combined after each.
Toss diced peaches in 1 tbsp flour, fold gently into batter with 6 to 8 strokes. Divide between pans and smooth tops.
Bake 28 to 34 minutes until a toothpick comes out clean. Cool in pans 15 minutes then transfer to a wire rack and cool completely before frosting.
Make frosting: melt butter in a saucepan, add brown sugar, stir and cook 2 minutes. Remove from heat, stir in cream carefully. Cool 20 minutes. Beat with powdered sugar, vanilla, and salt 3 to 4 minutes until smooth and spreadable.
Frost the cooled cakes, spreading frosting between the layers and across the top and sides. Garnish with fresh peach slices if desired. Rest 15 minutes before slicing.