1. Preheat the oven to 350F. Grease and flour two 8-inch round cake pans.
2. Combine the brown sugar, butter, eggs and vanilla in a large bowl. Beat with an electric mixer on low speed until blended. Beat on medium-high speed, scraping bowl occasionally, until light and fluffy, 2 to 3 minutes.
3. Combine the flour, ground walnuts, baking powder and baking soda in a medium bowl; mix well. Add the flour mixture alternately with the milk to the brown sugar mixture, beating on low speed until well blended. Divide the batter between the prepared pans.
4. Bake the cakes until a toothpick inserted in the centers comes out clean, 30 to 35 minutes. Cool in the pans on wire racks for 10 minutes. Remove the cakes from the pans; cool completely on wire racks.
5. Place 1 cake layer on a serving plate. Spread the top with about 2/3 cup of the frosting. Top with the second layer. Spread side and top of cake with the remaining frosting. Arrange the walnut halves on top of the cake.