Melt 1/4 cup of the butter. Pour 2 tablespoons of the melted butter into an 8-cup bundt pan; brush the butter over pan sides and bottom. Mix together the brown sugar, cinnamon, nutmeg, and almonds. Sprinkle bottom of pan with half the brown sugar mixture; combine the remaining mixture with the remaining melted butter; set aside.
In a large bowl, beat remaining 1/4 cup butter with granulated sugar until blended. Beat in eggs, 1 at a time, until blended. Beat in orange peel and banana.
Mix all-purpose and whole-wheat flours, baking powder, soda, and salt. Add to banana mixture along with the buttermilk; stir until well blended.
Pour half the batter into prepared pan. Spoon remaining brown sugar mixture evenly over top; cover with remaining batter.
Bake in a 350° oven until a long wood skewer inserted into the thickest part of the cake comes out clean, about 50 minutes. Cool the cake on a rack about 5 minutes, then invert cake onto a serving plate. Serve the cake warm or cool.