Preheat oven to 350°. Grease 2 loaf pans (each 8 1/2 by 4 1/2 in.).
Combine flour, baking powder, baking soda, and salt in a medium bowl; set aside.
Mix milk, eggs, butter, sugar, and guavas in the bowl of a stand mixer. Add flour mixture and mix until combined. Divide batter between prepared loaf pans.
Bake until golden brown and a toothpick comes out clean when inserted in center of each loaf, 50 to 60 minutes.
Cool loaves in pan for 15 minutes, then invert onto a cooling rack. Let cool until barely warm to the touch, about 10 minutes.
Mix powdered sugar and milk until smooth and spreadable. Spoon half of glaze onto the center of each loaf, letting it drizzle down sides.