In a 10- to 12-inch frying pan over medium-high heat, stir fennel until seeds are slightly darker, 1 to 2 minutes. Add oil, peel, chilies, and olives. Stir until olives are hot, about 2 minutes.
Serve olives hot, warm, or at room temperature. If making ahead, store airtight up to 2 weeks.