Use a vegetable peeler to peel a 2 1/2-in. by 1-in. strip of zest from orange; save fruit for another use. Cut zest strip crosswise into very fine (1/16 in.) slivers; set aside.
Heat oil and garlic in a 10-in. frying pan over medium heat, stirring often, until fragrant, about 1 minute. Add olives, orange zest, thyme, chile flakes, and fennel seeds. Cook, stirring occasionally, until warm, about 2 minutes.
Remove from heat, stir in vinegar, and transfer to a small serving bowl. Serve warm.
Make ahead: Up to 2 weeks, chilled. Rewarm to serve.