Heat grill to medium-low (350° to 400°).
Combine 1 tbsp. chopped rosemary, the salt, and pepper in a small bowl. Peel potatoes and put in a bowl of water. Coat a large cast-iron skillet with 1 tbsp. oil. Using a mandoline or knife, cut half the potatoes into 1/8-in. slices and arrange them in tightly overlapping circles in skillet. Sprinkle with half of salt mixture. Repeat with remaining potatoes and salt mixture, making a second layer. Drizzle remaining 3 tbsp. oil over potatoes, shaking pan to distribute it well. Press potato cake with hands to compact.
Cook potatoes with grill lid down until well browned on bottom (lift edge with a spatula to check), 35 to 40 minutes. Remove from grill. Loosen potatoes with a wide spatula. Invert a platter over pan, carefully hold both together (use oven mitts), and invert potato cake. Scatter a little more chopped rosemary over potatoes, plus rosemary flowers if you like. Cut into wedges.
Make ahead: Keep potato cake warm in pan or on platter in a 200° oven up to 1 hour.
* Percent Daily Values are based on a 2,000 calorie diet. Your daily value may be higher or lower depending on your calorie needs.