In a small saucepan over medium heat, whisk cocoa, sugar, and 2 tbsp. water to form a smooth paste. Add another 2 tbsp. water; stir until mixture is warm and sugar dissolves. Remove from heat; stir in milk.
Pour mixture into a metal bowl, cover, and chill until cold, about 1 hour. Process in an ice cream maker according to manufacturer's directions. Or, pour mixture into a shallow metal pan or ice cube tray and freeze 3 hours or up to overnight; then whirl in a food processor, adding up to 2 tbsp. water to smooth mixture if necessary.
Transfer mixture to a freezer-safe container, cover, and freeze until hard, about 30 minutes.
Note: Nutrition analysis is per 3/4-cup serving.