Prepare grill for medium-high heat. Place a medium skillet or cast-iron pan on the grill. Add oil and let heat 3 minutes. Add the bananas, cut side down, and sprinkle the brown sugar evenly over the top. Cook bananas until heated through and sugar is melted, 5 to 7 minutes. Add whiskey and flip bananas over. Add the walnuts and cook until the bananas are well caramelized and the whiskey has reduced to a syrup. Serve over pancakes with vanilla ice cream, if desired.