Preheat oven to 375°. In a bowl, mix granulated sugar, brown sugar, flour, cinnamon, nutmeg, salt, and ginger. Add pumpkin, milk, and eggs; whisk until well blended.
Pour mixture into unbaked 10-inch pie pastry in pan. Sprinkle walnut streusel evenly over filling.
Bake on the bottom rack until a knife inserted in center comes out clean, 60 to 65 minutes.
Set on a rack until cool to touch, about 2 hours. Cut into wedges.