In a large mixing bowl, combine the milk, peanut butter, vanilla extract, sugar, nutmeg, cinnamon, orange zest and eggs. Whisk until well mixed.
Butter an 11x7 baking dish. Add a layer of the bread and top with peanut butter chips, bananas and some of the custard mixture. Repeat another layer, adding any remaining custard to the top. Allow the bread to soak up the custard for 15 to 20 minutes.
Bake uncovered for 30 to 45 minutes or until a toothpick comes out clean. Allow the bread pudding to cool for a few minutes, then top with caramel sauce before serving.