See http://joshsymonds.com/blog/2014/06/12/shell-awesomeness-with-prezto/ for an example of a more involved configuration regarding this.
Some kind of nice keybind or something would be nice, in order to be able to move back and forth in the folder path history.