To those who never listen to albums in their entirey, it obviously won't matter as much. However, I do think they're missing out a bit. Many albums are much more than just collections of songs and have an extra layer of appreciation that can be tapped into if you listen to and take it in as a whole. Those who occasionally do appreciate albums in their entirety will know that the best albums leave you wanting for more when the last track finishes. To me, it's not good if I get tired of the album before it's over .. For an album to be really good, I think it should be enjoyable from start to finish without being boring.
So yeah, length matters.
Close to the Edge by Yes is an example of an album which I think is way too short .. in a good way.
edit :
If I put on an album that has been rereleased with extra tracks such as proto-versions of the songs, I often leave the extras out of the playlist. I also tend to leave them out when I write reviews .. The reason is they were not intended as part of the original product and including them as part of the whole can turn an otherwise nicely timed and exciting album into a long and repetitive one.