Good point, that does change my thinking about the nostalgic person being unable to achieve the heights of their past. Maybe the envy comes immediately after the experience. The traveler knows that others who experienced virtually the same thing that they did will recall the night with nostalgia, while it will fade into just another happening in a city for the traveler. The other person's reaction and the ensuing envy the traveler experiences becomes more important in recalling the city than the beautiful sight he shared with its inhabitants.

I could also see some reflexive angles there, with Calvino describing how people will receive his book. Some readers will come away with a significant experience of one of the cities he describes while for others the specificities fade as it becomes a part of an indistinct empire that the book builds. A bit out there but I wouldn't put it past him.