Men of a certain age or weight shouldn't wear Lycra, to be honest I don't get why your average cyclist going out for a ride needs to wear it, I guess if you are riding competitively you need it the rest for me are just Wiggins wannabes.
Speaking as an ex cyclist who used to wear lycra despite what I looked like wearing it - compared to wearing normal trousers and say a fleece, lycra adds about 5mph to your cruising speed due to the massive reduction in wind resistance. It's bad enough having to slog up hills but at least with lycra on you can zoom down the other side.
For a comparison try freewheeling a Landrover down a hill compared to your average saloon car.