OK so the results are in! I left the 2 beers for 2 weeks warm 2 weeks cold conditioning to give them a decent amount of time before tasting. In order to prevent any preconceptions influencing the tasting I covered the caps with gaffa tape and mixed the bottles around until I was unable to tell which was which and would only find this out after making tasting notes, both beers were poured into identical glasses that had been well rinsed prior to use. I drank water and ate a bit of a cracker in between tasting each beer to try and cleanse my palate as much as possible. I tried to assess the beers following the tasting suggestions given by Mr Mosher in Radical Brewing.
Before I get to the final verdict (and to build a bit of tension lol!) some general observations of the process overall. Its an easy area to get confused by and thanks to everyone who offered help and advice during this thread, I have found that the best approach is to keep it simple and if possible change only 1 or 2 things at a time in order to keep track of what effect if any you are having on the finished product. From my experience, the advice of others and what I have read the most important things to look at are de-chlorinating, reducing alkalinity (if your water is particularly hard) and adjusting the sulphate to chloride ratio to suit the style you are brewing. As such I would suggest a basic water treatment kit can consist of Campden Tabs, CRS or other suitable Acid, Gypsum and Calcium Chloride to get you started. The old forum water calculator has worked well for me and assuming your alkalinity figures are accurate it should get you into the right ballpark in terms of PH, I have used CRS on all my brews since and measured Mash PH and it generally lands between 5.2-5.3. It has overshot slightly a couple of times where there is a greater quantity of crystal/caramel malts involved, I put this down to the calculator giving the same CRS quantity for bitters as pale ales where I think it should be a touch lower for the bitters and this option should be used for pale beers with 10% + crystal or other speciality malts. Although I didn't see an increase in efficiency from the lower PH it has been consistently in the optimal window through using CRS.
Anyway on to the final verdict! The beer on the left was beer A and the right one beer B. Upon pouring both beers had identical colour, carbonation and clarity. The only difference was that the sediment in beer B appeared a little looser and more beer was left in the bottle to stop this going in the glass. My first impression of beer A was little obvious aroma apart from general beeriness, a lager like character which was quite dry and bitter and a slight astringent/sour finish. Straight away I thought this was the untreated batch as the astringency reminded me of the samples I tasted at bottling time. My first impression of beer B was the same lager like character but a very slightly more prominent hop aroma, also dry but a smoother overall taste and bittering, hop character was more obvious but not massively prominent with a floral spicy presence. Side by side the difference was pretty obvious although I think if you tried them in isolation you might not notice the issues with beer A as much, beer B was definitely my preference and had a cleaner more refreshing flavour overall. At this point I revealed the labels on the bottle caps and as I suspected Beer A was the untreated batch and Beer B was the treated, so a positive result for water treatment.
Conclusion:
Obviously whether it is worth treating your water depends entirely on your local water profile but for me I will be continuing to treat my brewing water from now on. I believe that reducing the alkalinity has had the biggest impact overall for me and especially for pale beers this is worth doing if you have hard water. The sulphate to chloride ratio of my water already slightly favours hoppy beer but there was a noticeable change through the addition of Gypsum and again where it suits the style I will be continuing to add this or Calcium Chloride for malt driven beers. Luckily the actual process of treating the water is pretty straightforward and inexpensive so for me its definitely been worth the extra effort.