I've had this a couple of times and have just bagged up some hops and dropped them into the barrel. The difference it makes is night and day.
I've experimented with pulling them out after a week or so (by stringing the bag using dental floss) so that you can remove it, and also just leaving it...