Playing Brahms is so different to any other concertos. The soloist needs to reflect upon the grandeur of Beethoven 'Emperor', simplicity of Schumann, passion of Grieg Piano concerto, and at times, the ability to simply go 'over-the-top' like in Tchaikovsky. The music is working at many different levels, and good soloists know this.
This recording is so special because it balances these different aspects so perfectly. Brahms himself would have been happy about the sensitivity with which Gilels treats the first few bars of the 1st movement of the Second Concerto. However, at the same time, Gilels is aggressive when he needs to (if you have this recording, I suggest you listen to the section starting at 7:05 of the first movement of Brahms 2). The balance of quiet first section and the strong, emotionally-charging block chord section is absolutely perfect.
Good soloist also needs a good orchestra, and here, Berlin Philharmoniker certainly delivers. I am particularly impressed with the horn section, which conveys the distinctive German Romantic atmosphere beautifully.