The first examples of theme and variations can be traced back to the 14th century and the form was popular in the Baroque era of music. The chaconne, passacaglia and groundbass are all forms of the variation and theme format. They can even be called variations on the format themselves.
Some of the masterpieces of the form of theme and variations have been written for keyboard from a theme that is far from complicated. The Diabelli variations of Beethoven derive from a simple march, the famous set of variations that comprise Paganini's 24th Caprice for Solo Violin also uses a fairly simple tune that has the distinction of not only inspiring the composer of the tune to vary it, but many other composers as well. And of course the famous Goldberg Variations by Bach that inspired many to write their own set of variations.
Brahms wrote his Variations And Fugue On A Theme Of Handel in 1861 when he was 28 years old and dedicated the work to his dear friend Clara Schumann, the widow of Robert Schumann. Brahms was not only a composer, but he was a scholar, particularly of older music. He had written other sets of variations before, but the Handel Variations came after his study of Baroque forms. It is this duality of Brahms, the scholar of older music that was none the less a product of the Romantic era he lived, that makes his music so interesting. Some have called him ultra conservative, but the ultra modern composer Schoenberg always considered Brahms a progressive. So there is much more to Brahms than appears to the ear. Some of his progressiveness is of a technical nature, such as his odd number of bars in his phrases for example. These technical devices are hard to explain to non-musicians, but they can certainly be heard as something different by the attentive listener.
The basis of the variations is an aria from Handel's Harpsichord Suite No. 1 in B-flat Major, a tune that is varied by Handel himself in the suite. It is a two-part tune that Brahms composed 25 variations for. Brahms used some Baroque forms for some of the variations in accordance with his study of the era but he didn't restrict himself to these forms at all. There are some free form variations also, and Brahms manages to keep things together as a structure by having sub-divisions within the whole where some of the variations are 'related' to one another, while also managing to keep the intensity level moving forward to the crown of the work, the fugue. It is a testament to Brahms' skill as a composer that the fugue is never played without the variations. The fugue is a complicated, contrapuntal masterpiece in its own right, but it is an organic growth of what has preceded it. To divorce it from its parent variations would make the fugue, despite all of its wonders, unintelligible.
The 28-year old Brahms was still perfecting has mastery of the piano when he wrote this piece. He played them in public on occasion and it was the piece that he played for his first meeting with Richard Wagner. Evidently Wagner was impressed enough to tell Brahms that it was a good example of what could still be done with the old forms with someone who knew how to use them.
It wasn't the last time Brahms wrote a set of variations. He continued to use the form, it expand and deepen his expression of his musical ideas. Ahead lay the Paganini Variations and the chaconne of the 4th symphony, but the Handel Variations marked an important period in the development of Brahms.