There really isn't a better time to visit Japan than the spring. During the first half of the year, a series of flowers bloom in a fashion as orderly as the Japanese themselves: narcissus and camellia in January; ume (plum) blossoms in February; peach blossoms in March; sakura (cherry) blossoms in late March or early April, depending on the weather; wisteria, azaleas, and peonies around Golden Week; hydrangea from late May; irises in June, and so on.
Before coming to Japan I couldn't have identified a peony had my life depended upon it, but nineteen years on I'm practically a botanist. Much of my knowledge of the flora Japanica has come to me passively, through dating women who either taught or were learning ikebana (flower arrangement). The rest has been filled in by students, many of whom are invariably trotting off to, say, Mount Kuju to view the wild azaleas in June, foraging their local woods for horsetails and bamboo shoots in early spring, or joining tours to see famous sakura tree. (Seriously.)
I must be turning Japanese, because I too willingly (and gleefully) partake in these flower viewing festivities. A few years ago I traveled to the Daikôzen temple (大興善寺) in Kiyama, Saga prefecture to see the azaleas there.