The spine of this truly charming region is the Balakan-Baku highway, a misnomer for what is mostly a quaint country lane traversing everything from desert to verdant forest. It offers by far the most interesting way to transit between Baku and Tbilisi.
In most cases the attractions (timeless villages, ancient Albanian church-ruins and very ruined castle remnants) lie to the north of this road in the thickly wooded foothills of the high Caucasus Mountains. The glorious scenery gets particularly impressive between the towns of Ismayilly and Sheki, especially in spring, when wildflowers and fields full of poppies add majestic splashes of colour to the woodland foreground. However, while the area is great for admiring the white-tops, if you really want to get among the highest peaks you'd do better to start from Quba or Qusar.
Note that by public transport it's easier to travel from west to east. That is because from any town there will be fairly regular Baku-bound marshrutkas, but coming the other way vehicles tend to be full on arrival at any intermediate town.