In the Old City (Icherisheher), right behind the Palace of the Shirvanshahs.

You can go without worries and without a permit. There is no border checkpoint in the Maloye Almaty Gorge.