The shrine at Mitokusan designated a National Treasure and the oldest such of its kind, isn’t easy to get to and a lot of people won’t want to make the arduous climb to see it close up.
But it is still worth the bus trip from Kurayoshi to view it through the (free) binocular telescope that the tourist bureau has set up, and look at it perched four-fifths of the way up a steeply wooded mountainside.
This temple has existed since the Nara Period (8th century) and looking at it will make you appreciate the ingenuity and perseverance of the Tendai monks who built it. The local government is currently lobbying to have the temple recognized as a World Heritage Site.
On your way back, why not stop in Misasa. Misasa itself is an onsen town, with dozens of hot springs lining the small and beautiful Mitoku river. There are also lots of specialty restaurants in the town, and you can dine on crab and the famous Tottori pears in the evening.
Mornings and afternoons are lazy. You can rent bicycles and take a shaded ride along the river, or have a rotenburo bath in one of the riverside pools that are established for this purpose. (While men generally bathe nude in these, women are expected to wear a bathing suit of some sort.)