The answer is actually 1/8, and it's not too hard to get, but the usual trick requires a detour through complex analysis.
Easy part first. We can turn this product of cosines into a sum by using an identity: cos A cos B = 1/2 (cos (A+B) + cos (A-B)). Note that if A-B is negative, we can just turn it positive since cos (-x) = cos x. So...
cos pi/7 cos 2pi/7 cos 3pi/7
= 1/2 (cos 3pi/7 + cos pi/7) (cos 3pi/7)
= 1/2 (cos 3pi/7 cos 3pi/7 + cos pi/7 cos 3pi/7)
= 1/2 (1/2 (cos 6pi/7 + cos 0) + 1/2 (cos 4pi/7 + cos 6pi/7))
= 1/4 (cos 2pi/7 + cos 4pi/7 + cos 6pi/7 + 1)
Now we get to the interesting part. A basic rule of complex analysis is that
e^(ix) = cos x + i sin x
Another one is that there are exactly n solutions to the equation x^n = 1, given by e^(i * 2pi (1/n)), e^(i * 2pi (2/n)), ..., up to e^(i * 2pi (n/n)), which is just 1 itself. In particular, there are 7 solutions to x^7 = 1 (we call them seventh roots of unity):
e^(i * 2pi/7), e^(i * 4pi/7), ..., e^(i*12pi/7), e^(i * 14pi/7)=1
These are roots of the degree-seven polynomial x^7 - 1, so they add up to the coefficient of x^6. Since there IS no x^6 term, they must add up to zero. In particular, their real parts add up to zero. And by the first thing I mentioned above, their real parts are as follows:
cos 2pi/7, cos 4pi/7, cos 6pi/7, cos 8pi/7, cos 10pi/7, cos 12pi/7, 1
We now bring one more identity into play: cos (2pi - x) = cos x. So what we have here is
2 cos 2pi/7 + 2 cos 4pi/7 + 2 cos 6pi/7 + 1 = 0
cos 2pi/7 + cos 4pi/7 + cos 6pi/7 = -1/2
And we can plug this into that first equation:
cos pi/7 cos 2pi/7 cos 3pi/7 = 1/4 ((-1/2) + 1) = 1/4 (1/2) = 1/8
Voila! There may be a simpler way (probably not), but this is the first thing any complex analysis student will try -- he's seen too many exercises like this.