Plugging Coefficient[TrigReduce[(a + b*Cos[x])^4],Cos[2*x]]
into Wolfram|Alpha produced the output you wanted it to. This leads me to suggest that your problem might have to do with how the expression is being evaluated as opposed to a problem with how you are mathematically thinking about it.
I do not have access to a copy of Mathematica so can not test this, but I would try changing :=
to =
in the second line of code.
I would also try putting it all in one line as
g[x_, k_] := Coefficient[TrigReduce[(a + b Cos[x])^4], Cos[k x]]
If that works, it is definitely something wrong with how/when Mathematica is assigning stuff.
See this link for more information.