Well that formula is not correct - it looks like a linear interpolation - but that does not work on sphere. One counterexample would be: latidude_A = latitude_B = 45 deg ... then latitude_C would also have to 45 deg (for all C-points) - but that is not a great circle.
To get the unique great circle through the two points you need the vector n perpendicular to this great circle which is in the plane defined by O, A and B. You can compute this vector with the "x-product" of the vectors a and b:
n = a x b
To get the equation for points (c) on the great circle you can cunstruct a new vector m which is perpendicular to a and n and is therefore in the plain of the great circle:
m = n x a
now m needs to be trimmed to the right length (the radius of the sphere r = |a|):
m_r = m * (r / |m|)
all points c may have an arbitrary angle "angle_COA" in the range of [0, 360] deg and can be calculated from:
c = a * cos(angle_COA) + m_r * sin(angle_COA)
note that "*" indicates the multiplication of a vector with a scalar
You can interpret a and m_r as the new coordinate axes in 3 dimensional space for a normal two axes circle.
Of course the according formulas with b would also work.
I hope that helps.