FractionalN Frequency Synthesizer
Negative feedback systems (e.g. PLLs) are unstable (they oscillate) if the total phase shift around the loop is 180° at the frequency where the magnitude of loop gain passes through unity (0 dB).
Two indicators of stability are:
 Phase Margin
How far short of 180° the phase shift is, at the unity gain crossover frequency.
 Gain Margin
How far below unity the gain has fallen, at the frequency where the phase does reach 180°.
These can be evaluated using a Bode plot.
The first step is to determine VCO gain k_{VCO}.
VCO Gain
k_{VCO} = 2.5 x 10^{4} radians per voltsecond at 4.360 MHz.
I've actually slightly increased the VCO tank capacity since this graph was plotted.
The frequency span is now 4180  4310 KHz.
I haven't bothered recharacterising the VCO because the gain still lies within the same spread of values.
The stability margin is quite wide.
Loop gain
The next step is to plug k_{VCO} and the other loop coefficients (k_{PD}, k_{N}) into the transfer function and play around with the RC filter until you get a phase minima at the unity gain crossover frequency like this:
Phase margin = 180120 = 60°, gain margin = 20dB.
This Bode plot was generated in SCILAB using the following script:
s = poly(0,'s');
// PoleZero model of OP42 from manufacturer's data sheet
A = 5e14 * (s+1e6*2*%pi)/(s+20*2*%pi)/(s+300e3*2*%pi)/(s+3e6*2*%pi);
// Loop coefficients
kvco = 2.5e4;
kpd = 0.55e3/2/%pi;
kn = 1/17;
// Active feedback network
c1 = 470e9;
c2 = 22e9;
r = 10e3;
z1 = 1/s/c1 + r;
z2 = 1/s/c2;
// Input RC (pole) 723 KHz
Ri=22;
Ci=10e9;
// Output RC (pole) 1.45 KHz
Ro=1000;
Co=110e9;
f = 1/(1+s*Ri*Ci) * 1/(1/z1 + 1/z2) * 1/(1+s*Ro*Co);
g = A/.(1) * kpd * f * kvco/s * kn;
xbasc(0);
xselect();
bode(syslin('c',g),1,9e4,.01);



As a rule of thumb, the C1/C2 ratio should be about 10:1; so I'm a bit off.
The transfer function of the opamp and the input/output poles could be omitted because they don't have much effect at the unity gain crossover frequency.
Co includes the 10n VCO input filter capacitor.
Loop bandwidth
3dB bandwidth is 330 Hz. Here's a normalised plot of closed loop gain:
bode(syslin('c',g/.(1)),1,5e2,.01);
Poles and zeroes
Having defined the loop filter transfer function in SCILAB, as shown above,
it's easy to check the pole/zero positions.
You can find roots of numerator and denominator polynomials as follows:
>f
f =
9.671E+13 + 4.545E+11s

2 3 4
47582205s + 15244.511s + 1.1033515s + 2.420E07s
>roots(numer(f))/2/%pi
ans =
 33.862754
>roots(denom(f))/2/%pi
ans =
! 0 !
!  757.29431 !
!  1446.8631 !
!  723431.56 !
>
As expected, we have: a zero at 34 Hz; a pole at the origin; and poles at 757 Hz, 1.45 KHz and 723 KHz.