Hi,
Thank you for your comment
We have added the -40 calibration feature to mitigate the imperfect region of -40dB due to the internal crosstalk of Pluto. The -40 calibration needs adding a good 40dB attenuator in the loopback after calibration of 0dB. SATSAGEN fire an error when read amplitude above the region of -40. Let aside for the moment the -40 calibration for your purpose of measuring return loss with coupler.
In my opinion, the problems you noticed should be due to the directional coupler. I suggest you find the best coupler for the characteristic of the DUT, in your case, the antenna. You should choose the coupler with the range frequency nearest your antenna, also this corresponds to the highest directivity value of the coupler.
If the directivity of the coupler is not sufficiently related to the typical return loss of the antenna, the results are errored.
The coupler directivity should be almost 10dB higher than the typical return loss of the antenna.
In my opinion, you should consider a directional coupler with about 23dB or higher of directivity.
In the picture of your post, the setup of Pluto and coupler is correct. With this setup (open), you can do a 0dB calibration, that it is sufficient to measure the return loss next.
The RX gain and TX power levels should be defined in the range where you use more of the dynamic of Pluto and less internal crosstalk.
I suggest you start with 30 of RX gain and -30 of TX pwr.
I suppose the antenna should be connected directly to the coupler because cables and connectors could affect the readings, but the antenna should be possibly free from interaction with other near objects. I know, it's not easy to do! Moreover, the Pluto without a metal case can be affected by antenna emissions!
In your picture, I see dipole antennas. The impedance matching of these antennas is it well done? If we ever manage to implement the VNA on SATSAGEN maybe we will measure this too!
73, Alberto