This paper presents a measurement recipe to characterize the out-of-band nonlinear behavior of RF devices when harmonic sampling is used as a digitizing technique. A major challenge to consider when using harmonic sampling is the overlapping of the aliased spectral bins of the digitized waveform. This challenge is more pronounced when using modulated excitation signals. In addition, the excitation signal used should mimic the real behavior of signals used in today's wireless systems. Hence, this paper provides the reader with a tool to select the proper excitation signal, sampling frequency and record length in order to achieve an accurate characterization of the nonlinear behavior of RF devices.