High-fidelity preparation of target spin states is fundamental to achieving high-sensitivity detection in nuclear magnetic resonance (NMR) spectroscopy. The traditional gradient ascent pulse engineering (GRAPE) algorithm is sensitive to initial guesses and prone to trapping in local extrema. Conversely, pure reinforcement learning (RL) often suffers from policy divergence due to sparse rewards when applied to complex quantum systems. To address these limitations, this paper proposes a hybrid optimization framework that cascades RL with GRAPE. This framework leverages the model-free nature of RL for global evolutionary trajectory search, followed by local continuous gradient fine-tuning using GRAPE. Liquid-state NMR experiments on a two-spin ¹H system of citric acid demonstrate that this method stably generates robust pulses with a theoretical fidelity exceeding 0.99, despite chemical shift drifts induced by pH variations. Furthermore, the significant compression of pulse duration effectively mitigates coherence loss caused by relaxation. This control strategy, which balances evolutionary time and parameter robustness, provides a reliable methodological foundation for the detection of complex metabolites in clinical spectroscopy.