Background: Jiao-Tai-Wan (JTW) is widely used for insomnia in traditional Chinese medicine. It has been found that berberine in JTW has a potential anti-insomnia effect, but the anti-insomnia effect of volatile oil, which is another major component in JTW, remains unclear. Objective: To comprehensively analyze the anti-insomnia effect of volatile oil through chemical analysis, pharmacological research and network pharmacology methods. Methods: Gas chromatography-mass spectrometry (GC–MS) analysis and network pharmacology platform were used to elucidate the material basis, effect and mechanism of JTW in the treatment of insomnia. A rat model of insomnia induced by p-Chlorophenylalanine (PCPA) was used to verify its anti-insomnia effect. The levels of Melatonin (MT) and Melatonin receptor type 1b (MTNR1B) in the brain and serum of rats were detected by ELISA, Western blot and Real-time PCR methods. Results: Eighteen volatile oil ingredients of JTW were identified by GC–MS, containing six new components which were found in JTW for the first time. Cinnamaldehyde, Behenic alcohol, Tetradecanal, and Gleenol can promote sleep by targeting MTNR1B according to the predicted results of network-pharmacology. Compared with the model group, JTW can increase the level of MTNR1B in rats’ prefrontal cortex and brain stem, while reducing the level of MT in rats’ brain stem. Conclusion: The volatile oil components, such as Cinnamaldehyde, Tetradecanal, and Gleenol, might also be playing a critical role in the anti-insomnia effect of JTW by target MTNR1B.