Background Sleep is essential for maintaining human health, and insomnia is a widespread problem. Traditional Chinese medicine (TCM) has been used for centuries to treat sleep disorders, with fewer reported side effects compared to conventional treatments. Objective This study seeks to investigate the sleep-promoting effects of the GSZ formula, which comprises γ-aminobutyric acid (GABA), Schisandrae Chinensis Fructus (Wuweizi in Chinese), and Ziziphi Spinosae Semen (Suanzaoren in Chinese). In addition, this study aims to explore the active ingredients and potential mechanisms underlying the sleep-enhancing effects of the formula. Methods The impact of GSZ on sleep was evaluated using two models, the complete sleep model and the subthreshold sleep model. Mice were randomly divided into five groups and orally administered GSZ solution (0.33 g/kg/day or 0.99 g/kg/day), positive drug diazepam (2.50 mg/kg) or a control solution for 30 days. Hypnosis model was established in mice using pentobarbital sodium. Sleep duration and incidence were measured by recording when the righting reflex of mice disappeared for more than 1 min. GABA and dopamine (DA) levels in mouse brain tissue were measured using ELISA kits. The ingredients of the GSZ formula were identified using mass spectrometry, and the targets of these ingredients and disease-related genes were retrieved from public databases. A network medicine approach was used to calculate the shortest path between ingredient targets and disease-related proteins. The expression levels of potential proteins, such as Akt, p-Akt, GSK-3β, and p-GSK-3β, were analyzed using Western blotting based on the predicted results. Results GSZ significantly prolonged sleep duration and enhanced the sleep rate in mice (P < 0.05). Furthermore, it elevated GABA levels and reduced DA levels in the mouse brain (P < 0.05). Network medicine analysis suggested that GABA, stearic acid, genistin, and coumestrol may be the most crucial active ingredients for sleep improvement. Western blotting analysis demonstrated that GSZ modulated the protein expression levels of p-Akt/Akt and p-GSK-3β/GSK-3β (P < 0.05). Conclusion Our study demonstrated that the GSZ formula could improve sleep, with key ingredients likely being GABA, stearic acid, genistin, and coumestrol. The mechanism might involve the regulation of the Akt/GSK-3β pathway, as revealed by the network medicine analysis and experimental validation. Our current new findings shed light on the potential mechanisms underlying the sleep-enhancing effects of the GSZ formula, which could provide experimental evidence to develop innovative treatments for insomnia.