Abstract: This paper mainly focuses on the beamforming training problem in cooperative non-orthogonal multiple access (NOMA) scenarios in millimeter-wave communication, extending the work from single-cell NOMA to multi-cell NOMA scenarios. To maximize system throughput while considering user locations and channel information, the beam configuration problem at the base station is modeled as a Markov cooperative-competitive game problem. And then the problem is solved by exploiting multi-agent deep deterministic policy gradient (MADDPG) based reinforcement learning algorithm. A multi-agent reinforcement learning-based beamforming training algorithm for cooperative NOMA in multi-cell scenarios is designed to effectively allocate resources such as beams and power in multi-base station systems, thereby enhancing system throughput. Numerical simulations demonstrate that the proposed MADDPG algorithm achieves better system throughput and user coverage.