Divorce finally taking toll on my four year old son. Need some advice.

My ex wife and I got a divorce when our son was two. He handled the separation fairly easily at a younger age but lately he's starting to get more emotional when we swap and lately has flat out cried and has not wanted to come home with me. She's a stay at home mom and I work full time so it makes sense he'd be more comfortable with her during the week based on my work schedule. I've compromised and let him stay with his mother more lately (It's 50/50 custody) just to make sure he's in the place he feels the most comfortable.Yesterday I went to pick him up from her and he had another breakdown. However, this time he said "I want you and mommy to be with me all the time. That would make me the happiest." He said this crying and it was the most heart wrenching thing I've ever had to endure.So my question is: Should I start doing activities with him and my ex wife? Lunch, a movie, anything. Her and I both have significant others and I prefer not to hang out with her, but if that's what my son needs I'll do it. I just don't want him to get confused and get used to the fact we're all three back together again.Just curious if that would make a difference getting that quality time with the three of us until he's old enough to move on to the next phase of his perspective on this situation. via /r/Parenting http://ift.tt/2hp7bBj