Casey Affleck remembers the birthdays he had as a child with his brother Ben Affleck

Casey Affleck and his older brother Ben Affleck had to combine their birthday spotlights as kids.

While Casey, who turns 49 on August 12, Entertainment tonight at the New York premiere of his new film The instigators On July 31, the actor pointed out that his family celebrated his and Ben’s birthdays together during their childhood, as Ben turns 52 on August 15.

“We had to do this growing up,” Casey recalled. “We both had a birthday party.”

The Oscar winner added that he and Ben haven’t necessarily kept up that tradition in adulthood. “When we had kids, my birthday didn’t become as important, you know? And it was always about them,” Casey said. “But now that they’re moving out of the house, maybe I should reclaim that. I’m throwing myself a birthday party!”

Casey has two sons, Indiana, 20, and Atticus, 16, with ex-wife Summer Phoenix, whom he was married to from 2006 until their separation in 2016. He is currently dating actress Caylee Cowan, who also stars with him and the Afflecks’ longtime friend Matt Damon in the new heist film.

Never miss a story — sign up for PEOPLE’s free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from celebrity news to compelling human interest stories.

“I still have two years at home with my youngest son, so I’m trying to spend that time with him,” Casey said ETwhen asked how he would like to spend his time before he turns 50 in 2025.

“Professionally, I really just want to keep working with people I love and some people I haven’t had the chance to work with yet. But I want to focus less on climbing the corporate ladder and achieving things. I want to focus more on my personal work and my own challenges that I want to work on and work with people I like and have fun doing it.”

Casey and Damon filmed The instigators — which also stars Hong Chau, Michael Stuhlbarg, Paul Walter Hauser, Ving Rhames, Alfred Molina, Toby Jones, Jack Harlow and Ron Perlman – in her hometown, which, according to Casey, led to some special moments during filming.

“I’m not going to lie, I’ve been doing this for 38 years now and I’ve had a lot of ups and downs and I’ve worked hard, but still, when I was on set, I felt really happy, you know?” he said ET. “I’m in my hometown, working with people I love, working with new people I’ve admired for a long time, doing what I love to do and making a living.”

The instigators premieres on Apple TV+ on August 9th.