You’d think that with their career kicking off at just nine months old, Mary-Kate and Ashley Olsen‘s parents had high hopes for their talented little tykes. But in their early years, nobody was more mystified by their success than father David “Dave” Olsen. “They utilize their personalities pretty well,” he told Washington Post back in 1991. “They’re just kind of funny. They were almost like chimpanzees when they were little. You just dress them up and they’re fun. I guess they’re just photogenic.”

“Photogenic” seems like an underwhelming label for one’s children, yet it’s also not an inaccurate observation. Nobody could deny that the cameras loved the Olsen twins. And though Papa Dave’s initial observations have more shock-factor in the comparisons — “chimpanzees” is definitely jarring —there’s something ironic about his initial doubts. For years, the sisters denied being ruled by stage parents. But it is interesting that Dave took the reigns on the careers just around the time the Olsens started showing franchise potential.

baby mary-kate and ashley

The SparkNotes version of how the Olsens got their Full House gig is that their mom, Jarnette “Jarnie” Olsen was pressured to volunteer the twins as tribute. Dave, on his end, remained ambivalent. “I didn’t even want to be bothered with it,” he said. “I said, ‘Honey, if you want to do this, fine.'” According to Dave, the parents wanted to pull the twins out after the first season, but the girls allegedly loved playing Michelle Tanner so much. So they co-managed their career…approximately until they decided to divorce.

In 1996, it was reported that Dave and Jarnie were divorcing, and that Dave was living the old cliché and marrying his secretary. “The reality is that the transition has been very smooth,” Dave told Entertainment Weekly. “Everyone sees eye to eye. The girls are fine.” But the tabloids reported otherwise, declaring the dramatic divorce was dividing the family with the twins at the forefront. Star even reported on Dave’s wedding day to wife McKenzie, Mary-Kate showed up as Maid of Honor while Ashley refused to attend, choosing to stay home with her sobbing mother.

olsen twins young

Whatever the case, it was Dave who was sitting in with the twins and managing their career throughout the next decade. He sat as a chairperson of their multi-million dollar company Dualstar. He told the LA Times he cut back on his work schedule to accommodate the girls’ shooting schedules. In 2001, he set the rules for Ashley’s Valentine’s Day date in front of the New York Times. The publication added as an aside, “their mother, Jarnette, does not speak to the press or participate in Mary-Kate and Ashley’s lives as public figures.” Hmmm.

Granted, all this doesn’t necessarily paint Dave as a cruel svengali. He did appear, however, to be pulling the strings as his daughters built a brand. “A child actor cannot be anything without a stage parent and it’s where the stage parent takes it,” talent agent Lorrie Tierney observed in 2013. “They were just cute early on, but I think their dad knew what he was doing and knew how to promote them.”

mary-kate and ashley family

The one thing that sticks out as suspicious is how in 2010, it was reported that Dave Olsen filed for bankruptcy. Mind you, Mary-Kate and Ashley seized control of their empire in 2004 when they turned 18, so in a legal sense, they no longer needed their father. But with said empire raking in approximately more than 100 million dollars and their father and stepfamily slated to lose their home, why wouldn’t the girls step in? Did they have residual resentment for their dad for pressing their careers?

Maybe not. Ultimately, there’s been a lot of haziness over the years when it comes to the twins’ current relationship with their father. Their sister, Elizabeth Olsen, for what it’s worth, seems to have a decent relationship with him. He’s not publically in their lives, but clearly, neither is Jarnie. Maybe the 2010 bankruptcy file is an isolated incident and maybe the Olsens twins helped their father out after all. There’s a lot that happens when cameras aren’t rolling.

somber olsen twins

Still. We can’t help but think back to what Mary-Kate told Marie-Claire that very same year. “I would never wish my upbringing on anyone,” she said, adding the girls were nothing more than “little monkey performers.”