After several women, including actress Ally Sheedy, called out James Franco during the Golden Globes on Jan. 7, the Best Actor in a Motion Picture, Musical or Comedy winner responded to the sexual harassment claims in a tense interview with Stephen Colbert.

“Things that I heard were on Twitter are not accurate, but I completely support people coming out and being able to have a voice,” he said, adding, “If there is restitution to be made I will make it. If I have done something wrong, I will fix it. I have to.”

He then addressed Ally’s tweets directly, which were deleted shortly after she posted them. She originally wrote, “Why is a man hosting? Why is James Franco allowed in? Said too much. Nite love ya.” She later said, “Ok wait. Bye. Christian Slater and James Franco at a table on @goldenglobes #MeToo,” before adding, “James Franco just won. Please never ever ask me why I left the film/tv business.” The 39-year-old had previously directed her in a 2014 off-Broadway production of The Long Shrift.

“I have no idea what I did to Ally Sheedy … I had nothing but a great time with her,” James said on The Late Show. “I have total respect for her. I have no idea why she was upset. She took the tweet down … I can’t speak for her.”

One person who didn’t delete her tweet is actress Violet Paley. She wrote on Sunday, “Cute #TIMESUP pin James Franco. Remember the time you pushed my head down in a car towards your exposed penis & that other time you told my friend to come to your hotel when she was 17? After you had already been caught doing that to a different 17 year old?”

Two days later, she added, “A couple weeks ago, James offered me & a few other girls an overdue, annoyed, convenient phone ‘apology’. I don’t accept, but maybe some other people’s lives would be made easier if he donated all of his earnings from ‘The Disaster Artist’ to @RAINN01.” RAINN is the nation’s largest anti-sexual violence organization.

Filmmaker and actress Sarah Tither-Kaplan also made her feelings about the Oscar nominee known. “Hey James Franco, nice #timesup pin at the #GoldenGlobes, remember a few weeks ago when you told me the full nudity you had me do in two of your movies for $100/day wasn’t exploitative because I signed a contract to do it? Times up on that!” she tweeted. Time’s up, indeed.