The two were seen leaving Los Angeles restaurant Le Petit Four on Feb. 3. The two were linked as early as 2010, following Houston's 2007 divorce from Bobby Brown.
Ray J is notorious for his relationship with Kardashian, which led to his own reality TV romance show, "For the Love of Ray J." He is also the brother of singer Brandy, who played Cinderella to Houston's fairy godmother in the television remake of the classic fairytale.
Ian Drew, senior editor at Us Weekly, told ABCNews.com that Ray J and Houston dated on and off for several years.
"He was her big eyebrow-raising relationship after Bobby [Brown]," Drew said.
Ray J appeared on "The Insider" on Friday to promote his new advice book "Death of the Cheating Man." In the interview, he was repeatedly pressed for information on his relationship with Houston.
"You know, they blow this thing out of proportion. I mean, that's my friend. I've been knowing her for years and years and years, and she's a friend of the family," he said on the show. "I have a lot of love for her and I have a lot of respect for her."
Ray J remained coy when pressed on whether his relationship with Houston was more that platonic.
"Our friendship is something that people might not understand, maybe because of the age difference, but we've always just been friends," Ray J said. "We hug, we laugh, we go out to dinner.… We talk about, you know, different things in our lives and she's a cool person."
When asked directly if his relationship with Houston was physical, he again deflected.
"We've always kept our relationship as private as we could, but when we go out, she's such an icon, it reaches the masses," Ray J said. "But, for the most part, we've always been friends and we've kept everything hush-hush and just quiet and we really don't focus on that."
Ray J's agents did not immediately respond to ABC News' requests for comment.
Ray J has not yet commented on Houston's death and his Twitter account has been silent since Saturday morning. His last tweet was a happy birthday message for his sister, Brandy, and the tweet before that included three photos and the message, "Im with Clive Davis helping with the party 2mrw nite."
In one photo, Ray J and Davis were sitting side-by-side in a ballroom, presumably at the Beverly Hilton, where the party took place and where Houston died. The two were facing the stage and surrounded by stacks of chairs and tables that had yet to be set.
The second photo is of musicians rehearsing on stage and the third is of Ray J and Davis smiling and posing with a woman.
Houston had seen Ray J as recently as Thursday night, when both attended Grammy nominee Kelly Price's party, "Kelly Price and Friends Unplugged: For the Love of R&B Grammy Party," at the Los Angeles nightclub Tru Hollywood.
Houston was there with her 19-year-old daughter, Bobbi Kristina Brown, and performed a duet with Price in the early hours of Friday morning, in what would become her last performance.
At the party, Houston reportedly clashed with singer and former "X Factor" finalist Stacy Francis, who had credited Houston as her inspiration and had been pulled onstage to sing with her in 1999 during a concert, according to The Hollywood Reporter.
Houston and Francis were re-introduced at the party and all seemed fine until Ray J entered the conversation.
"Whitney just got belligerent," a witness told The Hollywood Reporter. "Ray was trying to defend Stacy, telling Whitney, 'Stacy's family!', but Whitney was feeling crowded out and hands were raised."
The witness said that no punches were ultimately thrown, but that it was a tense exchange.
Photos taken later that night showed Houston disheveled, agitated and bloated as she left, and she reportedly had to be escorted from the club with blood dripping down her leg and scratches on her wrist.
Friends said she was simply sweating from the performance.
"There was nothing wrong. She was singing, she was dancing, she was laughing, she was playing," Price said. "I'm going to remember her just like that because that's the Whitney I always had the pleasure of being around."
On Saturday afternoon, Houston's lifeless body was discovered in her bathroom at the Beverly Hilton, where she was planning to attend friend and mentor Clive Davis' annual pre-Grammy party. A cause of death has not yet been determined.
