Huddersfield said on their official website: “The 27-year-old left-back will put pen-to-paper on a two-year-deal, with the club having the option of a further year’s extension.”
Ruffels made 48 appearances in all competitions for Oxford last season, scoring seven goals and making six assists.
He featured in over 300 games for the U’s in total having joined the club from Coventry in 2013.
Terriers’ head coach Carlos Corberan added: “He is an offensive left-back, who has also played in the middle of the pitch, which shows a good level of versatility, which will help the team.
“He has played many minutes and a lot of games in a row over the last few years, which shows he is physically ready to adapt to the Championship so we look forward to working with him.”