Former U.S. Rep. Barney Frank and his husband, James Ready, are suing a construction contractor who they said abandoned the building of their home in Maine.

Shaun Madden Contracting walked off the job in May after completing a little more than $200,000 worth of work, Frank and James Ready said in their lawsuit filed on Wednesday, the Bangor Daily News reported.

The couple paid $350,000 upfront for the construction of the home in Wells, Maine, the lawsuit states. They say Madden has returned $50,000, but they are asking a judge to order him to repay them more than $120,000, in part because he sited the home too close to a road, the newspaper reported.

Madden did not immediately return the newspaper's request for comment.

Frank, 80, served 16 terms in Congress as a representative of Massachusetts and was the first member to come out as gay in 1987. He and Ready married in 2012.