WASHINGTON — A federal judge has struck down a rule setting a cap on the fees that banks can charge merchants for handling debit card purchases. He said the Federal Reserve didn't have the authority to set the limit the way it did in 2011, improperly including data that made the cap too high.

The ruling by U.S. District Court Judge Richard Leon on Wednesday handed a victory to a coalition of retail groups — which are seeking a lower cap — and a setback to banks. The retail groups had sued the Fed over its setting the cap at an average of about 24 cents per debit-card transaction.

The previously unregulated "swipe" fee averaged 44 cents. The Fed initially proposed a 12-cent cap, and the retailers had argued that the Fed buckled under pressure from bank lobbyists when it set the cap at double that level.

The Fed now must craft a new rule. The current one will remain in effect in the meantime.

"We are reviewing the judge's opinion," Fed spokeswoman Barbara Hagenbaugh said.

The cap is the first-ever limit on debit card fees. Before it took effect in October 2011, banks had negotiated such fees with merchants. A big chain like Starbucks would likely get a better rate than a local coffee shop because it handles more customers. The fees were typically based on a percentage of the purchase price.

The Fed rule was called for by the 2010 financial overhaul law, which was enacted in response to the 2008 crisis. But Leon said in his ruling that the Fed disregarded Congress's intent in passing the law by "inappropriately inflating all debit-card transaction fees by billions of dollars and failing to provide merchants with multiple unaffiliated networks for each debit-card transaction."

The retailers' lawsuit maintained that the cap is an "unreasonable interpretation" that exceeds the authority given to the Fed by the 2010 law. It also asserted that the Fed wrongly interpreted a provision of the law that requires that merchants have a choice of which bank network handles their transactions.

Leon agreed with the retailers' complaint that the Fed had deviated from the law's intent by factoring banks' expenses into the cap that the law didn't allow. He noted in the ruling that the Fed changed its earlier view that the only costs that should be considered were those involved in the authorization, clearing and settlement of a transaction. Instead, the ruling said, the Fed added costs such as losses from fraud that were outside the scope of the law.

Including costs for losses from fraud was for the Fed "a blatant act of policymaking that runs counter to Congress's will," Leon wrote.

The Fed in June 2011 formally set the cap for what banks can charge merchants at 21 cents for each debit-card transaction, plus an additional 0.05 percent of the purchase price to cover the cost of fraud protection.

Sen. Richard Durbin, D-Ill., the author of the provision of the 2010 law mandating a cap on swipe fees, called Leon's ruling a "victory for consumers and small business around the country (that) will lead to lower interchange rates for billions of debit-card transactions each year."

Durbin, who is the assistant majority leader in the Senate, had filed a legal brief supporting the retail groups' suit in the case.

The National Retail Federation, one of the parties that brought the suit, said the Fed had "grossly misapplied" the law and failed to follow Congress's call to set "reasonable" standards for fees that are in proportion to banks' costs for transactions. "As a result, small-ticket transactions, such as those imposed on convenience stores and restaurants, skyrocketed under the misapplied law," the group's senior vice president and general Counsel, Mallory Duncan, said in a statement.

Banks had lobbied hard against the cap, saying the lower fees wouldn't cover the cost of handling transactions, maintaining their networks and preventing fraud. Attempts by some big banks to compensate by charging consumers monthly fees for using debit cards sparked a nationwide furor in late 2011, leading the banks to drop their plans.

The Consumer Bankers Association, which represents large U.S. banks and regional banks, said the new ruling "will create even more chaos for consumers and small banks."

"Congress ought to save families from this uncertainty by repealing this government mandated price-fixing," Richard Hunt, the group's president and CEO, said in a statement. "We certainly hope retailers return to their free-market principles as they did when opposing the proposed government ban on big gulp sodas in New York."

In addition to the National Retail Federation, the case was brought by the National Association of Convenience Stores, the National Restaurant Association, the Food Marketing Institute, Boscov's Department Store, a chain of 40 stores based in Reading, Pa.; and Miller Oil Co. of Norfolk, Va., which operates convenience stores and gas stations.

Miller Oil has said it used to pay about 16 cents per transaction in debit swipe fees.

The Fed rule doesn't apply to credit cards, government-issued debit cards, prepaid cards or cards issued by banks and credit unions with assets under $10 billion.