The lawsuit, filed in a state court in Illinois, said the exchanges had discovered last year that a prominent firm had been mismarking orders as originating from public customers when they were not.

This resulted in an unknown amount of improper fees being assessed on market makers such as Citadel that the exchanges have refused to reimburse, according to the plaintiffs, which also include Group One Trading LP, Ronin Capital LLC and Susquehanna International Group, LLP.

The firm that mismarked the trades is not named in the complaint. But the almost $6.39 million in fines assessed against the firm by the exchanges and listed in the complaint matches an amount paid last year to the five by Goldman Sachs Group Inc (>> Goldman Sachs Group, Inc.) as part of a settlement. A Goldman Sachs spokesman declined to comment.

The other exchanges named as defendants are International Securities Exchange, The Nasdaq OMX Group Inc's (>> NASDAQ OMX Group, Inc.) Nasdaq OMX PHLX, and NYSE Euronext-owned (>> NYSE Euronext) NYSE Arca Inc and NYSE MKT.

Representatives for the exchanges declined comment. Stephen Bedell, a lawyer for the investment firms at Foley & Lardner, did not respond to a request for comment.

The lawsuit centers on "payment for order flow" or "marketing fee" programs that the exchanges operated from at least January 2004 to June 2011.

Under the programs, the exchanges assessed fees of $0.10 to $1.00 per options contract on certain trades executed by the market makers. CBOE alone collected $637 million in these fees from 2004 to 2010, the lawsuit said.

Most of the orders for which fees were assessed were orders by public customers. Fees were not assessed on orders submitted by market makers as part of their market-making business, as well as most orders submitted for members' broker-dealer clients.

But by fall 2012, the exchanges became aware that the unnamed prominent firm had for seven years been mismarking its options orders as coming from its customers when they were not. That firm has paid the five exchanges nearly $6.39 million in penalties, the lawsuit said.

The conduct at issue and the penalties paid to the five exchanges match those indicated in consent orders and stipulations entered into between Goldman Sachs and the exchanges, documents show.

The lawsuit contends that market makers were wrongfully charged fees as a result of those mismarked orders.

The exchanges have countered that under their rules they are not responsible for improper fees they charge and collect, the lawsuit said. They have cited rules that the lawsuit said were designed to protect them against losses incurred by members as a result of their use of the exchanges' systems and procedures.

"The Exchanges' reliance on these rules is baseless," the lawsuit said.

The lawsuit seeks, among other things, a finding that the exchanges be required to reimburse the fees and an accounting of how much the market makers have been overcharged.

The case is Citadel Securities LLC, et al, v. Chicago Board Options Exchange Inc, et al, Circuit Court of Cook County, Illinois, No. 13CH13246.

(Reporting by Nate Raymond in New York; Editing by Edmund Klamann)

By Nate Raymond