WASHINGTON, June 27 (Reuters) - The U.S. Supreme Court ruled on Thursday in favor of a challenge to the Securities and Exchange Commission's in-house enforcement of investor protection laws in certain proceedings, dealing a setback to the agency amid the heavy scrutiny by the justices of federal regulatory power.

The 6-3 ruling, a setback for President Joe Biden's administration, upheld a lower court's decision siding with George Jarkesy, a Texas-based hedge fund manager who contested the legality of the SEC's actions against him after the agency determined he had committed securities fraud.

The case involved Texas-based hedge fund manager George Jarkesy, who the SEC fined and barred from the industry after determining he had committed securities fraud. He responded with a lawsuit challenging the legality of the SEC's system.

Jarkesy was supported in the case by numerous conservative and business groups, which long have complained about the regulatory reach of the federal "administrative state" in areas such as energy, the environment, climate policy, workplace safety and financial regulation.

Biden's administration had appealed a 2022 decision against the SEC by the New Orleans-based 5th U.S. Circuit Court of Appeals.

The SEC in recent years has faced a series of legal attacks even as the Supreme Court's conservatives show skepticism toward expansive federal regulatory power. The court in 2018 faulted the way the SEC selected its in-house judges. In 2023 rulings in cases involving the SEC and Federal Trade Commission, the court made it easier for targets of agency actions to mount challenges in federal court. The court has curbed the power of other agencies in recent years including the Environmental Protection Agency.

SEC critics have said the agency has an unfair advantage litigating cases before its home-turf judges rather than before a jury in federal court. The SEC, which enforces various U.S. laws that protect investors, pursued 270 new in-house proceedings in the fiscal year that ended on Sept. 30, compared to 231 in federal court.

However, since the Supreme Court's 2018 ruling, most of the SEC's administrative proceedings are now handled by the commission itself, with very few - as of March, just two - proceeding before an administrative law judge.

The SEC in 2011 began investigating Jarkesy, who founded two hedge funds with his Houston-based investment advisory firm, Patriot28 LLC. The funds had about 120 investors and roughly $24 million in assets under management.

An SEC administrative judge found that Jarkesy and his firm violated the Securities Act of 1933 and other U.S. laws including by misrepresenting the identity of the auditor of the funds and value of the holdings. The SEC then ordered them to pay a $300,000 civil penalty and Patriot28 to disgorge nearly $685,000 in ill-gotten gains.

The 5th Court threw out the SEC's decision. In addition to its conclusion about the right to a jury trial, the 5th Circuit found that Congress gave the SEC too much power to choose whether to bring cases in-house, and that job protections for its administrative judges make them too difficult to remove, infringing on presidential powers under the Constitution.

During November oral arguments in the case, conservative justices expressed concern that SEC administrative proceedings are conducted for certain charges, such as fraud, without a jury, when similar cases alleging fraud in federal court would have one.

Liberal justices said that existing Supreme Court precedent allowed Congress to leave regulatory enforcement to administrative tribunals without juries, and that Congress gave the SEC more power after the Great Depression of the 1930s and later financial crises to combat investor fraud.

The Supreme Court has restrained the power of federal agencies in major rulings in recent years. It is deciding multiple cases during its current term on the authority of agencies. The justices on May 16 upheld the Consumer Financial Protection Bureau's funding mechanism in a challenge brought by the payday loan industry.

The court is expected in the coming days to rule in other cases involving the Environmental Protection Agency and National Marine Fisheries Service. (Reporting by Andrew Chung; Editing by Will Dunham)