The FCA said McDermott, who has been at the regulator and its predecessor, the Financial Services Authority, for 15 years, pulled out of the race in December but would remain acting CEO until her permanent replacement was in post.
"Going through the recruitment process has made me reflect on what I want to do with the rest of my career. As a result I have decided that this is not the right job for me at this stage of my career," McDermott said in a statement.
"This was a decision taken after many months of careful thought and was not one that I took lightly."
McDermott was seen as the leading internal candidate for the job of replacing Martin Wheatley, the former hardline CEO who was ousted by British finance minister George Osborne last year, when he also called for a "new settlement" with banks.
Some members of parliament are concerned the government is putting pressure on the FCA to take a softer line on lenders and will grill the watchdog later this month.
In a previous role as chief enforcer, McDermott was instrumental in fining several banks for trying to rig the Libor interest rate benchmark and attempting to manipulate the foreign exchange market.
Trained as a lawyer, McDermott then moved to the FCA's supervisory arm, seen as giving her broader experience for taking on the top job.
Osborne is expected to name the FCA's new CEO in coming weeks, with regulators from outside Britain among the people mentioned in media reports.
(Reporting by Huw Jones; Editing by Mark Potter)
