Challenges for the holder of the sensitive post include Iran's expanding uranium enrichment programme, blocked investigations into alleged military nuclear activity in Iran and Syria, and North Korea's atomic tests.
Amano, supported largely by industrialised nations, defeated South Africa's Abdul Samad Minty in a sixth round of balloting after five inconclusive votes. It was his second try for the top job at the International Atomic Energy Agency (IAEA) following an election stalemate in March.
He succeeds Nobel Peace laureate Mohamed ElBaradei, who retires in November after 12 turbulent years.
Amano, 62, managed to win the required 2/3rds majority of the governing board members who expressed a preference, with 23 votes and one crucial abstention in the 35-nation meeting.
"Everyone looked a little glum when the result was announced," said one European diplomat in the closed-door session. "There was total silence," said another diplomat.
Diplomats said some European members voted for Amano only reluctantly, disappointed that a more inclusive, compromise candidate did not enter the field.
Poorer nations regarded Amano as a tool of major powers pushing the IAEA to get tougher on proliferators, while using such concerns as an excuse to hinder the sharing of nuclear technology for development purposes.
Diplomats said Russia told other board members privately it would be "unacceptable" if Amano were elected by the bare minimum margin. This would harden resentment that could undermine his authority, it said.
Amano said he would balance IAEA resources between anti-proliferation inspections and fostering the safe use of nuclear energy for modernisation and medicine.
"As a national from Japan, I will do my utmost to prevent the spread of nuclear weapons. To do that, the solidarity of all the member states, countries of the north, from the south, from east and west, is absolutely necessary," he told reporters.
Japan is the only country to have been attacked with nuclear weapons, by the United States, at the end of World War Two.
Woos Sceptical Developing Nations
"I will do my utmost to ... ensure sustainable development through the peaceful use of nuclear energy," he said in a nod to developing nations.
Iran has exploited rich-poor tensions over IAEA priorities by arguing Western pressure on it to scrap its nuclear energy campaign is aimed at stunting its development. But Iran's curbs on IAEA inspections have raised suspicions of a bomb agenda.
The soft-spoken Amano has specialised in multilateral nuclear disarmament and non-proliferation posts and negotiations over 36 years in Japan's foreign service.
His backers hope he will take a tougher line on cases of non-compliance with nuclear safeguards than ElBaradei who riled the United States and close allies by advocating negotiation rather than sanctions against Tehran.
They also believe he will "depoliticise" the IAEA leadership after the outspoken ElBaradei. One nuclear analyst said Amano's track record should make him a worthy IAEA director.
"Given the nuclear challenges facing the world and the divisions that have deepened between the 'haves and have nots', it's very important for the IAEA to have a leader who can bridge the differences and bolster its reputation for technical competence and political independence," said Mark Fitzpatrick of the International Institute for Strategic Studies in London.
The governors were expected to seal the election result by acclamation on Friday. This would be rubber-stamped by all 146 IAEA member states at their annual conference in September.