The missile, launched early on Tuesday Korean time, flew over Hokkaido island before crashing into the sea.
The UN Security Council is expected to hold an emergency meeting in response.
North Korea has conducted a flurry of missile tests recently, but this is the first time it has fired what is thought to be a ballistic weapon over Japan.
On the two previous occasions its rockets crossed Japan - in 1998 and 2009 - North Korea said they were for satellite launch vehicles, not weapons.
Tokyo says this latest launch appears to be the first of a missile powerful enough to potentially carry a nuclear warhead.
The South Korean military said the missile was fired eastward just before 06:00 local time (21:00 GMT) from near the North's capital, Pyongyang - which is rare.
No effort was made by Japan to shoot down the missile but it issued a safety warning telling citizens in Hokkaido to take shelter in "a sturdy building or basement".
US and Japanese forces have just finished a joint drill in Hokkaido while another annual military exercise involving tens of thousands of South Korean and US soldiers is still under way in South Korea.
The North sees these regular military drills involving the US as highly provocative, perceiving them as a rehearsal for an invasion.
South Korean President Moon Jae-in ordered a show of "overwhelming" force in response to the launch. Four South Korean jets staged a live bombing exercise on Tuesday.
Early analysis of the launch suggests the missile:
- flew a distance of more than 2,700km (1,678 miles)
- was likely to have been a Hwasong-12, a newly developed intermediate range weapon
- reached a maximum altitude of about 550km (342 miles), lower than most previous North Korean tests
- fell into the North Pacific Ocean 1,180km off the Japanese coast after breaking into three pieces