The two Koreas have failed to agreed on security guarantees for test runs of inter-Korean railways and prevention of naval clashes in disputed waters.
A South Korean Defense Ministry official said the two sides failed to reach an agreement in the fourth inter-Korean military talks.
The two sides also did not set a date for another round of military talks, with the North also rejecting an offer for a meeting of defense ministers.
North Korea would not back down from its demand for discussing the expansion of the inter-Korean maritime border to further south from the coast.
South Korea wanted to discuss security guarantees for test runs on inter-Korean railways and a joint fishing area, asking that a meeting of defense ministers discuss the sea border.
Former South Korean President Kim Dae-jung wants to travel by train to Pyongyang next month for a meeting with North Korean leader Kim Jong-il.
The North Korean military, however, is known to oppose an agreement on train operations for fear of exposure of the country to the outside world.