Steven Sinofsky, the head of Microsoft's Windows division, has left the company with immediate effect.
His departure comes just weeks after Microsoft launched Windows 8, the latest edition of its flagship product, seen as key to the firm's future.
Microsoft did not give any reason for Mr Sinofsky's departure.
However, one industry watcher suggested there had been talk of an internal "war" between Mr Sinofsky and chief executive Steve Ballmer.
Markets took the news badly, with Microsoft's shares ending the trading day on Tuesday 4% lower.
The company said Julie Larson-Green would be promoted to lead all Windows software and hardware engineering.
"This is shocking news. This is very surprising," said Brendan Barnicle, an analyst at Pacific Crest Securities.
He added that many observers saw Mr Sinofsky as a potential successor to Steve Ballmer, Microsoft's chief executive.
In a letter to all employees, published by Forbes, Mr Sinofsky set out to quell the rumours about his departure.
"Some might notice a bit of chatter speculating about this decision or timing. I can assure you that none could be true as this was a personal and private choice that in no way reflects any speculation or theories one might read - about me, opportunity, the company or its leadership," he said.
"It is impossible to count the blessings I have received over my years at Microsoft," he added.
Michael Gartenberg, an analyst with research firm Gartner, said that Mr Sinofsky's were big boots to fill.
"The reasons why he left don't matter all that much but the big question is about how Microsoft fills the void," he said.
"He did a lot more than head up a division, he had a unified vision of Microsoft as an ecosystem, tying together the PC, phone, tablet, Xbox and online services. The ramifications of his departure are yet to be felt."
He added that the immediacy of his departure was "strange".
"You don't often see that at that level," he said.
Mr Sinofksy's departure is the latest change at the top of some of the world's biggest technology companies.
Last month, Apple announced that Scott Forstall, head of its iOS software, and of John Browett, head of retail, would be leaving the firm.
The announcement followed problems with Apple's new mapping software and disappointing quarterly results.
Meanwhile, Yahoo - which has been trying to regain some of its lost market share - also hired a new chief operating officer in October.
In July the internet company appointed its third chief executive in a year.
Microsoft's Mr Ballmer said the changes in leadership were aimed at ensuring the firm continued to be a dominant player in the sector.
"The products and services we have delivered to the market in the past few months mark the launch of a new era at Microsoft," Mr Ballmer said.
"To continue this success it is imperative that we continue to drive alignment across all Microsoft teams, and have more integrated and rapid development cycles for our offerings."