Kanye West is saying sayonara to Yeezy's longtime Chief of Staff in what appears to be a mass exodus at the company ... and in this case, it's all about porn tainting the spirit.

Milo Yiannopoulos -- who's been the point man on all things Yeezy for nearly two years now --  confirms his split from Ye's enterprise to TMZ ... telling us, "I wish Ye every success in the future. I have some concerns about his new team, and hope he proceeds with caution."

It's unclear who exactly might be a part of Kanye's new team -- things are always kinda in flux there -- but clearly, they're not sitting well with Milo, and BTW ... that's not the only thing troubling him.

Milo provided TMZ a copy of his resignation letter, which cites major issues over the formation of "Yeezy Porn." In it, Milo spells it out, clear as day ... saying he just can't be involved in producing/distributing pornographic material for moral and religious reasons, and because it can be, "an imminent danger to my life as a recovering addict and an unacceptable risk to my spiritual and physical health as a former homosexual."

His words.

Still, Ye's now-former CoS isn't totally slamming the door ... saying if Kanye ditches the on-camera sex venture, "it would be an honor to serve you again."

TMZ broke the story ... Kanye's been in talks with Stormy Daniels' ex, Mike Moz, about creating a new porn biz ... which Moz said would be unlike anything the industry's ever seen. If it does happen, Milo won't be around to shepherd it.

The shakeup at Yeezy doesn't end with Milo ... multiple sources with direct knowledge tell TMZ key players among the company's leadership have exited in the past few weeks -- although it's unclear if their resignations are over the porn issue, or something else.