Ben Affleck's not taking questions about the alleged kiss he shared with Britney Spears ... with even the mention of the moment putting him in a seemingly sour mood.

The actor was walking down the street in Santa Monica Friday when a photog fired a question at him in Spanish ... a language in which Ben's pretty much fluent, and it sure seems like he fully understood this question.

Check out the vid ... Ben's face breaks into a big grin when he hears the pap greet him in Spanish -- possibly excited at breaking out his bilingual skills -- but then he quickly gets, "Is it true what Britney Spears said about you that you and she shared a kiss??"

Once the full question is out, Ben's expression totally changes ... smile gone and exasperatedly shaking his head as he walks swiftly to his car and drives off.

BA's clearly not happy about having this question directed at him ... all triggered by one of Britney's social media posts.

As we previously reported ... BS shared a throwback snap of herself with Ben and legendary songwriter Diane Warren, and claimed she made out with Ben on the same night the photo was taken back in 1999.

She also implied a lot more happened before they locked lips, but didn't share exactly what she meant by that with fans.

The comment raised a lot of eyebrows ... after all, Britney just released her memoir packed full of bombshell revelations -- so it's odd she seemed to have left this out. She's claiming she completely forgot about it which seems unlikely, to say the least.

She later took down the post -- an added layer of drama -- but it appears that move isn't quelling interest in what is, at this point, just a rumor she's created.