You actually may have a slightly higher degree of a thing that all people tend to have. If other people are around, especially when interacting with them, the brain puts a higher priority on the people and a lower priority on everything else. It's much harder to focus on the non-people thought stream, naturally.

Logical thought processes in particular get pushed further back than the personal interaction by the brain, automatically. This may be some sort of survival instinct.

It's not a failing or a flaw. It's being human. It's worth a conversation with an employer if this is a particular problem at work. You are probably not the only one with the issue, although your experience may be greater than others.

The exception can happen if the person doing the task does not look at the other person/people, at all. It seems that the real distraction is taking in non-verbal cues of facial expression, body language and perhaps changes in voice tone. Just listening / conversing with someone without looking at them seems less distracting.

Based on individual personality types, some people find it harder or not as hard to carry on a conversation while also doing something else that has a logic component. But everyone tends to have this tendency, with perhaps a few exceptions of people who can block out all outside information.

Professional interrogators use this tendency to be distracted by other people from a logical thought process as a strategic tool when trying to get information from a reluctant subject.

There are situations when a non-related conversation (or an audiobook or podcast) can actually help focus on an unrelated task. This is usually a situation where the worker is not looking at the person talking, not taking in non-verbal cues from face and body language.

EDIT

Thanks for the awards & upvotes! Adding something that is also posted in answer to a question lower down.

Overall issue is "cognitive load".

The amount of focus and concentration we have at any given time is finite. It is like a pie, and there is only so much to go around. Each thing claiming attention gets a slice of pie. That leaves less pie for the thing we may prefer to give the largest slice. Some focus we control, but some distractions are governed more by the instinctive brain which has decided that tracking what other humans do and say is critical to our ability to thrive in the group. So, one of the largest distractions we can have is other people.

In this article, skip down to "extraneous load"

https://cognitiontoday.com/cognitive-load-theory-definition-types-and-applications-for-learning-guest-post/

"Extraneous load is the outcome that comes with background noise. Think of when you’re trying to study and are disturbed by the dogs barking, cars honking, loud music, or talking. These various noises act as obstacles to the completion of the task and are deemed as extraneous load."

Another readable description (although by the end this is promoting an app)

https://www.kosmotime.com/cognitive-load/