"Connection ruined by hurtful remarks"

This happened years ago, around 2008-2009 and I still think about it now and then
I was new to the city of Toronto and my first language is French. I can understand and speak English quite well and at the time needed a bit of time to translate what others were saying. I was with friends in a group setting at a restaurant where I did not know everyone there, we were about 10 people of very mixed backgrounds and skin colors all on a big table. I was having trouble keeping up with the quick, witty banter that was going back and forth in English and was finding myself being quite silent as the group would move to another subject by the time I could translate in my head and have something to say. A new friend of a friend arrives late to the dinner and is introduced to the group as being from the country France and new to the city. After a bit I notice that she seems to be in the same predicament as me so I reach out across the table and ask her if she speaks French, she says yes and say that I do too, we switch to speaking French to each other and start connecting on the speed of translation issue and not fully being able to keep up with the group conversations. We probably talk a bit like that for 5mins and one of the people next to us says very loudly: "Could you please speak English! I can't understand what you're saying and frankly that's very rude of you two". I was shocked and taken aback and saw that the other person was too. We apologized and sat back quietly and didn't really talk to anyone for the rest of the night.