Manager Carlo Ancelotti asked for play to be suspended on three separate occasions but his demands fell on deaf ears.
The Italian later said his players could walk off the pitch next time and be prepared to take whatever punishment comes their way for taking a stand against the abhorrent behaviour.
Koulibaly, who was sent off after a second yellow card for sarcastic applause, posted a tweet after the game.
He wrote: ‘I’m sorry for the defeat and above all to have left my brothers! But I’m proud of the color of my skin. To be a French, Senegalese, Neapolitan: man.’
And after full time, Ancelotti weighed in with his verdict:
‘I asked the referee for information on the red cards and nothing else.
‘Those incidents rather decided the match, as we had done very well in the second half and were pushing hard.
‘We even had the chance to win it when down to 10 men, but tempers were a little raised at the end,’ the coach told Sky Sport Italia.
‘There was a strange atmosphere, as we asked three times for play to be suspended, they had announcements with the speaker three times.
‘Koulibaly was certainly irritable. Usually, he is very calm and professional, but he was subjected to monkey noises throughout the game.
‘We asked three times for some action to be taken, but the match continued. We keep being told play can be halted, but when? After four or five announcements?
‘Maybe we have to take matters into our own hands next time and stop play ourselves. They’ll probably make us lose the game if we walk off, but we are prepared to do it. It’s not good for Italian football, seeing this,’ he lamented.
The ill-tempered game was punctuated by violence among supporters outside the San Siro.
Four Napoli fans were stabbed with one being taken to hospital to treat an abdominal wound and the others treated on site for minor injuries.
During the clashes, one Inter Milan fan is reported to have been hit by a passing car and is believed to be in serious condition in hospital.