Dear Preeti,
Remember one thing in your life that never ever compromise. I guess yoi were saying sorry when it was not your fault. I did the same mistake & then realised its useless to serve such a person. Now just strict to the ICAI rules. Let him say whatever he wants to. After all even a dog gets tired after barking !!! Now become strictly professional. Develop an attitude to fight against such person. I know it might sound as a lecture but this is reality & my experience also says so.
All the best for your efforts!!!