I would say the brown sugar in your recipe might be the cause of the color. I only use brown sugar in butterscotch pudding.. I still can't figure out the grainy unless its the salt or the flour but I sifted mine it still turned out grainy.. I didn't use evaporated milk because its too rich I used whole milk and heavy whipping cream which I thought would get rid of the grainy but no such luck.. I am totally stumped.
You and I are having the same problem. My pudding did not turn out brown but it is grainy beats me .. I have tried and tried and I know there is a secret somewhere.. but no one is sharing that secret. LOL