Kenneth Colley was great! I've run into him a few times in my old telly watching, and he's always so good. (I also read somewhere that in rl he had a stammer and was v nervous, but it just vanished when he acted.)

I like your other choices too (and agree with Swordznsorcery that Eddie Marsan is great as Mr Norrell.)