Well this is a very touchy subject and I'll wade into it. The analogy sometimes made is that the show Avatar is a fantasy world based on Asia and there are no white characters, so why not set similar standards for LotR based on Europe? This makes a degree of sense but on the other hand, some would point out that medieval Europe was not monochromatically white, which is true. On the other other hand how widespread nonwhite minorities were especially outside Southern Europe is debatable, they would be mostly lighter-skinned by our standards, and it would not be the integrated melting pot that America is at least aspirationally (people like medievalPoc who argue black people would be a common sight throughout medieval Europe have some good points but are overplaying their conclusions with the evidence, IMO).
But setting historical debates aside here's another way to look at it: modifying the story for a modern audience and a different medium is something every adaptation does. Jackson did it too, such as making Arwen and to a lesser degree the other females more important, Aragorn more conflicted, or downplaying the element of social class in Frodo and Sam's relationship. (You can argue correctly that some adaptive changes are because of the film medium, not "modern concerns", but diverse casting is also prompted by the medium - in the book a character's ethnicity is mostly in the background and easily fades away unless you explicitly make it important to the plot, on film you always see it).
But there are definite limits to what you can change. What constitutes an acceptable fudging of the source and what violates the spirit of it? Well, that has to be treated on a case-by-case basis (Aragorn was fine IMO, Denethor was not). Most people are ok with downplaying the master-servant dynamics between Frodo and Sam because that is just one aspect that does not define a very rich relationship, for example. And I would say that a character's ethnicity is a pretty minor concern that will not make a big difference in fidelity.
If you're going to say it's necessary for historical accuracy, well all sorts of things in the Jackson films aren't historically accurate or realistic. A military historian went through Jackson's Pelennor and Helms Deep and found all sorts of
anachronisms and inaccuracies. Then are here are things like the Argonath being implausibly big or the beacon setup not actually making sense. A lot of people (myself included) are ok with this kind of thing because they work cinematically, or at least a large portion of them do. Now obviously this does not mean that just anything goes in not following history; verisimilitude is important. But again I would group making the cast more diverse under "minor fudge."
My personal preference would actually not be to just do diversity anywhere. I would do it as broadly as can be justified within the text itself or at least plausibly argued from it, which means probably more diverse than Jackson. Gondor, for example, is a multi-ethnic kingdom, has presumably intermarried with Harad over the centuries, and has regions (including Beregond's home) described as "swarthy." You could also give a very sympathetic treatment of Harad as Alatar said. Perhaps go for a mixed "Mediterranean" look for Númenor which could include North African and Arab actors. However a more generalized approach to diversity, while not what I would ideally do, is not that big a deal or something I'm going to get upset about. There are much better reasons to worry about the show.
I agree that not everyone who expresses reservations about this aspect is racist and there are some who are too dismissive in that regard. However, a lot of people online, especially those who complain most loudly about it, are.
