Nothing in the legendarium is more full of contradictions than the writings about the Istari. The quote you are thinking of, Jude, from the Istari section in
Unfinished Tales is this:
Indeed, of all the Istari, one only remained faithful, and he was the last-comer. For Radagast, the fourth, became enamoured of the many beasts and birds that dwelt in Middle-earth, and forsook Elves and Men, and spent his days among the wild creatures. Thus he got his name (which is in the tongue of Númenor of old, and signifies, it is said, "tender of beasts"). 4 And Curunír 'Lân, Saruman the White, fell from his high errand, and becoming proud and impatient and enamoured of power sought to have his own will by force, and to oust Sauron; but he was ensnared by that dark spirit, mightier than he.
Yet, in another text in that section, he writes that "that Curumo [Saruman] took Aiwendil [Radagast] because Yavanna begged him" suggesting that Radagast purpose was to support the beasts and birds.
As for the Blue Wizards, while it is true as Jude says that in one place Tolkien suggests that they went east and formed and cult and were not of any assistance against Sauron, in a text printed in
The Peoples of Middle-earth he tells a very different story (and gives them different names):
But the other two Istari were sent for a different purpose. Morinehtar and Romestamo.(28) Darkness-slayer and East-helper. Their task was to circumvent Sauron: to bring help to the few tribes of Men that had rebelled from Melkor-worship, to stir up rebellion ...
and after his first fall to search out his hiding (in which they failed) and to cause [? dissension and disarray] among the dark East ... They must have had very great influence on the history of the Second Age and Third Age in weakening and disarraying the forces of East ... who would both in the Second Age and Third Age otherwise have ... outnumbered the West
But perhaps most extraordinary, there is a reference to the Istari) in
The Nature of Middle-earth (edited by Carl Hostetter, aka Aelfwine) that gives them a completely different history. In a bundle of texts labeled by Carl as “Key Dates” that he says was probably written by Tolkien around 1959, we have this remarkable entry:
DB (“Days of Bliss”) 866/13 2175. Oromë remains for 12 years, and then is summoned to return for the councils and war-preparations. Manwë has decided that the Quendi should come to Valinor, but on urgent advice of Varda, they are only to be invited and are to be given free choice. The Valar send five Guardians (great spirits of the Maiar) - with Melian (the only woman, but the chief) these make six. The others were Tarindor (later Saruman), Olórin (Gandalf), Hrávandil (Radagast), Palacendo and Haimenar
If in fact Gandalf and the other Istari were actually sent to Middle-earth shortly after the awakening of the Eldar as Guardians as them make their free choice as to whether accept the invitation of the Valar to come to Valinor, the answer to Halifirien's question would be very different.
"Spirits in the shape of hawks and eagles flew ever to and from his halls; and their eyes could see to the depths of the seas, and pierce the hidden caverns beneath the world."