insights: hard errors from downstream APIs should be retried by default

Created by: leonore

work for #35227 included a change where we made API errors terminate by default. we should approach this the other way, and specify which cases should not be retried (i.e. invalid queries, network errors). we can then add on to those cases if we find any more (see #34990 (closed))