Fix references panel stuck loading

Created by: chrismwendt

This fixes the references panel, which would get stuck in a loading state:

This also simplifies the data model a bit.