From a terminology standpoint, usually when people talk about visual field deficits, they are referring to reductions in a visual field, rather than unusual visual phenomena within an otherwise functional visual field.

Anyway, visual field deficits can be brain-lesion-driven, as you say (e.g., the classic bitemporal hemianopsia in pituitary tumors).