Farm advocates say concerns over worker detentions and immigration enforcement are creating fear and instability ahead of peak growing season.