I remember listening to a discussion about how the advent of farming enabled the feeding (at least to a survivable level) of large groups of those loyal to the holder. This enabled militaries, which enabled military expansion, which snowballed into everything else. Refusing the farming means refusing a military, which leaves you defenseless against those who use it. Farming is a weapon.