Yes. And no.
Edaphagy is the practice of eating soil. This is not uncommon in some cultures, especially among those of African origin. There are a couple hypotheses about this: possibly the person is experiencing a micronutrient deficiency and is eating some soil as you might take a mineral supplement; or maybe they are taking it as an antacid to help with an upset stomach. And among kids who still live and play in the country, a bit of mud pie consumption may have occurred.
Historically the inert binding agent used by pharmacists in making tablets was kaolinite, a special kind of clay. And perhaps you got a really bad stomach bug and couldn’t keep anything in either end, and someone gave you Kaopectate to help calm your stomach. Pectate is a gel-like material, and Kao stands for kaolinite, that same clay mineral. So you may have ingested clay (soil) for medicinal purposes.
Soil can be baked to kill any bacteria or other pathogens, and most soils could be consumed with no ill effects, other than the excessive grinding down of the enamel on your teeth.
But remember, soil particles are minerals, not organic materials like foods, and soils are not digestible, so they would just pass through your digestive tract.
But there are certain clay minerals of the smectite family that should never be ingested in any quantity because they may swell 2 to 5 times when they start from completely dry to saturated, may absorb up to 5 times their weight in water, and are extremely plastic and sticky. These clays could have a devastating effect on your insides.