Every consumer in India has a stake in the food economy. And the biggest risk facing our food economy and food security is the lack of reliable big data on the availability of cereals, pulses, sugar, cooking oil, fruits and vegetables. Its absence is also the primary cause of rural distress.
There is no shortage of food in the world. So, even if the monsoon is poor, with a little bit of planning, we can easily import enough to prevent prices from spiraling up. Government agencies can prepare sale/purchase strategies. States can trigger food security measures. The RBI can correctly forecast inflation. Farmers can switch crops. Traders can book consignments in advance. And processors can better manage raw material inventories. Prices may still rise but not alarmingly. We would be prepared.
But planning requires prior knowledge. This is where we fall short year after year. Neither the private sector nor the government has reliable estimates of crop production, stocks in godowns, last season's leftover quantity and current shipments. Or rather, though there is no dearth of data, there's none we can bet on. As a result, the food market is continuously rocked by rumours, speculation, industry guesstimates and the government's revised estimates.
"If information is the oil of the 21st century, and analytics the combustion engine, our food and agricultural economy has to get off the bullock cart."
An outdated, unreliable system
How did we come to such a pass? India has a traditional decentralised agricultural statistics system. States collect primary statistics which are consolidated nationally by the Directorate of Economics & Statistics (DES), under the Union Ministry of Agriculture. Every agricultural year (July-June), the ministry releases four advance estimates of major crops, followed by final estimates of production.
At the heart of this endeavour is the overworked, under-paid village patwari or accountant. The post is often hereditary. He has to manually gather data about each crop in each village. The patwari's beat ranges from more than 10 villages in Bihar to 4600 acres in Karnataka. Supervisors physically verify the patwari's girdawari or record book by visiting a few villages that add up to 10,000 for the entire country.
States also conduct 500,000 crop-cutting experiments in 68 crops to estimate yields for calculating crop size. Here too, their results differ widely from similar experiments on 9 lakh fields done by the Commission for Costs & Prices (CCE), under the Agriculture Ministry. Between 2001 and 2009, CACP experiments showed cotton yields 302% higher than DES estimates. So, we don't have reliable estimates of cotton yields -- from either scheme -- in a country expected to topple China as the world's largest cotton grower.
Several solutions have been offered. The National Commission on Agriculture, for example, has recommended reduction in the patwari's area and intensive supervision by state revenue and statistical staff. But states can't afford more patwaris.
An IIM Bangalore study finds that in Karnataka this system of "eye-balling" the crop (prevalent in 18 states) blows a fuse when farmers do mixed cropping or shift to high-value crops. Modern remote sensing satellite technology and hand-held GPS devices are the alternative. Indeed, the IIM study found a 54% difference in estimate of area under millets in the same village done by the patwari and GPS technology.
But data from hand-held devices is susceptible to the same human problems faced by the patwari. Remote sensing technology is available for only eight out of the 27 major crops for which the Ministry of Agriculture provides estimates. There is no technology for fruits and vegetables. Use of microwave remote sensing, beyond rice and jute, is yet to be established. And it is unable to do impact assessment of damage from extreme events such as cyclones or hailstorms. Agricultural insurance companies giving farmer compensation are especially hampered by the absence of reliable crop yield data.
Not surprisingly, large trading houses and industry associations for sugar, basmati, oilseeds and pulses are conducting their own surveys that also depend on field tours and crop-cutting experiments. Therefore, industry estimates also remain non-scalable, subjective and prone to human error. Satellite imaging is usually beyond reach. Last year, there was a 3-million-tonne difference between trade and government estimates for chana. Instead of data becoming the new raw material of business, we are saddled with opinion.
"Our claim of Digital India will come true only when we have accurate and useful agricultural statistics that serve farmers and consumers."
Both industry and the government have no fix on stocks in the pipeline because the warehousing industry remains unorganised. Only the 400-odd warehouses registered with commodity exchanges are required to report stocks in real time. With cloudy data on size of crop and inventory, farmers, policymakers, traders and industry lurch in the dark. The cost is borne by us.
What do other countries do?
How do other agricultural giants handle the problem? China considers commodity stocks a state secret. World markets remain volatile while trying to guess supply levels in the world's biggest commodity importer. Global crop data officials from the Agricultural Market information System, an inter-agency platform designed to increase transparency in global food markets, and members of the International Grains Council and the UN Food and Agriculture Organization are trying to convince China to release the country's official grain stock data.
Brazil's sugarcane estimates are considered accurate by the world market. But the Big Daddy is the National Agricultural Statistics Service of the US Department of Agriculture. Its forecasts move global markets. Crop reporting ranks as one of the soundest, most objective and most scientific of all the data gathering undertaken by the US federal government.
Because the USDA data is based on a sample of 100,000 farmers, it is far from perfect. But overall, veracity is high. The USDA even provides American exporters data from other countries, including India, collected by the Foreign Agricultural Service Bureau in every embassy.
India's track record falls somewhere between China and the US. That is no longer enough. If information is the oil of the 21st century, and analytics the combustion engine, our food and agricultural economy has to get off the bullock cart.
So, what do we do next?
A good start would be money. A budgetary allocation of Rs 113 crore in 2014-15 to improve crop statistics can't be expected to make a dent in a country sowing 142 million hectares annually across 29 states. States have to be incentivised while the nascent National Crop Forecasting Centre gathers strength. Big agricultural data exists to find the signal among the noise. It is interested in needles, not hay. Right now, our data is just noise. Our claim of Digital India will come true only when we have accurate and useful agricultural statistics that serve farmers and consumers.
Contact HuffPost India