U.S. taxpayer-funded government reports on the U.S. economy and agriculture have generated comments from stakeholders and others, especially since the Internet has made it easier for anyone to comment.
We decided to check in on response rates for USDA’s National Agricultural Statistics Service (NASS) reports after a new study by the American Statistical Association warned that the reliability of U.S. economic data is at risk due to shrinking budgets, declining survey response rates, and potential political interference (this is not the case with NASS reports). Currently, government statistics remain dependable, but the study, authored by statisticians from various institutions including George Mason University and the Urban Institute, likens the statistical system to infrastructure that is often neglected until a crisis occurs. Link to report. Link to New York Times article.
Note: We contacted several current and former USDA officials and asked them to comment about the numbers and some of the responses in this special report. These individuals include Lance Honig, Director of Methodology Division, Chair, Agricultural Statistics Board, USDA-National Agricultural Statistics Service; current top USDA economist Dr. Seth Meyer; and former top USDA economist Dr. Joe Glauber. |
The following is a look at USDA NASS reports:
Response rates have been declining over time:
• In the early 1990s, response rates for NASS crop surveys were around 80-85%.
• By the late 2010s, response rates had fallen below 60% in some cases.
Recent response rates for specific NASS surveys:
• The 2022 Census of Agriculture had a response rate of 61%.
• The 2017 Census of Agriculture had a response rate of 71.5%.
• For monthly NASS surveys, response rates are around 75%.
• For quarterly or annual NASS surveys, response rates are around 50-60%.
Factors affecting response rates:
• Increasing difficulty in accessing households due to new telephone technologies.
• Rising refusal rates from respondents — NASS’ Honig says “it’s mostly inability to reach people that’s increasing. Actual refusals are fairly steady.”
• Concerns about data privacy and time constraints from farmers.
• Increased number of people requesting information from farmers.
Impact of declining response rates:
• Reduced statistical precision of estimates, especially at county levels.
• Fewer counties for which estimates can be published.
• Potential introduction of bias if non-respondents differ from respondents.
NASS efforts to address declining response rates:
• Offering online response options.
• Developing shorter questionnaires.
• Adjusting sampling and weighting procedures.
• Increasing follow-up efforts through multiple contact methods.
• Increased outreach efforts to build relationships/trust and increase transparency (i.e. #StatChat, Data Users’ Meetings, Visitors to Lockup, etc.)
Importance of response rates:
• High response rates are crucial for maintaining data quality and reliability.
• Lower response rates can lead to increased costs for data collection.
• Accurate data is essential for policymaking, research, and agricultural planning.
Bottom line: NASS continues to monitor response rates and implement strategies to improve participation in their surveys, recognizing the critical importance of high-quality agricultural data.
— Ag industry contacts respond to NASS reports and response rates:
• “Accurate data primarily allows transparency in the marketplace, otherwise you get misinformation on social media and conspiracy theories due to what analyst one listens to. It does not create a level playing field for all.”
• “While people complain in the U.S. that big companies have all the information, USDA’s NASS allows one point of solid information that everyone can trade off of.”
• “While there are shortcomings, NASS provides the best source of agriculture data anywhere in the world. It has allowed U.S. agriculture to thrive. Going forward, the policy makers need to understand they need to fund the service to help farmers, agribusinesses and consumers.”
• “I’m actually surprised monthly USDA NASS surveys are still at around 75%” (a figure we got from NASS).
• “This is a topic every year on the Pro Farmer Midwest Crop Tour, because it’s a debate on how accurate NASS is with their crop estimates. Farmers want to complain that NASS isn’t that accurate, but they don’t want to give NASS any insights, either (the trust issue).” Note: NASS publishes a report each year that shows how accurate its estimates are.
• Several growers said they don’t trust the government and they aren’t giving out their data. One said: “You just wonder with all the technology on planters and combines today, as well as all the satellite info, when NASS will have to change their approach.” Note: NASS’ Honig says “NASS is exploring the potential of using precision ag data, but significant hurdles currently exist, primarily around ownership/availability of the data. Satellite data are currently utilized to augment the survey and administrative data.”
• “Growers don’t want to give NASS any intel, and I don’t see that changing. So, I assume USDA’s NASS is going to have to find different methods to get their data.”
• “I think the response rate is way below 50%. Some of these guys are big farmers and do not want to share data. Plus, they all think NASS’ quality has declined-so why give data to a failing entity?”
• “Grain stocks on farm has been a big topic amongst the elderly grain traders I keep in touch with. Most think the basis is a much better indicator than NASS. For example: why is the cash corn basis so strong in the WCB this year, yet ECB stocks are reported by NASS to be huge. Meanwhile, Cn/Cu is trading an inverse during delivery and the delivery points are in the ECB.” Note: NASS’ Honig says, “NASS stocks estimates represent quantities stored by location, but do not indicate whether or not those quantities are all still available to be marketed (i.e. some/all may already be contracted/committed).”
• “Why can’t crop insurance yield data be used to compare to NASS plot data? I would argue crop insurance yield data is probably the most accurate data currently available. Since crop insurance is a federally subsidized program, let us see the data.” Note: Honig says “NASS can use these data for evaluation purposes, but timing is a big issue with these data. Data are only provided once for each season, and not available until late-spring/early-summer the year following harvest. NASS publishes yield forecasts throughout the growing season and provides final season estimates in early January for most major row crops (late-September for small grain crops).”
• “After a response rate gets to a certain level, the survey is no longer statistically accurate and should be dropped.” Note: Honig notes that “Response rates only tell part of the story. Statistical measures of precision (i.e. CVs) are closely monitored and adjustments to sampling & other methodologies are made to adjust as needed.”
• “Social media (SM) has turned everyone into an analyst that feels they get ‘enough’ data from SM to determine crop size. Then they go on a drive and look for confirmation of what they expect to see and, of course, they find it. Now they are armed with 4 hours of research on ‘X’ and what they saw on a 200-mile round trip, and they think they have the U.S. crop figured out. When NASS reports something different than they expected, the first thing they do is get back to SM and tell everyone how wrong NASS is. When NASS reports something in line with their expectations, the first thing they do is get back to SM to tell everyone how right they were before NASS put out its guess.” Note: Honig says “Agreed — and people commonly assume because NASS estimates don’t match what many expected that they are wrong. Expectations are often based on limited information.”
• “Regarding the falling response rate...
“Distrust not just of USDA’s NASS but of anything to do with or organized by the government. Some refuse to respond while others (very few, but it happens) falsely respond and then complain (loudly) about how wrong NASS is.” Note: Honig responds: “More data always leads to increased accuracy, so responding to surveys is the best way to make things better!”
“Few take time to understand the process and how results are generated at different times of the year. If they had a better understanding they might be more willing to participate in a constructive way.” Note: Honig says “NASS works hard to be transparent and provide details about our procedures. We make ourselves readily available to answer questions and address concerns.”
“Of course, it’s not just the NASS estimates that create the distrust... it’s the combination of the NASS estimate and the market reaction. So, the distrust is also toward ‘the markets.’ It’s the, ‘Let them figure it out on their own’ attitude. These are the same people that don’t want any crop estimates (private or public), will say ‘Let the market figure it out’ and then complain when the market doesn’t perform like they think it should.”
• One veteran industry analyst emailed:
“Conspiracy theorists are taking over... due to social media.
“Media isn’t meant to be social.
“Keyboard warriors… everyone thinks they’re an expert. They hide behind cute screen names, but no recourse for putting out wild/false claims.”
• “Going to be more important going forward! These new AI models scrape data and form conclusions. Without good data, the promise for AI may be limited, or even worse, misleading.”
• “USDA’s NASS should evaluate and determine what reports need to be sunset if they do not have confidence in the output.” Note: Honig responds: “NASS has strict quality standards to ensure that all published estimates are accurate and fit for use. We stand behind all of our published data.”
The following comments are from Lance Honig, Director of Methodology Division and Chair, Agricultural Statistics Board of USDA’s NASS: “As a Federal statistical agency, NASS has the unique ability to level the playing field by providing unbiased and accurate information to everyone involved in agriculture — free of charge and available to everyone at the same time. The work that we do is a partnership with farmers across the Nation. Every producer who receives a NASS survey has an opportunity to improve the accuracy of the results by completing it, which leads to better decisions, better policy, and increased market efficiency. That’s a win for everyone. Response to surveys has declined in recent years, but overall rates remain very strong at NASS relative to other organizations and entities conducting survey work — a tribute to the time farmers commit to this partnership. While surveys remain the backbone of our estimates, we incorporate additional information into our process, including administrative data from across USDA, geospatial information, and more. This helps to improve accuracy while reducing the volume of survey contacts we have to make, therefore reducing the burden placed on farmers. We continue to explore additional data sources as we look to the future, but remain committed to utilizing the most reliable information available today.” |
— Here are the comments we received from USDA top economist Dr. Seth Meyer on the topic:
“There is talk about crop insurance and use of other data sources. For RMA data, the issue is timeliness and when producers are required to report information to AIPs. By that point it has very little additive value. (Note: USDA’s Honig also commented on this as noted above.) NASS also makes extensive use of FSA data; they can pull what they need, and NASS has moved up when it more fully utilizes the FSA data as the FSA data have both improved quality and timeliness. I expect we will see more of this, and it will support crop production estimates. They will use any bit of data they think can contribute to an improved estimate.
“Indeed, earth observation (EO) data and analysis are improving. 20 years ago, EO often overpromised what it can do, but the reality is starting to meet the hype. We use EO extensively in the WASDE report, in particular this is helpful in countries which lack a strong statistical service or where data collection are challenging. However, I’m not yet willing to trade my NASS data for it and I’m going to want a couple of decades of overlapping data before I’d agree.
“Often when I’m overseas, people I meet report that they rely on USDA data more than their own government’s data. There is a level of trust that the data are unbiased, and we need to work to maintain that trust among our direct constituents in the U.S. I think important points are raised [in this report] about how to ensure that the quality of that data is maintained.
“I always try to explain to producers why responding [to surveys] is in their best interest. They might not always like the way prices move when the report is released (at least not half the time) but these reports level the playing field every 30 days. They are at a disadvantage to large grain traders able to accumulate more information; NASS reports resolve some of this information asymmetry each month.
“There is a lot of value in NASS reports, value that isn’t as flashy as a new program or initiative but lays the foundation for a lot of decision making across the country by producers and others. I think it is always important for us at USDA to make the case, and I think it is a good case, that these reports are a benefit to the sector and not only help market function but result in better policy formation in DC and in the state capitals.”
— Comments from former top USDA economist Dr. Joe Glauber:
“Earth observation data and AI technologies are becoming increasingly accurate at measuring area and predicting yields. While we still ground truth a lot of those data against NASS surveys, those methodologies will become increasingly prevalent and may ultimately become the gold standard. Earth observation technologies are already the standard for evaluating crop conditions in many countries (for example, GEOGLAM’s estimation of cropland and crop conditions in occupied areas of Ukraine). But moving beyond area, yield and production, it gets more difficult. NASS is one of the few national statistical agencies that attempts to measure grain stocks. Consumption estimates are even more difficult (There is a reason the WASDE corn balance sheet includes “Feed AND RESIDUAL.”) Lastly, NASS and ERS have provided long time series on farm sector well-being and as much as I am often critical of the farm income measure, my criticism is more about how the measure is (mis)interpreted.
“USDA’s original function was providing research and development for farmers (through seed development and distribution) and providing information on prices, production, etc. These remain public goods that I would argue are still relevant today as they were in the 1860s.”