In defense of survey methodology

Photo: Peter Bono

The recent tiff between the Supreme Court (SC) of Bangladesh and Transparency International, Bangladesh (TIB) is a watershed event for all professionals who conduct surveys and the organisations that use them to guide important decisions. In the business world, surveys often guide major strategic decisions. And the use of surveys for political opinion polling is ubiquitous.
What is surprising is that not one organisation in Bangladesh that makes a living using surveys, responded. Even from the universities and their esteemed researchers, there was not a single intellectual opinion in defense of survey methodology.
The TIB survey suggested that the judiciary is perceived as the most corrupt service sector in the country. According to one report, "…lawyers, court employees, court clerks and brokers take money to hasten or postpone hearings, to withdraw and destroy case documents, and influence the judgment."
Expectation and payment of "speed money" is not uncommon in the public sector. People face such practices routinely that in turn forms and reinforces their "perception" that the corruption menace is as healthy as ever. One might wonder why the use of speed money to facilitate court-related matters would be anything unusual to expect.
The visceral reaction to the suggestion that corruption exists in such a powerful institution was understandable. Similar reactions occur in the developed world too. In the United States when the rankings are revealed periodically about universities (in the US News and World Report), many "universities decry the commercialism of the rankings, [and] attack the methodology of the ranking process…."
Curiously, when the rankings are stellar, they are widely accepted and used liberally to bolster support from various stakeholders; when the rankings are not favourable, there is a big hue and cry.
The SC, given its extraordinary powers, publicly questioned the TIB report, concluding that it was baseless and without substance. It also opined: "The TIB report was just made for publicity as it could not help pinpoint people involved in corruption."
Surveys, to be clear, are not police investigations. They are not intended to identify specific episodes of corruption or corrupt persons. Rather, surveys are tools used to "estimate" certain attitudes, perceptions, propensities, behaviours and incidences that exist in a population. According to one source, "Surveys are a powerful, cost-effective tool for gathering important information and for identifying and diagnosing problems and opportunities." They provide statistical insight into a phenomenon -- in this case perceived corruption -- that exists in a defined population.
If the probe committee expected the study to identify specific cases of corruption, it must be respectfully submitted that it has little understanding of the purpose of surveys and their methodologies.
That does not mean, however, that all surveys are perfect. They can be flawed and sometimes their intent can be dubious. To refute the TIB's findings, the SC could have investigated the source of flaws by seeking clarifications in the following areas:
* Was there a design flaw, particularly in sampling and generalising to the entire population of Bangladesh?
* Was there a flaw in the way the questions were framed and the measurement scales used?
* Was there a flaw in data collection, especially in how fieldwork quality was ensured?
* Was there a flaw in analysis, especially the statistical procedures employed?
* Was there a flaw in the conclusions that were drawn?
From my understanding, the study used the Integrated Multipurpose Sampling Frame (IMPS) employed by the Bangladesh Bureau of Statistics. This is accepted protocol. The large sample size also suggests the study's representativeness.
The questions that needed to be asked of the respondents appear to be fairly straightforward. Actual data collection can be erroneous. And the type of analysis needed did not seem too esoteric to obfuscate the results. What remains is the conclusion: the interpretations here can be tricky because of such things as order, recency and halo effects. But that's technicality.
If the SC felt strongly that the study's conclusions were in error, it could have commissioned a replication study to compare the results. It may be cautioned, however, that replication studies will not produce identical results because of sampling and non-sampling errors that tend to creep in. But the confidence intervals can provide reasonable insights into the accuracy of the findings.
The SC's actions and conclusions could, thus, be quite disconcerting and raise a number of questions:
1) If the judiciary had been shown in a cleaner image (not leading), would the SC have displayed such a response?
2) If a replication study is conducted, the judiciary's ranking could fall from first place to second, third, or even fourth. Should that allay citizens' concerns about perceived corruption in the judiciary?
3) TIB reports have been called into question in the past too; but, was its credibility ever really destroyed? As part of an international effort to promote transparency, it seems to use credible methodologies;
4) Does the SC feel that surveys are inherently flawed and ought not to be used to assess public services?
5) Might the judiciary have strengthened the hands of the corrupt within its system? Will these elements now feel much safer to harass the citizenry, and in greater measure?
Transparency, today, is a crying need in the country. True development and alleviation of poverty can come about when the inner workings of the nation's core institutions are opened to the public for assessment. In this, TIB has played a vital role by seeking opinions of the key constituency: the beneficiaries of public services. A concern now is whether TIB is unlikely to conduct similar studies in the future for fear of being twice-burnt.
Instead of the angry reaction, the SC could have displayed a wiser and nobler stance by thanking TIB for helping unearth what goes on inside the judiciary's arcane world and its current state and health. This would reinforce the judiciary's clear intent to establish and deliver justice in greater measure, signaling hope for those who struggle with injustice almost daily.
A positive institutional attitude is what is needed today across various public sector services to better serve the nation in its quest for emancipation. In such an attitude lies the real possibility that appropriate measures will be taken by the higher ups to ultimately rid the people, whom they are duty-bound to serve, of their miseries. The present attitude comes across as ominous -- one that maintains the status quo, projects bureaucratic hegemony, and detracts from a people-centric ethos.

The writer is a Distinguished Professor of Marketing and Editor, Journal of Bangladesh Studies.