Why different surveys tell opposite stories on youth's wishes to leave Pakistan

Recent studies paint starkly different pictures of Pakistani youth's desire to emigrate. Still, a closer examination reveals that these contradictions tell us more about how we measure public opinion than actual migration desires.

The first narrative, espoused by Ipsos Research and Gallup Pakistan, suggests a population largely content to remain in their country. Ipsos's October 2024 study reports only 26% of youth want to leave, while Gallup's late 2023 survey shows a similar 25% figure.

However, this narrative sharply contrasts with the Pakistan Institute of Development Economics (PIDE) 2022 research, which presented a markedly different reality. PIDE's findings showed that 50% of youth aged 15-24 want to leave the country, with an even more striking 62% of young males in this age group expressing a desire to emigrate.

Such significant variations in findings demand scrutiny, as these differences go beyond normal statistical variance.

Contradictory findings

Even more concerning is Ipsos's approach to presenting their findings. Their research report boldly claims, "Contrary to popular belief, the acute majority of Pakistani youth (74%) desires to stay back in Pakistan," dismantling the prevailing propaganda without any evidence.

This statement notably fails to acknowledge PIDE's comprehensive 2022 study, which surveyed over 20,000 respondents - more than 10 times Ipsos's sample size - and found that close to a majority of youth indeed expressed a desire to leave.

Interestingly, in July 2024, Gallup Pakistan issued a press release addressing viral social media claims that 94% of Pakistanis wanted to leave the country. While denouncing this misinformation, Gallup referenced PIDE's 2022 study showing much higher emigration desires than their findings. This suggests Gallup considers PIDE's methodology credible, despite the dramatic differences in results.

History of migration measurement

Gallup Pakistan's Executive Director Bilal Gilani emphasizes his organization's long-term perspective on the issue. "The subject of migrating or moving abroad is a subject we at Gallup Pakistan have been studying since 1979 when the firm was set up," he explains. According to their data, overall intention to migrate fluctuated between 17% in 1984 and 38% in 2003.

In recent years, Gilani argues, the number has actually declined for the overall population, ranging between 25%-30% across various surveys and question formulations, suggesting PIDE's youth figures are outliers compared to their historical data.

Dr. Durr E Nayab, author of the PIDE research, challenges this characterization. "Are we going through a normal phase? Why are [our findings] considered outliers?" she responds. "Ask a group of young people sitting anywhere and you will get your answer. Ask visa offices and you will get your answer again."

A tale of two methodologies

The stark differences in findings stem from fundamentally different approaches to data collection. The PIDE study, which researcher Amar Latif Qazi considers methodologically superior, conducted face-to-face interviews with over 20,000 respondents across almost all districts of Pakistan using the PBS national sample framework.

In contrast, the Ipsos study relied on computer/telephone interviews with 1,627 respondents aged 18-34 in May 2022. While the sampling attempted to cover different provinces, education levels, and rural-urban populations, Qazi points out significant limitations in who could be reached through telephone interviews and notes that respondents are likely to give more socially acceptable answers in phone surveys.

Gilani acknowledges these methodological differences, saying, "We feel the variation is also explainable by the difference in how the question is worded and difference of mode - Gallup and IPSOS are phone surveys while PIDE is face to face."

He also notes that when comparing overall Pakistan-level figures instead of just the youth, the gap narrows somewhat - Gallup shows 28% while PIDE indicates 37%.

Gillani also points out a "cocoon effect" among Pakistan's educated urban class. "Since we encounter a phenomenon so much (for example people moving abroad or wishing to do so), we feel the rest of the population also behaves or feels similarly," he explains.

"As the income divide, education divide, urban-rural divide, social media divide, and many other cleavages widen, our own perception vs reality regarding the total population can see many surprising findings." This suggestion that educated urban Pakistanis might overestimate emigration desires due to their social bubble is compelling.

Dr. Durr E Nayab, author of the PIDE study, claims the contrary, "A bias is already there by only selecting those who have phones, can be connected, and are ready to respond," she says, arguing that phone surveys might create their own "reverse cocoon effect" by primarily reaching more accessible, phone-owning populations and missing less accessible populations.

Sample size considerations

Gilani argues the mathematical reality is more nuanced than just directly comparing sample sizes. "The reliability isn't dependent per se on the size," Gilani explains, pointing to statistical error margin tables. According to these tables, beyond a sample size of about 2,000 respondents, increasing the sample yields diminishing returns - typically less than 1%-1.5% difference in accuracy. This suggests the gap between PIDE's findings and other surveys can't be explained by sample size alone.

Gilani also emphasizes his organization's historical perspective. While individual Gallup surveys may have smaller samples, they've consistently measured migration intentions since 1980. This institutional memory, he argues, provides crucial context that newer organizations lack. "PIDE has done this exercise once, and I don't think they have institutional memory of previous survey work, therefore they don't have any trend," he notes.

Measuring different things?

The difference in findings may be better explained by who was surveyed rather than how many. While Gallup and Ipsos focused on respondents 18 and older, PIDE's inclusion of younger respondents starting at age 15 captured a demographic that typically shows stronger emigration desires.

Qazi notes that the 15-24 age group shows distinctly different trends, with the younger generation appearing more willing to leave. All three surveys agree that the younger the population segment surveyed, the higher the emigration desire.

The timing of these studies presents another crucial consideration in understanding their divergent results. When asked whether youth emigration desires could have changed so dramatically in the two years since PIDE's study, Dr. Nayab said, "The emigration desires have not shifted/reduced, there is no reason to believe so." She points to patterns visible at visa offices and among informal groups of young people as evidence that emigration desires remain high.

Official data presents a complex picture. Deportations of Pakistani nationals have declined from 58,758 in 2021 to 43,578 in 2023. However, senior Federal Investigation Agency (FIA) officials attribute this drop to stricter enforcement and deterrence rather than diminishing emigration desires.

The international context

Gilani brings an international perspective to the debate, pointing to Gallup International's research across 64 countries asking similar migration questions. "Pakistan ranks quite low in the list," he notes, "Lower than the global average." This, he suggests, provides an important context for understanding Pakistani migration desires relative to other countries.

However, this international comparison raises methodological questions. Given the significant variations found between different survey approaches within Pakistan, comparing across countries with different cultural contexts and survey methodologies becomes even more complex.

Qazi noted there appears to be a conflict between Gallup Incorporated and the Gallup International Association, with which Gallup Pakistan is affiliated, raising questions about institutional credibility.

The political dimension

Dr. Nayab emphasizes the multiple factors that influence survey results. "Survey design, its sample, tools used, training of enumerators, respondents' availability/willingness to give a genuine response, etc., etc. all have an impact on the results we get," she said, advocating for careful examination of methodology before judging any study's quality: "I would refrain from calling any study good/bad, robust/weak, till I exactly know the details."

This measured approach contrasts with Ipsos's dismissal of contradictory findings as "propaganda". Ipsos did not respond to multiple requests for comment from Nukta.

Gilani welcomes this methodological scrutiny, seeing it as beneficial for Pakistan's survey research industry. He particularly appreciates evidence-based critique that goes beyond "drawing room conversation or pseudoscience," which he says unfortunately characterizes much media commentary on survey work.

Ultimately, these methodological debates matter because the findings directly influence policy discussions about youth employment, education, and economic opportunity. When government officials selectively cite lower figures from phone surveys while dismissing more concerning findings from comprehensive studies, they risk basing policy decisions on incomplete information.

The way forward might involve combining different methodological approaches – using quick phone surveys for trend tracking while conducting regular comprehensive face-to-face studies to provide deeper insights. What's clear is that understanding Pakistani youth's emigration desires requires more than just headline figures; it demands careful attention to methodology, context, and the complex social realities these surveys attempt to measure.