Watch Introduction to Sampling and Randomization, (Register 2020)
Read Working-Class Households in Reading, (focus on the method and approach, rather than the specific results) (Bowley 1913)
Read On the Two Different Aspects of the Representative Method, Parts I “Introduction”, III “Different Aspects of the Representative Method”, V “Conclusion” and Bowley’s discussion, (Neyman 1934)
Read Guide to the Census of Population, Chapter 10 “Data quality assessment”, (Statistics Canada 2017)
Before there can be a dataset, there must be measurement, and this brings a whole host of challenges and concerns. One dataset that is designed to be complete, at least in certain respects, is a census. While not perfect, governments spend a lot of money on censuses and other official statistics, and they are a great foundational data source.
However even when we cannot obtain such a dataset, we can use sampling to ensure that we can still make sensible claims. There are two varieties of this—probability and non-probability. Both have an important role. Key terminology and concepts include: target population, sampling frame, sample, simple random sampling, systematic sampling, stratified sampling, and cluster sampling.
As we think of our world, and telling stories about it, one of the most difficult aspects is to reduce the beautiful complexity of it into a dataset that we can use. We need to know what we give up when we do this. And be deliberate and thoughtful as we proceed. Some datasets are so large that one specific data point does not matter—it could be swapped for another without any effect (Crawford 2021, 94); but how different would your life be if you had a different mother?
We are often interested in understanding the implications of some dataset; making forecasts based on it or using that dataset to make claims about a broader world. Regardless of how we turn our world into data, we will usually only ever have a sample of the data that we need. Statistics provides formal approaches that we use to keep these issues front of mind and understand the implications. But it does not provide definitive guidance for broader issues, such as considering who profits from the data that were collected, and whose power it reflects.
In this chapter we first discuss measurement, and some of the concerns that it brings. We then turn to censuses, in which we typically try to obtain data about an entire population. We also discuss other government official statistics, and long-standing surveys. We describe datasets of this type as “farmed data”. Farmed datasets are typically well put together, thoroughly documented, and the work of collecting, preparing, and cleaning these datasets is mostly done for us. They are also, usually, conducted on a known release cycle. For instance, most developed countries release unemployment and inflation datasets monthly, GDP quarterly, and a census every five to ten years.
We then introduce statistical notions around sampling to provide a foundation that we will continually return to. Over the past one-hundred years or so, statisticians have developed considerable sophistication for thinking about samples, and dealt with many controversies (Brewer 2013). In particular we consider probability and non-probability sampling and introduce certain key terminology and concepts.
Data are not neutral. Thinking clearly about who is included in the dataset, and who is systematically excluded, is critical. Understanding, capturing, classifying, and naming data is an exercise in building a world and reflects power (Crawford 2021, 121), be that social, historical, financial, or legal.
For instance, we can consider the role of sex and gender in survey research. Sex is based on biological attributes and is assigned at birth, while gender is socially constructed and has both biological and cultural aspects (Lips 2020, 7). We are likely interested in the effect of gender on our dependent variable. But the move toward a nuanced concept of gender in official statistics is something that has only happened recently. Surveys that insist on a binary gender variable that is the same as sex, will not reflect those respondents who do not identify as such. Kennedy et al. (2020) provide a variety of aspects to consider when deciding what to do including: ethics, accuracy, practicality, and flexibility. But there is no universal best solution. The most important job is to ensure utmost respect for the survey respondent (Kennedy et al. 2020, 16).
Why do we even need classifications and groupings if it causes such concerns? Scott (1998) positions much of this as an outcome of the state, for its own purposes, wanting to make society legible and considers this a defining feature of modern states. For instance, Scott (1998) sees the use of surnames as arising because of the state’s desire for legible lists to use for taxation, property ownership, conscription, and censuses. The state’s desire for legibility also required imposing consistency on measurement. Metrology, which is “the study of how measurements are made, and how data are compared” (Plant and Hanisch 2020), begins in its modern form in the French Revolution when various measurements were standardized and this later further developed as part of Napoleonic state building (Scott 1998, 30). Prévost and Beaud (2015, 154) describe the essence of the change as one where knowledge went from being “singular, local, idiosyncratic… and often couched in literary form” to generalized, standardized, and numeric. It is difficult to consider how we could collect data without categorizable, measurable, scales.
All datasets have shortcomings, but, as Dr Laura Bronner, Senior Applied Scientist, Public Policy Group and Immigration Lab, ETH Zürich, describes, there tends to be a considerable difference between datasets such as censuses and surveys that are developed specifically for the purpose of being used as datasets, and those that we gather ourselves and use as data even if they were not designed for this. For instance, there are special concerns around measurement and consistency. In this chapter we develop comfort with “farmed data”—those data where some of those concerns have been taken care of for us and where a dataset has been developed specifically for the purpose of being analyzed.
8.2 Measurement
Measurement is an old concern. Even Aristotle distinguished between quantities and qualities (Tal 2020). Measurement, and especially, the comparison of measurements, underpins all of data science due to its foundations in statistics. But deciding what to measure, and how to do it, is especially challenging. Measurement is trickier than it seems. For instance, what makes a one-hit wonder? A surprising number of artists that may immediately come to mind, turn out to have at least one or two other songs that did reasonably well in terms of making it onto charts (Molanphy 2012). Should an analysis of all one-term governments include those that did not make it through a full term? How about those that only lasted a week? How do we even begin to measure the extent of government transfers when so much of these are in-kind benefits (Garfinkel, Rainwater, and Smeeding 2006)? And how can we measure how well represented a person is in a democracy despite that being the fundamental concern (Achen 1978)?
Philosophy brings more nuance and depth to their definitions of measurement (Tal 2020), but the International Organization Of Legal Metrology (2007, 44) define measurement as the “process of experimentally obtaining one or more quantity values that can reasonably be attributed to a quantity”, where a quantity is a “number and reference together”. It implies “comparison of quantities, including counting of entities”, and “presupposes a description of the quantity commensurate with the intended use of a measurement result, a measurement procedure, and a calibrated measuring system…”. This definition of measurement makes clear that we have a variety of concerns including: instrumentation and units, and that we are interested in measurements that are valid and reliable.
Instrumentation refers to what we use to conduct the measurement. Thorough consideration of instrumentation is important because it determines what we can measure. For instance, Morange (2016, 63) describes how the invention of microscopes in the sixteenth century led to the observation of capillaries by Marcello Malpighi in 1661, cells by Robert Hooke in 1665, and bacteria by Antonie van Leeuwenhoek in 1677 (Lane 2015). And consider the measurement of time. Again we see the interaction between instrumentation and measurement. With a sundial it was difficult to be much more specific about elapsed time than an hour or so. But the gradual development of more accurate instruments of timekeeping would eventually enable sports like F1 to differentiate cars to the thousandth of the second, and through GPS, allow navigation that is accurate to within meters.
A common instrument of measurement is a survey, and we discuss these further in Chapter 10. Another commonly-used instrument is sensors. For instance, climate scientists may be interested in temperature, humidity, or pressure. Animal movement analysis, such as Leos-Barajas et al. (2016), is especially interested in accelerometers. Sensors placed on satellites may be particularly concerned with images, and such data are easily available from the Landsat Program. Interestingly, physicists are very concerned with measurement, and are constrained not only by their instrumentation, but also storage capacity. For instance the ATLAS detector at CERN is focused on the collision of particles, but not all of the measurements can be saved because that would result in 80 TB per second (Colombo et al. 2016). And in the case of A/B testing, which we discuss in Chapter 10, extensive use is made of cookies, beacons, system settings, and behavioral patterns. Another aspect of instrumentation is delivery. For instance, if using surveys, then should they be mailed or online? Should they be filled out by the respondent or by an enumerator?
The definition of measurement, provided by metrology, makes it clear that the second fundamental concern is a reference, which we refer to as units. The choice of units is related to both the research question of interest and available instrumentation. For instance, in the Tutorial in Chapter 1 we were concerned with measuring the growth of plants from seeds, and hair growth over the course of a week. Neither of these would be well served by using kilometers or miles as a unit. If we were using a ruler, then we may be able to measure millimeters, but with calipers, we might be able to consider tens of micrometers.
Valid measurements mean that the quantity that we are measuring is actually related to the estimand and research question of interest. It speaks to appropriateness. Recall, from Chapter 5, that an estimand is the actual effect, such as the unknowable actual effect of smoking on life expectancy. It can be useful to think about estimands as just the parameters of interest. This means that we need to ensure that we are measuring relevant aspects of an individual. For instance, the number of cigarettes that they smoked, and the number of years they lived, rather than, say, their opinion about smoking. We would find that measuring opinion about smoking would not help us answer the relationship between smoking and life expectancy and so it would not be a valid measure.
For some units, such as a meter or a second, there is a clear definition. And while that definition evolves over time (Mitchell 2022) it is widely agreed on. But for other aspects that we may wish to measure it is less clear and so the validity of the measurement becomes critical. At one point in the fourteenth century attempts were made to measure grace and virtue (Crosby 1997, 14)! More recently, we try to measure intelligence or even the quality of a university. That is not to say there are not people with more or less grace, virtue, and intelligence, than others, and there are certainly better and worse universities. But the measurement of these is problematic.
U.S. News tries to quantify university quality based on aspects such as class size, number of faculty with a PhD, number of faculty who are full-time, and student-faculty ratio. But an issue with such constructed measures, especially in social settings, is that it changes the incentives of those being measured. For instance, over the past 20 years, Columbia University has increased in this ranking from 18 in 1988 to 2 in 2022. But Michael Thaddeus, Professor of Mathematics, Columbia University, showed how there was a difference, in Columbia’s favor, between what Columbia reported to U.S. News and what was available through other sources (Hartocollis 2022).
Such concerns are of special importance in psychology because there is no clear measure of many fundamental concepts. Fried, Flake, and Robinaugh (2022) review the measurement of depression and find many concerns including a lack of validity and reliability. This is not to say that we should not try to measure such things, but we must ensure transparency about measurement decisions. For instance, Flake and Fried (2020) recommend answering various questions, designed to bring clarity, whenever measurements have to be constructed. These include specifying the underlying construct of interest, the decision process that led to the measure and what alternatives were considered, the quantification process, and detailing the scale. These questions are especially important when the measure is being constructed for a particular purpose, rather than being adopted from elsewhere, because of the concern that the measure will be constructed in a way that provides a pre-ordained outcome.
Reliability draws on what is implied by the part of the definition of measurement that reads “process of experimentally obtaining…”. It implies some degree of consistency and means that multiple measurements of one particular aspect, at one particular time, should be, ceteris paribus, essentially the same. If two enumerators independently count the number of shops on a street, then we would hope that their counts are the same. And if they were different then we would hope we could understand the reason for the difference. For instance, perhaps one enumerator, misunderstood the instructions and incorrectly considered only shops that were open. To consider another example, demographers are often concerned with the migration of people between countries, and economists are often concerned with international trade. It is concerning the number of times that the in-migration or imports data of Country A from Country B, do not match the out-migration or exports data of Country B to Country A.
Oh, you think we have good data on that!
It is common for the pilot of a plane to announce the altitude to their passengers. But the notion and measurement of altitude is deceptively complicated, and underscores the fact that measurement occurs within a broader context (Vanhoenacker 2015). For instance, if we are interested in how many meters there are between the plane and the ground, then should we measure the difference between the ground and where the pilot is sitting, which would be useful for the announcement, or to the bottom of the wheels, which would be useful for landing? What happens if we go over a mountain? Even if the plane has not descended, such a measure–the number of meters between the plane and the ground–would claim a reduction in altitude and make it hard to vertically separate planes. So we may be interested in a comparison to sea level. But sea level changes because of the tide, and is different at different locations. As such, a common measure of altitude is flight level, which is determined by the amount of air pressure. And because air pressure is affected by weather, season, and location, the one flight level may be associated with very different numbers of meters to the ground over the course of a flight! Despite this, the measures of altitude used by planes serve their purpose of enabling relatively safe air travel.
Measurement error is the difference between the value we observe and the actual value. Sometimes it is possible to verify certain responses. If the difference is consistent between those responses that we can verify and those that we cannot, then we are able to estimate the extent of overall measurement error. For instance, Sakshaug, Yan, and Tourangeau (2010) considered a survey of university alumni and compared responses about a respondent’s grades, with university records. They find that the mode of the survey (telephone interview conducted by a human, telephone interview conducted by a computer, and an internet survey) affected the extent of the measurement error.
Such error can be particularly pervasive when an enumerator fills out the survey form on behalf of the respondent. And this especially manifests itself when there are racial differences. For instance, Davis (1997, 177) describes how Black people in the US may limit the extent to which they describe their political and racial belief to White interviewers.
Another example is censored data, which is when we have some partial knowledge of the actual value. Right-censored data is when we know that the actual value is above than some observed value, but we do not know by how much. For instance, immediately following the Chernobyl disaster, the only available instruments to measure radiation had a certain maximum limit. When the radiation was measured to that (maximum) level, it was considered manageable, even though the implication was that the actual value was much higher.
Right-censored data are often seen in medical studies. For instance, say some experiment is conducted, and then patients are followed for 10 years. At the end of that 10-year period all we know is whether a patient lived at least 10 years, not the exact length of their life. Left-censored data is the opposite situation. For instance, consider a thermometer that only went down to freezing. Even when the actual temperature was less than that, the thermometer would still register as freezing. A slight variation of censored data is winsorizing data. This occurs when we do observe the actual value, but we change it to a less extreme one. For instance, if we were considering age then we may change the age of anyone older than 100 to be 100. We may do this if we are worried that values that were too large would have too significant of an effect.
Truncated data is a slightly different situation in which we do not even record those values. For instance, consider a situation in which we were interested in the relationship between a child’s age and height. Our first question might be “what is your age?” and if it turns out the respondent is an adult, then we would not continue to ask height. Truncated data are especially closely related to selection bias. For instance, consider a student who drops a course – their opinion is not measured on course evaluations.
To illustrate the difference between these concepts, consider a situation in which the actual distribution of newborn baby weight has a normal distribution, centered around 3.5 kg. Imagine, there is some defect with the scale, such that any value less than or equal to 2.75kg is assigned 2.75kg. And imagine there is some rule such that any baby who would weigh more than 4.25kg is transferred to a different hospital to be born. These three scenarios are illustrated in Figure 8.1. We may also be interested in considering the mean weight, which highlights the bias (Table 8.1).
Table 8.1: Comparing the means of the different scenarios identifies the bias
Measurement
Mean
Actual
3.521
Censored
3.530
Truncated
3.455
Regardless of how good our data acquisition process is, there will be missing data. When we talk of this it is important to remember that, in a sense, a variable has to be measured, or at least thought about and considered, in order to be missing. This is why it is so important to really think about the situation, and work with subject-matter experts. Non-response could be considered a variant of measurement error whereby we observe a null, even though there must be an actual value. But it is usually considered in its own right. And there are different extents of non-response: from refusing to even respond to the survey, through to just missing one question. Non-response is a key issue, especially with non-probability samples, because there is usually good reason to consider that people who do not respond are systematically different to those who do. And this serves to limit the extent to which the survey can be used to speak to more than just survey respondents. For instance, Gelman et al. (2016) go so far as to say that much of the changes in public opinion that are reported in the lead-up to an election are not people changing their mind, but differential non-response. The use of pre-notification and reminders may help address non-response in some circumstances (Koitsalu et al. 2018; Frandell et al. 2021).
Data might be missing because a respondent did not want to respond to one particular question, a particular collection of related questions, or the entire survey, although these are not mutually exclusive nor collectively exhaustive (Newman 2014). In an ideal situation data are missing completely at random. This rarely occurs, but if it does, then the missing data can just be ignored with little concern because the sample should still be reflective of the broader population. It is more likely that data are missing at random and the extent to which we have to worry about that differs. For instance, if we are interested in the effect of gender on political support, then it may be that men are less likely to respond to surveys, but this is not related to who they will support. If that differential response is only due to being a man, and not related to political support, then we may be able to continue, provided we include gender in the regression, or are able to post-stratify based on gender. That said, the likelihood of this independence holding is low, and it is more likely, as in Gelman et al. (2016), that there is a relationship between responding to the survey and political support. In that, more likely, case we have a more significant issue. One approach would be to consider additional explanatory variables. It is tempting to drop incomplete cases, but this may further bias the sample, and requires extensive justification. Data imputation should be considered only as a last resort, and again is likely to further bias the sample. Far better than either option is to rethink, and improve, the data collection process.
8.3 Censuses and other government data
There are a variety of sources of data that have been produced for the purposes of being used as datasets. One thinks here especially of censuses of population. Whitby (2020, 30–31) describes how the earliest censuses that we have written record of are from China’s Yellow River valley. One motivation for censuses was taxation, and Jones (1953) describes census records from the late third or early fourth century A.D. which enabled a new system of taxation. Detailed records, such as censuses, have also been abused. For instance, Luebke and Milton (1994) sets out how the Nazis used censuses and police registration for data gathering to support their eugenic agenda. The Allies also used census records, with Bowen (2022, 17) describing how the US Census Bureau provided information that contributed to the internship of Japanese Americans; an event for which President Clinton would later apologize.
Another source of data deliberately put together to be a dataset include official statistics like surveys of economic conditions such as unemployment, inflation, and GDP. Interestingly, Rockoff (2019) describes how these economic statistics were not actually developed by the federal government, even though governments typically eventually took over that role. Censuses and other government-run surveys have the power, and financial resources, of the state behind them, which enables them to be thorough in a way that other datasets cannot be. For instance, the 2020 US Census is estimated to cost US$15.6 billion (Hawes 2020). But this similarly brings a specific perspective. Census data are not unimpeachable, and common errors include under- and over-enumeration, as well as misreporting (Steckel 1991) and there are various measures and approaches used to assess quality (Statistics Canada 2017).
Oh, you think we have good data on that!
Censuses of population are critical, but not unimpeachable. Anderson and Fienberg (1999) describe how the history of the US census is one of undercount, and that even George Washington complained about this in the 1790s. The extent of the undercount was able to be estimated due to the selective service registration system used for conscription in World War II. These records were compared with census records, and it was found that there were about half a million more men recorded for conscription purposes than in the census. This was race-specific, with an average undercount of around 3 per cent, but an undercount of Black men of draft age of around 13 per cent (Anderson and Fienberg 1999, 29). This became a political issue in the 1960s, and race and ethnicity related questions were of special concern in the 1990s. Nobles (2002, 47) discusses how counting by race first requires that race exists, but that this may be biologically difficult to establish. And certainly not something that is “fixed” and “objective” but instead due to class, social legal, structural, and political constructs (Nobles 2002, 48), despite it being fundamental to the US census.
Another, similarly, large and established source of data are from long-running large surveys. These are conducted on a regular basis, and while not usually directly conducted by the government, they are usually funded, one way or another, by the government. For instance, here we often think of electoral surveys, such as the Canadian Election Study, which has run in association with every federal election since 1965, and similarly the British Election Study which has been associated with every general election since 1964. More recently there has been a large push toward open data in government. The underlying principle—that the government should make available the data that is has—is undeniable. However, the term has become a little contentious because of how it has occurred in practice; the government is only providing data that it wants to provide, and will not make it look bad. One excellent source of data that the government has, but does not necessarily want to provide is that gathered through Freedom of Information (FOI) request (Walby and Luscombe 2019). And these are used to excellent effect by Cardoso (2020), who shows systematic racism in Canadian prisons.
While farmed datasets have always been useful, they were developed for a time when much analysis was conducted without the use of scripts and programming languages. Many R packages have been developed to make it easier to get these datasets into R. In this chapter we cover a few that are especially useful.
8.3.1 Canada
The first census in Canada was conducted in 1666. This was also the first modern census, whereby every individual was recorded by name (Godfrey 1918, 179), although it is worth pointing out that it does not include Aboriginal peoples. There were 3,215 inhabitants that were counted, and the census asked about age, sex, marital status, and occupation (Statistics Canada 2017). In association with Confederation, in 1867 a decennial census was required so that political representatives could be allocated for the new Parliament. Regular censuses have occurred since then, the most recent in 2021.
We can explore some data on languages spoken in Canada from the 2016 Census using canlang(Timbers 2020). This package is not yet available on CRAN, and so we install it from GitHub, using devtools(Wickham, Hester, and Chang 2020).
We can quickly see the top-10 most common languages to have as mother tongue.
can_lang |>slice_max(mother_tongue, n =10) |>select(language, mother_tongue)
# A tibble: 10 × 2
language mother_tongue
<chr> <dbl>
1 English 19460850
2 French 7166700
3 Mandarin 592040
4 Cantonese 565270
5 Punjabi (Panjabi) 501680
6 Spanish 458850
7 Tagalog (Pilipino, Filipino) 431385
8 Arabic 419890
9 German 384040
10 Italian 375635
We could combine two datasets together “region_lang” and “region_data”, to see if the five most-common languages differ between the largest region, Toronto, and the smallest, Belleville.
region_lang |>left_join(region_data, by ="region") |>slice_max(c(population)) |>slice_max(mother_tongue, n =5) |>select(region, language, mother_tongue, population) |>mutate(prop = mother_tongue / population)
# A tibble: 5 × 5
region language mother_tongue population prop
<chr> <chr> <dbl> <dbl> <dbl>
1 Toronto English 3061820 5928040 0.516
2 Toronto Cantonese 247710 5928040 0.0418
3 Toronto Mandarin 227085 5928040 0.0383
4 Toronto Punjabi (Panjabi) 171225 5928040 0.0289
5 Toronto Italian 151415 5928040 0.0255
region_lang |>left_join(region_data, by ="region") |>slice_min(c(population)) |>slice_max(mother_tongue, n =5) |>select(region, language, mother_tongue, population) |>mutate(prop = mother_tongue / population)
# A tibble: 5 × 5
region language mother_tongue population prop
<chr> <chr> <dbl> <dbl> <dbl>
1 Belleville English 93655 103472 0.905
2 Belleville French 2675 103472 0.0259
3 Belleville German 635 103472 0.00614
4 Belleville Dutch 600 103472 0.00580
5 Belleville Spanish 350 103472 0.00338
We can see a considerable difference between the proportions, with a little over 50 per cent of those in Toronto having English as their mother tongue, compared with around 90 per cent of those in Belleville.
In general, data from Canadian censuses are not as easily available through the relevant government agency as in other countries, although IPUMS, which we discuss later, provides access to some. Statistics Canada, which is the government agency that is responsible for the census and other official statistics, freely provides an “Individuals File” from the 2016 census as a Public Use Microdata File (PUMF), but only in response to request. And while it is a 2.7 per cent sample from the 2016 census, this PUMF provides limited detail.
Another way to access data from the Canadian census is to use cancensus(von Bergmann, Shkolnik, and Jacobs 2021). This package can be installed from CRAN. It requires an API key, which can be requested by creating an account and then going to “edit profile”. The package has a helper function that makes it easier to add the API key to an “.Renviron” file, which we will explain in more detail in Chapter 9.
We can use get_census() to get census data. We need to specify a census of interest, and a variety of other arguments. For instance, we could get data from the 2016 census about Ontario, which is the largest Canadian province by population.
Data from the 1996, 2001, 2006, 2011, and 2016 censuses are available, and list_census_datasets() provides the metadata that we need to provide to get_census() to access these. Data are available based on a variety of regions, and list_census_regions() provides the metadata that we need. And finally, list_census_vectors() provides the metadata about the variables that available.
8.3.2 USA
The requirement for a US Census is included in the US Constitution, although births and deaths were legally required to be registered in what became Massachusetts as early as 1639 (Gutman 1958). We can use tidycensus to get started with access to US census data (Walker and Herman 2022). As with cancensus, we first need to obtain an API key from the Census Bureau API and store it locally using a helper function.
Having set that up, we can use get_decennial() to obtain data on variables of interest. As an example, we could gather data about the average household size, in 2010, overall, and by whether owner or renter (Figure 8.2).
Figure 8.2: Comparing average household size in DC, Utah and Massachusetts, by household type
The definitive book about analyzing US census data with R is Walker (2022).
The US is in the enviable situation where there is usually a better approach than using the census and there is a better way than having to use government statistical agency websites. IPUMS provides access to a wide range of datasets, including international census microdata. In the specific case of the US, the American Community Survey (ACS) is a survey whose content is comparable to the questions asked on many censuses, but it is available on an annual basis, compared with a census which could be quite out-of-date by the time the data are available. It ends up with millions of responses each year. Although the ACS is smaller than a census, the advantage is that it is available on a more timely basis. We access the ACS through IPUMS.
Shoulders of giants
Steven Ruggles is Regents Professor of History and Population Studies at the University of Minnesota and is in charge of IPUMS. After taking a PhD in historical demography from the University of Pennsylvania in 1984, he was appointed as an assistant professor at the University of Minnesota, and promoted to full professor in 1995. The initial IPUMS data release was in 1993 (Sobek and Ruggles 1999). Since then it has grown extensively and it now provides integrated social and economic data from across the world.
Go to IPUMS, then “IPUMS USA”, and select “Get Data”. We are interested in a sample, so go to “SELECT SAMPLE”, and un-select “Default sample from each year” and instead select “2019 ACS” and then “SUBMIT SAMPLE SELECTIONS” (Figure 14.21 (a)).
(a) Selecting a sample from IPUMS USA and specifying interest in the 2019 ACS
(b) Specifying that we are interested in the state
(c) Adding STATEICP to the cart
(d) Adding additional demographic variables that are available on an individual basis
(e) Beginning the checkout process
(f) Specifying that we are interested in CSV files
(g) Reducing the sample size from three million responses to half a million
Figure 8.3: An overview of the steps involved in getting data from IPUMS
We might be interested in data based on state. So we would begin by looking at “HOUSEHOLD” variables and selecting “GEOGRAPHIC” (Figure 14.21 (b)).
We add “STATEICP” to our “cart” by clicking the plus, which will then turn into a tick (Figure 14.21 (c)).
We might then be interested in data on a “PERSON” basis, for instance, “DEMOGRAPHIC” variables such as “AGE”, which we should add to our cart. Still on a “PERSON” basis, we might be interested in “INCOME”, for instance, “Total personal income” “INCTOT” and we could add that to our cart (Figure 14.21 (d)).
When we are done, we can “VIEW CART”, and then “CREATE DATA EXTRACT” (Figure 14.21 (e)). At this point there are two aspects that we likely want to change:
Customize the sample size as we likely do not need three million responses, and could just change it to, say, 500,000 (Figure 14.21 (g)).
Finally, we want to include a descriptive name for the extract, for instance, “2022-02-06: Income based on state and age”, which specifies the date we made the extract and what is in the extract. After that we can “SUBMIT EXTRACT”.
We will be asked to log in or create an account, and after doing that will be able to submit the request. IPUMS will email when the extract is available, after which we can download it and read it into R in the usual way. It is critical that we cite this dataset when we use it. For instance we can use the following bibtex entry for Ruggles et al. (2021).
@misc{ipumsusa,
author = {Ruggles, Steven and Flood, Sarah and Foster, Sophia and Goeken, Ronald and Pacas, Jose and Schouweiler, Megan and Sobek, Matthew},
year = 2021,
title = {IPUMS USA: Version 11.0},
publisher = {Minneapolis, MN: IPUMS},
doi = {10.18128/d010.v11.0},
url = {https://usa.ipums.org},
language = {en},
}
Incredibly, full count data, that is the entire census, are available through IPUMS for the US censuses conducted between 1850 and 1940, with the exception of 1890. Most of the 1890 census records were destroyed due to a fire in 1921. One per cent samples are available for all censuses through to 1990. And ACS data are available from 2000. We return to the ACS in Chapter 12 in relation to discussing synthetic data and differential privacy.
8.4 Sampling essentials
Statistics is at the heart of telling stories with data because it is almost never possible to get all the data that we would like. Statisticians have spent considerable time and effort thinking about the properties that various samples of data will have and how they enable us to speak to implications for the broader population.
Let us say that we have some data. For instance, a particular toddler goes to sleep at 6:00pm every night. We might be interested to know whether that bedtime is common among all toddlers, or if we have an unusual toddler. We only have one toddler so our ability to use his bedtime to speak about all toddlers is limited.
One approach would be to talk to friends who also have toddlers. And then talk to friends-of-friends. How many friends, and friends-of-friends, do we have to ask because we can begin to feel comfortable that we can speak about some underlying truth of toddler bedtime?
Wu and Thompson (2020, 3) describe statistics as “the science of how to collect and analyze data and draw statements and conclusions about unknown populations.” Here “population” is used in a statistical sense and refers to some infinite group that we can never know exactly, but that we can use the probability distributions of random variables to describe the characteristics of. We discuss probability distributions in more detail in Chapter 14. Fisher (1928, 41) goes further and says:
[t]he idea of an infinite population distributed in a frequency distribution in respect of one or more characters is fundamental to all statistical work. From a limited experience,… we may obtain some idea of the infinite hypothetical population from which our sample is drawn, and so of the probable nature of future samples to which our conclusions are to be applied.
Another way to say this is that statistics involves getting some data and trying to say something sensible based on it even though we can never have all of the data.
Three pieces of critical terminology are:
“Target population”: The collection of all items about which we would like to speak.
“Sampling frame”: A list of all the items from the target population that we could get data about.
“Sample”: The items from the sampling frame that we get data about.
A target population is a finite set of labelled items, of size \(N\). For instance, in theory we could add a label to all the books in the world: “Book 1”, “Book 2”, “Book 3”, …, “Book \(N\)”. There is a difference between use of the term population here, and that of everyday usage. For instance, one sometimes hears those who work with census data say that they do not need to worry about sampling because they have the whole population of the country. This is a conflation of the terms, as what they have is the sample gathered by the census of the population of a country. While the goal of a census is to get every unit, and so if this was achieved then sampling error would not be an issue, there are many other issues. Even if a census was done perfectly and we got data about every unit in the target population, there are still issues, for instance due to measurement error, and it being a sample at a particular time, and Groves and Lyberg (2010) provides an interesting discussion of the evolution of total survey error.
In the same way that we saw how difficult it can be define what to measure, it can be difficult to define a target population. For instance, say we have been asked to find out about the consumption habits of university students. How can we define that target population? If someone is a student, but also works full time, then are they in the population? What about mature-age students? Some aspects that we might be interested in are formally defined to an extent that is not always commonly realized. For instance, whether an area is classified as urban or rural is often formally defined by a country’s statistical agency. But other aspects are less clear. For instance, Andrew Gelman discusses the difficulty of how do we classify someone as a “smoker”? If a 15-year-old has had 100 cigarettes over their lifetime, then we need to treat them differently than if they have had none. But if a 90-year-old has had 100 cigarettes over their lifetime, then are they likely different to a 90-year-old who has had none? At what age, and number of cigarettes, do these answers change?
Consider if we want to speak to the titles of all the books ever written. Our target population is all books ever written. But it is almost impossible for us to imagine that we could get information about the title of a book that was written in the nineteenth century, but that the author locked in their desk and never told anyone about. One sampling frame could be all books in the Library of Congress Online Catalog, another could be the 25 million that were digitized by Google (Somers 2017). And then finally, our sample may be the tens of thousands that are available through Project Gutenberg, and that we can access using gutenbergr(Robinson 2021).
To consider another example, consider wanting to speak of the attitudes of all Brazilians who live in Germany. The target population is all Brazilians who live in Germany. One possible source of information would be Facebook and so in that case, the sampling frame might be all Brazilians who live in Germany who have Facebook. And then our sample be might all Brazilians who live in Germany who have Facebook who we can gather data about. The target population and the sampling frame will be different because not all Brazilians who live in Germany will have Facebook. And the sampling frame will be different to the sample because we will likely not be able to gather data about all Brazilians who live in Germany and have Facebook.
8.4.1 Sampling in Dublin and Reading
To be clearer, we consider two examples: a 1798 count of the number of inhabitants of Dublin, Ireland (Whitelaw 1805), and a 1912 count of working-class households in Reading, England (Bowley 1913).
In 1798 the Reverend James Whitelaw conducted a survey of Dublin, Ireland, to count its population. Whitelaw (1805) describes how population estimates at the time were something of a curate’s egg. For instance, the estimated size of London at the time ranged from 128,570 to 300,000. Whitelaw expected that the Lord Mayor of Dublin could compel the person in charge of each house to affix a list of the inhabitants of that house to the door, and then Whitelaw could simply use this.
Instead, he found that the lists were “frequently illegible, and generally short of the actual number by a third, or even one-half”. And so instead he recruited assistants, and they went door-to-door making their own counts. The resulting estimates are particularly informative (Figure 8.4). And the total population of Dublin in 1798 was estimated at 182,370.
Figure 8.4: Extract of the results that Whitelaw found in 1798
One aspect worth noticing is that Whitelaw includes information about class. It is difficult to know how that was determined, but it played a large role in the data collection. Whitelaw describes how the houses of “the middle and upper classes always contained some individual who was competent to the task [of making a list]”. But that “among the lower class, which forms the great mass of the population of this city, the case was very different”. It is difficult to know how Whitelaw could have known that the upper and middle classes were not representing their number, while the lower class was. It is also difficult to imagine Whitelaw going into the houses of the upper class and counting their number, as he and his assistants did for the lower classes, so there may have been some data truncation. As always, the issue of defining the target population is a difficult one, and it seems that there may have been slightly different approaches to each class.
Whitelaw attempted to construct a full sample of the inhabitants of Dublin without using much in the way of statistical machinery to guide his choices. We will now consider a second example, conducted in 1912, where they were able to start to use sampling approaches that we still use today.
A little over one hundred years after Whitelaw (1805), Bowley (1913) was interested in counting the number of working-class households in Reading, England. Bowley selected the sample using the following procedure (Bowley 1913, 672):
One building in ten was marked throughout the local directory in alphabetical order of streets, making about 1,950 in all. Of those about 300 were marked as shops, factories, institutions and non-residential buildings, and about 300 were found to be indexed among Principal Residents, and were so marked. The remaining 1,350 were working-class houses… [I]t was decided to take only one house in 20, rejecting the incomplete information as to the intermediate tenths. The visitors were instructed never to substitute another house for that marked, however difficult it proved to get information, or whatever the type of house.
Bowley (1913) says that they were able to obtain information about 622 working-class households. For instance, they are able to estimate how much rent is paid each week (Figure 8.5).
Figure 8.5: Extract of the results that Bowley found about rent paid by the working-class in Reading, England
Then, having judged, based on the census that there were about 18,000 households in Reading, Bowley (1913) applies a multiplier of twenty-one to the sample, resulting in estimates for Reading overall. The key aspect that ensures the resulting estimates are reasonable is that the sampling was done in a random way. This is why Bowley (1913) was so insistent that the visitors go to the actual house that was selected, and not substitute it for another.
8.4.2 Probabilistic sampling
Having identified a target population and a sampling frame, we need to distinguish between probability and non-probability sampling:
“Probability sampling”: Every unit in the sampling frame has some, known, chance of being sampled and the specific sample is obtained randomly based on these chances. The chance of being sampled does not necessarily need to be same for each unit.
“Non-probability sampling”: Units from the sampling frame are sampled based on convenience, quotas, judgement, or other non-random processes.
Often the difference between probability and non-probability sampling is one of degree. For instance, we usually cannot forcibly obtain data, and so there is almost always an aspect of volunteering on the part of a respondent. Even when there are penalties for not providing data, such as the case for completing a census form in many countries, it is difficult for even a government to force people to fill it out completely or truthfully. The most important aspect to be clear about with probability sampling is the role of uncertainty. This allows us to make claims about the population, based on our sample, with known amounts of error. The trade-off is that probability sampling is often expensive and difficult.
We will consider four types of probability sampling:
simple random sampling;
systematic sampling;
stratified sampling; and
cluster sampling.
To add some more specificity to our discussion, in a way that is also used by Lohr (2022, 27), it may help to consider the numbers 1 to 100 as our target population. With simple random sampling, every unit has the same chance of being included. In this case let us say it is 20 per cent. That means we would expect to have around 20 units in our sample, or around 1 in 5 compared with our target population.
# A tibble: 2 × 2
simple_random_sampling n
<chr> <int>
1 Included 14
2 Not included 86
illustrative_sampling
# A tibble: 100 × 2
unit simple_random_sampling
<int> <chr>
1 1 Not included
2 2 Not included
3 3 Not included
4 4 Not included
5 5 Not included
6 6 Not included
7 7 Not included
8 8 Not included
9 9 Not included
10 10 Not included
# … with 90 more rows
With systematic sampling, as was used by Bowley (1913), we proceed by selecting some value, and we then sample every fifth unit to obtain a 20 percent sample. To begin, we randomly pick a starting point from units 1 to 5, say 3. And so sampling every fifth unit would mean looking at the third, the eighth, the thirteenth, and so on.
# A tibble: 2 × 2
systematic_sampling n
<chr> <int>
1 Included 20
2 Not included 80
illustrative_sampling
# A tibble: 100 × 3
unit simple_random_sampling systematic_sampling
<int> <chr> <chr>
1 1 Not included Included
2 2 Not included Not included
3 3 Not included Not included
4 4 Not included Not included
5 5 Not included Not included
6 6 Not included Included
7 7 Not included Not included
8 8 Not included Not included
9 9 Not included Not included
10 10 Not included Not included
# … with 90 more rows
When we consider our population, it will typically have some grouping. This may be as straight-forward as a country having states, provinces, counties, or statistical districts; a university having faculties and departments; and humans having age-groups. A stratified structure is one in which we can divide the population into mutually exclusive, and collectively exhaustive, sub-populations called “strata”.
We use stratification to help with the efficiency of sampling or with the balance of the survey. For instance, the population of the US is around 335 million, with around 40 million people in California and around half a million people in Wyoming. So even a survey of 10,000 responses would only expect to have 15 responses from Wyoming, which could make inference about Wyoming difficult. We could use stratification to ensure there are, say, 200 responses from each of the 50 US states. We could use random sampling within each state to select the person about whom data will be gathered.
In our case, we will stratify our illustration by considering that our strata are the 10s, that is, 1 to 10 is one stratum, 11 to 20 is another, and so on. We will use simple random sampling within these strata to select two units from each.
# A tibble: 2 × 2
stratified_sampling n
<chr> <int>
1 Included 20
2 Not included 80
illustrative_sampling
# A tibble: 100 × 4
unit simple_random_sampling systematic_sampling stratified_sampling
<int> <chr> <chr> <chr>
1 1 Not included Included Included
2 2 Not included Not included Not included
3 3 Not included Not included Not included
4 4 Not included Not included Not included
5 5 Not included Not included Not included
6 6 Not included Included Not included
7 7 Not included Not included Not included
8 8 Not included Not included Not included
9 9 Not included Not included Included
10 10 Not included Not included Not included
# … with 90 more rows
And finally, we can also take advantage of some clusters that may exist in our dataset. Like strata, clusters are collectively exhaustive and mutually exclusive. Our examples from earlier of states, departments, and age-groups remain valid as clusters. However, it is our intentions toward these groups that is different. Specifically, with cluster sampling, we do not intend to collect data from every cluster, whereas with stratified sampling we do. With stratified sampling we look at every stratum and conduct simple random sampling within each strata to select the sample. With cluster sampling we conduct simple random sampling to select clusters of interest. We can then either sample every unit in those selected clusters or use simple random sampling, within the selected clusters, to select units. That all said, this difference can become less clear in practice, especially ex post. Interestingly, Rose et al. (2006) gather mortality data for North Darfur, Sudan, in 2005. They find that both cluster and systematic sampling provide similar results, and they point out that systematic sampling require less training of the survey teams. In general, cluster sampling can be cheaper because of the focus on geographically close locations.
In our case, we will cluster our illustration again based on the 10s. We will use simple random sampling to select two clusters for which we will use the entire cluster.
# A tibble: 2 × 2
cluster_sampling n
<chr> <int>
1 Included 20
2 Not included 80
illustrative_sampling
# A tibble: 100 × 5
unit simple_random_sampling systematic_sampling stratified_sampling cluste…¹
<int> <chr> <chr> <chr> <chr>
1 1 Not included Included Included Included
2 2 Not included Not included Not included Included
3 3 Not included Not included Not included Included
4 4 Not included Not included Not included Included
5 5 Not included Not included Not included Included
6 6 Not included Included Not included Included
7 7 Not included Not included Not included Included
8 8 Not included Not included Not included Included
9 9 Not included Not included Included Included
10 10 Not included Not included Not included Included
# … with 90 more rows, and abbreviated variable name ¹cluster_sampling
At this point we can illustrate the differences between our approaches (Figure 8.6). We could also consider it visually, by pretending that we randomly sample using the different methods from different parts of the world (Figure 8.7).
Figure 8.6: Illustrative example of simple random sampling, systematic sampling, stratified sampling, and cluster sampling over the numbers from 1 to 100
(a) The world
(b) Systematic sampling
(c) Stratified sampling
(d) Cluster sampling
Figure 8.7: Illustrative example of simple random sampling, systematic sampling, stratified sampling, and cluster sampling across different parts of the world
The best way to appreciate the differences between these approaches is to consider them in practice and Au (2022) provides a number of examples. In particular, Fuller and Mosher (1987) provides a fascinating comparison of simple random sampling, stratified sampling, systematic sampling and cluster sampling, as well as additional considerations, in the context of counting raptors.
Having established our sample, we typically want to use it to make claims about the population. Neyman (1934, 561) goes further and says that “…the problem of the representative method is par excellence the problem of statistical estimation. We are interested in characteristics of a certain population, say \(\pi\), which it is either impossible or at least very difficult to study in detail, and we try to estimate these characteristics basing our judgment on the sample.”
In particular, we would typically be interested to estimate a population mean and variance. We introduced the idea of estimators, estimands, and estimates, in Chapter 5. We can construct an estimator to estimate the population mean and variance. For instance, if we were using simple random sampling with a sample of size \(n\), then the sample mean and variance could be constructed to produce estimates of the population mean and variance:
We can use the approaches that we have used so far to simulate various types of survey designs. There are also packages that can help, including DeclareDesign(Blair et al. 2019) and survey(Lumley 2020).
Scaling up estimates can be used when we are interested in using a count from our sample to imply some total count for the target population. We saw this in Bowley (1913) where the ratio of the number of households in the sample, compared with the number of households known from the census, was 21 and this information was used to scale up the sample.
To consider an example, perhaps we were interested in the sum of the numbers from 1 to 100. We know that our samples are of size 20, and so need to be scaled up five times (Table 8.2).
Table 8.2: Sum of the numbers in each sample, and implied sum of population
Sampling method
Sum of sample
Implied population sum
simple_random_sampling
840
4,200
systematic_sampling
970
4,850
stratified_sampling
979
4,895
cluster_sampling
910
4,550
The actual sum of the population is 5,050. We can obtain this using a trick attributed to Leonhard Euler, the eighteenth-century mathematician, who noticed that the sum of 1 to any number can be quickly obtained by finding the middle number and then multiplying that by one plus the number. So, in this case, we have \(50 \times 101\). Alternatively we can use R: sum(1:100).
While the specifics are unique to this sample, our estimate of the population sum, based on the scaling, are especially revealing. The closest is stratified sample, closely followed by systematic sampling. Cluster sampling is a little over 10 per cent off, while simple random sampling is a little further away. To get close, it is important that our sampling method gets as many of the higher values as possible. And so stratified and systematic sampling, both of which ensured that we had unit from the larger numbers did particularly well. The performance of cluster and simple random sampling would depend on the particular clusters, and units, selected. In this case, stratified and systematic sampling ensured that our estimate of the sum of the population, would not be too far away from the actual population sum. Here, we might think of implications for the construction and evaluation of measures, such as GDP and other constructions that are summed, and the effect on the total of the different strata based on their size.
This approach has a long history. For instance, Stigler (1986, 163) describes how by 1826 Adolphe Quetelet, the nineteenth century astronomer, had become involved in the statistical bureau, which was planning for a census. Quetelet argued that births and deaths were well known, but migration was not. He proposed an approach based on counts in specific geographies, which could then be scaled up to the whole country. The criticism of the plan focused on the difficulty of selecting appropriate geographies, which we saw also in our example of cluster sampling. The criticism was reasonable, and even today, some two hundred years later, something that we should keep front of mind, (Stigler 1986):
He [Quetelet] was acutely aware of the infinite number of factors that could affect the quantities he wished to measure, and he lacked the information that could tell him which were indeed important. He… was reluctant to group together as homogenous, data that he had reason to believe was not… To be aware of a myriad of potentially important factors, without knowing which are truly important and how their effect may be felt, is often to fear the worst… He [Quetelet] could not bring himself to treat large regions as homogeneous, [and so] he could not think of a single rate as applying to a large area.
We are able to do this scaling up when we know the population total, but if we do not know that, or we have concerns around the precision of that approach then we may use a ratio estimator.
Ratio estimators were used in 1802 by Pierre-Simon Laplace to estimate the total population of France, based on the ratio of the number of registered births, which was known throughout the country, to the number of inhabitants, which was only know for certain communes. He calculated this ratio for the three communes, and then scaled it, based on knowing the number of births across the whole country to produce an estimate of the population of France (Lohr 2022).
A ratio estimator of some population parameter is the ratio of two means. For instance, imagine that we knew the total number of hours that a toddler slept for a 30 day period, and we want to know how many hours the parents slept over that same period. We may have some information on the number of hours that a toddler sleeps overnight, \(x\), and the number of hours their parents sleep overnight \(y\) over a 30 day period.
And given the toddler slept 185 hours over that 30-day period, our estimate of the number of hours that the parents slept is \(185 \times 0.8 = 148\). This turns out to be almost exactly right, as the sum is 147. In this example, the estimate was not needed because we were able to sum the data, but say some other set of parents only recorded the number of hours that their toddler slept, not how long they slept, then we could use this to estimate how much they had slept.
One variant of the ratio estimator, that is commonly used is capture and recapture, which is one of the crown jewels of data gathering. It is commonly used in ecology where we know we can never gather data about all animals. Instead a sample is captured, marked, and released. The research return after some time to capture another sample. Assuming enough time passes that the initially captured animals had time to integrate back into the population, but not so much time has passed that there are concerns around births, deaths and migration, then we can use these values to estimate a population size. The key is what proportion in this second sample have been recaptured. This proportion can be used to estimate the size of the whole population.
8.4.3 Non-probability samples
While acknowledging that it is a spectrum, much of statistics was developed based on probability sampling. But a considerable amount of modern sampling is done using non-probability sampling. One approach is to use social media and other advertisements to recruit a panel of respondents, possibly in exchange for compensation. This panel is then the group that is sent various surveys as necessary. But think for a moment about the implications of this. For instance, what type of people are likely to respond to such an advertisement? Is the richest person in the world likely to respond? Are especially young or especially old people likely to respond? In some cases, it is possible to do a census. Nation-states typically do one every five to ten years. But there is a reason that it is only nation states that do them—they are expensive, time-consuming, and surprisingly, they are sometimes not as accurate as we may hope because of how general they need to be.
Non-probability samples have an important role to play because they are typically cheaper and quicker to obtain than probability samples. They have been found to be less accurate (Yeager et al. 2011). But Beaumont (2020) further describes a variety of factors in favor of non-probability samples including declining response rates to probability samples, and increased demand for real-time statistics. Further, as we have discussed, the difference between probability and non-probability samples is sometimes one of degree, rather than dichotomy. In any case, non-probability samples are legitimate and appropriate for some tasks provided one is clear about the trade-offs and ensure transparency (Baker et al. 2013).
Convenience sampling involves gathering data from a sample that is easy to access. For instance, one often asks one’s friends and family to fill out a survey as a way of testing it before more wide-scale distribution. If instead we were to analyze such a sample, then we would likely be using convenience sampling.
The main issue with convenience sampling is that it is unlikely to be able to speak to much of a broader population than those who filled out the survey. There are also tricky ethical considerations, and typically a lack of anonymity which may further bias the results. On the other hand, it can be useful to cheaply get a quick sense of a situation while rolling out sampling approaches likely to be more broadly useful.
Quota sampling occurs when we have strata, but we do not use random sampling within those strata to select the unit. For instance, if we again stratified the US based on state, but then instead of ensuring that everyone in Wyoming had the chance to be chosen for that stratum, just picked people at Jackson Hole. There are some advantages to this approach, especially in terms of speed and cost, but the resulting sample may be biased in various ways. That is not to say they are without merit. For instance, the Bank of Canada runs a non-probability survey focused on the method that is used to pay for goods and services. They use quota sampling, and various adjustment methods, and this use of non-probability sampling enables them to deliberately focus on hard-to-reach aspects of the population (H. Chen, Felt, and Henry 2018).
As the saying goes, birds of a feather flock together. And we can take advantage of that in our sampling. Although Handcock and Gile (2011) describe various uses before this, and it is notoriously difficult to define attribution in multidisciplinary work, snowball sampling is nicely defined by Goodman (1961). Following Goodman (1961), to conduct snowball sampling, we first draw a random sample from the sampling frame. Each of these is asked to name \(k\) others also in the sample population, but not in that initial draw, and these form the “first stage”. Each individual in the first stage is then similarly asked to name \(k\) others who are also in the sample population, but again not in the random draw or the first stage, and these form the “second stage”. We need to have specified the number of stages, \(s\), and also \(k\) ahead of time.
Respondent-driven sampling was developed by Heckathorn (1997) to focus on hidden populations, which are those for which:
there is no sampling frame; and
being known to be in the sampling population could have a negative effect.
For instance, we could imagine various countries in which it would be difficult to sample from the gay population or those who have had abortions because it is illegal. Respondent-driven sampling differs from snowball sampling in two ways:
In addition to compensation for their own response, as is the case with snowball sampling, respondent-driven sampling typically also involves compensation for recruiting others.
Respondents are not asked to provide information about others to the investigator, but instead recruit them into the study. Selection into the sample occurs not from sampling frame, but instead from the networks of those already in the sample (Salganik and Heckathorn 2004).
8.5 Exercises and tutorial
Exercises
(Plan) Consider the following scenario: Five friends—Ash, Jacki, Matt, Mike, and Rol–each measure the height of 20 of their friends. Each of the five use a slightly different approach to measurement and so make slightly different errors. Please sketch what that dataset could look like and then sketch a graph that you could build to show all observations.
(Simulate) Please further consider the scenario described and simulate the situation with every variable independent of each other.
(Acquire) Please describe a possible source of such a dataset.
(Explore) Please use ggplot2 to build the graph that you sketched using the data that you simulated.
(Communicate) Please write two paragraphs about what you did.
Please identify three other sources of data that you are interested in and describe where they are available (please include a link or code).
Please focus on one of those sources. What steps do you have to go through in order to get a dataset that can be analyzed in R?
In a paragraph or two, and using your own words, please define measurement error and provide an example from your own experience.
Let us say you take a job at RBC (a Canadian bank) and they already have some quantitative data for you to use. What are some questions that you should explore when deciding whether that data will be useful to you?
Write three points (you are welcome to use dot points) about why government data may be especially useful.
Please pick a government of interest and find their inflation statistics. To what extent do you know about how these data were gathered?
With reference to W. Chen et al. (2019) and Martı́nez (2022) to what extent do you think we can trust government statistics? Please mention at least three governments in your answer.
The 2021 census in Canada asked, firstly, “What was this person’s sex at birth? Sex refers to sex assigned at birth. Male/Female”, and then “What is this person’s gender? Refers to current gender which may be different from sex assigned at birth and may be different from what is indicated on legal documents. Male/Female/Or please specify this person’s gender (space for a typed or handwritten answer)”. With reference to Statistics Canada (2020), please discuss the extent to which you think this is an appropriate way for census to have proceeded. You are welcome to discuss the case of a different country if you are more familiar with that.
Pretend that we have conducted a survey of everyone in Canada, where we asked for age, sex, and gender. Your friend claims that there is no need to worry about uncertainty “because we have the whole population”. Is your friend right or wrong, and why?
Please consider Kennedy et al. (2020). What are some statistical facets when considering a survey focused on gender, with a post-stratification survey that is not (select all)?
Impute all non-male as female
Estimate gender using auxiliary information
Impute population
Impute sample values
Model population distribution using auxiliary data
Remove all non-binary respondents
Remove respondents
Assume population distribution
Please consider Kennedy et al. (2020). What are some ethical facets when considering a survey focused on gender, with a post-stratification survey that is not (select all)?
Impute all non-male as female
Estimate gender using auxiliary information
Impute population
Impute sample values
Model population distribution using auxiliary data
Remove all non-binary respondents
Remove respondents
Assume population distribution
Please consider Kennedy et al. (2020). How do they define ethics?
Respecting the perspectives and dignity of individual survey respondents.
Generating estimates of the general population and for subpopulations of interest.
Using more complicated procedures only when they serve some useful function.
With reference to the Introduction of Beaumont (2020) do you think that probability surveys will disappear, and why or why not (please write a paragraph or two)?
Please use IPUMS to access the 2020 ACS. Making use of the codebook, how many respondents were there in California (STATEICP) that had a Doctoral degree as their highest educational attainment (EDUC)?
4,684
5,765
2,007
732
Please use IPUMS to access the 1940 1% sample. Making use of the codebook, how many respondents were there in California (STATEICP) with 5+ years of college as their highest educational attainment (EDUC)?
1,789
1,056
532
904
What are some types of probability sampling, and in what circumstances might you want to implement them (write two or three pages)?
There have been some substantial political polling “misses” in recent years (Trump and Brexit come to mind). To what extent do you think non-response bias was the cause of this (write a page or two, being sure to ground your writing with citations)?
It seems like a lot of businesses have closed since the pandemic. To investigate this, we walk along some blocks downtown and count the number of businesses that are closed and open. To decide which blocks to walk, we open a map, start at the lake, and then pick every 10th street. This type of sampling is (pick one)?
Cluster sampling.
Systematic sampling.
Stratified sampling.
Convenience sampling.
Please name some reasons why you may wish to use cluster sampling (select all)?
Balance in responses.
Administrative convenience.
Efficiency in terms of money.
Underlying systematic concerns.
Estimation of sub-populations.
Write R code that considers the numbers 1 to 100, and estimates the mean, based on a cluster sample of 20 numbers. Re-run this code one hundred times, noting the estimate of the mean each time, and then plot the histogram. What do you notice about the graph? Add a paragraph of explanation and discussion.
Tutorial
With reference to Dean (2022), please discuss the difference between probability and non-probability sampling, and the implications of each. Strong answers would compare actual data, for instance, the UK Coronavirus (COVID-19) Infection Survey with self-reported data. Please write at least two pages.
Achen, Christopher. 1978. “Measuring Representation.”American Journal of Political Science 22 (3): 475–510. https://doi.org/10.2307/2110458.
Baker, Reg, Michael Brick, Nancy Bates, Mike Battaglia, Mick Couper, Jill Dever, Krista Gile, and Roger Tourangeau. 2013. “Summary Report of the AAPOR Task Force on Non-Probability Sampling.”Journal of Survey Statistics and Methodology 1 (2): 90–143. https://doi.org/10.1093/jssam/smt008.
Beaumont, Jean-Francois. 2020. “Are Probability Surveys Bound to Disappear for the Production of Official Statistics?”Survey Methodology 46 (1): 1–29.
Blair, Graeme, Jasper Cooper, Alexander Coppock, and Macartan Humphreys. 2019. “Declaring and Diagnosing Research Designs.”American Political Science Review 113 (3): 838–59. https://doi.org/10.1017/S0003055419000194.
Chen, Heng, Marie-Hélène Felt, and Christopher Henry. 2018. “2017 Methods-of-Payment Survey: Sample Calibration and Variance Estimation.” Bank of Canada. https://doi.org/10.34989/tr-114.
Chen, Wei, Xilu Chen, Chang-Tai Hsieh, and Zheng Song. 2019. “A Forensic Examination of China’s National Accounts.”Brookings Papers on Economic Activity, 77–127. https://www.jstor.org/stable/26798817.
Colombo, Tommaso, Holger Fröning, Pedro Javier Garcı̀a, and Wainer Vandelli. 2016. “Optimizing the Data-Collection Time of a Large-Scale Data-Acquisition System Through a Simulation Framework.”The Journal of Supercomputing 72 (12): 4546–72. https://doi.org/10.1007/s11227-016-1764-1.
Crawford, Kate. 2021. Atlas of AI. New Haven: Yale University Press.
Crosby, Alfred. 1997. The Measure of Reality: Quantification in Western Europe, 1250-1600. Cambridge: Cambridge University Press.
Davis, Darren. 1997. “Nonrandom Measurement Error and Race of Interviewer Effects Among African Americans.”The Public Opinion Quarterly 61 (1): 183–207. https://doi.org/10.1086/297792.
Fisher, Ronald. 1928. Statistical Methods for Research Workers. 2nd ed. London: Oliver; Boyd.
Flake, Jessica, and Eiko Fried. 2020. “Measurement Schmeasurement: Questionable Measurement Practices and How to Avoid Them.”Advances in Methods and Practices in Psychological Science 3 (4): 456–65. https://doi.org/10.1177/2515245920952393.
Frandell, Ashlee, Mary Feeney, Timothy Johnson, Eric Welch, Lesley Michalegko, and Heyjie Jung. 2021. “The Effects of Electronic Alert Letters for Internet Surveys of Academic Scientists.”Scientometrics 126 (8): 7167–81. https://doi.org/10.1007/s11192-021-04029-3.
Fried, Eiko, Jessica Flake, and Donald Robinaugh. 2022. “Revisiting the Theoretical and Methodological Foundations of Depression Measurement.”Nature Reviews Psychology 1 (6): 358–68. https://doi.org/10.1038/s44159-022-00050-2.
Garfinkel, Irwin, Lee Rainwater, and Timothy Smeeding. 2006. “A Re-Examination of Welfare States and Inequality in Rich Nations: How in-Kind Transfers and Indirect Taxes Change the Story.”Journal of Policy Analysis and Management: The Journal of the Association for Public Policy Analysis and Management 25 (4): 897–919.
Gelman, Andrew, Sharad Goel, Douglas Rivers, and David Rothschild. 2016. “The Mythical Swing Voter.”Quarterly Journal of Political Science 11 (1): 103–30. https://doi.org/10.1561/100.00015031.
Godfrey, Ernest. 1918. “History and Development of Statistics in Canada.” In The History of Statistics–Their Development and Progress in Many Countries. New York: Macmillan, edited by John Koren, 179–98. Macmillan Company of New York.
Groves, Robert, and Lars Lyberg. 2010. “Total Survey Error: Past, Present, and Future.”Public Opinion Quarterly 74 (5): 849–79. https://doi.org/10.1093/poq/nfq065.
Gutman, Robert. 1958. “Birth and Death Registration in Massachusetts: II. The Inauguration of a Modern System, 1800-1849.”The Milbank Memorial Fund Quarterly 36 (4): 373–402.
Hawes, Michael. 2020. “Implementing DifferentialPrivacy: Seven LessonsFrom the 2020 UnitedStatesCensus.”Harvard Data Science Review 2 (2). https://doi.org/10.1162/99608f92.353c6f99.
Heckathorn, Douglas. 1997. “Respondent-Driven Sampling: A New Approach to the Study of Hidden Populations.”Social Problems 44 (2): 174–99. https://doi.org/10.2307/3096941.
Jones, Arnold. 1953. “Census Records of the Later Roman Empire.”The Journal of Roman Studies 43: 49–64. https://doi.org/10.2307/297781.
Kennedy, Lauren, Katharine Khanna, Daniel Simpson, and Andrew Gelman. 2020. “Using Sex and Gender in Survey Adjustment.”https://arxiv.org/abs/2009.14401.
Koitsalu, Marie, Martin Eklund, Jan Adolfsson, Henrik Grönberg, and Yvonne Brandberg. 2018. “Effects of Pre-Notification, Invitation Length, Questionnaire Length and Reminder on Participation Rate: A Quasi-Randomised Controlled Trial.”BMC Medical Research Methodology 18 (3): 1–5. https://doi.org/10.1186/s12874-017-0467-5.
Lane, Nick. 2015. “The Unseen World: Reflections on Leeuwenhoek (1677) ‘Concerning Little Animals’.”Philosophical Transactions of the Royal Society B: Biological Sciences 370 (1666): 20140344. https://doi.org/10.1098/rstb.2014.0344.
Leos-Barajas, Vianey, Theoni Photopoulou, Roland Langrock, Toby Patterson, Yuuki Watanabe, Megan Murgatroyd, and Yannis Papastamatiou. 2016. “Analysis of Animal Accelerometer Data Using Hidden Markov Models.”Methods in Ecology and Evolution 8 (2): 161–73. https://doi.org/10.1111/2041-210x.12657.
Lips, Hilary. 2020. Sex and Gender: An Introduction. 7th ed. Illinois: Waveland Press.
Luebke, David Martin, and Sybil Milton. 1994. “Locating the Victim: An Overview of Census-Taking, Tabulation Technology, and Persecution in Nazi Germany.”IEEE Annals of the History of Computing 16 (3): 25–39. https://doi.org/10.1109/MAHC.1994.298418.
Neyman, Jerzy. 1934. “On the Two Different Aspects of the Representative Method: The Method of Stratified Sampling and the Method of Purposive Selection.”Journal of the Royal Statistical Society 97 (4): 558–625. https://doi.org/10.2307/2342192.
Nobles, Melissa. 2002. “Racial Categorization and Censuses.” In Census and Identity: The Politics of Race, Ethnicity, and Language in National Censuses, edited by David Kertzer and Dominique Arel, 43–70. Cambridge: Cambridge University Press.
Plant, Anne, and Robert Hanisch. 2020. “Reproducibility in Science: A Metrology Perspective.”Harvard Data Science Review 2 (4). https://doi.org/10.1162/99608f92.eb6ddee4.
Prévost, Jean-Guy, and Jean-Pierre Beaud. 2015. Statistics, Public Debate and the State, 1800–1945: A Social, Political and Intellectual History of Numbers. Routledge.
R Core Team. 2022. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing. https://www.R-project.org/.
Rockoff, Hugh. 2019. “On the Controversies Behind the Origins of the Federal Economic Statistics.”Journal of Economic Perspectives 33 (1): 147–64. https://doi.org/10.1257/jep.33.1.147.
Rose, Angela, Rebecca Grais, Denis Coulombier, and Helga Ritter. 2006. “A Comparison of Cluster and Systematic Sampling Methods for Measuring Crude Mortality.”Bulletin of the World Health Organization 84: 290–96. https://doi.org/10.2471/blt.05.029181.
Ruggles, Steven, Sarah Flood, Sophia Foster, Ronald Goeken, Jose Pacas, Megan Schouweiler, and Matthew Sobek. 2021. “IPUMS USA: Version 11.0.” Minneapolis, MN: IPUMS. https://doi.org/10.18128/d010.v11.0.
Sakshaug, Joseph, Ting Yan, and Roger Tourangeau. 2010. “Nonresponse Error, Measurement Error, and Mode of Data Collection: Tradeoffs in a Multi-Mode Survey of Sensitive and Non-Sensitive Items.”Public Opinion Quarterly 74 (5): 907–33. https://doi.org/10.1093/poq/nfq057.
Salganik, Matthew, and Douglas Heckathorn. 2004. “Sampling and Estimation in Hidden Populations Using Respondent-Driven Sampling.”Sociological Methodology 34 (1): 193–240. https://doi.org/10.1111/j.0081-1750.2004.00152.x.
Scott, James. 1998. Seeing Like a State. Yale University Press.
Sobek, Matthew, and Steven Ruggles. 1999. “The IPUMS Project: An Update.”Historical Methods: A Journal of Quantitative and Interdisciplinary History 32 (3): 102–10. https://doi.org/10.1080/01615449909598930.
Steckel, Richard. 1991. “The Quality of Census Data for Historical Inquiry: A Research Agenda.”Social Science History 15 (4): 579–99. https://doi.org/10.2307/1171470.
Stigler, Stephen. 1986. The History of Statistics. Massachusetts: Belknap Harvard.
Vanhoenacker, Mark. 2015. Skyfaring: A Journey with a Pilot. Alfred A. Knopf.
von Bergmann, Jens, Dmitry Shkolnik, and Aaron Jacobs. 2021. cancensus: R package to access, retrieve, and work with Canadian Census data and geography. https://mountainmath.github.io/cancensus/.
Walby, Kevin, and Alex Luscombe. 2019. Freedom of Information and Social Science Research Design. Routledge.
Walker, Kyle, and Matt Herman. 2022. tidycensus: Load US Census Boundary and Attribute Data as “tidyverse” and “sf”-Ready Data Frames. https://CRAN.R-project.org/package=tidycensus.
Whitby, Andrew. 2020. The Sum of the People. New York: Basic Books.
Whitelaw, James. 1805. An Essay on the Population of Dublin. Being the Result of an Actual Survey Taken in 1798, with Great Care and Precision, and Arranged in a Manner Entirely New. Graisberry; Campbell.
Wickham, Hadley, Mara Averick, Jenny Bryan, Winston Chang, Lucy D’Agostino McGowan, Romain François, Garrett Grolemund, et al. 2019. “Welcome to the Tidyverse.”Journal of Open Source Software 4 (43): 1686. https://doi.org/10.21105/joss.01686.
Wu, Changbao, and Mary Thompson. 2020. Sampling Theory and Practice. Springer.
Yeager, David, Jon Krosnick, LinChiat Chang, Harold Javitz, Matthew Levendusky, Alberto Simpser, and Rui Wang. 2011. “Comparing the Accuracy of RDD Telephone Surveys and Internet Surveys Conducted with Probability and Non-Probability Samples.”Public Opinion Quarterly 75 (4): 709–47. https://doi.org/10.1093/poq/nfr020.
