28oct2000

Outline

UCLA Soc. 210A, Topic 6, Study Design

Professor: David D. McFarland

Web Pages for Fall 2000

Topic 6: Study Design

Topic 6 deals with design of empirical studies, in particular, those designs that involve the investigator's use of random numbers for such things as determining which cases will receive one or another treatment. Probabilities considered previously, in Topics 4 and 5, often pertained to the social phenomena under study, doing such things as determining whether the next birth would be male or female. Here in Topic 6, the probabilities are involved in the research process itself, doing such things as determining which potential respondents are chosen to actually be interviewed.

Populations and Samples

With each study design, we need to consider how it makes the connection between the theoretically relevant universe, the universe from which data are actually sampled, and the actual sample.

Example: "Postindustrial society" is instantiated by "The U. S. in 1994", whose population is "the quarter billion or so U. S. residents in 1994", which is sampled by "the three thousand or so respondents in the 1994 General Social Survey" (albeit with slippage from such matters as nonresponse). Part of that chain involves random selection, and that part can be traced backward using statistical inference. "Postindustrial society" is not a well-defined universe from which it would be possible to select a random sample, not least because it presumably includes not just past and present instances, but also future instances that do not yet exist.

Example: Sometimes, and particularly to scholars who live in or are otherwise strongly affected by it, the universe that is actually sampled is the theoretically relevant universe. For many researchers, "The Contemporary U. S." per se is of considerable interest, quite apart from its being an instance of some more general phenomenon such as "postindustrial society".

Causality

Another important aspect of study designs is how they deal with causality issues, what assurance is provided that the effects found are due to the variables considered, and not to some other variables. (This really goes beyond univariate statistics, the main emphasis this quarter, but the rationale for study designs would be a mystery without some consideration of causal relations among variables.)

"Controlling For" Other Variables by Exclusion, by Regression, or by Randomization

Example: How does one know that observed differences in labor force participation are really due to differences in social networks, rather than, say, effects of gender? Stoloff et al. remove the possibility that they are gender effects by exclusion: they consider only females. Similarly, various other variables, including rural-urban differences, school enrollment, and retirement status, are controlled by exclusion.

Other variables were controlled by incorporating them into the prediction equation, along with social networks. Examples include immigrant status, welfare experience, work experience, and age. More on controlling by regression in 210B and C.

Controlling by randomization would have required reorganizing study participants' lives, randomly assigning them to two groups, installing the one group into new social networks, and removing the other group from any networks in which they were already involved. For very good reasons, nothing along these lines was attempted.

Stoloff, Jennifer, Jennifer L. Glanville, and Elisa Jayne Bienenstock. 1999. "Women's Participation in the Labor Force: The Role of Social Networks." Social Networks 21 (#1, January): 91-108. (Online in sciencedirect.)

Random Sample of What?

In the intro stat course you would have studied "simple random sampling", with and without replacement. That served the important purpose of providing a rationale for statistical inference procedures.

Here we will build on that, in a couple different directions. For one thing, many important datasets, including censuses, are not random samples from larger populations of theoretical interest; yet we use probability considerations to analyze the internal variability in such datasets. Our coverage will include examination of the rationale for the common practice of treating such datasets "as if" there were some larger population from which they were random samples.

This is a matter of some controversy among statisticians who concern themselves with the foundations of that discipline. My own take on it is that conventional statistics tell us something useful about the stability of the dataset, whether or not it is a random sample from some larger population about which one wishes to make inferences; and a dataset not large enough to give stable estimates of population parameters isn't large enough to have stable statistics of its own either.

Complex Probability Samples

Even datasets which are random samples seldom are 'simple' random samples, and we shall consider what needs modified when data come instead from more complex probability samples, such as stratified cluster samples commonly used in actual large-scale surveys.

At the production end, a complex sample does not require preparation of a list of all members of the population, from which to choose respondents. At the first stage one needs only a list of 'primary statistical units' (PSUs) such as counties, and some of those are chosen, with probabilities proportional to size. At the second stage, smaller units such as city blocks are listed, only for the selected PSUs, and a random selection is again made, with probabilities proportional to size. Successive random selections choose housing units within each chosen block, and choose one respondent within each chosen housing unit. At each stage, only the chosen units need to be subdivided and listed for the next stage selection. This is thus feasible when SRS would not be for lack of a list of the population.

One effect of this type of sampling is that the respondents are more geographically clustered than an equal number of respondents in a SRS would be. That, in turn, means that observed values of variables that also cluster geographically show less variability than they would in a SRS with the same number of respondents. The 'design effect' of the GSS varies with the particular variable, but over a mixture of demographic and attitudinal variables averages about 1.5, which means that the approximately 3,000 cases in 1994 are equivalent to approximately 2,000 cases in a simple random sample, as far as sampling variability is concerned. A huge sample is still huge, whether 2,000 or 3,000; where it matters is rare subpopulations, such as 90 cases being equivalent to only 60 SRS cases. For rare subpopulations, the Census public use data are better suited (see below).

Another place it matters is in significance tests, which will be covered in a couple weeks. One of the several reasons not to mechanically apply computer-generated significance tests is that they may not take design effects into account.

Survey Designs and Other Designs

Although important, surveys are not the only types of social research. We will briefly consider the features, and relative advantages, of different study designs, notably randomized experiments. (Soc. 212C is an entire course devoted to these matters.)