|Methodological Appendix||-Download NJPS Report PDF|
-Download PowerPoint Summary
|Planning for the National Jewish Population Survey 2000-01 was carried out by UJC research staff in conjunction with the National Technical Advisory Committee, UJC Pillar and other professional and lay leadership, NJPS Trustees, the Jewish federation system and a broad range of Jewish communal organizations.|
Interviewing for NJPS 2000-01 took place from August 21, 2000 to August 30, 2001. RoperASW, a survey research firm with headquarters in New York City, conducted the fieldwork, and in conjunction with UJC research staff and its consultants produced the survey weights, tabulations and electronic data files.
Interviewing was conducted by telephone. All telephone numbers in all 50 states of the United States plus the District of Columbia were eligible to be called. The sample of telephone numbers called was randomly selected by a computer through a Random Digit Dialing (RDD) procedure, thus permitting access to both listed and unlisted phone numbers.
The United States was divided into seven strata, based upon pre-survey estimates of Jewish population density. Telephone calls were made to all seven strata. To more efficiently locate Jews -- whom survey researchers call a rare population -- RoperASW moderately over-sampled strata with higher estimated levels of Jewish density and under-sampled strata with lower estimated levels of Jewish density. Over-sampling in some strata and undersampling in others meant that some people had a greater chance of being called for an interview than others. This difference in the chance of being called was adjusted in the weighting process in order to provide a representative sample of U.S. and Jewish households and populations (see below for more on weighting).
The sample was divided into 22 distinct replicates, each in itself a representative sample of the Jewish and U.S. populations. Replicates are used to increase the efficiency and quality of the sampling process. During the first five replicates, up to 16 calls were made to a telephone number to obtain a disposition, i.e. a determination that the number was a residence, business, not working, or other categories. After conducting a formal test of callback efficiency, up to 8 calls were made in later replicates.
No telephone interviewing for NJPS was conducted on Shabbat or Jewish holidays.
Over 175,000 households were screened for possible inclusion in NJPS. A series of screening questions was asked about all adults in the households contacted:
1. What is your (other adult's) religion, if any? (If not Jewish, then ask:)
2. Do you (Does other adult) have a Jewish mother or a Jewish father? (If no, then ask:)
3. Were you (Was other adult) raised Jewish?
4. (Ask all if not Jewish/Judaism in Q.1): Do you (Does other adult) consider yourself (him/herself) Jewish for any reason?
Based on answers to the screening questions, all household adults were initially classified into 19 Sample Allocation Codes, which were further consolidated into three groups: Jews, People of Jewish Background (PJBs) and non-Jews.
In households with one qualified Jewish adult, that person was selected for a full NJPS interview. In households with two or more qualified adult Jews, one was randomly selected for a full NJPS interview. The questionnaire administered to Jewish respondents consisted of over 300 questions, though no single respondent received every question. Questionnaires are available on the UJC website, www.ujc.org/njps and the North American Jewish Data Bank website, www.jewishdatabank.org.
Among PJB households (defined as having no adult Jews, but at least one adult PJB), a random subsample was selected for an interview. Within selected PJB households, one qualified PJB adult was randomly selected from among all qualified PJB adults in the household (in households with one PJB adult, that person was selected for the interview). Based on initial, pre-survey assumptions that PJBs were not Jewish, PJB respondents were administered a short-form questionnaire consisting of a subset of approximately 40% of the questions administered to Jewish respondents. The PJB questionnaire excluded many but not all questions on Jewish topics.
A subsample of non-Jewish households, defined as households with no Jewish or PJB adults, was randomly selected for a survey entitled the "National Survey of Religion and Ethnicity." Completion of non-Jewish interviews was kept at approximately the same rate as the completion of Jewish/PJB interviews across the entire course of the field phase. Within selected non-Jewish households, one qualified adult was randomly selected from among all qualified adults in the household (in households with one adult, that person was selected for the interview). The NSRE interview consisted of 41 questions. Non-Jews were interviewed for two reasons: to collect data necessary for weighting and thus estimating the size of the Jewish population, and to providecomparative data to Jews and PJBs on socio-demographic topics.
A total of 9,175 adults (age 18 or older) completed the NJPS (Jewish and PJB) and NSRE (non-Jewish) questionnaires. Weighted data from this combined sample are representative of the total, U.S. household population. Table A-1 displays the total number of respondents in each sample, the range of interview length and the median time to complete an interview.
During data analysis, 264 respondents initially classified and interviewed as Jews were re-classified as non-Jews of Jewish background because they said they were Christians, and in one case Muslim. At the same time, 303 respondents initially classified and interviewed as PJBs were re-defined as Jews. The final unweighted number of Jewish respondents is 4,523 and the final unweighted number of respondents who are non-Jews of Jewish background is 625. This report addresses the 4,523 respondents defined as Jews according to the post-survey classification; it does not include any of the 625 respondents who are non-Jews of Jewish background under the post-survey classification, except in the case of calculating the intermarriage rate under the "born Jewish" definition first used by 1990 NJPS researchers. None of the 4,027 initial non-Jewish respondents who were administered the National Survey of Religion and Ethnicity were reclassified during analysis of the data. Table A-2 displays initial and post-survey classifications of respondents.
The American Association for Public Opinion Research (AAPOR), the leading professional association of survey researchers, provides several alternative formulas for calculating response and cooperation rates. Using the formulas most commonly reported in the research industry (known as RR3 and COOP3), the response rate to the screening interview for NJPS/NSRE is 28% and the cooperation rate is 40%.
Weights were applied to the data to correct for the unequal probability of household and respondent selection into the sample. These weights adjust for the number of telephone lines in the household, sampling rates within each of the seven strata, subsampling rates for PJB and NSRE households, number of qualified adults in the household, and the number of qualified children in the household for a battery of questions asked about one randomly selected child. Post-stratification weights were also applied to bring sample household and respondent data to U.S. Census totals for strata, age, gender, and region. Final weights provide estimates of households, adults and children. Further information on weighting is available from UJC research staff.
For small numbers of households, data on the number of telephone lines in the households, which were needed for constructing the weights, were not available. Instead, they were imputed based on data from similar households. In addition, small numbers of cases that were screened and selected for a full NJPS/NSRE interview, but then never successfully completed the interview, were not retained in the survey firm's computer system. Because the total number of these missing screener cases was necessary for computing the weights, estimates of their number were made based on screened cases that were retained.
A sensitivity analysis was conducted to test whether imputing data for telephone lines and estimating the number of missing screener cases in the weighting process had an effect on findings from the data. Very extreme alternatives were tested with respect to both the imputed data on telephone lines and the estimates of missing screener cases. The results indicated that though the alternatives could affect the total number of Jews, they did not affect descriptions of the characteristics of Jews based on percentages. No estimated percentages on Jewish variables (e.g., synagogue and denominational members, charitable donations to Jewish causes, volunteering under Jewish auspices, or ritual observances) or socio-demographic variables (e.g., region, income or assessments of health) changed by more than 1-2% when applying vastly different weights. As a result, NJPS researchers are confident that the selective imputations and estimates used to calculate the weights do not affect descriptions of characteristics of the Jewish population emerging from NJPS.
All surveys are subject to sampling variability, the margin of error associated with taking a sample from a population rather than a census of the entire population. Margins of error are a function of both the sample design and sample size. In theory, in 19 out of 20 cases, the margin of error for statistics such as percentages and proportions for all Jewish adults is +/- 2.0%, for Jewish households +/- 2.4%, and for Jewish children +/- 4.0%. Statistics from subsamples will have larger margins of error. For example, a subsample of 50% of Jewish adults has a margin of error of +/- 2.9%, and a subsample of 10% of Jewish adults has a margin of error of +/- 6.4%.
In addition to margins of error around percentages and proportions, the weighted population estimates are subject to sampling variability. The household population estimate of 5.1 million Jews may vary by as much as +/- 2.8%, and the household estimate of 4.3 million Jews in the more engaged population may vary by as much as +/- 3.0%. No statistical margins of error can be calculated for the non-sampling estimate of 100,000 Jews in institutional settings.
All surveys are subject to the possibility of other kinds of errors, called nonsampling errors. Researchers conducting Jewish population studies -- including those who conducted the 1990 NJPS -- have long recognized that a major nonsampling concern is the potential miscount of the Jewish household population, due for example to respondents not accurately reporting or denying their current Jewishness or Jewish background, or to differential response rates of Jews and non-Jews.
Limited studies of this issue, none of which are methodologically rigorous, suggest that errors in estimating the Jewish population tend to be in the direction of undercounts, although the size of the undercounts seem to be small. Two recent studies by UJC -- one on respondents inaccurately reporting their Jewishness and the second on differential rates of distinctive Jewish names between survey cooperators and non-cooperators -- may point to a small undercount of the Jewish population, but both studies have methodological limitations that make their findings inconclusive.
Furthermore, Jewish population studies, be they national or local, have used different screening questions to identify Jews, and they have placed the screening questions in different orders. Both of these factors may cause estimates of the Jewish population to vary across studies. Some have suggested that the NJPS 2000-01 opening screener question on religion (what is your religion, if any?) may have dissuaded some Jews, especially secular or ethnic Jews, from affirming their Jewish identity. Others have suggested that the open-ended format of the NJPS 2000-01 screener question on religion and its placement at the very beginning of the screening interview may have lowered the Jewish population estimate as well.
In sum, many researchers believe that the methodologies of survey research may yield undercounts of the Jewish population. However, no tests to date provide firmly established data to accurately estimate the potential undercount of the Jewish population or the effects of different screening questions on population estimates. Lacking such an empirical foundation, no adjustments to the NJPS 2000-01 population estimate or weights were made for these factors.
The UJC Research staff has conducted a series of tests and analyses of important methodological issues related to NJPS 2000-01. Several of these tests and analyses were part of the project's original research plan developed in consultation with the National Technical Advisory Committee. Others arose in response to concerns following the initial release of data in October 2002. In addition, UJC commissioned an external review of NJPS methodological issues. Call the NJPS information line at 1-888-711-4490 or email firstname.lastname@example.org for more detailed information on methodological tests and analyses, the external review, or the following topics: