Methods note

Methods note “There is space for everyone: Climate crisis messages across the political spectrum”

This brief note sum­ma­rizes the meth­ods used to col­lect the data for this project. The research was coor­di­nat­ed by d|part in col­lab­o­ra­tion with the Open Soci­ety Euro­pean Pol­i­cy Insti­tute (OSEPI) and the data col­lec­tion was under­tak­en by the sur­vey com­pa­ny Bilendi.

Ques­tion­naire design
The ques­tion­naire used for the sur­vey was devel­oped fol­low­ing an exten­sive lit­er­a­ture review of exist­ing stud­ies into cli­mate change atti­tudes and per­cep­tions. Some ques­tions were adapt­ed from exist­ing projects and new ques­tions devel­oped, where required. Themes were agreed between d|part and OSEPI, an ini­tial ques­tion­naire draft­ed and feed­back sought from sev­er­al experts to enhance the ques­tion­naire to address all the goals of the project. The mas­ter ques­tion­naire was cre­at­ed in Eng­lish and sub­se­quent­ly trans­lat­ed by pro­fes­sion­al inter­preters. The trans­la­tions were then checked by native speak­er researchers from the respec­tive coun­tries to ensure the word­ings reflect­ed the orig­i­nal ques­tions accu­rate­ly, while being under­stand­able for gen­er­al audi­ences in each country.

Pro­gram­ming and piloting
The ques­tion­naire was pro­grammed by Bilen­di using instruc­tions by d|part to ensure the ques­tion­naire could be accessed through a range of devices and browsers. Cor­rect­ness and user friend­li­ness were checked on desk­top com­put­ers, lap­tops, mobile phones and tablets and for Win­dows, Apple and Android sys­tems respec­tive­ly (where appro­pri­ate). After prepa­ra­tion of the mas­ter pro­gram­ming, the approved trans­la­tions were used to cre­ate coun­try-spe­cif­ic ver­sions, which were checked again before com­menc­ing data col­lec­tion. Ini­tial­ly, a soft launch pilot with 50 par­tic­i­pants in each coun­try was con­duct­ed to check response times and any poten­tial prob­lems. After mak­ing small adjust­ments, the full data col­lec­tion was car­ried out. The medi­an com­ple­tion time for the sur­vey was 18 min­utes and 10 seconds.

Sam­pling and data collection
All respons­es were col­lect­ed online through Bilen­di between 7 and 25 August 2020. In each coun­try, just over 1000 respon­dents were sur­veyed, except for Ger­many, where over 2100 were sur­veyed. The age range of respon­dents was 18 to 74, except for Ger­many, where 16- and 17-year olds were also includ­ed for some addi­tion­al analy­ses (for com­par­a­tive analy­ses, only the 18–74-year old sam­ple in Ger­many was includ­ed). The upper age lim­it was capped at 74, because in some of the coun­tries includ­ed online data col­lec­tion is not fea­si­ble in old­er age groups than those, if the goal is to achieve sam­ples that are rep­re­sen­ta­tive of the respec­tive pop­u­la­tion. To ensure com­pa­ra­bil­i­ty across coun­tries, the same upper age lim­it was used.

Unit­ed King­dom: Age range: 18–74; Sam­ple size: 1039
USA: Age range: 18–74; Sam­ple size: 1004
Spain: Age range: 18–74; Sam­ple size: 1007
Swe­den: Age range: 18–74; Sam­ple size: 1031
Czech Repub­lic: Age range: 18–74; Sam­ple size: 1031
Poland: Age range: 18–74; Sam­ple size: 1043
Italy: Age range: 18–74; Sam­ple size: 1003
France: Age range: 18–74; Sam­ple size: 1017
Ger­many: Age range: 16–74; Sam­ple size: 2112

The sam­pling strat­e­gy aimed at achiev­ing rep­re­sen­ta­tive­ness to over­all pop­u­la­tion data for the 18- to 74- year olds in each coun­try. To do this, quo­tas were set in each coun­try for age groups, gen­der, region and edu­ca­tion lev­els of the respon­dents, reflect­ing the dis­tri­b­u­tions of those char­ac­ter­is­tics in the gen­er­al pop­u­la­tion (based on offi­cial records). Addi­tion­al­ly, to ensure dis­tri­b­u­tions were bal­anced in the sam­ple, cross-quo­tas were applied for edu­ca­tion lev­els with­in each region, as well as age groups with­in each region. To make sure that respon­dents were recruit­ed across age groups in as bal­anced a way as pos­si­ble, quo­tas and cross quo­tas were applied com­pre­hen­sive­ly for as long as fea­si­ble and invi­ta­tions to sur­vey par­tic­i­pants were stag­gered over a peri­od of 19 days. Where spe­cif­ic cross-quo­tas could not be filled per­fect­ly, those restric­tions were only relaxed grad­u­al­ly in a tar­get­ed way towards the end of the data col­lec­tion process.

Weight­ing and qual­i­ty checks
After com­plet­ing the data col­lec­tion, devi­a­tions from pop­u­la­tion char­ac­ter­is­tics for each of the quo­tas and cross-quo­tas aimed for were assessed. Over­all, the dis­tri­b­u­tions were in many instances very close to pop­u­la­tion char­ac­ter­is­tics. Addi­tion­al­ly, devi­a­tions in the dis­tri­b­u­tion for a cross-quo­ta not pro­grammed (gen­der with­in each age group) were com­put­ed to account for poten­tial unob­served dis­tor­tions result­ing from the unbal­anced par­tic­i­pa­tion pat­terns. Two sets of weights were pro­grammed: those adjust­ing only for main quo­tas and those that adjust­ed for the unpro­grammed cross-quo­ta (gen­der and age) in addi­tion to the main quo­tas for region and edu­ca­tion groups. The dis­tri­b­u­tion of key vari­ables used in the analy­ses were com­pared between the results for the unweight­ed sam­ple, the sam­ple weight­ed by main quo­ta vari­ables and the sam­ple weight­ed by the addi­tion­al cross-quo­ta with remain­ing main quo­tas. This was done for each coun­try. The dif­fer­ences between out­comes was very small. For most vari­ables test­ed and in most coun­tries, dif­fer­ences in esti­mates between the three weight­ing approach were less than one per­cent­age point. In no instance exceed­ed the dif­fer­ences two per­cent­age points. So even when adjust­ing for devi­a­tions from pop­u­la­tion char­ac­ter­is­tics, the weights only had a lim­it­ed impact on the results. The sam­ple over­all reflect­ed pop­u­la­tion char­ac­ter­is­tics well. In the analy­ses used in the reports, the sim­ple weights, adjust­ing for main vari­ables have there­fore been applied.

Fur­ther information
Fur­ther details about the method­ol­o­gy, in terms of ques­tion­naire design, sam­pling and analy­sis can be obtained upon request from d|part.

Down­load