Basic principles and methods of conducting a pilot study
After the selection of psychological methods for carrying out PPE events, a check of their predictability should be carried out, that is, an assessment of their ability to solve the tasks assigned to them should be carried out: differentiation of all subjects according to the severity of the PVK and predicting the probability of success in professional activity. For this purpose, a so-called pilot study is being carried out.
The concept of "pilot study" (sometimes also called a pilot study) is quite widespread in psychology. Moreover, these concepts very often mean completely different activities carried out with different goals. However, if we generalize everything that is understood by this concept, then its meaning is reduced to conducting a "trial", "evaluative" or "indicative" study.
The essence of the pilot study, conducted in order to assess the predictive value of psychodiagnostic techniques, is to examine a group of people already performing professional activities for which it is planned to select. It is assumed that people working in this specialty have individual psychological and personal characteristics that either hinder or, conversely, contribute to the successful performance of professional duties. Therefore, if the professionally important qualities were identified correctly, if the methods were selected correctly, then the research results should have a close relationship with the performance indicators.
When conducting a pilot study, it is always necessary to solve two problems:
- how to determine the criteria for the success of the activity, in relation to which the predictability of the selected methods will be assessed;
- how to form a group of persons to be examined.
As a rule, when forming a group of subjects to check the predictability of the selected methods, one of two fundamental decisions is made: to examine two groups of people with polar indicators of the success of professional activities, or to conduct a survey based on the principle of random sampling, i.e., examine everyone who is possible.
If the assessment of the predictive value of the selected methods is carried out on the basis of the results of the survey of polar groups, then the experimental work is carried out in the following order.
- Based on clearly defined criteria for the success of activities, two groups of workers are formed: successful and unsuccessful, and the surveyed groups should be comparable in age and gender.
- At the final stage, a mathematical and statistical analysis of the results is carried out.
To implement this approach in practice, first of all, it is necessary to ensure the representativeness of the sample, which will require a sufficiently large group of people. When talking about the size of groups, one should recall the requirements mentioned above (see Chapter 8). Based on these requirements, it can be argued that to ensure the statistical reliability of the results, both groups of successful and unsuccessful workers compared with each other should be at least 30 people. However, it is extremely rare to achieve such a sample size, if only because the management of an organization or enterprise is unlikely to put up with such a large number of unsuccessful employees. Therefore, although the survey of polar groups is an ideal option, most often when conducting a pilot study during the development of an OSS system, developers survey a random sample of workers in the enterprise.
When conducting a pilot study based on the principle of random sampling, as a rule, all workers performing the same professional duties, that is, representatives of the same profession, are surveyed. In this case, the order of work is as follows:
- a survey is carried out of employees of the enterprise performing professional duties in the position in respect of which it is supposed to carry out professional psychological selection;
- at the final stage, a mathematical and statistical analysis of the results obtained is carried out.
At the same time, it should be noted that there are cases when none of the two above-mentioned methods of conducting a pilot study is suitable for their implementation in specific conditions. For example, this happens when a system of professional selection for newly created professions is being developed. In addition, there are situations when it is not possible to select the number of people necessary for conducting a pilot study. In this case, the psychologist has to model the sample of subjects; then the stages of the experimental study will be as follows:
- at the first stage, an experimental sample of the surveyed is modeled and psychological testing is carried out;
- at the second stage, a mathematical and statistical analysis of the results is carried out.
The most difficult stage in this version of the pilot study is the stage of modeling the experimental sample, since it determines the accuracy of assessing the compliance of the selected methods with the conditions for performing the tasks of the application software. When modeling the experimental sample, one should take into account the estimated age of people who will have to perform the relevant professional duties, as well as their gender and education. At the same time, it is desirable that the professional activities that they actually carry out present very similar requirements in terms of content to the presence of the relevant professionally important qualities in employees that are necessary for successful work in a new position. It is also desirable to take into account that the size of the simulated sample should be at least 55–60 people, and in the ideal case, at least 100.
It should be noted that all the options for conducting a pilot study to assess the predictability of methods that are supposed to be used in PPE activities have a very similar stage - a mathematical and statistical analysis of the results. However, the content and order of implementation of this stage may be different. Let's consider this stage in more detail.
^
Mathematical and statistical analysis of the results of the pilot study
When conducting a mathematical and statistical analysis of the results of a pilot study carried out on the basis of testing polar (contrast) groups, as a rule, the normal distribution of the results is not assessed, since it is assumed that the results of the two surveyed groups will differ significantly. Therefore, first of all, the search for differences in the results of the tests performed by the representatives of the studied groups is being conducted. For this, it is most often used t- Student's test. At the same time, the degree of difference in general across groups is first assessed, and then - depending on age and gender. In the course of this analysis, methods are identified according to which the surveyed groups have sufficiently clear differences, as well as those for which no significant differences were found. It is quite natural that the methods that emphasize the differences between the compared groups of successful and unsuccessful workers should be included in the list of methods recommended for use in professional selection events.
It is somewhat more difficult to make a decision on the use of professional selection methods in activities that did not allow us to detect differences in the compared groups of workers. What should you look for when analyzing the results of such tests? First of all, for the presence in the list of tested tests of two or three methods that assess the same psychological quality. If this was done and differences were established for some methods, but not for others, then in the list of methods that will be recommended for professional selection, only those methods are left that confirmed the existence of differences between the compared groups. Accordingly, those methods for which no differences were found should be excluded from the list.
If, in order to study some professionally important quality, only one technique was included in the list of techniques tested during the pilot study and, according to the results of its implementation, no differences were found between the compared groups of successful and unsuccessful workers, then making a decision on the appropriateness of its use in selection activities is further complicated. First, the degree of importance of this professionally important quality in the general structure of the internal control system is assessed. If this quality is secondary, then in some cases this quality can be neglected and the test, which did not allow to establish the differences between the groups, can be excluded from the list of methods recommended for use in professional selection events. Secondly, the value of the deviation of the obtained t- criterion on the values of significant differences. As you know, in statistics there are very different levels of reliability of values that characterize the degree of probability of an event. According to the majority of authors, developing a methodology for using various methods of statistical analysis in psychology, as reliable indicators t- Student's test, characterizing the differences between two groups of people, one should take those that correspond to at least 95% probability level, i.e. R < 0.05. Accordingly, indicators below this value should be classified as unreliable.
This point of view is quite justified in relation to various psychological studies. However, when developing a PPE system, one should take into account the nature of a professionally important quality. So, there are such PVCs that can be fully manifested only when performing real professional activities. For example, many methods for assessing a person's stress tolerance demonstrate the best performance in differentiating subjects for the development of this quality only when the subjects are in a stressful situation. Therefore, it should always be borne in mind that when conducting psychological testing, the subject is in conditions of testing only, and not real activity. It is this circumstance that determines the need to assess the deviation value of the obtained indicator t- criterion from values characterized as reliably significant. In certain cases, indicators t- criteria that have not even reached the level of reliably significant indicators can be assessed as quite sufficient. For example, when comparing the results of two groups of 30 people in each, the indicator is considered to be reliably significant t- criteria from 2,042 or more, but how to relate to t = 1,711 or 1,800? This value of the indicator t- the criterion does not reach the level of significance at which it is customary in psychology to speak of high reliability. However, from the point of view of statistics, it is quite significant. In the case of comparing the results of a psychological examination of two compared groups of 30 people in each value t- a criterion of more than 1.697 and less than 2.042 indicates that there are 90% (R < 0.1) the probability that these groups really have differences in the severity of the corresponding psychological characteristics. In this case, in psychology, it is customary to talk about existing trends. Since a tendency has been identified, it is not possible to refuse to use or, conversely, to recommend such a technique for use in professional selection events without additional analysis.
Further analysis is to identify the relationship between test scores and performance indicators. For this, correlation analysis is used - a method of mathematical statistics that allows you to detect the relationship between individual characteristics of one object. In this case, it is assumed that if the success of the activity is really determined by some psychological qualities, then there should be a close relationship between the indicators of the development of this quality (assessed using tests) and the indicators of the success of the activity. If, in the course of the correlation analysis, such a relationship was established for the indicators of the test, for which only a tendency to achieve significant differences was revealed (i.e., the differences in the indicators according to this method between the two groups correspond to R < 0.1), then it can be used in professional selection events. If the correlation coefficient has not reached the level of significant significance, then it is not recommended to use such a test in professional selection events.
It should be noted that the results of the correlation analysis act as decisive in assessing which of two or three methods characterizing the level of development of the same psychological quality, it is advisable to use in the structure of PPE activities. As mentioned above, when conducting a pilot study, it is possible, and sometimes quite advisable, to use several psychological methods (parallel tests) that assess the same quality, since psychological tests sometimes demonstrate different reliability and validity on different samples and under different conditions. If in the course of the mathematical and statistical analysis of the results of the pilot survey it was found that parallel tests have equal indicators when comparing polar groups, then the preference is given to the method that has higher indicators in the correlation analysis. In this case, it is assumed that all coefficients (and t- criterion and correlation coefficient) must reach the level of significant significance, otherwise there is no point in comparing the methods with each other. It should always be remembered that statistical confidence is the primary consideration in the selection of methodologies.
In contrast to the previous version of the pilot study, when examining a random sample of employees of an enterprise or organization, using mathematical and statistical analysis of the results, the closeness of the test indicators obtained to the normal distribution is first and foremost assessed. If this requirement is met, then it can be argued that the technique used allows people to be ranked depending on the level of development of the psychological quality, for the assessment of which it is intended. If a normal distribution is not obtained, then the question of the possibility of using the tested test in the structure of professional selection measures is questioned. Therefore, further analysis is reduced to assessing the nature of the distribution of the obtained test indicators.
As a rule, when analyzing the distribution of test indicators, you can face one of three options:
- distribution close to normal;
- the top of the distribution graph is shifted to the right or left (right or left asymmetry);
- the distribution graph has two or more peaks.
The first distribution option is ideal and, as noted above, removes at this stage of the analysis all questions about the expediency of using the test in the structure of professional selection events.
In the second variant, when there is a left or right asymmetry of indicators, it is necessary to take into account the design features of the test and the characteristics of the sample of the surveyed. For example, with a high level of PVK development, the majority of the subjects will have right asymmetry. Often this phenomenon is observed if at the stage of recruitment some kind of selection was carried out. Therefore, when faced with the right asymmetry, it is necessary to analyze the experience of the subjects, the effectiveness of their activities, the level of professional preparedness, etc. As a rule, such an analysis reveals the pattern of asymmetry occurrence, due to the characteristics of the sample of the subjects.
At the same time, the asymmetry can also be due to the design features of the test. From this point of view, it is necessary to analyze the possibility of eliminating this asymmetry by developing a normalized scale, that is, such a scale that is formed when converting "raw" indicators into standard ones (stans, stanins, etc.). For example, asymmetric results were obtained when using the normalized scale. In this case, an attempt is made to make adjustments to the existing normalized scale. If the asymmetry was obtained on "raw" indicators, then an attempt is made to eliminate it by developing an appropriate standard normalized scale.
Further, the decision on the use of the test in the activities of professional selection is made by mutual comparison of the results of the analysis of the characteristics of the sample and the design features of the test. However, if attempts to create a normalized scale or make changes to an existing scale have not been crowned with success and the asymmetry cannot be compensated for, then it is still inappropriate to recommend such a test for use in professional selection events.
One of the most difficult cases of distribution analysis is the case with the presence of several peaks in the distribution graph of test scores. This is the so-called variant of the distribution with negative kurtosis, which indicates a shift in most of the results to polar values. There may be explanations for this distribution. First, all the same features of the sample. However, if the negative kurtosis is really caused by the characteristics of the sample, then it should be inherent not in one technique, but in several at once. It is this phenomenon that can be observed when analyzing the results obtained when examining well-matched polar groups. If negative kurtosis is inherent in only one or two methods, then, most likely, it is due to the design features of the test. In this case, one should either refuse to use the test in professional selection events, or increase the size of the sample of subjects and re-examine the subjects.
After analyzing the nature of the distribution of test indicators during a pilot survey of a random sample, a correlation analysis is carried out. At this stage, those methods are eliminated that do not have reliably significant correlations with indicators of the success of professional activity.
At the final stage of the mathematical and statistical analysis of the results of a pilot study carried out on a random sample, the ability of tests to differentiate all those examined in relation to the criteria for the success of activities is assessed. For this, among the surveyed, three groups of people are distinguished - with high, medium and low success rates. Next, a comparison of test results is made between the groups with high and low success rates. Tests for which significant differences were obtained between the compared groups, and tests that showed the presence of positive trends, can be considered recommended for inclusion in the list of methods planned for use in professional selection activities.
However, most often, when examining a random sample, psychologists are faced with a situation where it is not possible to form control groups with high or low performance indicators. This is sometimes due to insufficient sample size. Therefore, it is possible to use reverse analysis. For this purpose, all the subjects are divided into several groups in relation to the indicators of any one test, after which the indicators of the success of the activity between the selected groups are analyzed. If differences in performance indicators are found between groups of persons (at least at the level of regularities, i.e. R < 0.1 or higher), then the test can be recommended for use in professional selection events.
During the pilot study of a test, similar work should be done for each test subject. In addition, it should also be borne in mind that in the final list of methods recommended for solving the problems of professional selection, there should not be parallel tests. Therefore, if parallel methods are used in the pilot study, at the final stage, one should select from them those that showed the best predictive value.
Another option for a pilot study is a survey of an experimental simulated sample. This option is carried out in situations where it is not possible to survey people performing professional activities for which selection is planned. The most significant drawback in this situation is the lack of performance indicators. As a result, it is not possible to obtain highly reliable analysis results characterizing the predictive value of the evaluated tests. Therefore, whatever the results are obtained, they can be regarded as preliminary and after the accumulation in the course of the practical use of the tests of the material, the entire analysis should be repeated and in full. Nevertheless, in this variant of the pilot study, methods of mathematical and statistical analysis are used. It is considered advisable to analyze the distribution of indicators of the tests used. Techniques, according to the results of which the indicators of the subjects do not correspond to the normal distribution, are, as a rule, excluded from the list of techniques recommended for use in PPE.
Concluding the conversation about the analysis of the results of the pilot study, which is one of the stages in the development of the PPO system, it should be emphasized that in the presented review only the most common techniques and methods of analyzing the results of the examination of experimental groups are considered. In the specific conditions of creating a system of professional psychological selection, other methods of analysis can also be used. Therefore, a psychologist who solves the problems of professional selection must not only know psychological tests, but also be well versed in mathematical statistics.
After the selection of psychological methods for carrying out PPE events, a check of their predictability should be carried out, that is, an assessment of their ability to solve the tasks assigned to them should be carried out: differentiation of all subjects according to the severity of the PVK and predicting the probability of success in professional activity. For this purpose, a so-called pilot study is being carried out.
The concept of "pilot study" (sometimes also called a pilot study) is quite widespread in psychology. Moreover, these concepts very often mean completely different activities carried out with different goals. However, if we generalize everything that is understood by this concept, then its meaning is reduced to conducting a "trial", "evaluative" or "indicative" study.
The essence of the pilot study, conducted in order to assess the predictive value of psychodiagnostic techniques, is to examine a group of people already performing professional activities for which it is planned to select. In this case, it is assumed that people working in this specialty
have individual psychological and personal characteristics that either hinder or, conversely, contribute to the successful performance of professional duties. Therefore, if the professionally important qualities were identified correctly, if the methods were selected correctly, then the research results should have a close relationship with the performance indicators.
When conducting a pilot study, there is always a need to solve two problems: how to determine the criteria for the success of the activity, in relation to which the predictability of the selected methods will be assessed; how to form a group of persons to be examined.
As a rule, when forming a group of subjects to check the predictability of the selected methods, one of two fundamental decisions is made: to examine two groups of people with polar indicators of the success of professional activities, or to conduct a survey based on the principle of random sampling, i.e., examine everyone who is possible.
If the assessment of the predictive value of the selected methods is carried out on the basis of the results of the survey of polar groups, then the experimental work is carried out in the following order. On the basis of clearly defined criteria for the success of activities, two groups of workers are formed: successful and unsuccessful, and the surveyed groups should be comparable in age and gender. Further, an examination is carried out and the processing of the primary results is carried out. At the final stage, a mathematical and statistical analysis of the results is carried out.
To implement this approach in practice, first of all, it is necessary to ensure the representativeness of the sample, which will require a sufficiently large group of people. When talking about the size of groups, one should recall the requirements mentioned above (see Chapter 8). Based on these requirements, it can be argued that to ensure the statistical reliability of the results, both groups of successful and unsuccessful workers compared with each other should be at least 30 people. However, it is extremely rare to achieve such a sample size, if only because the management of an organization or enterprise is unlikely to put up with such a large number of unsuccessful employees. Therefore, although the survey of polar groups is an ideal option, most often when conducting a pilot study during the development of an OSS system, developers survey a random sample of workers in the enterprise.
When conducting a pilot study based on the principle of random sampling, as a rule, all workers performing the same professional duties, that is, representatives of the same profession, are surveyed. In this case, the order of work is as follows: a survey of the employees of the enterprise performing professional duties in the position in respect of which it is supposed to carry out professional psychological selection is carried out; further, the criteria for the success of professional activity are determined and, if possible, an integral indicator of success is developed; at the final stage, a mathematical and statistical analysis of the results is carried out.
At the same time, it should be noted that there are cases when none of the two above-mentioned methods of conducting a pilot study is suitable for their implementation in specific conditions. For example, this happens when a system of professional selection for newly created professions is being developed. In addition, there are situations when it is not possible to select the number of people necessary for conducting a pilot study. In this case, the psychologist has to model the sample of subjects; then the stages of the experimental research will be as follows: at the first stage, the experimental sample of the surveyed is modeled and psychological testing is carried out; at the second stage, a mathematical and statistical analysis of the results is carried out.
The most difficult stage in this version of the pilot study is the stage of modeling the experimental sample, since it determines the accuracy of assessing the compliance of the selected methods with the conditions for performing the tasks of the application software. When modeling the experimental sample, one should take into account the estimated age of people who will have to perform the relevant professional duties, as well as their gender and education. At the same time, it is desirable that the professional activities that they actually carry out present very similar requirements in terms of content to the presence of the relevant professionally important qualities in employees that are necessary for successful work in a new position. It is also desirable to take into account that the size of the simulated sample should be at least 55-60 people, and in the ideal case, at least 100.
It should be noted that all the options for conducting a pilot study to assess the predictability of methods that are supposed to be used in PPE activities have a very similar stage - a mathematical-statistical analysis of the results. However, the content and order of implementation of this stage may be different. Let's consider this stage in more detail.
More on the topic Basic principles and methods of conducting a pilot study:
- 3. Basic concepts and experience of marketing research 3.1. Content and directions of marketing research
A pilot study can be carried out for various purposes: testing the main hypotheses of the program, identifying individual characteristics of the object in order to determine the most stable ones for their further in-depth development, clarifying the effectiveness of the technique as a whole or its individual parts, preliminary verification of the reliability of the survey results, etc.
There are various forms of conducting a pilot study. It can be done on small or large samples. If the research program requires it, the questionnaire can be piloted with a large representative sample. In essence, such aerobatics sometimes acts as an independent empirical research, which takes no less effort and time than the main research.
However, usually a pilot study is conducted on a small area. The exact number of the surveyed in this case is not established, since in each case it may be different. Practice shows that the number of respondents most often does not exceed 50-100 people. This number of respondents is enough to solve the problems facing the pilot study. A necessary condition is that the object of the study should be those people who, in their main characteristics, correspond to the main object of the study. In this case, the aerobatic survey is carried out not at the main object, but at another "field" that is close in its characteristics to the main one. This is done in order not to spoil the main "field" of research, since any, even the smallest, research gets resonance, and in one way or another can have an impact on the main research. So, if workers of an industrial enterprise act as the main object of research, then in the pilot survey the object should be workers of an industrial enterprise, too, but of another enterprise.
The order in which the pilot survey is conducted is very important. Despite the fact that the questionnaire is being prepared for a correspondence survey, in a pilot study it is better to present it in the form of a formalized interview. The advantage of a formalized interview lies in the fact that, in addition to the fact that the results of the pilot survey will be analyzed in the aggregate of answers, it also provides direct observations of interviewers over the response of respondents to a particular question and to the entire questionnaire as a whole. The interviewers are most often members of the research group or people specially trained for this. At the end of the survey, interviewers write short reports on the results of each interview. Although such written reports take a lot of time and effort and are not pleasant to every interviewer, they play a significant role in the pilot study. They are of great interest as a direct observation document, and their analysis is extremely important for finalizing the questionnaire. This is how one interviewer describes his experience of the survey in the pilot study.
Intelligence research is often referred to as aerobatic research. Although it is more correct to consider it an independent type of sociological research. Reconnaissance and aerobatic research on the walk in two aspects:
♦ Target - to obtain approximate data on a particular phenomenon, or to test the application of the methodology to larger-scale research.
« An object - both studies require a small set of objects, so they are carried out in a limited time frame.
But unlike a reconnaissance study, a pilot study is intended to work out technical procedures and techniques, most often to test a questionnaire. According to J. Mannheim and R. Rich, preliminary testing of the survey instruments is as important for the success of the study as a test drive is for the successful purchase of a used car. It helps to identify problems that can only fully manifest themselves in the field.
When conducting a pilot study, it is necessary to take into account the following rules:
4 The sample size in the pilot study does not have a clear methodological basis. Usually it is considered sufficient to interview about 30 respondents. It is only important that they represent all the essential categories of the planned research object. "
A small pilot sample does not have to be representative of the general population. Much more important is its other quality - diversity: it should include those groups of respondents who are able to respond to the toolkit differently. If people with little education will make up an insignificant proportion of the future sample, they should be interviewed in the first place precisely because they may not answer your smart questions at all as expected.
“The pilot sample includes those respondents who represent the most important socio-demographic characteristics for the survey topic (gender, age, education, work experience, content of work, etc.).
In social psychology, aerobatic research (the working term is "probing") is used to establish the required sample size, clarify the content and number of questions in the questionnaire, survey time, etc. serves as a means of identifying some of the standards of the main test.
In sociology, aerobatics is carried out before the main research and acts as a way to test the validity of hypotheses and tasks, as well as the professional level and methodological development of the toolkit. Pilotage helps to assess the correctness of the sample model and to make appropriate corrections, if necessary; clarify some characteristics of the object and subject of research, justify the financial costs and timing of the main research. Pilotage is also useful for training a group of interviewers (questionnaires).
Modern sociologists adhere to the rule: before giving a questionnaire to a large circulation, it is advisable to test it in a small-circulation (pilot) survey in order to clarify whether the respondents understand the questions correctly, whether it is too long and tedious, how long it takes to fill out (or an interview), and in the case of mailing - how many days elapsed from the moment the questionnaire was sent to the receipt of answers.
Shmerlina I.A. Procedural principles of research sociology. Textbook. manual // http: // www.unn.ac.ru/ras/fl4/k2/courses/shmern.htm
A pilot study is a study according to a "shortened" design - it uses small samples, information collection is not fully carried out, the information obtained is analyzed only for the most significant criteria. If a pilot study is carried out in a cycle of researching a well-studied problem, it allows you to "hone" research tools, to identify and eliminate its defects and defects in approaches to mathematical analysis. Conducting a pilot study in this case allows avoiding the collection of information on “empty” questions, the respondents' answers to which cannot be analyzed, or are inadequately (ambiguous) interpreted by the respondents themselves. For exploratory research (research in an underdeveloped area), a pilot study provides information to clarify many points of the program, which further provides significant resource savings.
Pilotage is useful for: a) testing the instrumentation in which the researcher is quite confident, b) improving the instrumentation in a situation where the research subject is less known to the researcher. In the first case, the toolkit is preliminary tested in its final version. In the second, the researcher may want to experiment with different versions (layouts) of the tool to find out which one is more convenient for his work. This type of pretesting may include:
1. Testing of various verbal formulations and question forms.
2. Testing the questionnaire, intended for self-filling by the respondent, in a personal interview, so that in the process of live communication with the respondent, possible difficulties in filling it out were revealed.
3. The use of open-ended questions to identify typical answers, which can then be included in a set of answers to closed-ended questions in order to use the latter in the final version.
4. Testing of various types of instruments (mail questionnaires, face-to-face interviews, telephone interviews) in order to determine the suitability of each of them 2.
Rice. 28. Testing different types of tools
Aerobatics can be considered a dress rehearsal for the main exploration. It allows you to make the first conclusions about how successfully the standard of preparation for it has passed, what the expected results may be. In the course of such a survey, methodological information is collected, i.e. information on the quality of the developed tools. All organizational and methodological conditions for future research are checked, and above all: the availability of the necessary
Mannheim J.B., Rich R.K. Political science. Research methods. M .: Ves mir, 1997.S. 211-212.
documents, the attitude of respondents to the survey and their reaction to the questionnaire. Assessing the quality of the survey methodology, the sociologist registers any difficulties of the respondent caused by the filling technique, misunderstanding of the meaning of the questions, individual words. All comments and observations of the questionnaires about the methodological shortcomings that they managed to find are also taken into account.
After returning from the survey, it is better to immediately sit down at your questionnaire, look through the answers to all the questions, mark the most problematic ones, and then go to the object and talk with the respondents.
Aerobatics is carried out in two versions. In the first, the entire group of respondents is invited to a separate room, where questionnaires are laid out on the tables. The respondents are “introduced” to the situation of aerobatics, i.e. explain its goals and objectives, instruct on the technique of filling out the questionnaire and ask them to express critical comments after filling out, talk about ambiguities and those issues that make it difficult to work with the questionnaire. The completed questionnaires are dropped into sealed ballot boxes, after which a group discussion of the content of the survey as a whole begins. Indicators that make it possible to assess the suitability of the questionnaire are the number of those who did not answer the question and the nature of the distribution of answers on the scale used in the question.
The second version of the pilot survey involves inviting a group of 3-4 respondents to discuss the questionnaire as it is completed: the respondents make notes about the quality of the questions. Before the start of aerobatics, they are introduced to the situation of an expert survey, explaining that the content of the answers is not analyzed, since the survey pursues only methodological goals.
What is a pilot study? What is the purpose of it? What tasks is it aimed at?
general information
First, let's define what a pilot study is. This designation is used to refer to pilot or small search (reconnaissance) checks of the status quo. So, if you need to clarify the problem, more correctly formulate the problem and put forward sound hypotheses, then a pilot study is the best option for this. There may be a special need for it in cases where there is no literature on the topic of interest. Then a pilot study is conducted to fill the information gap.
What is it?
Pilot research is a type of analysis in sociology, in which the range of tasks is significantly limited, the number of people interviewed is small, the data is unrepresentative, and the toolkit and program are extremely simplified. Because of this, the researcher receives only approximate information about what the object of research is. This knowledge is used for general orientation. The main slogan of aerobatic research is cheap, fast and approximate. Therefore, they are used in cases where the problem is either not studied at all, or knowledge about it is very miserable.
How is it done?
So, we already know that a pilot study is a case study. But how is it done? There are many different options here. The most optimal is an informal interview with potential respondents. But alas, people's subjectivity can influence their answers. To correct the data, they use supervision from the side of specialists. For this, a focus group can be formed. But then you should concentrate as much as possible on something specific. The survey of experts is also very popular. These include specialists or ordinary people, but who must necessarily have a certain attitude to the problem area that interests the researcher. As a supplement, you can study the documentation and statistical data, where there is the necessary information to confirm / disprove a hypothesis or solve a problem. Express polls are also very popular. True, they, as a rule, despite their literacy, do not set themselves the solution of deep scientific problems and the development of fundamental science. With their help, the momentary importance of something for society is recognized. At the same time, it does not matter what acts as an object: Trump's election as President of the United States, the prohibition of abortion, or something else. Be that as it may - data acquisition is carried out in order to superimpose them on larger-scale processes.
About reliability
How much can you trust the information received? Considering that the pilot study is a pilot study, this very fact implies a high share of risks. And if it is still carried out and not by specialists, but by a group of amateurs (which can be the personnel department, magazine, circle, owner of the website), then in this case, although there is fresh and necessary information, it, nevertheless, is unrepresentative, and its reliability is highly questionable. At first glance, it can be quite reliable. But if you approach it from a scientific point of view, then its flaws will open up. Therefore, it makes sense to use pilot studies only if there are no strict requirements for reliability. The sample should be affected initially. There are no clear methodological requirements here. As a rule, it is believed that a survey of 3 dozen respondents will provide the required information. But at the same time, you should make sure that among them there are representatives of all categories of people who fall under the study. In this case, you need to strive for maximum diversity. In addition, you should make sure that among the respondents there are people for whom the topic is of at least some importance. Sex, education, age, work experience and other similar criteria are used as qualifying criteria.
The importance of pilot studies
In general and in general, this aspect was considered earlier. Now let's dwell on this in more detail. From the name itself it is clear that the pilot is conducted before the start of the main research. It is necessary in order to check the validity of problems and hypotheses. Although it can also be used for methodical development of tools. If necessary, a pilot study helps to make adjustments to the model that will improve its performance, to clarify the characteristics, subject matter, justify financial spending and completion dates. After all, if a full-fledged monitoring of the mood in society is carried out and an error creeps in somewhere, then its presence will be fraught with significant troubles. This approach has a beneficial effect on saving resources. Pilot studies can also be conducted to test the effectiveness and feasibility of using the available tools. They are also suitable as a dress rehearsal for the main study. In this case, the success of the first stage is checked and the results are evaluated. Also, when researching a new object, this allows you to develop methodological material. Organizational conditions are also checked at the same time: how the respondents relate to the survey being conducted, whether all the necessary documents are available and the quality of the material is assessed. At the same time, all the difficulties that arise in the course of the case are recorded.
Conclusion
The pilot study itself takes place, as a rule, in groups. The only question is how big they are. There are two of the most popular options. The first involves inviting all respondents to a separate room, where they fill out the questionnaire. Before that, people are informed about aerobatics, reported and explained its tasks and goals, inspected the nuances of filling out the questionnaire and asked to express all the comments. The second option relies on small groups of 3-4 people. In this case, the questionnaires are discussed as they are completed. The quality of the questions asked is of the greatest interest to researchers. In such cases, the most interesting are, as a rule, methodological goals.
