To read Part 1, click here.
1) Trust Your Observations. Even if you haven't proved them yet, know that your own observations in the field are important. Sometimes what you are saying will be discounted, but most great breakthroughs in science, economics, or business have begun with observations or hunches. So, have confidence in what you have observed and believe. Keep track of your hunches. Keep a daily journal. View your work as pioneering, that no-one has ever seen it before. Your observations can lead to valuable insights -- if not now, then for future generations. In 1992, I had a meeting with arguably the greatest entrepreneur of his time: Ewing Marion Kauffman. Unnerved by his rapid-fire questions about NFTE, I began to hedge on my dedication to my life's work. Without missing a beat, he looked at me and said: "Don't back down, Steve; you know you are right on this." I will never forget that moment: it gave me courage.
2) Research the Research. Before meeting with a potential donor, ask to see the data of the research that person is funding. Does it make sense? Do you understand it? Does the donor understand it? Ask questions. Learn from others in your field. View the research of others as an opportunity to learn and build on. If a donor is holding up research by another organization as the reason he or she is funding that organization and not yours, ask questions. Ask if your colleague's research uses random assignment, for example (if not, it is not state-of-the-art). Ask if the research is independent. Always ask to see the research instruments used and talk to the researchers directly, if possible.
Competitors in your field are also striving for resources -- money, talent, recognition, and media attention. Often, a competitor will be a foundation or corporation that is conducting its own program. Study them carefully and stay friends, so you can share insights and data. If it is a foundation, be aware of the pressure on the employees that are running their programs. Think of ways to help them, so they will not feel threatened by your success.
Finally, think through what information you want to collect and why? How will you use it to demonstrate the social return on investment of your program? Who will be interested in the information? Why? If the answer is that you want to collect the information, "just to have it," ask yourself if you have the capacity to work with all the data you will collect.
3) Find the Right Research Team: Internal or External? Research your research team! When you are ready to hire outside researchers, check their references carefully (if you hire the wrong team, all will be lost!). When you begin, make sure you partner with a well-established research partner. Ours were Brandeis University and then Harvard. Graduate students at Stanford collected our original data. People will take the results much more seriously if you're working with an established and distinguished institution.
Sometimes funders and researchers will belittle your research efforts. Don't be intimidated. Remember, they have million-dollar budgets and endowments that you do not. They are not in the trenches. They are supposed to be truth seekers, not self-important critics. Again, ask to see their research. Look them in the eye and stand strong and fearless. Your opinion matters.
Smaller organizations may not have the capacity to hire large, well-known research entities. A solution is to hire a reputable research or evaluation firm to help you design a project you can manage internally, and then analyze the results yourself. While this does not necessarily ensure scientific validity, it can provide a cost-effective way of collecting information. One of the keys to conducting internal evaluation using your own program staff is to design workflow to incorporate data collection, rather than making it an additional task or department. If you are dealing with students, an intake form designed to capture the information you want to gather, paired with an exit interview collected right before the student finishes the program, can be a simple yet powerful tool to measure change.
Hiring a skilled internal evaluation person can also be a cost-effective way to conduct scientifically sound research and evaluation. One of the benefits of hiring such an evaluator is that he or she can learn the workings of the program without being involved directly, which will decrease the risk of bias in measuring program outcomes. Another risk is that, unless you have a clear plan of how to use one, an internal evaluator can become unfocused or misdirected. It is important when hiring an internal evaluator to make sure there is adequate training for an effective partnership in deciding what to evaluate and how.
4) Participant Impact. There has been discussion recently about participants in a study being involved in the research evaluation. For example, a prison-education program could be evaluated through a study designed and conducted by the participating inmates, as they have the greatest access to the information. This kind of research is called Participatory Action Research, or PAR, and is becoming popular. The premise of PAR is the belief that participants are better equipped to ask and answer questions about the effectiveness of a project than those who are not directly connected.
There is obviously a concern that objectivity may be compromised in this kind of study. However, there are obviously valuable lessons to be learned from participants of a study. Some researchers argue that participatory action research can be conducted with scientific validity and rigor. Ray Tebout, a re-entry education expert and co-contributor to several PAR projects on criminal justice and re-entry issues has said, "My firsthand experience with the topics that I am researching gives me a unique insight on what to ask and where to look to find the information pertinent to the question at hand. I knew what and who to ask in a way an outsider to the topic never would have guessed."
Never forget that the voices of the people you are serving are essential. Ultimately, what they say about your program is as important as any research study. Be sure to collect their personal stories and testimonials. Follow up with alumni. At NFTE, we have an office dedicated to following up with our graduates for many years, to see how they are doing. In my opinion, what our alumni say about us is the best proof we have that our programs work.
5) Get Down in the Trenches. Require your research team to take your program themselves, and experience it thoroughly before designing a study for you. This aspect is often overlooked, so that researchers may not understand the intricacies of your program; the research will suffer accordingly. If the study is being done through a professor at a university, make sure you interview the graduate students who will be collecting the data. Since they will be doing much of the work, it is important that you meet and are comfortable with them. Be aware of group dynamics. In one case, our team of researchers had one "superstar" on it, but the other, less productive, members did not like her, so they worked against her until she resigned. After she left, the quality of the research declined immediately. You should be aware of any issues in the relationships between team members.
6) The Importance of Research Design. There are seven things you will need to think about in designing the research: a) What questions do you want the study to answer? b) What group(s) do you want to study c) What are you going to compare the results with? d) What outcomes are you looking for? e) What research are you going to use? f) What will your sample size be? g) Should you do a pilot study first? Listen to the individual you hired to do the research for advice on these issues. Read as much as you can on research design and ask questions about anything you do not understand. Never be afraid to admit that you do not know.
7) Pay Attention to the Research Environment. Question the validity of paper-and-pencil tests. I found this to be especially true for low-income young people, who are often in environments where it is hard to focus. I remember walking into the gym of a New York City high school and seeing students taking state exams while a gym glass was being conducted. The noise was overwhelming and the atmosphere chaotic -- not an environment conducive for children to give an accurate idea of their knowledge on a test!
8) Can Qualitative Data Help You? Do not discount the value of qualitative data. Anecdotes and stories are powerful motivators, and are a form of evidence that will be valuable to your program. Focus groups and narrative surveys are powerful tools for getting at information about where your program is working and where it needs to improve. Qualitative, or non-numerical, data is often a good guide for designing quantitative data research as well. These narrative responses can help you to identify categories you or your consultant may have overlooked. Additionally, the data collected in qualitative research can be coded and evaluated quantitatively (for example, measuring the number of times a certain work, concept, or phrase comes up). Key insights will come out of both quantitative and qualitative methodologies.
9) Look Further than Two Variable Linear Relationships. Many of the goals for research in youth work are finding two variable linear relationships. An example would be: if we provide an after-school tutor, graduation rates will go up. Beware of using this paradigm in the social sciences, particularly in education. I have found that dealing with youth in the field of entrepreneurial education is complex, multivariable, and nonlinear; there are many things that cannot be measured. In my field, the goal is to seek ways to change a child's life through teaching entrepreneurship and the concept of ownership.
10) Keep Track of the Research. This is critical: Never become disengaged from the data research of your organization. It is easy to delegate this task and not put your personal time into it, perhaps hoping that it will take care of itself. Emphasize to your organization from the top down how critical research is to getting your programs funded and achieving your goals. It may be time-consuming and expensive but stick with it!
11) Use the Data to Improve Your Impact. As you collect data, look for relationships. Spend time every month studying the data and thinking how it is related to previous observations and data. Always be in "pilot mode," so that you are constantly looking at research as a way to improve your organization. Even if the results seem to be neutral, or negative, think of research as an opportunity to improve yourself and your program. Just the fact that you are engaged in doing research gives your organization a reason to exist, and your observations and data collection could prove important in the future.
12) Sell, Sell, Sell! Try to sell your program to others. Ultimately, the best reflection of value is if other people are willing to pay money for what your organization does.
Note: Special thanks to Vanessa Beary of the Harvard Graduate School of Education for reviewing this article.