Goodwill Industries

e-Evaluation Technology & Process Solution Johns Hopkins University – Carey Business School IT Capstone Graduate Project Graduate Applied Project in Technology Executive Summary The JHU Capstone Team was approached by the Member Services Center (MSC) of Goodwill Industries to help them select an online survey application to eliminate their current paper-based event evaluation system. The paper-based system was not only time consuming and inefficient, but it was also not fulfilling their ‘Going Green’ environmentally-friendly strategic initiative.
The JHU Capstone Team began addressing this business problem by evaluating internal processes, collecting and analyzing specific functional requirements, and completing external market research and industry analysis. The internal assessment allowed the team to truly understand Goodwill’s needs, while the external assessment allowed the team to understand the various technological solutions available on the market. Although numerous requirements were identified, there were three that were of significant importance to Goodwill.
These requirements included the fact that the solution needed to: be compliant with 508 and the rehabilitation act; have minimal resource impacts on their IT department; and be low cost. It was these three requirements that helped the team assess and evaluate all three architectures. During analysis, it was quickly determined that Software as a Service (SaaS) solutions had multiple advantages over Commercial-of-the-Shelf (COTS) and Open Source Solutions (OSS) solutions. First, COTS products were either too expensive or required too many technical resources to implement.

Additionally, while OSS products provided best cost effectiveness, they normally required even more technical configuration than the commercial products. Finally, both COTS and OSS solutions may have required the purchase of hardware on which to install i Graduate Applied Project in Technology the software, therefore adding to the overall cost of these solutions. Although COTS and OSS solutions had been eliminated due to high costs and IT impacts, there were still a number of SaaS solutions that had the potential of fulfilling Goodwill’s requirements.
Each potential SaaS candidate was further evaluated against Goodwill’s top requirements. The JHU Capstone Team then narrowed the possible solutions to SurveyMonkey, SurveyGizmo, and Survey Methods. Each of these finalists was then further evaluated in terms of ease of use, professional presentation, and special features. The team selected SurveyGizmo due to its special features that would allow Goodwill to streamline their ‘thank you’ letter process. SurveyGizmo provided the capability to generate hidden questions that would tag responses based on speakers and sessions.
This hidden information would allow Goodwill to sort and quantify the ratings by speaker and/or session, thereby assisting with the generation of ‘thank you’ letters. In summary, the JHU Capstone team recommended that the MSC consider SurveyGizmo as the service provider for their survey evaluation process. Although Goodwill had in-house knowledge of SurveyMonkey, the JHU Capstone Team believed that the special features offered by SurveyGizmo would make a positive impact on the analysis and creation of ‘thank you’ letters. ii Graduate Applied Project in Technology Table of Contents
Executive Summary ………………………………………………………………………………….. i  1. 0  Overview ………………………………………………………………………………………….. 1  1. 1  Background Information ……………………………………………………………….. 1  2. 0  Business Problem ……………………………………………………………………………… 2  3. 0  Project Objectives & Scope ……………………………………………………………….. 3  3. 1  4. 1  4. 2  5. 1  5.   5. 3  Project Constraints ……………………………………………………………………….. 3  Internal Assessment Team …………………………………………………………….. 5  External Assessment Team ……………………………………………………………. 6  Paper-Based Process …………………………………………………………………….. 7  SurveyMonkey Electronic-Based Process ……………………………………… 10  ‘Thank You’ Letter Process …………………………………………………………. 2  4. 0  Team Methodology …………………………………………………………………………… 4 5. 0  Current Processes …………………………………………………………………………….. 7 6. 0  Project Requirements & Success Criteria ………………………………………… 14  7. 0  Perceived Benefits and Improvements ……………………………………………… 15  8. 0  Industry Analysis ……………………………………………………………………………. 16  8. 1  8. 2  8.   Industry Market Research ……………………………………………………………. 16  Incorporating Surveys into the Organization ………………………………….. 21  Selecting a Solution ……………………………………………………………………. 22 9. 0  Industry Evaluation Process ……………………………………………………………. 26  9. 1  Non-profit Industry Evaluation …………………………………………………….. 26  9. 1. 1  Purple Heart ………………………………………………………………………….. 1  9. 1. 2  The Salvation Army …………………………………………………………………. 32  9. 1. 3  NISH……………………………………………………………………………………… 33  9. 1. 4  Cal Ripken, Sr. Foundation ……………………………………………………… 35  9. 2  For-profit Industry Evaluation ……………………………………………………… 37  9. 2. 1  American Airlines …………………………………………………………………… 39  9.   Government Industry Evaluation ………………………………………………….. 42  10. 0  Breakdown of Technological Solutions …………………………………………. 43  10. 1  COTS Solutions …………………………………………………………………………. 44  10. 1. 1  Select Survey ………………………………………………………………………….. 45  10. 1. 2  SurveyTracker ………………………………………………………………………… 46 Graduate Applied Project in Technology 10.   Software as a Service (SaaS) ……………………………………………………….. 47  10. 2. 1  SurveyMonkey ………………………………………………………………………… 49  10. 2. 2  SurveyGizmo ………………………………………………………………………….. 50  10. 2. 3  Survey Methods ………………………………………………………………………. 51  10. 2. 4  VOVICI and Cvent ………………………………………………………………….. 52  10.   Open Source Solutions………………………………………………………………… 53  10. 3. 1  Advance Poll ………………………………………………………………………….. 55  10. 3. 2  Lime Survey……………………………………………………………………………. 55  10. 3. 3  phpESP………………………………………………………………………………….. 56  11. 0  SWOT / TOWS Analysis ……………………………………………………………… 56  11. 1  11.   11. 3  11. 4  Strengths …………………………………………………………………………………… 58  Weaknesses ……………………………………………………………………………….. 58  Opportunities……………………………………………………………………………… 59  Threats………………………………………………………………………………………. 60 12. 0  Solution Matrix & Analysis………………………………………………………….. 61  13.   Cost Analysis ………………………………………………………………………………. 65  13. 1  Direct Costs……………………………………………………………………………….. 65  13. 2  Labor Costs ……………………………………………………………………………….. 67  14. 0  Recommendations ……………………………………………………………………….. 68  14. 1  14. 2  14. 3  14. 4  14. 5  Product Recommendation ……………………………………………………………. 8  Phased Approach ……………………………………………………………………….. 69  Phase I: Improved Process …………………………………………………………… 70  Phase II: Automation ………………………………………………………………….. 70  Phase III: Advanced Recommendations ………………………………………… 73 15. 0  Conclusions …………………………………………………………………………………. 74  16.   References …………………………………………………………………………………… 76  Appendix A – Statement of Work …………………………………………………………… 80  Appendix B – Full Requirements List……………………………………………………… 98  Appendix C – Survey Matrix ………………………………………………………………… 101  Appendix D – Cost Analysis ………………………………………………………………….. 104  Appendix E – ‘Thank You’ Letter Templates………………………………………… 06  Appendix F – Survey Best Practices………………………………………………………. 108 Graduate Applied Project in Technology List of Figures Figure 1 – JHU Capstone Team Methodology ………………………………………………… 4  Figure 2 – Paper-Based Learning Event Quality Improvement Process ……………… 9  Figure 3 – SurveyMonkey Learning Event Quality Improvement Process………… 11  Figure 4 – ‘Thank You’ Letter Process ………………………………………………………… 3  Figure 5 – Type/Purpose Chart……………………………………………………………………. 19  Figure 6 – Feedback Percentages ………………………………………………………………… 21  Figure 7 – Survey Vendor Decision Chart …………………………………………………… 24  Figure 8 – Non-Profit Industry Segments……………………………………………………… 27  Figure 9 – 2008 Non-Profit Contributions ……………………………………………………. 8  Figure 10 – Non-Profit – Internet Goals ……………………………………………………….. 30  Figure 11 – Non-Profit Internet Strategy………………………………………………………. 31  Figure 12 – SWOT Matrix………………………………………………………………………….. 57  Figure 13 – TOWS Matrix………………………………………………………………………….. 57  Figure 14 – Recommended Phased Approach……………………………………………….. 70
Graduate Applied Project in Technology List of Tables Table 1 – Percentage of Workforce in Non-Profit Key Functional Areas …………………… 29  Table 2 – Survey Matrix ………………………………………………………………………………………. 64  Table 3 – Estimate Cost Savings (Goodwill Estimates) ……………………………………………. 66  Table 4 – Estimate Cost Savings (Conservative Estimates) ………………………………………. 66 Graduate Applied Project in Technology 1. 0 Overview
Goodwill Industries International, Inc. (Goodwill) is a nonprofit organization dedicated to providing education and training services to people who may be disadvantaged by a financial situation, lack of work experience, or a disability. The non-profit organization offers a chance for individuals to enhance their lives through work opportunities. Goodwill’s mission is to “enhance the dignity and quality of life of individuals, families, and communities by eliminating barriers to opportunity and helping people in need reach their fullest potential through the power of work. (Goodwill, 2009) In this endeavor, Goodwill constantly strives to keep people relevant to the job market by offering a multitude of yearly training sessions. As part of its strategic initiative, Goodwill has also become a proponent of ‘Going Green’ – which initiative involves becoming more environmentally-conscious and ties nicely into the non-profit’s set of values, including: respect; stewardship; ethics; learning; and innovation. (Goodwill, 2009) 1. 1 Background Information Goodwill was founded in 1902 under the name of Morgan Memorial Goodwill Industries in Boston by Edgar J. Helms, a Methodist minister and early social innovator.
Helms collected used household goods and clothing in wealthier areas of the city, then trained and hired those who were poor to mend and repair the used goods. After these goods were repaired, he then resold them or gave them to the people who repaired them. Goodwill operates as a network of 184 independent organizations in the U. S. , Canada and 14 other countries. In 2007, Goodwill organizations collectively earned more than $3. 16 billion, 1 Graduate Applied Project in Technology and used 84 percent of that revenue to provide employment and training services to more than 1,113,000 individuals.
Today, Goodwill has placed 140,287 people in competitive employment jobs earning about $2. 3 billion dollars. It operates 2,246 retail stores, has 65. 4 million donors, has generated $3. 2 billion dollars in revenue and serves 1. 1 million individuals through employment and training programs. One of the many departments in the Goodwill organization is the Member Services Center (MSC). The MSC is responsible for organizing and administering employee learning events and sessions across the non-profit organization.
They host approximately five large events and many small events each year. These events can have anywhere from 100 to 500 attendees. Most, if not all events occur at hotels. On average, a small event consists of 10-15 sessions while a large event consists of 60-80 sessions with approximately 50-60 speakers over a 4-day time period. There are approximately four timeslots per day with seven available sessions per timeslot. 2. 0 Business Problem In order to ensure continued improvement and success, the MSC conducts evaluations ollowing the completion of each session and event. Although the goal of these evaluations is to collect feedback on a number of items, the main focus is to analyze the performance of the speakers and the conditions of the event facilities. The resulting analysis provides the MSC with the ability to determine which speakers and facilities they should consider utilizing for future events. The business problem investigated focuses on the process for the event surveys. Until very recently, each event evaluation was prepared, completed, and analyzed using a manual and 2
Graduate Applied Project in Technology paper-based approach. The lack of automation, coupled with the significant dependence on paper, results in a process that is costly, time-consuming, and inefficient. 3. 0 Project Objectives & Scope In order to increase efficiencies and support Goodwill’s current ‘Going Green’ initiative, the MSC has engaged The Johns Hopkins University Carey Business School Graduate Capstone Project Team (JHU Capstone Team) to explore automated, paperless, and cost effective methods for collecting, reporting on, and analyzing event evaluation data.
The following project objectives have been identified: • Analysis of current manual and paper-based process of distributing, collecting, and analyzing event evaluation and survey forms; • Identification of ways to improve the current process such that it is more efficient and streamlined (this includes the evaluation of survey content); • Identification and evaluation of electronic and web-based solutions that provide for an improved and more environmentally-friendly process; • • Identification of a streamlined solution for distributing ‘thank you’ letters; and Development of documentation to describe the new process.
Note: The implementation and installation of a solution is considered out of scope for the purpose of this project. 3. 1 Project Constraints Two significant constraints have been identified for this project. The first constraint pertains to the cost of the recommended solution. Because no actual budget has been identified for this project, the Goodwill MSC has indicated that the solution must be low-cost. This places a 3 Graduate Applied Project in Technology limitation on the number of products that can be evaluated by the JHU Capstone Team.
Furthermore, it should be noted that a low-cost solution may not be able to fulfill all of Goodwill’s needs and requirements. The second constraint pertains to the lack of available resources from the Information Technologies (IT) Department for installation and maintenance support for the recommended solution. This constraint not only reduces the list of possible products for evaluation, but it could also mean that the final solution may not fulfill all of Goodwill’s needs and requirements. 4. 0 Team Methodology The JHU Capstone Team was broken into several sub-teams for the purposes of focusing n different aspects of the project. Figure 1 provides an overview of the overall methodology. Figure 1 – JHU Capstone Team Methodology 4 Graduate Applied Project in Technology 4. 1 Internal Assessment Team Following the establishment of the project’s objectives and goals, the Internal Assessment Team completed further analysis of Goodwill’s business problem through the use of interviews and questionnaires. An additional meeting with Goodwill was used for information gathering purposes related to clarifying project requirements, expectations, and objectives.
More specifically, the following items were completed: 1. The Internal Assessment Team completed interviews and surveys with Goodwill to determine the functional and non-functional requirements for the project. 2. The Internal Assessment Team confirmed the project objectives, assumptions, risks, and scope with the client. 3. The Internal Assessment Team evaluated the current questionnaire content, ‘thank you’ letters, historical data retention process, and current analysis reports in order to assess Goodwill’s internal processes (i. e. distribution, collection and analysis of surveys). 4.
The Internal Assessment Team conducted hands-on testing of the current survey tool (SurveyMonkey) to better understand the process currently being performed. This testing also gave the team a better understanding of the evaluation form content. 5. The Internal Assessment Team attempted to gather non-functional requirements from Goodwill’s IT Department. Unfortunately, due to resource constraints at Goodwill, the team was unable to gather this information. 6. The Internal Assessment Team researched and evaluated the Goodwill environment in order to determine its current strengths and weaknesses. Graduate Applied Project in Technology 4. 2 External Assessment Team The External team was tasked with exploring the options that other companies and organizations have chosen for their survey needs by conducting extensive market research and industry analysis. This research pned both similar and non-similar companies to get an overall feel of what industries were using and what product would be the best fit for Goodwill. The External Assessment Team broke into three groups to further refine research.
The first group was tasked with collecting data on industry standards and practices as well as developing a market snapshot of what companies should look for when choosing a survey product. The second group contacted a number of companies to find out how they are handling the task of survey distribution and analysis. These company included in-industry and out, so that a clear picture and possible trends could be identified. The third group focused on specific product research. An exhaustive product matrix was developed to document investigated products and services.
This was used to analyze compatibility with Goodwill’s requirements. Many of these companies were engaged to explore financing and compliance requirements. The major deliverables met by the External Assessment Team can be found below: • Conducted industry and market research to gain knowledge and understanding of available industry solutions and study corporate direction, market space, consumer benchmarking, and competitive analysis. • Created an opportunity and threat matrix for Goodwill from the aspect of having a survey system as requested by the sponsor.
Data collected during market research and industry analysis was used to identify opportunities and threats affecting the Goodwill event team. • Identified, evaluated, and contacted many survey vendors to find an acceptable fit 6 Graduate Applied Project in Technology with the requirements. A detailed matrix has been developed showing all vendors and products contacted and how their solution meets or does not meet the defined requirements. • Developed a report based on the analysis of research conducted on similar organizations and the market.
The JHU project team provided a written recommendation for changes, modifications, additions, and/or replacement of current evaluation process. 5. 0 Current Processes This section provides an overview of the manner in which event evaluations are currently being conducted at Goodwill. For instance, event evaluations are currently sent out daily to all event participants. Furthermore, a final survey is distributed at the end of the event to collect generalized information about the event itself. Although these evaluations are currently sent out via an online application, SurveyMonkey, this was not always the case.
Previously, all surveys were paper-based which required event attendees to complete these surveys before departing for the day and Goodwill would manually tabulate the results and enter them into a spreadsheet for calculations. 5. 1 Paper-Based Process Goodwill’s Learning Event Quality Improvement Process is broken down into nineteen steps, as seen in Figure 2. Goodwill has expressed dissatisfaction with the overall time it takes them to complete this process, in addition to the amount of printed material it generates.
They understand that manual data entry is not only very time-consuming and prone to errors, but that it is also very costly and inefficient. Additionally the current evaluation process can take upwards 7 Graduate Applied Project in Technology of 3 months to complete. Goodwill would ultimately prefer that the evaluation process, including ‘thank you’ letter generation, be completed within one to two weeks of the event. Figure 2 depicts a very high-level overview of the paper-based Learning Event Quality Improvement Process. The process consists of evaluating events to obtain ratings on sessions, speakers and the overall event.
The evaluations are used to determine the type of ‘thank you’ letter that should be sent out to event speakers. Furthermore, the following bullet points summarize the steps of the paper-based process, which is depicted in Figure 2. • Goodwill staff members plan a conference featuring individual sessions; session times and titles are finalized 2 – 3 weeks prior to the event. • Goodwill staff members create surveys using a template document and then send them to OfficeMax for copying; labels are made for session survey envelopes. • Goodwill staff members sort evaluations by session and distribute them to session monitors. Conference attendees complete evaluation form and return them to the Goodwill registration desk. • • Goodwill staff members tabulate the results of the evaluations; copies are archived. Goodwill staff members send ‘thank you’ notes to the conference speakers, along with evaluation results. • Goodwill staff members discuss the survey results and contact approximately 20% of the conference attendees for additional feedback. • Final survey results are presented at Goodwill staff meeting prior to being archived. 8 Graduate Applied Project in Technology Figure 2 – Paper-Based Learning Event Quality Improvement Process (Goodwill Industries, 2009) Graduate Applied Project in Technology 5. 2 SurveyMonkey Electronic-Based Process Recently, Goodwill subscribed to SurveyMonkey as an interim solution to the problem. These actions were taken after contracting with JHU for the Capstone project, but prior to the first team meeting. SurveyMonkey is an online, web-based survey tool that can be subscribed to for as little as $20 per month. This tool allows Goodwill to distribute surveys to employees electronically via e-mail, allowing the participants to either complete the surveys at the event’s computer kiosks or at a later date when it is more convenient to them.
Furthermore, SurveyMonkey assists the MSC by removing the need for them to manually convert paper-based results to an electronic format. Figure 3 displays Goodwill’s Learning Event Quality Improvement Process, which has been updated for the inclusion of SurveyMonkey. The following bullet points summarize the steps of Goodwill’s SurveyMonkey process, which is further depicted in Figure 3. • Goodwill staff members plan a conference which will feature individual sessions; session times and titles are finalized 2 – 3 weeks prior to the event. • Goodwill Staff create a daily and overall survey form in SurveyMonkey. Goodwill staff members send evaluations at the end of each day and then again at the conclusion of the event. • • Goodwill staff members download reports and meet to discuss results. Goodwill staff members contact approximately 20% of conference attendees for further feedback. • • Goodwill staff members print and archive reports and analysis. Goodwill staff members send results and ‘thank you’ notes to session speakers. 10 Graduate Applied Project in Technology
Figure 3 – SurveyMonkey Learning Event Quality Improvement Process (Goodwill Industries, 2009) 11 Graduate Applied Project in Technology 5. 3 ‘Thank You’ Letter Process As described in the previous processes, Goodwill sends out a ‘thank you’ letter from the CEO to all the speakers of the event. Goodwill is currently frustrated by the length of time that this process takes to complete. Currently it takes Goodwill 5-8 weeks from the time the event is complete to the time the first ‘thank you’ letter is sent out. Goodwill expressed that these letters are very important.
Therefore, a streamlined and efficient process is needed. Figure 4 shows the process Goodwill takes to create and send out ‘thank you’ letters. After all the results from the evaluations are tabulated, Goodwill determines which version of letter should be used. Ideally Goodwill uses two types of ‘thank you’ letter templates. One template is used for speakers that have mostly all positive evaluations, and another template is used for the remaining speakers. The below bullet points summarize the steps of the ‘thank you’ letter process, which is further depicted in Figure 4. Goodwill staff members review and edit comments from each session and produce one to two sentences to use as feedback in the ‘thank you’ letter. • • Goodwill staff members find Session Speaker names and contact information. Goodwill staff members edit, print, and mail ‘thank you’ notes. 12 Graduate Applied Project in Technology Figure 4 – ‘Thank You’ Letter Process (Goodwill Industries, 2009) 13 Graduate Applied Project in Technology 6. 0 Project Requirements & Success Criteria In order to properly identify an appropriate solution, a list of requirements was necessary.
Goodwill provided the JHU Capstone Team with an extensive list of requirements. The full list of requirements and rankings can be found in Appendix B. This list of requirements was used to measure the qualifications of any specific tool that can potentially meet the needs of the MSC. Given the extensive number of requirements gathered for this project, the JHU Capstone Team asked Goodwill to provide them with priority requirements so that an appropriate solution could be found. Goodwill provided the following 6 requirements that they consider to be the most important.
It is therefore vital that the recommended solution fulfill, at a minimum, the following prioritized requirements: • • • • • • Comply with 508 and rehabilitation act; Require minimal resources from the Information Technology Department; Be low cost; Be able to complete automated analysis of the survey results; Allow for the creation, collection and analysis of multiple styles of questions; and Provide export formats compatible for Microsoft Office. In addition to these priority requirements, the JHU Capstone Team selected another six from the master list to create additional differentiation in the evaluation matrix.
These include: • • • Provide speaker ‘thank you’ letters as an output; Allow the sharing of results with others via a webpage; Allow users to create, review, update, and delete an unlimited number of surveys and survey questions; 14 Graduate Applied Project in Technology • Maximum number of responses allowed per month meets maximum number of responses possible from largest conference; • Allow users to create graphical representations of analytical results within the application; and • Convenient product support and hours of operation. 7. 0
Perceived Benefits and Improvements Some of the perceived benefits and improvements of an electronic-based survey tool include: • Increased efficiency – The new electronic-based solution will automate the distribution, collection, and analysis of surveys and results. It will reduce the amount of manual work required by the MSC. It will also make surveys easier to complete for employees by providing them the option to complete the survey when it is more convenient for them. Additionally, it provides the opportunity to automate the ‘thank you’ letter process. Cost reduction – The new electronic-based solution will reduce the cost of required man-hours associated with the organization and administration of the surveys. This solution will also reduce the costs for paper by eliminating the current paper-based solution. • ‘Going Green’ – The new electronic-based solution eliminates the need for paper copies. Given that over 5000 copies are made for the largest conference, this will make a significant impact on Goodwill’s green initiative. This solution also provides better security for the collected information than the paper version. 5 Graduate Applied Project in Technology • Better analytical results – The new electronic-based solution will provide a mechanism to provide instant access to results or store historical data. It will also have the ability to quickly tabulate results as needed for reports. 8. 0 Industry Analysis To properly evaluate the possible alternatives, the JHU Capstone Team determined it was necessary to understand how surveys were addressed in other sectors of industry. The team began by investigating general trends on the application of surveys and current technology developments. . 1 Industry Market Research In recent years, the use of survey applications has gained interest within industry. This is partially due to the fact that employee and customer satisfaction have become paramount for success in today’s highly-competitive market. To satisfy this growing desire to collect survey data, the technology used for analyzing collected data has had to expand and mature. Enterprise Feedback Management is a catch phrase that many technology circles are using to describe the latest evolution of surveys.
High-end solutions can provide such options as: permission levels to data access; expansive reporting capabilities; and historic data collection and analysis, etc. As can be seen, today’s survey applications come in varying shapes and forms – from the simplistic “bare-bones” versions to the ultra sophisticated enterprise model. In order to effectively and efficiently implement a survey solution, there are specific considerations that must be addressed. Engagement To begin, the scope of survey-specific requirements must be gathered and examined. As 16
Graduate Applied Project in Technology noted in a 2007 Gartner report, organizations must identify four main categories before being capable of implementing a successful survey (Kolsky, 2007). The first category consists of having the organization identify their target survey participants. By identifying target groups and / or individuals to take the survey, it becomes possible for the organization to create relevant questions. The second category requires that the organization define the purpose of their survey. This step is often overlooked even though it is considered extremely valuable in the survey creation process.
By exploring the purpose of a survey, organizations can better focus their efforts on improving their reputation and maintaining and /or improving customer satisfaction. The third category consists of determining the type of data and information the organization wishes to gather. It is not enough to simply find out if the survey participants are satisfied or not. There should be more thought put into the survey. For instance, organizations may wish to obtain data to backup and / or confirm known issues or problems that need to be investigated. The final category relates to timing.
Surveys should not be sent out weeks after an event because the information being requested will no longer be fresh in the recipients mind. Depending on the experience, this could considerably skew the data collected. Analysis Although most surveys appear to be similar, this is not necessarily the case. The way to distinguish a good survey from a bad one is based on the level with which an analysis can provide useful data. A major pitfall in the implementation of surveys surrounds the fact that organizations do not have a clearly defined goal and objective in mind for their survey.
It is absolutely vital that survey goals and objectives be defined prior to implementation. Note that 17 Graduate Applied Project in Technology goals can be defined relative to how the information collected will be utilized, whereas objectives can be defined relative to how the information that must be collected from the survey in order to meet the goal. It is important to understand that analyzing customer feedback is not an objective skill. According to Gartner, 75% of people polled in surveys provide high scores to an organization even if the organization did a poor job.
This can be attributed to many reasons including: people being afraid of letting the organization down; people simply not caring; or people simply seeing no real benefit in providing ‘true’ feedback. These issues make it even more important to identify the organization’s objectives and goals in order to match them to specific feedback. This will ensure that the collected data will be beneficial to the organization and will provide them with a better understanding of topics and issues directly affecting them (Kolsky, 2007).
Know What You Need, Ask What You Want Another major issue surrounds surveys that do not have questions tailored to collecting the data necessary to fulfill the survey’s goals and objectives. To avoid this pitfall, organizations need to ensure that they create questions that have very specific purposes and / or actions. This can be done by identifying expectations prior to creating questions, thereby ensuring that the questions are directed towards collecting the desired data. This methodology helps to confirm or quantify the data and information being pursued and collected.
Finally, by using this approach, organizations will reduce the amount of time analyzing data and therefore increase the amount of time they have available to take actions on the results. Simplicity is the Key Many times, surveys are created with rating scales of 1-x, with the inclusion of comment boxes at the end to allow for respondents to leave more personalized remarks. According to 18 Graduate Applied Project in Technology Gartner, these types of surveys tend to return lower than adequate response rates. Additionally, they can take longer to analyze and review because of the personalized comments that are obtained.
This may result in information that can no longer be used for the intended goal of the survey. On average, surveys that ask directed questions and avoid open-ended feedback questions tend to get much better response rates. Furthermore, these types of surveys can be more easily analyzed. Again, with the reduction of the amount of time spent doing analysis, more time is freed up to take action on the results of the survey. A best practice approach to creating effective surveys is to follow a three-phase model.
Figure 5 – Type/Purpose Chart (Kolsky, 2007) As seen in Figure 5, this model consists of three distinct tiers. The bottom tier begins with the Point of Delivery methodology. A survey developed using this method is short and to the point. They allow the organization to quickly identify issues and concerns for resolution. 19 Graduate Applied Project in Technology Moving up the chart, the next level is the Customer Satisfaction survey tier. With this tier, an organization would take a random sample of respondents from the Point of Delivery surveys and ask a larger set of questions.
These questions would be used to confirm the organization’s interpretation of the initial smaller surveys. The top level of this model, the Planning tier, would consist of surveyors asking detailed questions on what specific action the organization can take to improve within the next year or two. This tier allows the collection and analysis of specific data, thereby allowing organizations to act in an educated and expedited manner. This three-phase model has the potential to be highly beneficial to Goodwill. Specifically, the MFC can use these methodologies to help plan future training events.
Using data collected from past surveys, Goodwill can then select specific individuals to interview with a more specific and in-depth questions. Feedback After the survey has been completed, the final step involves informing the survey participants of what was done with their feedback. As a best practice, it is very helpful when organizations inform survey participants with the actions taken based on the results of the survey feedback. This step helps solidify the fact that the organization cares about the survey participant’s opinion and may have an added benefit of increasing future response rates. 0 Graduate Applied Project in Technology As seen in Figure 6 below, it is clear that the surveyed participants were more receptive to working on the surveys if they were made aware of how their opinion was being used and that appropriate actions were being taken. As can be seen in the diagram, both Business-to-Customer (B2C) and Business-to-Business (B2B) were greatly boosted by this feedback. Figure 6 – Feedback Percentages (Kolsky, 2007) 8. 2 Incorporating Surveys into the Organization Now that the basis for a strong survey implementation has been defined, it is necessary to iscuss the method that will be used to introduce the survey environment into the organization. As previously discussed, it is important to accurately align the goals and objectives of the organization into the survey questions. By taking these steps, the organization can create a clear path of expected benefits. In turn, this can be quantified by real dollar savings or by process improvements through the implementation of actions based on the survey’s results. The best approach to successfully implement survey results into the organization is to disseminate the results to all stakeholders.
By integrating the data into all affected business areas, the organization entrenches the usefulness of the information directly into its core business 21 Graduate Applied Project in Technology processes. It is important to remember that a survey is not a solution but rather a measurement device that indicates whether the solution is working. (Kolsky, 2007) As discussed previously, Goodwill has identified a need for collecting and analyzing valuable associate feedback pertaining to the organization’s hosted events.
By implementing a solid survey system, and identifying the appropriate goals and objectives, the MSC will be able to better understand the needs of its associates pertaining to these events. This will allow them to focus on developing and organizing the events, instead of evaluating survey results. Additionally, the ability to share the results with participants, subject matter experts (SME), and management will result in greater participation with future surveys. 8. 3 Selecting a Solution After a solid survey business case has been identified and approved, the next step nvolves the selection of an appropriate application and vendor. To successfully choose a solution that will work best in a particular environment, it is recommended that a few simple steps are followed: 1. All surveys tools and evaluation applications currently in use within the organization should be listed and catalogues for evaluation purposes. 2. A long term plan should be identified to determine the immediate and future survey needs in order to align the solution with the business case. 3. All survey-specific requirements should be gathered, documented, and analyzed.
It will also be necessary to clearly determine a list of what is absolutely necessary, as opposed to what items would be nice to have in the solution. 22 Graduate Applied Project in Technology Vendors The survey application industry has grown significantly in the past 15 years: It has grown from a few million dollars to approximately $7 billion dollars a year. Many companies around the world have begun exploring this untapped market to better serve the needs of those desiring survey applications. Currently there are more than 300 vendors in the survey marketplace.
Selecting an appropriate solution can, therefore, be a daunting task for any organization. Each of the vendors varies in depth, breadth, and price. Gartner, a leader in market research, has published six core criteria to help organizations select appropriate survey solutions. The six criteria consist of: 1. Architecture, 2. Cost, 3. Functionality, 4. Service, 5. Viability, and 6. Vision. Figure 7 displays these six criteria along with specifically assigned weights. The weights deal directly with what the market demands and what the vendors provide.
For instance, more weight is given to architecture because more vendors are moving towards service-oriented architecture due to the higher demand for customization. Additionally, functionality is given less weight because of today’s increased desire for commoditization. 23 Graduate Applied Project in Technology Figure 7 – Survey Vendor Decision Chart (Davies, 2008) Gartner suggests that these six criteria be used as a starting point, and that further discussions with internal users, finance, and information technology will be necessary. Below is a summarized explanation of each of the six criteria.
Architecture Architecture pertains to the medium that an organization chooses for their deployment, and consists of the interrelationships between interfaces of the solution to those already found within the organization’s infrastructure. Software as a Service (SaaS), Commercial-off-the-Shelf (COTS), and Open Source are three types of architectures that will be discussed in more detail later in the report. As expected, there are pros and cons to each, but all should be considered closely when choosing a final solution for an organization. 24
Graduate Applied Project in Technology Cost As can be expected, pricing for products can run a full range of the spectrum, and should considered when evaluating a particular a solution. It should also be noted that, just because a product may not have an initial upfront cost, it does not mean there is no cost associated with it. Items like licensing and support fees must be included in total cost comparisons. Functionality Functionality describes the desired function of a particular solution and is typically described in terms of inputs and outputs.
Functional requirements describe what a system is supposed to do. Features such as: survey design, delivery, and analysis; supportability; frequency management; event management sampling; and security are just a few of the areas that fall under functionality. Functional requirements are defined when gathering and analyzing requirements and should be listed in prioritized order. Service Being able to call for service and support may not be an import feature for everyone, but it may make or break others, particularly those with limited technical support in-house.
Support contracts and level of support should always be considered when looking at enterprise solutions. Viability Viability, or the “capacity for survival”, also needs to be taken into consideration when selecting a solution. If the solution is expected to last for many years, organizations will want to ensure that the company providing the solution will have a strong likelihood of being around just as long. 25 Graduate Applied Project in Technology Vision While looking for a specific product and / or vendor, it is important to remember that most solutions on the market today have a specific focus.
For instance, one vendor might focus their product on compliance management, while another one might focus on customer satisfaction. It is important to align the objectives of the organization with the product to ensure a good fit now and in the future. As time goes on, vendors will begin to focus more on specific industries, allowing their products to be tailored to fit the needs of their customers. 9. 0 Industry Evaluation Process In order to truly understand which products need to be evaluated for use by Goodwill, the
JHU Capstone Team needed to find out not only what types of products were currently on the market, but also what some of Goodwill’s “competitors” and other industry segments were using to achieve their feedback. The JHU Capstone Team investigated the three main industry segments: Non-Profit, For-Profit and Government. The team conducted both a general discovery of segment trends and closer investigation of selected organizations. 9. 1 Non-profit Industry Evaluation The non-profit sector, also known as the voluntary sector, includes organizations that are independent from the government and not part of the for-profit business sector.
This sector is also called the third sector, in reference to the public and private sectors. The non-profit sector includes many diverse groups such as advocacy/interest groups, think tanks, social movements, political parties, charitable organizations and religious organizations. As shown in Figure 8, the non-profit industry is segmented in different areas: 26 Graduate Applied Project in Technology Foundation (public); 3% Animal Related; 2% Environmental; 3% Religious; 7% Community and youth; 8% School; 9% Foundation (private); 2% Aquarium or zoo; 1% Healthcare; 22%
Higher Education; 10% Arts and Cultural; 10% Other; 1% Social Service; 21% Figure 8 – Non-Profit Industry Segments (Blackbaud, 2008) Income Despite the current economic downturn, half of the industry received contributions of more than $1 million in 2008, which is similar to 2007. The industry segments that receive the largest contributions were healthcare and social services. The complete breakdown of contribution among non-profits for 2008 can be seen in Figure 9. 27 Graduate Applied Project in Technology Figure 9 – 2008 Non-Profit Contributions (Blackbaud, 2008)
Although donations are expected to grow, the uncertain economic situation will affect how non-profits operate. Non-profits have experienced changes in the areas of total revenue and an increase in the demand for their services. In 2007, 75% expected increases in charitable donations versus 57% in 2008. Increases in the demands for the organizations’ services were expected to increase by 76% in 2008, but only a small percentage of non-profits are planning to have an increase in their hiring efforts to accommodate the high demand. Most organizations expect their staffing levels to remain the same. Blackbaud, 2008) While non-profit organizations have different sources for revenue, the sources that provide the highest revenues are individual donations, corporate donations, special events, and grants. Income from investments is the source that provides the least revenue. Non-profits, like any other organization, have many priorities and how they react to these 28 Graduate Applied Project in Technology greatly impact the organization’s performance. The highest areas of importance are donor acquisition and retention, information organization, operational efficiency, staff retention, and competition with other non-profits.
The top driver for non-profits is donor donations. Direct mail and special events are the most used methods for raising donations. Staffing Today’s non-profits face many challenges such as the increased need and the constant competition for donations to ongoing staff turnover. Even though some organizations have the lead in dealing with these challenges, other ones are still struggling on ways to successfully manage a non-profit in the 21st century. Many of the non-profits have attributed their success to three main factors: the board of directors, technology, and performance metrics.
As shown in Table 1, there is a small percentage of the workforce dedicated to IT compared to other key functional areas in non-profit organizations: Full-time 2008 Position Accounting Data/Computer Systems Marketing Major Gifts Grant Writing Internet/Web Direct Mail Prospect Research Planned Giving / Legacies Endowments 60% 37% 31% 25% 20% 16% 12% 10% 14% 7% Part of Someone’s Job 19% 37% 52% 59% 58% 59% 67% 67% 58% 61% Part-time Position 7% 5% 3% 3% 5% 4% 2% 3% 2% 2% Volunteer Outsourced Position 4% 5% 5% 4% 4% 6% 3% 4% 4% 4% 9% 15% 5% 1% 6% 12% 6% 4% 2% 2%
Table 1 – Percentage of Workforce in Non-Profit Key Functional Areas (Blackbaud, 2008) 29 Graduate Applied Project in Technology The majority of non-profits view the use of technology as vital to meeting their organization’s mission-critical objectives. However, only half have a technology budget and very few have a written technology plan. In terms of technology and Internet usage, most nonprofit organizations have a unified database that contains information about its donors and other essential components related to their operations.
Non-profit organizations have also realized that the use of the Internet will contribute to efficiency and versatility of the daily operations. Almost 98% of non-profits have a presence of the World Wide Web. Their Internet goals are to market the organization, educate the public, and raise funds. As shown in Figure 10, different functional areas in non-profit organizations have the responsibility of managing the content and driving the direction of the organization’s website. 40% 40% Percentage of Internet Usage 35% 30% 25% 20% 15% 18% 18% 11% 10% 5% 0%
Marketing Executive DevelopmentTechnology Services 10% 6% 2% Other Program Membership execution Non-Profit Functional Areas Figure 10 – Non-Profit – Internet Goals (Blackbaud, 2008) Non-profits also use other online tools as part of their Internet strategy. Some of these 30 Graduate Applied Project in Technology online tools are primarily used for fundraising purposes. As shown in Figure 11, 27% of nonprofit organizations use online survey tools as powerful research tools to obtain feedback from members, donors, event participants, employees, etc. Paper-based surveys can be costly and time-consuming.
Non-profits use online survey tools because they tend to be easier to use and because they can provide valuable feedback. Finally, non-profits use the collected data to determine: the kinds of activities that their donors are likely to fund; if their members’ needs are being met; and if they are able to improve 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 78% 72% 61% 52% 49% 33% 29% 27% 26% 23% 20% 19% 17% 17% 12% in the quality of their events and activities. 9. 1. 1 The Purple Heart “is a military decoration awarded in the name of the President to members of the armed forces of the U. S. ho are wounded or killed during war” (Purple Heart, 2009). Originally established in 1932, and now known as the “Military Order of the Purple 31 Em ai Fu ec l nd tro ra ni is c W in Ne eb g w sit sle e tte us rs ag e M Ev re em po en rts be tr eg ra ist cq ra ui t io si tio n Ad n an vo d ca m cy an ag Co em ns Po en t it lls t ue /s nt ur pr ve Vo of ys i le lu nt up ee da Re rm te ta an s il ( ag se em Di ll g sc en oo us t ds sio G on ro n up gr li n ou e) Fu ps nd /m ra es isi sa ng ge bo W ar eb ds lo gs (b lo gs ) Au ct io ns El Figure 11 – Non-Profit Internet Strategy (Blackbaud, 2008) Purple Heart
Graduate Applied Project in Technology Heart”, the organization’s mission was to protect all decoration holders as well as their families. The organization receives funds through the sale of used household items, the operation of thrift stores, the donation of automobiles, and the annual sale of the organization’s official flower: the Purple Heart Viola. (Purple Heart, 2009) It was discovered that the Purple Heart organization will use surveys to better understand public opinion and where to focus some of its efforts. One of the surveys that Purple Heart has online is the “Should Mental Wounds Qualify for the Purple Heart Medal”.
This survey was created using a content management website from the Pacific Bell Internet company. It allows users to create their own web pages without having to do any programming or without having to use / edit HTML by providing basic templates to create home pages. The company also offers a set of tools that allow users to edit, upload, and delete files from their own Home Page. It should be noted that the organization’s pages will have the http://home. pacbell. net/ web address followed by the user id and that Pacific Bell does not do web hosting. The cost for the first 15MB of storage is provided at no cost.
Additional space can be purchased at a cost of $1 per MB. Finally, it should also be noted that the results of the survey are displayed on a static web page as percentages and that there are no sophisticated reporting tools since the results are hard-coded into the web page. 9. 1. 2 The Salvation Army  The Salvation Army is one of the world’s largest providers of social aid. It is an international non-profit organization that assists the community with different programs such as missing persons, adult rehabilitation, Christmas charity, disaster relief, elderly services, combating human traffic, community care and recreation.
The Salvation Army operates in 118 countries worldwide, and provides services in 175 languages with expenditures including 32 Graduate Applied Project in Technology operating costs of $2. 6 billion in 2004. The organization helps more than 32 million people in the United States alone. The Salvation Army has a contract with Cvent for their event management. Cvent is an online software program that allows the organization to market, organize, and manage its events. The Salvation Army uses Cvent to allow attendees to register online to different events.
They also use it to send e-mail invitations, get confirmation on attendance, and to send reminders. It also helps the planning committee with budgeting, reporting, and the processing of payments for those events that are fee-based. Even though The Salvation Army uses Cvent for their bigger events, surveys for smaller events are usually done using SurveyMonkey, since it is free. Cvent contracts can be costly; especially if the organization wishes to take full advantage of the features such as the “Web Surveys” feature. 9. 1. NISH   NISH is “one of two national, nonprofit agencies that provide training and employment opportunities for people who are blind or have other severe disabilities” (NISH, 2009). This non-profit agency has offices in all 50 states, as well as the District of Columbia, Puerto Rico and Guam. In 2008, NISH reported a record 43,344 jobs for people with severe disabilities and more than 136,000 jobs for people with other disabilities. (NISH, 2009) NISH offers many learning opportunities for people with disabilities through their AbilityOne Program.
This program includes online classes as well as face-to-face class meetings. For their training sessions, NISH uses a learning management system called GeoLearning. This solution helps the organization in assessing their training needs, planning and delivering their training courses, and measuring their training quality. 33 Graduate Applied Project in Technology Even though NISH uses a learning solution for their training program, the measuring of the impact of their training events is contracted out to eLearning – MTM (Measures that Matters).
For the online classes, the evaluation forms are part of the course; however, for the face-to-face courses NISH uses paper forms to get feedback from training attendees. The questions on the paper forms come from eLearning – MTM. NISH also has a contract with Scantron who prints the evaluation forms, scans them to collect the data, and then provides the results to NISH in a way that they can use for analysis and reporting (NISH’s staff is responsible for further analysis). Once a year, NISH compares trainers and courses. The cost of the contract with Scantron is on average $777. 4 per year; however, the cost depends on how many events / courses are conducted, in addition to the number of evaluations that are submitted. NISH hosts many different events throughout the year; one of which is the Annual National Training and Achievement Conference. For this event, they use Race Management Software, which offers industry-leading software technology and marketing solutions to help event organizers better plan, organize, and market their events to participants. Their event evaluation process is designed to get feedback from participants on each session, in addition to feedback on the overall event.
They also use paper forms for this process, in addition to Scantron for the collection of the data. Currently, NISH’s manual process fits their needs for the following reasons: • They have an ability to read each evaluation individually, which gives them a flavor of what people really think about their events and / or training classes. They claim that by going electronically they will be able to only look at averages; with their manual process they can identify who is providing the feedback (upper management, middle management, etc) and determine how they can improve. 34
Graduate Applied Project in Technology • Their attempt to automate their evaluation process has failed in the past: Out of 1400 conference participants, less than 100 filled out their online survey. They feel that once the event or class is complete, participants have the tendency to ignore their request for feedback. • There are no computers at their events, and their training courses are not computer related – they are more for leadership skills, communication skills, etc. Because of this, participants would not be able to give feedback directly following a training session. Their ideal scenario consists of being able to implement a future computer-based evaluation process that will have the necessary hardware and software to allow feedback to be submitted directly after each event. It will also allow for the distribution of follow-up surveys. 9. 1. 4 Cal Ripken, Sr. Foundation  The Cal Ripken, Sr. Foundation (CRSF) is a national non-profit business that uses baseball and softball-themed programs to help build character and teach critical life lessons to disadvantaged youth residing in America’s most distressed communities.
CRSF is a 501 (c) (3) organization, founded in 2001 by baseball Hall of Famer Cal Ripken, Jr, 12-year Major League Baseball veteran Bill Ripken and members of the Ripken family. The Foundation honors the legend and spirit of Cal Ripken, Sr. who succumbed to lung cancer in 1999. During his 37 year career with the Baltimore Orioles organization, Cal, Sr. was a pioneer for his way of teaching the basics of the game as well as the basics of life to both big leaguers as well as their little league counterparts. The traits and lessons passed on by Cal, Sr. leadership, work ethic, responsibility and healthy living – are brought to life for youth using the 35 Graduate Applied Project in Technology power of baseball and softball. CRSF served youth in over 48 states and passes through over $3 million in grants to community-based organizations each year. To date, the Foundation has served over 650,000 under-resourced youth through their after-school character education programming. In order to assess their overall effectiveness, and to continue to refine their work with youth, the Ripken Foundation began contracting their evaluation program to outside consultants.
In 2007, CRSF began using pre- and post-surveys given to youth program participants in order to assess their change in attitudes after the duration of the program. With mixed results and low program attrition rates, the Foundation moved toward a measurement of attitudes, using online pre and post survey tools. The consulting company that CRSF paired with in 2008, EXACT Sports Inc (EXACT), utilized SurveyGizmo as their online survey software product. SurveyGizmo enabled EXACT and CRSF to customize the online surveys for each organization that the youth participants were from.
This allowed them to test for two different types of program implementations. Although EXACT created the online survey through SurveyGizmo, CRSF was able to have their own user name and password on EXACT’s account in order to access reports and check on the status of the evaluation surveys. This external and independent access to EXACT’s account was highly important to CRSF to ensure they were getting a high return rate on the surveys. It also allowed them to follow-up with youth organizations that were not in compliance.
EXACT was satisfied with the SurveyGizmo product and used the tool with all of their sports psychology evaluation clients because of its low-cost and professional appearance. EXACT utilized the many customization features to setup logical question sequences and filtered reports, and to change the colors and graphics based on their clients’ needs. 36 Graduate Applied Project in Technology Results for the Ripken Foundation evaluation were downloaded from SurveyGizmo into CSV files and then uploaded into statistical software packages like Clementine’s SPSS product for further data analysis, data mining, and assessment.
Results were then analyzed and reported back to CRSF for use in federal reporting and promotional materials to distribute to current and future funders. The ease in downloading the data from SurveyGizmo into compatible file formats also became a key factor for EXACT and CRSF when choosing the online survey software product. 9. 2 For-profit Industr