Evaluation of Business Productivity and Growth

Executive Summary

1.0 Introduction

Business Productivity and Growth is one of three sub-programs supporting Western Economic Diversification’s (WD) Business Development and Innovation Program, which ultimately contributes to the department’s strategic outcome of growing and diversifying the western Canadian economy. Business Productivity and Growth assists western Canadian businesses, business service providers, industry, and research organizations in enhancing business productivity, competitiveness, and growth of western Canadian small and medium-sized enterprises (SMEs). The sub-program’s three activities promote: 1) access to business development services, capital, and leveraged capital; 2) adoption of sound management and business practices and technology; and 3) access to skilled labour.

This document reports the findings, conclusions and recommendations of the evaluation of the Business Productivity and Growth programming delivered by WD over the five fiscal years spanning 2009–10 to 2013–14. In accordance with Treasury Board of Canada Policy on Evaluation, the evaluation methodology was designed to assess the programming’s relevance and performance. It included document/ literature review, file and database review, 74 key informant interviews, 101 survey respondents and three focus groups. A total of 87 Grants and Contributions projects totalling $90 million were approved during the evaluation period. As of December 2015, 63 (79%) of projects were complete. The evaluation findings are presented according to the programming’s relevance and performance.

Although they are part of the Business Productivity and Growth sub-program, the members of the Western Canada Business Service Network (WCBSN) were not included in this evaluation given that their program delivery and performance issues differ substantially from the funded projects. Furthermore, two members of the WCBSN were recently evaluated.¹

Since 2014, WD has introduced several significant changes including: implementation of a new streamlined Program Alignment Architecture; introduction of the Western Innovation Initiative (WINN) to assist western businesses to bring innovative technology-based products, processes and services to market; launch of the Western Diversification Program (WDP) call for proposals process; and adoption of new strategic priorities to promote economic growth and the diversification of the western Canadian economy. The introduction of the call for proposals process for both WINN and WDP has enhanced WD’s responsiveness to industry needs by establishing a more regular intake schedule and clarifying the criteria used to assess applications under these programs. In addition, WD has leveraged the call for proposal process to enhance its outreach and engagement with external stakeholders. Consequently, some of the evaluation findings, which covered projects under the continuous intake process have been addressed or resolved as a result of the introduction of the call for proposal process.

2.0 Relevance

2.1 Relevance – Continued Need for Programming

Respondents indicated that the Business Productivity and Growth programming fills an important gap by funding small and medium-sized enterprises (SMEs) to attract and retain skilled labour through improved access to capital, management training, building capacity and strengthening collaboration across stakeholders. There is a continued need for the programming as research shows Canada continues to underperform in terms of productivity, SMEs play an important role in the Canadian economy and could benefit from assistance. Furthermore, focus group participants indicated that WD could do more to promote economic diversification and develop emerging sectors.

Although there are many federal and provincial SME support programs, the department’s Business Productivity and Growth programming is needed and does not overlap or duplicate other similar programs.

• Out of 51 proponents consulted, 69% of proponents estimated that there was 0 to 25% chance that their project would have proceeded in some form without WD assistance and only 12% indicated that their project would have proceeded in the absence of WD funding.
   
• Without departmental support, proponents estimated only 5% of projects would have proceeded as planned; most projects would have been cancelled (39%), reduced in scope (31%), or delayed (15%).
   
• Of the 18 projects that were not approved for departmental funding, two proceeded as originally planned, four (22%) were cancelled, seven (39%) were reduced in scope and five (28%) were delayed.

2.2 Relevance – Alignment with Departmental and Federal Government Priorities

Business Productivity and Growth programming aligns with the federal government’s objectives to stimulate economic growth, job creation and broad-based prosperity and supports the department’s strategic outcome of growing and diversifying the western Canadian economy. Key informants agree that the programming supports Government of Canada and WD priorities particularly with respect to economic growth, job creation, innovation, trade, and economic development for Indigenous Peoples.

2.3 Relevance – Consistency with Federal Roles and Responsibilities

The federal government has a key role to play with respect to promoting economic growth, job creation and broad-based prosperity. Support for business growth and productivity creates new jobs and industries and, therefore, contributes to improving prosperity.

Forty-eight (81%) proponents and 15 (79%) representatives of unfunded projects indicated that the federal government’s support for small and medium-sized enterprises (SMEs) is appropriate and that the federal government can act as a catalyst in business development by helping SMEs address barriers.

In summary, WD’s Business Productivity and Growth sub-program continues to be relevant and is well positioned to support economic growth, increased productivity and job creation in Western Canada. It is also well aligned with departmental and federal government priorities.

To enhance relevance and increase programming impacts, the literature identifies several best practices for the department to consider including: focusing on high growth businesses; adopting advanced technology; and promoting international trade.

Recommendation 1. The department should continue to identify and adopt best practices to enhance the ongoing relevance of the business productivity and growth programming.

3.0 Performance

3.1 Performance – Achievement of Expected Outcomes

Although some of the projects included in the evaluation were still ongoing, the survey reported that the projects were realizing short term results primarily related to skills training or employment.

Performance information reported to WD indicated that most completed projects met or exceeded their target expectations. Within the period of the evaluation, 3,398 businesses were created, maintained and/or expanded, 5,596 jobs were created or maintained, and business sales increased by close to $154 million. Reasons for completed projects not meeting or only partially meeting targets included unforeseen delays, poor project administration or in the case of one project, inclement weather.

Progress reports from projects that were in progress indicated that these projects were on course to either meet or exceed their target expectations.

• Client satisfaction with projects that involved participation in and/or delivery of community workshops, seminars, presentations and special events ranged from 76% to 100% exceeding set targets.
   
• Sixty-two percent of project proponents reported that other individuals, businesses or organizations also benefitted from their project by improving their market share/competitive position, obtaining information for future endeavours and maintaining/creating partnerships.
   
• There were some positive unintended impacts such as increased awareness of business productivity among SMEs and increased awareness of departmental programming among other potential proponents.

Success factors included collaboration between industry and other project partners, projects that were well timed in terms of demand or general economic conditions, leveraging other sources of funding, alignment with provincial priorities, and programming flexibility in terms of both type and implementation of projects.

Factors limiting success included limited time frames to complete projects, types of costs eligible for WD funding and annual project budgets. Some focus group members commented on WD’s recent narrower focus on skills development to the exclusion of other types of Business Productivity and Growth projects.

In terms of quality and usefulness of performance measurement, results on the 87 projects are primarily reported at the immediate and intermediate outcome levels. In addition, there are 165 unique indicators which cannot be captured systematically.

• The majority of participants agreed that the performance measures and reporting requirements are appropriate and that WD’s performance measurement system is effective in tracking and reporting on outcomes.
   
• Most departmental staff perceive that an appropriate risk management strategy exists and that the main risks relate to projects not achieving results or not being implemented as planned. Risk is managed throughout the lifecycle of the project and addressed primarily through the monitoring, reporting and claims payment processes.
   
• Eighty-five of the 87 projects were classified as low risk (68 projects) or medium risk (17 projects), showing the department tends to fund low risk projects.

In summary, the 87 projects covered in the evaluation are realizing short term objectives, however, it is not possible to determine longer term impacts. Nevertheless, proponents reported many direct project results as well as positive unintended impacts.

Recommendation 2. The department should improve how it measures and reports on longer term outcomes in order to enhance the business productivity and growth programming’s ability to tell its performance story.

3.2 Performance - Demonstration of Efficiency and Economy

• The department recently adopted an efficiency indicator for Business Productivity and Growth, which is stated as “operating costs per $1,000 in Grants and Contributions (Gs&Cs) expended”. The total Gs&Cs invested by the department in Business Productivity and Growth were $23,390,759 in 2013–14. Operating expenditures for the period were $3,060,317 resulting in an efficiency indicator of $131. Overall, operating costs as a proportion of Gs&Cs expenditures for Business Productivity and Growth activities compare favourably to other departmental Business Development sub-programs.
   
• The 87 projects leveraged $1.67 per dollar of departmental G&C funding approved. This leverage rate compares favorably with several similar federal programs.
   
• The projects reporting outcomes during the study period cost $7,953 in G&C funding per business created, maintained, or expanded and $2,206 per job created or maintained. The completed projects also generated $26.00 in increased sales per dollar cost.

The programming is well delivered, however, it could be more flexible and responsive.

• Although the new call for proposal process improves response times, suggested improvements included more predictability in the intake schedule and exploring ways to reduce the level of effort spent by applicants in preparing initial proposals.
   
• Other suggested improvements included continuing to strengthen and leverage partnerships in delivering programs and exploring funding for emerging and riskier sectors (e.g., new media, clean energy).

In summary, the efficiency of the Business Productivity and Growth sub-program is comparable to other similar federal programs. The programming is well delivered although it could be more flexible and responsive to client needs.

Recommendation 3. The department should review its business productivity and growth programming to improve flexibility and responsiveness.

[1] 1) Evaluation of the Women's Enterprise Initiative; 2) Evaluation of the Canada Business Network: www.ic.gc.ca/eic/site/ae-ve.nsf/eng/h_03696.html

Section 1: Introduction

1.1 Acknowledgement

Western Economic Diversification Canada (WD) would like to thank all of the key informants and survey participants who generously gave of their time and knowledge to take part in the Business Productivity and Growth Evaluation. Without their participation and their insights, this report would not have been possible. The evaluators acknowledge the work done by Ference Weicker & Company Ltd. (the consultants) in collecting key informant, focus group and survey data.

1.2 Background

Business Productivity and Growth is one of three sub-programs supporting Western Economic Diversification’s (the department’s) Business Development and Innovation Program, which ultimately contributes to the department’s strategic outcome of growing and diversifying the western Canadian economy. Business Productivity and Growth assists western Canadian businesses, business service providers, industry, and research organizations in enhancing business productivity, competitiveness, and growth of western Canadian small and medium-sized enterprises (SMEs). The sub-program’s three activities promote: 1) access to business development services, capital, and leveraged capital; 2) adoption of sound management and business practices and technology; and 3) access to skilled labour.

Business Productivity and Growth has been one of the department’s organizational priorities from 2009–10 to 2013–14. During this period, 87 Business Productivity and Growth projects, totalling $90 million in departmental funding, were approved under the Western Diversification Program Authority (WDP).

Historically, the department has supported business productivity and growth because:

• Productivity and growth is crucial for ensuring long-term economic growth and improving the overall standard of living of western Canadians;
   
• Western Canadian SMEs need to produce goods and services in an increasingly efficient and cost-effective manner to compete in a global marketplace, or risk losing market share in both domestic and international markets; and
   
• Western Canada lags in business productivity when compared to other major industrialized economies, largely “due to differences in the rate of adopting new technologies, business practices and innovations, research and development expenditures, as well as machinery and equipment investments”.²

The department has implemented this sub-program by investing in productivity initiatives that encourage the development and adoption of innovative business technologies, processes and practices, including efficiencies in strengthening regional supply chains and by supporting capacity-building initiatives, including skills training in critical sectors which were facing labour shortages in Western Canada.

1.3 Evaluation Context

Under the Government of Canada’s Financial Administration Act and the Treasury Board of Canada Secretariat (TBS) Policy and Directive on Evaluation, departments are required to undertake an evaluation of all ongoing programs of Grants and Contributions every five years. The department’s Business Productivity and Growth programming was last evaluated as part of the 2008 Western Diversification Program Evaluation. WD’s approved 2013–18 Evaluation Plan committed to evaluating the Business Productivity and Growth sub-program. This evaluation covers the five fiscal years 2009–10 to 2013–14.

Although they are part of the Business Productivity and Growth sub-program, the members of the Western Canada Business Service Network (WCBSN) were not included in this evaluation because their program delivery and performance issues differ substantially from the funded projects and combining them with the projects could obscure important issues. Furthermore, two members of the WCBSN were recently evaluated.³

Evaluation Scope and Objectives

The evaluation focused on the relevance and performance of the department’s Business Productivity and Growth programming. The purpose of the evaluation was to assess the extent to which the Business Productivity and Growth programming supported western Canadian small and medium-sized enterprises and contributed to developing and diversifying the western Canadian economy. Key evaluation issues are outlined in the table below.

[2] Western Economic Diversification Canada. 2013–14 Report on Plans and Priorities.

[3] 1) Evaluation of the Women’s Enterprise Initiative 2) Evaluation of the Canada Business Network.

Section 2: Evaluation Approach, Design and Methodology

The logic model underlying this theory-based evaluation⁴ was developed by the department’s performance measurement group, based on input from an inter-regional team with expertise in business productivity and growth. This logic model supported the refinement of the Performance Measurement Framework that was subsequently approved by the Deputy Minister. A non-experimental research design with multiple lines of evidence was considered appropriate and sufficient to meet the study objectives. A consulting firm contracted by the department gathered data from key informant interviews, surveys and focus groups; the evaluation unit analysed and integrated findings from all lines of evidence into a final report. To maximize the objectivity and relevance of the conclusions, the evaluation was guided by a steering committee, led by an Assistant Deputy Minister and representatives from three regional offices, and conducted in consultation with senior management.

2.1 Evaluation Study Activities

Preliminary Consultations

Preliminary consultations were conducted with the steering committee and senior management to finalize the evaluation methodology and framework.

Documents and Literature Review

Three groups of documents were reviewed as part of the evaluation:

• General background documentation (i.e., Treasury Board Submissions, the programming Performance Measurement Strategy, websites, documents that describe the programing rationale, history and theory);
   
• Departmental databases and website;
   
• Literature on trends and best practices in business productivity and growth.

File Review

The evaluation reviewed all financial and performance information contained in the department’s databases (Project Gateway and the GX financial system). Eighty-seven projects, totalling $90 million in departmental funding, were approved between April 1, 2009 and March 31, 2014. As of December 2015, 63 (79%) were complete or had their last claim approved. The majority of the projects were approved under the Western Diversification Program Authority (57 projects totalling $55 million in departmental funding) or the Western Economic Partnership Agreements (25 projects totalling $32 million in departmental funding). Most projects identified Improve Business Productivity (59 projects), Business Productivity and Growth (15 projects) or Technology Adoption and Commercialization (10 projects) as the sub-activity (Table 2.1).

Note: numbers may not add up due to rounding.

Client Survey

A total of 99 project proponents were contacted to complete a survey to collect outcome and economic information not captured in the departmental databases; 18 could not be reached and 59 (73%) completed the survey. Project proponents were asked for contact information for key stakeholders (participants) deriving direct benefits from the funded projects such as, individuals trained as part of a project; 23 (53%) participants completed a survey. The sample also included 19 of 65 clients (37%) that applied but did not receive funding; these clients represented a range of organization types such as local chambers of commerce and professional or industry associations. The regional distribution of survey respondents ranged from 43% (British Columbia), 26% (Alberta), 22% (Saskatchewan) and 10% (Manitoba).

Interviews with Key Informants

The consultants developed and pre-tested the questionnaires and then conducted individual key informant interviews by telephone. Most key informants were selected based on their familiarity and level of involvement with the programming. Of 160 key informants invited to participate, 74 completed interviews. There was a target of 40% (or minimum of 25 interviews) interviews from individuals least likely to have strong personal interest in the programming; 28 interviews (38%) were actually completed with this subgroup. Key informants were evenly distributed across regions: 21 (British Columbia), 16 (Alberta), 18 (Saskatchewan), 17 (Manitoba) and the remainder from headquarters or Ottawa. The selection criteria and sample sizes are summarized below.

• Project Proponents and Unfunded Applicants. All proponents and applicants were contacted to complete an online survey. A subgroup of survey participants was then chosen for interviews based on the survey results and their involvement in business productivity and growth-related programming. A total of 20 (12 project proponents and 8 unfunded applicants) were interviewed;
   
• Funding Partner Organizations. Selected organizations were those that: 1) represented a cross-section of types of funding partners (federal, provincial, private sector); 2) were involved in the greatest number of projects or amounts of funding; 3) were involved with business productivity and growth programming; and 4) were not Funded Project Proponents or Unfunded Applicants for this evaluation. A total of 10 organizations were interviewed;
   
• Other Government and Community Organizations. Selected organizations were those that: 1) represented provincial, federal or community based organizations responsible for the delivery of business productivity and growth programming; and 2) were not Project Proponents, Unfunded Applicants or Funding Partner Organizations for this evaluation. A total of 18 organizations were interviewed;
   
• Other Stakeholders and Experts. Chosen individuals were those that: 1) were stakeholders and experts in business productivity and growth (academic, industry associations, other); 2) were not Project Proponents, Unfunded Applicants or Funding Partner Organizations for this evaluation; and 3) other recommendations and feedback from departmental representatives. A total of 10 stakeholders and experts were interviewed; and
   
• Departmental representatives. Staff and management were recommended to the consultant by departmental representatives and chosen to be those: 1) representing a cross-section of areas of specialization and seniority; and 2) most knowledgeable about the programming. A total of 16 staff were interviewed.

Focus Groups

The consultants conducted three focus groups, one in each of British Columbia, Alberta and Saskatchewan/Manitoba in January 2015. The Saskatchewan/Manitoba focus group was held in-person in Saskatoon with concurrent videoconference and teleconference in Manitoba. The participants were selected based on their knowledge and involvement in the programming and familiarity with respect to the needs of small and medium-sized enterprises in the region. In total, 28 representatives, including seven departmental staff, participated in the three focus groups: eight from British Columbia, eight from Alberta and 12 from Saskatchewan/Manitoba.

A consultant presented the field research findings at the focus groups and then facilitated group discussions. The objectives of the focus groups were to review and validate the preliminary findings. The topics under discussion varied somewhat from site to site, depending upon the composition of the groups, the interests of the participants, and the relevance of particular questions to that jurisdiction.

Economic Analysis

The consultant obtained expenditure and full-time equivalent (FTE) data for the 87 projects under review from the department. This data was used to calculate operating and maintenance (O&M) costs and Grants and Contribution (G&C) costs as a percent of total expenditures; G&C costs per FTE; and O&M expenditure per project approved.

Leverage rates were estimated by subtracting total project funding from total costs and dividing that amount by the total project funding. To assess outcomes relative to funding costs, the results of the project and administrative data and survey results were used to determine the aggregate impacts of the projects to date in terms of leading indicators such as, for example, the number of businesses created/maintained/expanded, the number of jobs created or maintained, and the dollar increase in sales. The leverage ratios and other measures were then benchmarked against other Grant and Contribution programs⁶ from similar economic development and diversification initiatives.

2.2 Limitations of the Methodology

Key informant interviews: Key informant groups directly involved in the programming were often knowledgeable about the impacts of one or more projects but lacked a complete picture of the projects and areas funded under the sub-program. Departmental representatives also struggled with separating impacts of the Western Canada Business Service Network partners⁷ from the projects funded under the sub-program. To mitigate this constraint, representatives were asked to provide specific project examples to support their ratings; key informants less familiar with the department’s programming and impacts were asked to identify major needs and challenges facing SMEs and the department’s role in addressing those needs. To minimize key informant response bias, 38% of the key informants comprised individuals least likely to have strong personal interest in the programming such as, other government/community representatives and experts.

Surveys: The evaluation findings are based, in part, on the views of those with a vested interest in the programming and potentially biased in their responses regarding programming outcomes. To reduce the impact of respondent bias and validate interview results, the survey questionnaire and letter communicated the purpose of the evaluation, its design and methodology, and strict confidentiality of responses clearly to participants. Moreover, the respondents were asked to provide a rationale for their ratings including a description of specific activities which contributed to the reported outcomes. Many projects had no key stakeholders (i.e. participants) associated with them and, therefore, the number of surveyed participants was low. To address this challenge, the responses of participants were used to complement responses from project proponents and add examples and details of areas where some projects have achieved particular outcomes or follow-on investments or projects.

Focus Groups:  The focus groups were used to validate and interpret evaluation findings. The focus group discussion reflects the opinion of some focus group participants and may not be representative of all those involved in the program.

Efficiency and Economy: The change in the department’s Program Alignment Architecture (PAA) in 2013 meant that the 2009–13 expenditures would need to be estimated and, therefore, would not be directly comparable with 2013–14. For this reason, the analysis examines only the 2013–14 expenditures. Using only one year of data limits the overall validity and usefulness of the findings in terms of stability (i.e., large G&C expenditures in one year vs. another year) and trends. There was also limited available data on outcome level results for ongoing projects, recently approved projects and completed projects that were not followed up to determine their longer term impacts. Moreover, because there was limited information on the context and rationale underlying observed differences, the analysis focused on benchmarking, rather than explaining, variations in economy and efficiency. Where possible, differences in economy and efficiency were interpreted based on key informant insights or programming considerations. Finally, there were difficulties identifying similar departmental sub-programs for comparison purposes because the size and complexity of projects vary considerably across sub-programs.

Attribution:  Ideally, projects would be followed over the long term to capture spin-off benefits and additional outcomes as projects evolve. However, this would not entirely solve the attribution challenge because long term outcomes reflect a myriad of interacting factors. As such, this study used a theory of change approach, assuming the achievement of short and medium term impacts would eventually lead to the longer term outcomes over time. The use of standardized questions improved response validity and reliability. Furthermore, both objective and subjective indicators (and therefore quantitative and qualitative data collection methods) were used to accurately address multidimensional concepts.

[4] Treasury Board of Canada Secretariat. "Theory-Based Approaches to Evaluation: Concepts and Practices". 2012. This document defines a theory-based approach as one based on a theory of change. A theory of change involves describing and explaining causal linkages between outputs and outcomes in a logic model.

[5] Both projects were under the Community Futures Capitalization Program and supported Community Futures investment funds.

[6] Benchmarking data was obtained from evaluations completed by Atlantic Canada Opportunities Agency, Economic Development Agency for the Regions of Quebec, Employment and Social Development Canada, Federal Economic Development Agency for Southern Ontario, Industry Canada, National Research Council and Western Economic Diversification Canada.

[7] Western Canada Business Service Network includes the following organizations: Canada Business, Women's Enterprise Initiative offices, Aboriginal Business Service Network and Francophone Economic Development Organizations.

Section 3: Relevance

3.1 Continued Need for Programming

Canada underperforms

Canada continues to underperform in terms of productivity which is defined as “the amount of value (as measured in contribution to GDP) created by the average worker in an hour of their time.”⁸ Many economists believe per capita GDP is the most important indicator of a country’s standard of living.

Canada’s productivity levels began to lag those of the United States (U.S.) in the 1980’s, creating a gap that persists today. “From 2009–2011, U.S. productivity continued to grow at a healthy 2.1% per year, while Canadian productivity grew an average of 1.1% per year. By 2011, Canada’s output per worker was only 78.3% that of the United States.”⁹

Productivity disparities are also evident across Canada. Although labour productivity in Western Canada, particularly in Alberta and Saskatchewan, exceeded the national average over the 2009–13 time period, Canada’s labour productivity remains a concern as it lags behind that of many other countries. Between 2001–2009, Canada’s annualized labour productivity growth was in the bottom quartile of the OECD and far below comparators like Australia, Austria and Israel.

Through its Business Productivity and Growth sub-program, the department supports initiatives that enhance productivity and growth of small and medium-sized enterprises (SMEs), an important input into Canada’s GDP and, by extension, Canada’s productivity.

Small and medium-sized enterprises need support

Research¹⁰ shows that Canada’s small and medium-sized enterprises (SMEs) are struggling and need assistance. SMEs, defined as companies with less than 100 employees, comprise 99% of companies in Canada and are significant drivers of economic growth.

There are recent trends that point to the need for greater support of SMEs in Canada:

• Unemployment in Canada is on the rise and SMEs represent a major source of employment. In 2012, 89.9% of Canada’s employees (10 million individuals) worked for SMEs: 69.7% (7.7 million) worked for small businesses and 20.2% (2.2 million) worked for medium-sized businesses¹¹.
   
• In terms of job creation, small businesses created, on average, 100,000 jobs per year (2002–12), accounting for almost 78% of all private jobs created. Medium sized businesses account for 1.6% of all firms and created 12.5% of net new jobs over the same period. In total, SMEs, therefore, were responsible for creating 90.5% of all private jobs between 2002 and 2012¹².
   
• A 2013 study by the Business Development Bank indicates that mid-sized firms are on the decline. This shifts the burden of job creation onto small firms. There was a 17% decrease (i.e., from 9370 to 7814) in the number of mid-sized firms between 2006 and 2010¹³.
   
• Business Development Bank’s Index of New Entrepreneurial Activity shows entrepreneurial activity has decreased since 2006 across all regions of Canada¹⁴.
   
• Over 6000 goods producing firms declined between 2000 and 2010. “Given that goods-producing firms are responsible for the largest share of R&D in Canada, their decline is tantamount to a decrease in the country’s innovative capacity.”¹⁵

Canadians support federal government assistance to SMEs

Canadians support entrepreneurship and business creation. The Global Economic Monitor (GEM) Report on entrepreneurship in Canada reported: 1) the majority of Canadians consider entrepreneurship to be a good career choice; and 2) experts knowledgeable on entrepreneurship in Canada agree, to some extent, with the statement: “the support of new and growing firms is a federal government high priority.”¹⁶ Despite federal government support, there is an opinion among Canadian executives of high-growth firms that Canadian entrepreneurs lack motivation to grow their businesses, citing examples of small Canadian firms “selling out to larger firms, rather than opting to build their company into a viable international competitor.”¹⁷

The Business Productivity and Growth programming fills an important gap

Respondent comments support the literature findings and confirm a continued need for the Business Productivity and Growth programming. Survey respondents indicated there is a need for programming which helps SMEs to attract and retain skilled labour and other workers, to improve access to capital and other investment attraction, to access markets and to strengthen internal marketing capacity. In addition to access to labour and capital, some key informants and focus group participants listed issues such as business management training and capacity building. Focus group participants also highlighted the need to create and strengthen collaboration across government and industry stakeholders. All departmental representatives confirmed the continued need for the programming, primarily because there is significant demand for this type of programming relative to the available funding and given that western Canadian SMEs lag in terms of business productivity and growth compared to other jurisdictions.

The programming effectively meets needs

As further confirmation of the need to support SMEs, 81% of project proponent respondents indicated that the department’s assistance met their needs and they were very satisfied with the assistance. Surveyed proponents and representatives of unfunded projects rated the department’s effectiveness in addressing their needs as 3.5 on a 5-point scale. Key informants thought the programming addressed needs and issues to some extent, particularly in regards to increasing access to skilled labour, adopting sound management practices and advanced technology, and business management training and capacity building. Focus group participants agreed that the department has been successful in responding to major needs.

However, some participants commented that more could be done to promote economic diversification and develop emerging sectors, particularly in British Columbia and Alberta where the economies remain largely dependent on resources. The department could develop other industries which, although risky, may ultimately contribute to diversifying the western Canadian economy. Departmental representatives suggested that the programming could address issues such as increased focus on adoption of sound management practices and advanced technology.

The programming complements other programming

The department’s programming forms an important component of the total SME support network in Western Canada. Key informants identified over 143 organizations and initiatives (24 national or federal, 24 in Alberta, 50 in BC, 24 in Manitoba, and 21 in Saskatchewan) that support business productivity and growth in Western Canada. The types of organizations and activities supported are described in Table 3.1.

Most key informants (86%) perceive that the department’s Business Productivity and Growth activities do not overlap or duplicate other similar programs, since the support is coordinated and leveraged with other sources of assistance. Western Economic Diversification also funds different types of costs (e.g., equipment) and has a different mandate and focus than the other organizations. The department coordinates and partners with other organizations at the project level (both directly and indirectly through contribution requirements in funding applications) and sub-program level (regular meetings with partners to find areas of complementarity and leverage support).

Out of 51 proponents consulted, 69% of proponents estimated that there was 0 to 25% chance that their project would have proceeded in some form even without WD assistance and only 12% indicated that their project would have proceeded in the absence of WD funding. In the absence of WD support, proponents estimate only three projects would have proceeded as planned; most projects would have been cancelled (23 projects or 39%), reduced in scope (18 projects or 31%), or delayed (9 projects or 15%). Of the 18 projects that were not approved for departmental funding, two (11%) proceeded as originally planned, four (22%) were cancelled, seven (39%) were reduced in scope and five (28%) were delayed. When asked why their projects were not funded, four indicated they did not know and six had been told their project did not align with the department’s funding priorities or criteria.

3.2 Alignment with Departmental and Federal Government Priorities

WD’s Business Productivity and Growth programming aligns with the federal government’s priorities to help Canadian businesses grow, innovate and export so that they can spur economic growth, create good quality jobs and broad-based prosperity for Canadians in all regions across the country. Small and medium sized enterprises play an important role in filling the gap between research and the marketplace and creating jobs for Canadians.

The Business Productivity and Growth programming supports the department’s strategic outcome of growing and diversifying the western Canadian economy. Key informants agreed that the programming supports the current priorities of the department and of the federal government, particularly with respect to economic growth, job creation, innovation, trade, and economic development for Indigenous Peoples.

3.3 Consistency with Federal Roles and Responsibilities

The federal government is committed to increasing prosperity across Canada. Increases in business growth and productivity create new jobs and industries, improving prosperity. Key informants noted that the federal government has a role in assisting the economy to grow and there is a need to invest in SMEs, innovation and productivity for various reasons including increased ability to compete successfully in international markets.

Forty-eight (81%) proponents and 15 (79%) representatives of unfunded projects indicated that the federal government’s support for small and medium-sized enterprises (SMEs) is appropriate. SMEs contribute to the economic growth of the country, provide employment, and contribute to economic diversification. SMEs also face particular barriers to improving productivity and attaining growth. The federal government can act as a catalyst in business development by helping SMEs address these barriers.

[8] Deloitte. "The future of productivity: Clear choices for a competitive Canada". 2012.

[9] Deloitte. "The Future of Productivity: A wake-up call for Canadian companies". 2013.

[10] Centre for Digital Entrepreneurship + Economic Performance (deepcentre). "Driving Canadian Growth and Innovation: Five Challenges Holding Back Small and Medium-Sized Enterprises in Canada". 2013.

[11] Industry Canada. "Key Small Business Statistics". 2013

[12] Industry Canada. "Key Small Business Statistics". 2013.

[13] Business Development Bank. "What's Happened to Canada's Mid-sized firms?" 2013

[14] Business Development Bank. "BDC Index of New Entrepreneurial Activity". 2013.

[15] Centre for Digital Entrepreneurship + Economic Performance (deepcentre). "Driving Canadian Growth and Innovation: Five Challenges Holding Back Small and Medium-Sized Enterprises in Canada". 2013.

[16] Global Economic Monitor (GEM). "Driving wealth creation and social development in Canada". 2013 GEM Canada National Report. As part of the research presented in the GEM report, an adult population survey of randomly selected Canadians aged 18-99 years and 42 experts were interviewed. The experts were from different professional perspectives where they gain considerable knowledge of entrepreneurial activities. The expert questionnaire was an instrument developed for the global GEM project. The questionnaire asked the expert panel for their views on various aspects favorable to entrepreneurship in Canada including: that the support of new and growing firms is a federal government high priority (on a 5 point scale of completely false, partially false, neither true nor false, partially true, completely true).

[17] Centre for Digital Entrepreneurship + Economic Performance (deepcentre). "Canada's Billion Dollar Firms: Contributions, Challenges and Opportunities." 2014. Interviewees included executives from fast growing billion-dollar firms, stalled and growing mid-tier firms and fast-growing young firms.

Section 4: Performance: Achievement of Expected Outcomes

4.1 General Findings

Although many of the projects were ongoing, most reported significant progress towards achieving their performance targets. Several projects addressed skills training and improving access to well-trained human resources, including skills training and employment for Indigenous Peoples. Many project proponents reported that other individuals, businesses or organizations also benefitted from their project; these other beneficiaries (e.g., participants) were involved in, for example, industrial marine training, mining, the oil and gas sector and small business development for Indigenous Peoples. When questioned about the success of their interaction with proponents, 95% of participants indicated they achieved their objectives and that their organization benefitted from improved market share/competitive position, information for future endeavours and maintaining/creating partnerships or contracts.

Departmental representatives identified a few unintended positive impacts such as increased awareness of business productivity among SMEs (i.e., lean manufacturing) and increased awareness of departmental programming (i.e., successful projects in skills development have led to increased requests from other groups). Few department representatives could identify any negative impacts resulting from the programming other than the narrower focus on, for example, skills development which limited investments in other types of projects focusing on SMEs.

Factors facilitating or impeding success

Survey participants mentioned success factors such as collaboration between industry and other project partners, projects that were well timed (i.e., the project fit with demand and aligned with government priorities) and assistance from various levels of government. Key informants noted a number of additional success factors including: leveraging other sources of funding; industry and partner engagement; effective project due diligence, selection and monitoring; and alignment with provincial priorities. Focus group participants listed similar success factors (i.e., leveraging, partner engagement and alignment) as well as flexibility in both type and implementation of projects. Specifically, the ease of communication with departmental staff in adjusting projects increased project success and programming responsiveness to industry needs.

Survey participants reported that the most common factors constraining success included: limited time frames to complete the project; narrow definition of eligible costs; and challenges related to evaluating, acquiring, and learning about a new technology. Some key informants mentioned factors such as lack of willingness among businesses to invest in business productivity or growth and the weak economy following the 2008 slowdown. Focus group participants noted the difficulty in accessing longer-term (i.e., two to five years) follow-on funding limited the success of some projects. Shorter projects (i.e., 1 to 2 years) hindered their ability to build momentum or generate longer-term impacts. Focus group participants indicated that some communities or organizations are unwilling to partner and that some proponents are unable to pay staff and up-front costs prior to departmental reimbursement.

A number of the comments raised by the key informants and focus group participants related to the delivery of WD’s programs prior to the introduction of the WDP call for proposal process. This new process has addressed some of the concerns raised.

4.2 Performance Measurement

To better focus the department’s activities, WD’s Program Alignment Architecture (PAA) was revised in April 2013. Therefore, the business productivity and growth projects span two PAAs with different programs and sub-programs. The strategic outcome in both PAAs focuses on developing/growing and diversifying the western Canadian economy.

Programs. Projects approved between April 2009 and April 2013 addressed the Business Development Program whereas projects approved between April 2013 and April 2014 addressed the new PAA Program - Business Development and Innovation. Both programs support western Canadian businesses, industry, and research organizations to enhance business productivity and global engagement. The expected result of both programs is western SMEs that are engaged in international business, competitive, and strong.

Sub-programs. Projects approved between April 2009 and April 2013 addressed four sub-programs under WD’s former PAA: 1) Improve Business Productivity; 2) Access to Capital; 3) Industry Collaboration; and 4) Technology Adoption component of the Technology Adoption and Commercialization sub-program. Projects approved between April 2013 and April 2014 addressed the Business Productivity and Growth sub-program under WD’s new PAA. The descriptions and numbers of projects corresponding to the business productivity and growth sub-programs are summarized below:

2009/10 to 2012/13 PAA

• Improve Business Productivity Sub-Program (59 projects). Support western Canadian businesses, business service providers, industry and research organizations to undertake initiatives to enhance business productivity and competitiveness.
   
• Access to Capital Sub-Program (2 projects)¹⁸. Improve access to risk capital for entrepreneurs and small businesses through financing and services offered in conjunction with other business services organizations and associations.
   
• Industry Collaboration Sub-Program (1 project). Supports industry collaboration, joint venturing and strategic alliances that support entrepreneurial development and growth of the business sector in Western Canada.
   
• Technology Adoption component of the Technology Adoption and Commercialization Sub-Program (10 projects). Supports small and medium-sized enterprises to increase adoption of technologies, products, processes and services in the marketplace.

2013/14 PAA

• Business Productivity and Growth Sub-Program (15 projects). Supports western Canadian businesses, business service providers, industry, and research organizations to undertake initiatives to enhance business productivity, competitiveness, and growth of western Canadian SMEs.

4.2.1 Immediate Outcomes

The department tracks the sub-program level immediate outcomes through the Performance Alignment Architecture (PAA) performance indicators . Table 4.1 provides the results of projects over the time period of the evaluation. The results provided include indicators that aligned with WD’s new PAA effective from 2013. Results aligned with WD’s former PAA effective from 2009 to 2012 have been included at the bottom of the table.

• About sixty three out of eighty seven (79%) projects were complete as of December 2015. Most completed projects met or exceeded their target expectations. Reasons for completed projects not meeting or only partially meeting targets include unforeseen delays, poor project administration or in the case of one project, inclement weather.
   
• Progress reports from projects that were in progress indicated that these projects were on course to either meet or exceed their target expectations. Results for ongoing projects will be included and assessed in future evaluations of Business Productivity and Growth programming.
   
• Examples of key results achieved through WD support for Business Productivity and Growth initiatives over the evaluation timeframe include: 3398 businesses created/ maintained/ expanded; 771 skilled workers hired as a result of training or skills development; 5596 jobs created or maintained; and close to $154 million increase in business sales.
   
• About 10 projects collected data on client satisfaction with business services provided. Most of the projects involved participation in and/or delivery of community workshops, seminars, presentations and special events to clients. Client satisfaction targets established for events ranged from 75% to 96%. The percentage of clients satisfied by participating in the events ranged from 76% to 100% thus exceeding set targets.

4.2.2 Intermediate Outcome

The intermediate goal of the Business Productivity and Growth programming is “Western SMEs that are engaged in international business, competitive, and strong.”

WD’s Performance Measurement Framework identifies Program Alignment Architecture (PAA) indicators at the program and sub-program levels. However, given the difficulty of measuring indicators at the program level, reporting is currently mandatory only at the sub-program level and few projects report at the program level. Of the 87 projects included in this evaluation, only four identified program level indicators: number of SMEs that increase gross margins (2 projects), SME employment growth (1 project) and number of highly qualified personnel (HQP) jobs created (1 project).

Literature, however, supports a link between the immediate outcomes and the intermediate outcome²⁰. Most economists agree that productivity growth is fuelled by “skilled workers, capable managers, scientific and engineering talent, and competitive pressure.”²¹

Although it is not possible to identify and assess the intermediate outcomes of WD’s investments in Business Productivity and Growth, interviews with key informants and reviews of individual projects provided examples of projects that demonstrated intermediate level results that allowed SMEs to add value and increase competitiveness. For example, WD support to two projects in film and television production in Manitoba contributed to increased competitiveness of the industry, job creation and foreign investment to the region.

Specifically, the department funded the On Screen Manitoba Incorporated project to support an economic development strategy for Manitoba’s screen-based media industry and the New Media Manitoba Incorporated project to support business development in Manitoba’s interactive digital media sector. Both projects have been completed and results included 204 businesses created, maintained or expanded. In addition, between 2009 and 2013, the co-production volume in Manitoba’s film and television production industry grew by 50% from $157.5 million to $227.8 million. Manitoba companies now own more intellectual property (IP) and the trend is rising. IP ownership represents a greater return on investment, stronger companies leading to better business opportunities, increased business capacity and job creation.

4.2.3 Long-term Outcome

Evidence that Business Productivity and Growth programming contributes to growing and diversifying the western Canadian economy

Although it is too early or not possible to measure long term impacts of WD’s Business Productivity and Growth projects, research shows that supporting SMEs is important to economic development. SMEs, defined as companies with less than 100 employees, comprise 99% of companies in Canada and play an important role in Canada’s economy:

• As of 2008, SME’s accounted for 52.5% of Canada’s private sector GDP;
   
• In 2011, SMEs were responsible for $150 billion or about 41% of Canada’s total value of exports;
   
• In 2012, SMEs in Canada employed almost 90% of Canada’s labour force; and
   
• In 2009, 31% of total research and development expenditures were attributable to small businesses²².

SMEs are particularly important to the western Canadian economy. Relative to the working age population, Alberta and Saskatchewan have more SMEs than other provinces, with ratios of 50.3 and 47.8 per 1000 population respectively in 2012. Both provinces also have a relatively high GDP per SME compared to other provinces across Canada. With respect to employment, Saskatchewan and British Columbia had the highest percentage of employees working for SMEs in 2012, at 93.3% and 93.7% respectively. Manitoba and Alberta’s rates were also relatively high at close to 92%²³.

Consequently, continued support to strengthen and grow SMEs is important in order to continue to grow the western Canadian economy.

[18] Both projects were under the Community Futures Capitalization Program and supported Community Futures investment funds.

[19] Both projects for this indicator have been completed. As at the time of the evaluation, the final results for one of the projects was not available in the database.

[20] Immediate outcomes: 1) access to business development services, capital, and leveraged capital; 2) adoption of sound management and business practices, and technology; 3) access to skilled labour. Intermediate outcome: Western SMEs are engaged in international business, competitive, and strong.

[21] Institute for Competitiveness and Prosperity. "Canada's innovation imperative. Report on Canada 2011".

[22] Industry Canada. "Key Small Business Statistics". 2013.

[23] Industry Canada. "Key Small Business Statistics". 2013.

Section 5: Performance: Demonstration of Efficiency and Economy

5.1 Economy

Efficiency Indicator

In 2013-14, WD implemented a revised Program Alignment Architecture (PAA) with Business Productivity and Growth as a separate sub-program within the Business Development and Innovation program. This revision to the PAA enabled the department to track operating expenditures related to the delivery of programming for Business Productivity and Growth. Since then, the department also introduced a common efficiency indicator at the sub-program level - “operating costs per $1,000 in Grants and Contributions (Gs&Cs) expended”.

With respect to the Business Productivity and Growth sub-program, the total Gs&Cs invested by the department was $23,390,759 in 2013–14 with $3,060,317 in operating and maintenance (O&M) expenditures. This results in an efficiency indicator of $131. The department therefore spends roughly 13.1% of its budget to deliver $1000 of programming in Business Productivity and Growth. Going forward, the department will be able to report on trends related to this new indicator.

O&M expenditures include staff costs related to G&C project management and the cost of activities related to stakeholder engagement and pathfinding. Although full-time equivalent (FTE) data by sub-program was not available for each year, FTEs dedicated to Business Productivity and Growth decreased from 43 (2009–10) to 39 (2012–13) and 29 (2013–14). For the 2013–14 fiscal year, G&C expenditure per FTE was $806,578 and O&M expenditure per project approved was $35,176.

Comparison to other programming.

WD’s efficiency indicator for the Business Productivity and Growth sub-program compares favourably to other sub-programs under WD’s Business Development and Innovation program. Efficiency indicators for sub-programs ranged from 2.2% (Innovation Capacity Building) to 13.1% (Business Productivity and Growth) to 14.3% (Technology Commercialization) to and 48.2% (Trade, Investment, and Market Access). Sub-program cost differences are primarily attributable to variations in G&C expenditures (for example, Business Productivity and Growth projects are usually of lower dollar value, however, the project assessment, management and stakeholder engagement can require the same level of effort as for much higher valued innovation projects).

Key informants and survey respondents were also asked for their opinions on programming economy. Seventy-nine percent of project proponents and 37% of representatives of unfunded projects agreed that the Business Productivity and Growth projects provide good value for money; 83% of project proponents could not identify opportunities to generate the intended outputs and outcomes at a lower cost. Of the 57% of key informants that responded, most agreed that the programming provides good value for money with respect to the use of public funds. Forty-two percent of representatives of unfunded projects did not know or did not answer the question on value for money.

Design of the programming.

Among key informants who were aware of the design of the department’s support for Business Productivity and Growth activities, most agreed that it was appropriate (i.e., target groups, types of activities funded, delivery mechanisms). Focus group participants were also generally satisfied with the approval, monitoring, reporting and claims processes. However, participants expressed some concern about the department’s recent change in the proposal intake process from continuous to call for proposals (CFP) process. Although the new CFP process improved response times, participants raised a few disadvantages in terms of unpredictable intake periods and more limited communication between staff and applicants. Another issue raised in the focus groups related to limitations with respect to transferring project budgets from one year to the next given annual budget allocations.

Although the department is addressing client needs, a few key informants suggested design and delivery improvements such as strengthening communication with proponents regarding program offerings (requirements, priorities and approval processes) (9 key informants) and strengthening and leveraging partnerships in delivering programming (6 key informants). The focus group discussions led to a variety of suggestions including supporting projects which promote diversification and strategic investments and developing new mechanisms of support (e.g., for smaller projects in the $100,000 range).

5.2 Efficiency

Leverage Rate

A total of $90.3 million in departmental G&C funding was approved for the 87 projects and $150.6 million was leveraged from other sources, yielding a leverage rate of $1.67 leveraged per departmental G&C dollar of funding approved. This leverage rate compares favourably with several similar federal programs but is lower than for others²⁴ as illustrated in the table below.

Return on Investment

WD’s return on investment compares favourably with similar federal programs reviewed. WD projects that reported results at the outcome level during the study period cost $7,953 in G&C funding per business created, maintained, or expanded and $2,206 per job created or maintained; these projects also generated $26.00 in increased sales per dollar cost. These figures compare favourably to cost per job estimates for FedNor’s Northern Ontario Development Program ($5,229 in G&C cost per job created or maintained²⁹ ) and for the Federal Economic Development Agency for Southern Ontario (FedDev Ontario) CME-SMART Program, which provides assistance to businesses to undertake activities to improve their productivity and competitiveness, ($7,751 per job created and $3,545 per job maintained³⁰).

Key informants and survey respondents were asked for their opinions on programming efficiency. Forty-two percent of proponents and unfunded project representatives had no suggestions for improving the programming efficiency or economy. Thirteen key informants regarded the programming financial and staff resources as commensurate with its objectives. Most key informant respondents perceived that the resources allocated to Business Productivity and Growth were being used efficiently (i.e., delivery costs are minimized) since funding was leveraged, the due diligence process is rigorous and staff costs have already been minimized. A few suggestions were provided to further improve efficiency such as allowing regional approval for smaller projects and easing rules around types of expenditures covered. Key informants provided a few suggestions with respect to alternative models or methods that would result in more efficient delivery of the activities, such as leveraging federal, provincial, and industry partners and other third parties in delivering programming. A few survey participants suggested improvements to the timeliness and communication of decisions and the department’s online presence (i.e., claims reporting).

5.3 Quality and Usefulness of Performance Measurement

The Performance Measurement Strategy for the Western Diversification Program Authority (WDP) outlines the performance measures, accountability and evaluation strategy. There is also a Data Collection Strategy for the WDP that defines each performance indicator and measurement approach which, for the Business Productivity and Growth performance indicators, involves program officers entering the data into the departmental database (Project Gateway). The performance measurement team routinely checks and corrects for data entry errors as part of their reporting process.

Access to accurate data is, at times, difficult given the challenges of the department’s project management system. However, this problem should improve as the department moves to a new, more accessible system.

In terms of data completeness, the database contained at least one sub-program level Program Alignment Architecture (PAA) indicator for each project. At the program level, however, only four projects identified a PAA indicator. There were 165 unique indicators, which cannot be captured systematically and used for the purposes of evaluation or decision-making. For example, almost one quarter (39) of the unique indicators pertained to number of people trained or job creation, however, because the indicators were made very specific (e.g., highly qualified personnel, Aboriginal, students), this information cannot be generated by the system or captured at the departmental level.

The majority of surveyed project proponents (71%) agreed that the performance measures and reporting requirements set by the department were appropriate. Key informants, and particularly proponents, agreed that the performance measurement system was effective in tracking and reporting on outcomes. Some departmental key informants noted that the performance measurement data captured by the PAA indicators has been useful for decision-making, particularly in terms of informing investment decisions and risk management, undertaking departmental reporting and planning and identifying needs. Focus group participants noted that attribution can be a challenge since it is difficult to isolate project impacts resulting from the department’s funding from what would have occurred in the absence of funding. It is also difficult to capture longer-term impacts since some impacts do not occur until well after the project is complete.

5.4 Risk Management

The department assesses individual project risk as part of the due diligence process. There is also a “Control Based Monitoring (CBM) Framework” for the Western Diversification Program. This CBM Framework outlines a risk based approach to manage projects for compliance to the Contribution Agreement and for maintaining appropriate financial controls. Risks are monitored throughout the lifecycle of the project and the adequacy of the controls, last assessed in 2011, were found to be appropriate.

Most departmental representatives perceived that an appropriate risk management strategy exists and that the main risks related to projects not achieving results or not being implemented as planned. Risks are addressed primarily through monitoring and due diligence processes. The department also tends to fund low risk projects. Risk ratings in the due diligence reports for the 87 projects showed 68 (78%) were low risk, 17 (20%) were moderate risk and two were high risk.

[24] Includes the following evaluations: 1) Economic Development Agency for the Regions of Quebec (CedQ). 2012. Summative Evaluation of the Business and Regional Growth Program; 2) Industry Canada. 2011. Final Evaluation for the Northern Ontario Development Program; 3) Western Economic Diversification Canada. 2008. Evaluation of the Western Diversification Program; 4) Western Economic Diversification Canada. 2013. Evaluation of Trade and Investment Activities; 5) Atlantic Canada Opportunities Agency. 2010. Evaluation of the Atlantic Canada Opportunities Agency Trade and Investment Program Sub-activity; 6) Western Economic Diversification Canada. 2012; Evaluation of the Innovation Activity; 7) Atlantic Canada Opportunities Agency. 2010. Evaluation of the Atlantic Canada Opportunities Agency's Entrepreneurship and Business Skills Development Program Sub-activity; 8) Employment and Social Development Canada. 2013. Evaluation: Aboriginal Skills and Training Strategic Investment Fund.

[25] CedQ: Economic Development Agency for the Regions of Quebec.

[26] Only includes programs that leveraged funds: AB/SK Centenaries and Canada Celebrates SK (ASC/CCS), Western Economic Partnership Agreements (WEPA), Urban Development Agreements (UDA) in Regina, Saskatoon, Winnipeg, Vancouver, and Canada/SK Northern Development Agreement.

[27] Includes: Conference Support Payments (2 projects), WEPA Round III (2008-12) (16 projects), WDP (41 projects), and WEPA Round II (2003-08) (4 projects).

[28] Includes: WEPA Round II and III, and WDP.

[29] Industry Canada. 2011. Final Evaluation for the Northern Ontario Development Program. The evaluation stated: "With respect to NODP costs per outcome, 23 of the 40 projects reviewed as part of the file review resulted in temporary and/or permanent jobs created and/or maintained. Those 23 projects created or maintained a total of 1,637 temporary and permanent jobs at a total NODP funding of $8,560,499, and a total project cost of $30,138,317. Therefore, based on the sample files, the cost to the NODP per temporary and permanent job created and maintained was $5,229."

[30] Federal Economic Development Agency for Southern Ontario. "Interim Evaluation of FedDev Ontario Programs. 2012. Cost per job calculated based on $38.85 million in FedDev Ontario contributions for the CME-SMART Program from 2009 to 2012 which resulted in 5,012 jobs created and 10,960 jobs maintained.

Section 6: Trends, Best Practices and Possible Improvements

Research shows there are over 500 federal and provincial programs that fund research, technology or firm development in Canada.³¹ The programs differ according to a variety of factors including focus, region, financial instrument (e.g., grants, contributions, loans, tax credits) or sector. Internationally, some research³² shows policy makers across the OECD countries are recognizing the importance of high growth firms (HGFs)³³, particularly high growth start-up firms. Supports for high growth start-up firms include grants, tax incentives, accelerators, incubators, access to funding and proof-of-concept funds.

Federal programs are delivered through Western Economic Diversification (WD) and other regional development agencies³⁴, other departments such as Industry Canada³⁵ (Canada Small Business Financing Program) or National Research Council (Industrial Research Assistance Program) and Crown Corporations such as the Business Development Bank. Regionally, one report³⁶ found that all Canadian provinces and territories have programs to support businesses; financial instruments include Grants and Contributions (non-repayable, repayable and conditionally repayable), loans, financing, tax credits and deductions and scholarship programs.

Best practices and possible improvements

The literature and the evaluation yield the following summary list of best practices and opportunities for improvements.

Best practices. Processes that are working well for the department and proponents include partnering and collaboration with industry and other organizations, aligning with provincial priorities, the rigorous project assessment and due diligence process, and the flexibility in both type and implementation of projects.

Possible Improvement: focus on high growth businesses. Research shows that high growth firms of all sizes drive productivity growth.³⁷ A study by Industry Canada³⁸ using Canadian business data (2000–2010) found that many SMEs have the financial resources to realize higher levels of growth than what they achieved. Potential actions to increase the numbers of high growth firms include management training for high growth and increased access to capital for businesses with high growth potential. Business leaders in Canada are considered to be more risk averse than their counterparts in the United States. Although many Canadian firms realize high growth rates in their first five years, they do not sustain high growth over the long term. In fact, one study³⁹ reported that 70% of surveyed Chief Executive Officers (CEOs) of emerging technology companies plan to exit their business within the next six years through acquisition or an IPO. Research⁴⁰ suggests SMEs could benefit from conditional loans and grants to encourage investment in training, research and development or technology and thus enhance their potential for higher growth.

WD focuses, to some extent, on management training under the expected result “increased adoption of sound management and business practices, and technology”. For example, some key informants indicated that the department supports this area through the Western Canada Business Service Network and Community Futures organizations which offer business counselling, seminars and workshops. A few key informants noted, however, that the department’s focus on this outcome has decreased recently as it did not align directly with WD’s strategic priorities. In terms of access to capital, some key informants commented that this has not been a big focus for the G&C projects, only a small amount of capital is provided and the department does not provide direct funding to SMEs. However, since the launch of the Western Innovation Initiative in 2013, WD is now providing loans to SMEs to advance efforts to bring technology based products, processes and services to the commercial marketplace. Furthermore, the department supports both business creation and expansion although it does not specifically target firms with high growth potential.

Possible improvement: increased adoption of advanced technology. Research⁴¹ shows that technology adoption increases productivity, innovation and competitiveness. SMEs tend to be slow adopters of technology, hindering their growth and productivity. Successful firms in all industries increasingly rely on technology and governments need to collaborate with industry to finance and encourage investment in technology.

Key informants noted that the department tends to fund more technology commercialization rather than technology adoption projects and that, in the past, the department has discontinued project funding too early and has been unwilling to reimburse operating costs required to advance the project. Going forward, the department could increase its focus on technology adoption projects and provide longer term, more flexible funding to these projects.

Possible improvement: promote international trade. Research⁴² shows that Canada must increase exports to raise competitiveness and productivity, however, only 3.6% of Canadian companies currently export. About 90% of Canadian exporters were small businesses in 2011, a small increase over 2008 (85%) and 1999 (87%). However, only 10.4% of SMEs were exporters that year⁴³.

Although trade is a federal government and WD priority, research⁴⁴ shows that organizations and businesses that need this support are frequently not aware of government programs. The federal government needs to increase awareness and accessibility of their programs to exporters.

WD’s programming supports international trade through the intermediate outcome “Western SMEs are engaged in international business, competitive, and strong”. When questioned about the programming impact related to this outcome, key informants indicated some projects have allowed SMEs to add value and increase competitiveness. On the negative side, some key informants were of the opinion that SMEs in Canada tend to rely on resources, are risk averse and inward focused, and, therefore, do not export. Furthermore, although trade is a departmental priority, some key informants noted difficulties separating federal from provincial responsibilities and the best mechanisms for support.

The department could review its programming to determine how to improve programming impacts related to international trade. Possible improvements include enhancing SME access and awareness of current programming and the benefits of international trade. Through greater focus on international trade, the Business Productivity and Growth projects can reinforce and augment WD’s current activities under its Trade, Investment and Market Access sub-program.

[31] PriceWaterhouseCoopers. “Response to R&D Review Panel Consultation Questions”. 2011.

[32] Mason C., Brown R. "Entrepreneurial ecosystems and growth oriented entrepreneurship." 2014. This paper references the following publication: Organization for Economic Cooperation and Development. "High-growth enterprises: what governments can do to make a difference". 2010.

[33] Eurostat – OECD Manual on Business Demography Statistics, Chapter 8. The OECD definition of high growth enterprise: "All enterprises with average annualised growth greater than 20% per annum, over a three year period should be considered as high-growth enterprises. Growth can be measured by the number of employees or by turnover."

[34] Programs include, for example: 1) Atlantic Canada Opportunities Agency: Business Development Program; 2) Canadian Economic Development for Quebec Regions: Quebec Economic Development Program; 3) Federal Economic Development Agency for Southern Ontario: Investing in Business Growth and Productivity; 4) FedNor: Northern Ontario Development Program.

[35] As of November 4, 2015, the Ministries of Science and Technology and Industry Canada were combined to form Innovation, Science and Economic Development Canada.

[36] Acacia Policy Consulting Inc. "Policies and Programs of Canadian Provinces and Territories. Mechanisms to Support SMEs and Established Aerospace Firms." 2012.

[37] Deloitte. A series reports including: 1) "The future of productivity: An eight step game plan for Canada"; 2) "The future of productivity: Clear choices for a competitive Canada"; 3)" The future of productivity: A wake-up call for Canadian companies"; 4) "The future of productivity: Smart exporting for Canadian companies".

[38] Industry Canada. "Small and Medium-Sized Enterprises Growth Study: Actual vs. Sustainable Growth". 2013.

[39] Pricewaterhousecoopers. "Building tomorrow's giants. 2014 Emerging companies' survey". There were 150 interviews completed with Canadian technology CEOs.

[40] Centre for Digital Entrepreneurship + Economic Performance (deepcentre). "Driving Canadian Growth and Innovation: Five Challenges Holding Back Small and Medium-Sized Enterprises in Canada". 2013.

[41] Centre for Digital Entrepreneurship + Economic Performance (deepcentre). "Driving Canadian Growth and Innovation: Five Challenges Holding Back Small and Medium-Sized Enterprises in Canada". 2013.

[42] Deloitte. "The future of productivity: Smart exporting for Canadian companies". 2014.

[43] Industry Canada. "Key Small Business Statistics". 2013.

[44] Deloitte. “The future of productivity: Smart exporting for Canadian companies”. 2014.

Section 7: Conclusions and Recommendations

7.1 Conclusions

Combining results from all lines of evidence collected during this evaluation led to the following conclusions by core issue.

Relevance

Evidence. Respondents indicated that the Business Productivity and Growth programming fills an important gap by funding small and medium enterprises (SMEs) to attract and retain skilled labour through improved access to capital, management training, building capacity and strengthening collaboration across stakeholders. Research shows Canada underperforms in terms of productivity, that SMEs play an important role in the Canadian economy and would benefit from assistance in this area. Moreover, Canadians support federal government assistance to SMEs. Key informants agreed that WD assistance to SMEs aligns with federal government responsibilities related to prosperity, economic growth and job creation. There are a large number of federal and provincial programs available to western Canadian SMEs. The evaluation found that the department’s programming does not overlap or duplicate other similar programs and effectively meets proponent needs.

Suggested improvements. The literature and the interviews support the relevance and ongoing need for Business Productivity and Growth programming. The department can continue to review the literature and programs of other organizations and departments to identify best practices to improve the programming and to better meet the needs of western Canadian SMEs. The evaluation identified some best practices such as focusing on high growth businesses, increasing the adoption of advanced technology and promoting international trade.

Performance: Achievement of Expected Outcomes

Evidence. The interviews and departmental impact assessments show that WD’s projects are realizing short term results related to creating/maintaining/expanding businesses, job creation and skills development. The projects also helped other participants to improve market share/competitive position, obtain information for future endeavours and form partnerships or contracts.

Performance information reported to WD indicated that most completed projects met or exceeded their target expectations. Within the period of the evaluation, about 3,398 businesses were created, maintained and/or expanded exceeding set targets. Reasons for completed projects not meeting or only partially meeting targets include unforeseen delays, poor project administration or in the case of one project, inclement weather.

Progress reports from projects that were in progress indicated that these projects were on course to either meet or exceed their target expectations. Client satisfaction with projects that involved participation in and/or delivery of community workshops, seminars, presentations and special events ranged from 76% to 100% exceeding set targets.

Although it is too early or not possible to measure longer term impacts, the literature confirms the link between the short term outcomes and positive longer term impacts related to strengthening western Canadian SMEs and diversifying the economy.

Success factors included partnering and collaboration with industry and other organizations, aligning with provincial priorities, the rigorous project assessment and due diligence process and the flexibility in both type and implementation of projects. Factors constraining success included limited timeframes to complete project and the more recent narrower focus on skills development to the exclusion of support for other types of business productivity and growth projects.

The evaluation found the quality of the performance measurement data and the risk management strategy to be appropriate. However, some limitations were identified with respect to measuring the performance of business productivity and growth projects: 83 of the 87 projects used sub-program rather than program level indicators which measure longer term impact; and there are 165 unique indicators that cannot be aggregated to tell a performance story. With respect to risk, the department tends to fund lower risk projects with 78% of the 87 projects assessed as low risk.

Suggested improvements. The projects are realizing short term objectives and the department can continue with processes that are working well such as partnering and collaboration with industry and other organizations, aligning with provincial priorities, the rigorous project assessment and due diligence process and the flexibility in both type and implementation of projects. The department can improve performance measurement by encouraging staff to identify program level indicators which report on longer term outcomes and minimize the use of unique indicators.

Performance: Demonstration of Efficiency and Economy

Evidence. The department has adopted an efficiency indicator for Business Productivity and Growth, which is stated as “operating costs per $1,000 in Grants and Contributions (Gs&Cs) expended. The total Gs&Cs invested by the department in Business Productivity and Growth was $23,390,759 in 2013–14. Operating expenditures for the period was $3,060,317 resulting in an efficiency indicator of $131. The department therefore spends roughly 13.1% of its budget to deliver $1000 of programming in Business Productivity and Growth. The projects leveraged $1.67 per dollar of departmental Grants and Contributions (G&C) funding. The projects reporting outcomes during the study period cost $7,953 in G&C funding per business created, maintained, or expanded and $2,206 per job created or maintained. The completed projects also generated $26.00 in increased sales per dollar cost. These measures compare favourably with those for similar federal programs.

Although the programming is well delivered, there remain opportunities to continue to make it more flexible and responsive. The new call for proposal process improves response times, however, 29% of interviewees see disadvantages in terms of intake periods which are not predictable, limited communication between staff and applicants and the tendency to fund existing sectors rather than emerging riskier sectors (e.g., new media, clean energy).

The literature reveals other potential improvements or best practices including: 1) focusing on high growth businesses; 2) increasing the adoption of advanced technology; and 3) promoting international trade.

Suggested improvements. To potentially improve the call for proposal process, the department can explore ways to work with applicants earlier in the proposal application process to reduce time spent in preparing proposals. The department can also review the programming to identify further opportunities to adopt best practices and increase programming impacts related to business growth, technology adoption and international trade and explore opportunities to support projects focusing on high growth businesses and emerging sectors.

7.2 Recommendations

Although the evaluation concludes that Western Economic Diversification’s programming with respect to Business Productivity and Growth is in large part well delivered and achieving short term objectives, the evaluation also identifies possible improvements to enhance the efficiency and effectiveness of delivery and to strengthen achievement of longer term outcomes. The following recommendations are based on evidence gathered and conclusions discussed in this study:

1. The department should continue to identify and adopt best practices to enhance the ongoing relevance of the business productivity and growth programming.
    
2. The department should improve how it measures and reports on longer term outcomes in order to enhance the business productivity and growth programming’s ability to tell its performance story.
    
3. The department should review its business productivity and growth programming to improve flexibility and responsiveness.

ANNEX A – Response Rates by Participant Group

*Contacts were unavailable for reasons such as invalid contact information or the contact was no longer with the organization

**The number of targeted WD Representatives was reduced after the initial group was contacted so there was less follow-up for this group.

ANNEX B - Business Productivity and Growth Logic Model

ANNEX B - Business Productivity and Growth Logic Model (PDF)