Independent Summative Evaluation of the Office of the Privacy Commissioner of Canada's Personal Information Protection and Electronic Documents Act Contributions Program

Final Report

August 21, 2014

Alison Kerry
1270665 Ontario Inc.
241 Pleasant Park Rd.
Ottawa, Ontario, Canada K1H 5M4
Phone: 613-731-5331
E-mail: akerry@rogers.com

---

Executive Summary

This report summarizes the independent summative evaluation conducted on the Office of the Privacy Commissioner of Canada's (OPC) Contributions Program. Established under the Personal Information Protection and Electronic Documents Act (PIPEDA), this Contributions Program was established in 2004-2005 to enable the OPC to draw on expertise within institutions and organizations with an interest in privacy protection, to encourage research into a broad range of issues and to support initiatives that contribute to raising public awareness and promoting best practices.

This current evaluation covers the second five years of the Program, from 2009-2010 to 2013-2014, and has been completed in accordance with and to meet the requirements of Treasury Board's Policy on Evaluation (2009) and the Financial Administration Act.

The evaluation methodology included a document and file review, collection and assessment of performance information, interviews with key internal and external stakeholders, an assessment of economy and efficiency, and case studies of eight funded projects.

The evaluation concluded that:

1. The Program continues to address a demonstrated need based on public survey findings, the demand for the program, its unique niche in promoting privacy issues, and universal support from those consulted.
2. The Program is aligned with the OPC priorities as it meets the intent of PIPEDA, contributes to the OPC's outcomes and funds projects directly aligned with the OPC's priorities.
3. The Program is aligned with the roles and responsibilities of the OPC as it falls within the Office's mandate and is the only Canadian program with a sole focus on privacy research and related knowledge translation.
4. The Program is achieving its immediate outcomes to increase and enhance both the production and sharing of privacy information, knowledge and best practices. Since the last evaluation, there has been a greater diversity of projects funded across priority areas and recipient groups as well as more knowledge translation activities. While making an impact, continued work is needed to measure and document the degree to which the Program is enhancing policy development and contributing to raising public awareness.
5. The Program is generally being implemented as planned, with improvements made to the application review process and internal controls. However, the Terms and Conditions need to be updated to reflect current practices. Potential applicants who are not privacy specialists but have an interest in privacy issues are generally not aware of the Program and continued efforts to translate research into results are desired by all.
6. The Program is seen to be efficient overall with a high level of project output for the funds invested. The Program has a relatively high overhead percentage due to the requirements and initiatives put in place to achieve the intended outcomes, contribute to the OPC's larger mandate, and to strengthen internal controls.
7. The Program is seen as economical, producing good value for money. The Program spends the vast majority of its contribution funding each year, with most of the funds being spent in the Ontario Region. While alternative delivery options exist that may be more economical, these options would likely reduce the benefits accrued to the OPC by managing the program.

The following recommendations are provided for the OPC's consideration in renewing the Program (further details are provided in Section 7):

1. Continue knowledge translation efforts to translate research into results.
2. Encourage new innovative partnerships to extend the reach of the Program.
3. Balance Program requirements and efforts to maximize efficiency and economy.
4. Revise the Program Terms and Conditions to reflect current and desired requirements.

Acknowledgements

The evaluator would like to thank François Cadieux, Contributions Program Manager, for providing evidence and assistance in completing this evaluation and Chantale Roussel, Director Business Planning and Management Practices, for her guidance and oversight in implementing this evaluation. Thanks are also provided to all the individuals who provided their insights and input to this evaluation through interviews.

Acronyms

CMC – Certified Management Consultant
FTE – Full Time Equivalent
G&Cs – Grants and Contributions
IC – Industry Canada
N/A – Not applicable
OPC – Office of the Privacy Commissioner of Canada
PAA – Program Alignment Architecture
PIPEDA – Personal Information Protection and Electronic Documents Act
SSHRC – Social Sciences and Humanities Research Council
T&Cs – Terms and Conditions
TB – Treasury Board

1. Context

This report summarizes the independent summative evaluation conducted on the Office of the Privacy Commissioner of Canada's (OPC) Contributions Program.

Pursuant to the program/legislative authority given the OPC under the Personal Information Protection and Electronic Documents Act (PIPEDA), a Contributions Program was established in 2004-2005 to enable the OPC to draw on expertise within institutions and organizations with an interest in privacy protection, to encourage research into a broad range of issues and to contribute to raising public awareness and promoting best practices.

The current Terms and Conditions (T&Cs) for the PIPEDA Class Contributions Program (Jan. 5, 2010) expire on March 31, 2015 and require an evaluation to inform potential renewal of the Program. In accordance with the Financial Administration Act, an evaluation of the performance and ongoing relevance of all Government of Canada Contribution Programs is required every five years. This evaluation provides the basis for program renewal, revision or completion. An initial evaluation of the Contributions Program was completed in October 2009 to cover the first five years of the program. This current evaluation covers the second five years, from 2009-2010 to 2013-2014, and has been completed in accordance with and to meet the requirements of Treasury Board's (TB) Policy on Evaluation (2009).

2. Program Description/Profile

Background

The Privacy Commissioner of Canada is an Officer of Parliament who reports directly to the House of Commons and the Senate. The Commissioner is an advocate for the privacy rights of Canadians and the powers include:

• Investigating complaints, conducting audits and pursuing court action under two federal laws;
• Publicly reporting on the personal information-handling practices of public and private sector organizations;
• Supporting, undertaking and publishing research into privacy issues; and
• Promoting public awareness and understanding of privacy issues.^{Footnote 1}

Part of the OPC's role is implementation of the Personal Information Protection and Electronic Documents Act (PIPEDA). This Act sets out ground rules for how organizations may collect, use or disclose personal information in the course of commercial activities. PIPEDA applies to personal information collected, used or disclosed in the course of commercial activities by all organizations, except in those provinces that have enacted legislation that is deemed to be substantially similar to PIPEDA. PIPEDA continues to apply throughout Canada to federally regulated organizations engaged in commercial activities and to personal information personal information collected, used and disclosed across borders.

In 2004, under PIPEDA, a new Contributions Program with a value of $500,000 annually was initiated by the OPC to support privacy related non-profit research and knowledge translation. In the first 10 years, the Program has allocated close to $4 million to more than 100 initiatives in Canada.

Authority

The OPC derives its authority to administer the Class Contribution Program from sections (b) and (d) of PIPEDA:

"The Commissioner shall:

(b) undertake and publish research that is related to the protection of personal information, including any such research that is requested by the Minister of Industry […]

(d) promote, by any means that the Commissioner considers appropriate, the purposes of this part."

Under the Terms and Conditions for the PIPEDA Class Contribution Program, the Commissioner may provide contributions to eligible recipients to encourage research in privacy protection and related issues as well as to support initiatives that contribute to raising public awareness and promoting best practices in information protection.

Objectives

The Program's objectives are to:

1. Capitalize on existing privacy research capacity in academic, not-for-profit sectors and other sectors to generate new knowledge and support the development of expertise in selected areas of privacy and data protection;
2. Increase awareness and understanding among individuals and organizations of their privacy rights and obligations.^{Footnote 2}

Program Details

The intent and details of the Program have been articulated in the Performance Measurement Strategy drafted in December 2009. The logic model delineates the intended elements or activities and outputs of the program, as well as the associated results or outcomes (see the following page).

The PIPEDA Class Contributions Program is managed and delivered internally within the OPC. The Program operates by setting research priority areas (see Text box 1) and launching an annual campaign to attract applicants. Academic institutions and not-for-profit organizations, including industry associations and trade associations, are eligible under the Program for funding. Eligible applicants include consumer, voluntary and advocacy organizations.^{Footnote 3}

The Program provides funding for research projects aimed at promoting privacy and the protection of personal information in the private sector; and also for related knowledge translation initiatives aimed at disseminating research results and enabling their uptake and application among relevant stakeholders.^{Footnote 4} As the Contributions Program finds its authority under PIPEDA which governs the collection, use or disclosure of personal information by organizations in the course of commercial activities, only research and/or related knowledge translation proposals that address privacy issues in the private sector or at the interface between the private and public sectors are eligible for funding.^{Footnote 5}

Text version

Figure 1: Logic Model – Office of the Privacy Commissioner of Canada (OPC) Contribution Program

This figure is a logic tree illustrating how the objectives, inputs, activities, outputs, and outcomes of the OPC Contributions Program all flow into the Program’s identified strategic outcome: the privacy rights of individuals are protected.
Inputs, activities, outputs and outcomes are broken into sub classes illustrating the type of organization or individual that influences and controls them. Outcomes are further broken into sub classes illustrating their type: immediate, intermediate, ultimate or strategic.
The logic tree begins by identifying the Program’s two objectives, which guide all program activities and outcomes:

• To capitalize on existing research capacity in academic, not-for-profit and other sectors to generate new knowledge and support the development of expertise in selected areas of privacy and data protection.
• Increased awareness and understanding among individuals and organizations of their privacy rights and obligations.

The logic tree then breaks the program down into three main stages, and identifies the inputs, outputs, activities and outcomes for each stage. It also identifies how certain elements flow into others. The following three tables (labeled Stage 1, Stage 2 and Stage 3) describe the three main stages illustrated in the logic tree.

Stage 1: Summarizes the inputs, activities and outputs within OPC influence and control.

Inputs*	$500,000 (Contribution Budget) and .25 to .50 Full Time Employee
Activities**	(1) Prepare Call for Proposals	(2) Assess and Evaluate Proposal	(3) Award Promise of Contributions	(4) Monitor Projects	(5) Evaluate Deliverables for Value for Money	(6)Prepare and Issue Payment
Outputs***	(1) National Security	(2) Identity Integrity & Protection	(3) Information Technology	(4) Genetic Privacy	(5) Other (outside 4 priorities)	(6) Public Education	(7) Regional Outreach Initiatives

*The inputs flow into all activities
**Only Activity 3 flows into all outputs
***Outputs 1 to 5 flow into Activity 1 in Table 2 (see below). Outputs 6 and 7 flow into Activity 2 in Stage 2 (see below).

Stage 2: Summarizes the activities and outputs within individuals’ and organizations’ influence and control.

Research outcomesOutreach outcomes
Activities undertaken by recipients*	(1) Undertake Research Projects	(2) Organize Public Education and Outreach
Outputs produced by recipients**	(1) Reports/Academic Papers/Recommendations/Participation in Workshops	(2) Workshops, Conferences, Informational Materials (toolkits, web pages, etc.)

*Activity 1 flows into both Output 1 and Output 2. Activity 2 flows into Output 2.
**Both Outputs flow into both Immediate Outcomes identified in Stage 3 (see following table).

Stage 3: Summarizes the immediate, intermediate and ultimate outcomes outside OPC Influence and control.

Research outcomesOutreach outcomes
Immediate Outcome: Knowledge*	(1) Increased and Enhanced Information, Knowledge and Best Practices	(2) Increased and Enhanced Sharing and Dissemination of Information, Knowledge and Best Practices
Intermediate Outcome: Attitude**	(1) Increased and Enhanced Policy Development Capacity	(2) Increased and Enhanced Public Awareness
Ultimate Outcome: Behaviour***	(1) Enhance Privacy Legislation, Regulations, Policies and Practices	(2) Improve Individual (public) Capacity to Guard Against Threats to Personal Information

*Immediate Outcome 1 flows into both Immediate Outcome 2, as well as into both Intermediate Outcomes.
**Intermediate Outcome 1 flows into Intermediate Outcome 2 and Ultimate Outcome 1. Intermediate Outcome 2 flows into Ultimate Outcome 2.
***Ultimate Outcome 1 flows into both the Ultimate Outcome 2 and the Strategic Outcome. Ultimate Outcome 2 flows into the Strategic outcome.

Ultimately, all three stages of the program, as described above, lead to the program’s identified Strategic Outcome: the privacy rights of individuals are protected.

Proposals are submitted by various applicants, and these proposals are assessed and evaluated by OPC and external^{Footnote 8} evaluators based on their merit and the degree to which they address the priorities. Successful applicants are awarded a Contribution Agreement, and funds are released when deliverables are provided. The OPC oversees the projects, providing responses as required, and, at completion, evaluates if the final deliverable(s) meets the terms of the Agreement, and then provides final payment.

The maximum amount that can be awarded for each research or knowledge translation project is $50,000. The maximum any single organization can receive is $100,000. No matching funds are required from applicants. Projects are to be completed within the fiscal year that they are funded. Specific T&Cs are in place to set out the factors that govern this Contributions Program^{Footnote 9}.

Governance

The Commissioner has prime responsibility for conducting the work of the OPC and for decisions related to the approval of contributions under the Program.

The Director, Policy, Parliamentary Affairs and Research, which reports to the Senior General Counsel and Director General Legal Services, Policy and Research, is accountable and responsible for administering the PIPEDA Class Contribution Program^{Footnote 10} which includes providing strategic orientation for the Program, assessment of applicants and management of contribution agreements. The day-to-day management of the Program is carried out by the Senior Research Analyst in Policy and Research (the program manager), with support from a program officer and administrative assistant. The program is mainly supported by:

• Communications Branch, in respect to dissemination and communication of information flowing from Contribution Program; and
• Financial and Administrative Services, in respect to financial management of the contribution program.

Other branches may occasionally provide support and expertise, such as during the proposal assessment process and the evaluation of deliverables.

Resources

The maximum amount available under the OPC's Contribution Program is $500,000 per fiscal year. The funding supports about 10 projects per year of approximately $20,000 to $50,000 in value. Over last 5 years (2009-2010 to 2013-14), the program has allocated almost $2.3 million to 57 initiatives in Canada.

Funding for 1.4-1.6 Full Time Equivalent (FTE) positions is now provided from within the OPC reference levels^{Footnote 11} to manage and support the Program (increased since the last evaluation). This includes one Program Manager, 0.3-0.5 FTE for a Program Officer, and 0.1 FTE for an Administrative Assistant. In addition, it is estimated that 0.3 FTE from Communications and 0.1 FTE from Finance supports that program. Any additional costs associated with the management of the Program are also covered from within existing OPC reference levels (e.g., Director and Director General's time and the cost of the evaluation).

3. Approach to the Evaluation

Evaluation Objectives

The objectives of this evaluation were to:

1. 1. Determine the ongoing relevance of the Program;
2. 2. Assess the Program's performance (effectiveness, efficiency and economy); and
3. 3. Provide recommendations for renewing the Program in light of the evaluation's findings.

Evaluation Scope

The scope of this evaluation included the management, activities and products related to the Contributions Program in the OPC for the following 5 years: 2009-10, 2010-11, 2011-12, 2012-13, and 2013-14.

Evaluation Issues

As per TB's Policy on Evaluation, this summative evaluation assessed the following issues:

Relevance
Issue #1: Continued Need for program	Assessment of the extent to which the program continues to address a demonstrable need and is responsive to the needs of Canadians
Issue #2: Alignment with Government Priorities	Assessment of the linkages between program objectives and (i) federal government priorities and (ii) OPC's strategic outcomes
Issue #3: Alignment with Federal Roles and Responsibilities	Assessment of the role and responsibilities for OPC and the federal government in delivering the program
Performance (effectiveness, efficiency and economy)
Issue #4: Achievement of Expected Outcomes	Assessment of progress toward expected outcomes with reference to performance targets and program reach, program design, including the linkage and contribution of outputs to outcomes
Issue #5: Demonstration of Efficiency and Economy	Assessment of resource utilization in relation to the production of outputs and progress toward expected outcomes

Based on the program needs and evaluation plan for the OPC's Contributions Program, the following evaluation questions were used to evaluate the Program's performance in the 5 issue areas:

Issue #1: Continued Need for Program

1. Does the Program continue to address a demonstrated need?

Issue #2: Alignment with Government Priorities

2. Does the Program continue to be aligned with OPC priorities?

Issue #3: Alignment with Federal Roles and Responsibilities

3. Does the Program continue to be aligned with the roles and responsibilities of the OPC?

Issue #4: Achievement of Expected Outcomes

4. Is the Program achieving its intended outcomes?
5. Is the Program implemented as planned to achieve its intended outcomes?

Issue #5: Demonstration of Efficiency and Economy

6. Is the Program producing its intended outputs in an efficient manner?
7. Is the Program achieving its intended outcomes in an economical manner?

Evaluation Methodology

The approach to this evaluation was designed to add value to the OPC in its intent to renew the program, and is based on:

1. Application of a professional methodology, in line with Treasury Board Standards;
2. Maintenance of a flexible and a collaborative working relationship with the client; and
3. Provision of independent evaluation services that comply with the Values and Ethics Code for the Public Service and professional standards associated with designation as both a Credentialed evaluator (Program Evaluation Standards) and Certified Management Consultant (CMC Code of Professional Conduct).

The following lines of inquiry provided evidence for this evaluation:

1. Review of program documentation including material on the OPC website, a file review of funded and unfunded projects in theOPC's offices and review of other documents provided byOPC management and/or stakeholders. Key documents included:
   • Program Terms and Conditions
   • 2009 Summative Evaluation Report
   • Tracking Chart showing implementation of 2009 summative evaluation recommendations
   • Evaluation Committee Meeting Minutes
   • Evaluation Process material relating to the assessment of proposals
   • Program Financial Reports
   • Impact Analysis of Research Supported by the OPC Contributions Program (i.e., Science Metrix bibliometric study, 2012)
   • Internal controls chart and documents
   • Communications Strategy used to promote the Program more broadly
   • Performance Measurement Strategy (2009)
   • Performance Information collected by the Program
   • "Real Results" Publication which promotes funded Contributions Program Projects and the Program itself
   • Recipient Audit Reports
   • The "Next Generation" OPC Contributions Program Strategy
   • Program proposals, proposal assessments, progress/final reports, communications, tracking and financial accounts from 8 selected funded projects (~15% of the total projects funded over the evaluation timeframe)
   • Program proposals, proposal assessments and communications from 2 unfunded projects
2. Interviews with program personnel and external stakeholders , including:
   • Senior Management (4 senior managers to inform evaluation planning)
   • Program Management staff (Contributions Program Manager, Program Officer, Administrative Assistant)
   • Program Financial Management (one representative from Financial & Administrative Services)
   • Program Communications (two Communications Advisors)
   • Proponents/Leads from the seven of the eight funded projects
   • Two collaborating organizations (Industry Canada (Contribution Program for Non-Profit Consumer and Voluntary Organizations) and Social Sciences and Humanities Research Council (SSHRC))
   • Two representatives who help evaluate proposals (one internal and one external reviewer)
3. Assessment of efficiency and economy, including:
   • Comparison of planned to actual resources over the five years
   • Trend analysis of spending over the five years by region and priority addressed
   • Comparison of number of applications versus funded projects over time
   • Percentage of overhead/administrative costs to total program costs
   • Comparison of overhead/administrative cost percentage to other similar programs or best practices
   • Views from key stakeholders regarding efficiency and economy

Evidence collected from all lines of evidence was analyzed against the evaluation issues and questions, and the results have been documented in this report.

Evaluation Limitations

1. Potential for insufficient performance information to fully assess the achievement of outcomes.
   
   Performance information was not initially available to the evaluation but was collected and compiled by the Program as part of the evaluation. However, while the Program did have performance data at the project level, and this was compiled for the evaluation, it was not possible within the resources and timeframe available to collect sufficient evidence to fully assess the achievement of intermediate and ultimate outcomes (N.B. the Performance Measurement Strategy noted this was not required due to the Program's low risk and resources). When evidence was limited, the evaluation commented (to the degree possible) on the likelihood that the outcomes would be achieved.
2. Potential for insufficient financial information to fully assess efficiency and economy.
   
   The nature of financial information in the Government of Canada (where objects are not individually costed) limits the ability to assess efficiency and economy (e.g., by activity/function). This limitation is not specific to the OPC and influences all federal government evaluations. As well, while comparisons were made, similar programs do not exist to allow for an accurate cost comparative approach to inform the assessment. To address economy and efficiency, the evaluation conducted a basic comparative and trend based assessment and also included views related to the Program's efficiency and economy.
3. Potential for insufficient participation by key stakeholders to fully inform the evaluation.
   
   The evaluation required full and open participation by program personnel and external stakeholders. While program personnel all participated fully in the evaluation process, there were some gaps with external stakeholders. Despite multiple attempts, a representative from one of the eight case studies was unavailable to be interviewed, and neither of the unfunded applicants was available to be interviewed. As such, documentary evidence alone was relied upon to fill these gaps.

4. Implementation of Recommendations from the last Evaluation

As a prelude to presenting the findings from this current evaluation, the actions taken by the OPC in response to the 2009 Evaluation recommendations are presented. This provides an overview of how the Program has evolved and matured over the last 5 years.

Most of the previous recommendations have been implemented over the last 5 years - 77% have been implemented and the rest are ongoing or partially implemented. The following table summarizes the action taken by the OPC, as of July 11 2014.

2009 Evaluation Recommendations	Status	Comments
1. The application review process is internal to OPC and could benefit from more formal linkages with external experts.	Implemented	External peer reviewers from academia and civil society have been involved in reviewing funding proposals for the past three years.
2. The Program is seen as cost effective; however, greater administrative efficiencies would be beneficial (e.g., allowing recipients more time to complete their projects in the fiscal year).	Implemented	Various administrative efficiencies have been introduced over the past years with a view to improving on administrative efficiency. The Program has continuously refined its tracking mechanisms in an effort to support greater due diligence and delivery of projects as per the contribution agreements. A new SharePoint page was put in place some two years ago, containing all of the management tracking and historical documents related to the Program, allowing more efficient and effective tracking. The Program has also been put on an annual cycle, allowing greater predictability and planning both internally at OPC and externally on the part of stakeholders/potential applicants. Annual call for proposals are now planned for every September; as a result, contribution agreements can be signed earlier, thus allowing recipients more time in the new fiscal year to complete their projects (N.B. However, some agreements are being signed in May and June).
3. The OPC should establish an earlier launch date, and a more streamlined review, approval and contribution agreement process to ensure projects have a full year for project implementation. As well, the OPC should consider allowing a small number of 2 year projects, as justified by specific proposals.	Implemented	Launch date for annual call is now September (see above), and Program is now on an annual cycle to ensure regularity and predictability in implementation of key milestones (e.g. issuing calls, issuing contribution agreements, recipient reporting dates, etc.). A more streamlined proposal review and approval process is in place, which includes an internal and external peer review, and has been continuously refined ever since. A more streamlined contributions agreement process is also in place, to ensure that agreements are issued in a timely fashion and recipients have more time to complete their projects under their agreements. A small number of multi-year projects have been funded virtually every year since the last summative evaluation (typically one or two multi-year projects per funding year).
4. The OPC should draft new Terms and Conditions for the Program to align with the new Treasury Board Policy on Transfer Payments.	Implemented	New Terms and Conditions were drafted as a result of the 2009 summative evaluation; these align with the new Treasury Board Policy on Transfer Payments.
5. The OPC should consider defining specific priorities for the Contributions Program that are derived from the overall OPC priorities, but are more focused to direct project submissions. The OPC should consider reserving some funds (e.g., 10%) for projects in areas that are outside the priorities but relate to the overall goal of the program and PIPEDA.	Implemented	Program priority areas are aligned with the OPC's four policy priorities. As part of our annual call for proposals, we suggest to potential applicants to focus on these overall OPC priorities in developing their proposals. We further give examples in our Applicants Guide of more targeted research questions that fall within the broader policy priority areas. The annual call also indicates that the OPC welcomes proposals that fall outside these priorities. (No specific amounts - e.g. 10% - are set aside per se for projects falling outside the priority areas, as proposals are reviewed on the basis of merit and OPC does not want to restrict itself in its funding decisions.)
6. The OPC should consider mechanisms to promote the Program more broadly to the appropriate public education and outreach community.	Implemented	The OPC developed a new five year strategy for the Program, which in part focused on promoting the Program more broadly to the appropriate public education and outreach community. OPC Communications Branch also developed a new communications strategy. A key component of this strategy is how to better reach out to key stakeholders, to promote the Program's existence as a source of funding, and to promote the new knowledge generated under the Program. The Pathways to Privacy Symposium was created, the purpose of which is to enable uptake and application of research results by relevant stakeholders. Here too, a key component of this Symposium is to promote the Program more broadly to the appropriate stakeholders, notably the public education and outreach community and encourage uptake of research results. The OPC published "Real Results," a magazine style publication that celebrates some of the projects funded under the Program. This publication has been widely distributed across Canada (in paper and online), and plans are in place to publish a second Real Results in 12 to 18 months, once distribution of the first issue is complete. Finally, the OPC leverages its stakeholder community - e.g. other funding agencies, university grants and research offices, external peer reviewers, etc. - to promote the call for proposals and new knowledge generated under the Program.
7. Require proposals to identify in-kind costs being provided and encourage (but not require) other sources of funding to the project.	Ongoing	Applicants in their proposals - and ultimately recipients in the contribution agreements - are required to identify all other sources of funding for their projects, cash and in-kind. However, in-kind costs are not always identified.
8. Require successful applicants to report on their performance and work with OPC staff at the contribution agreement stage to ensure applicants understand what should be tracked and reported to OPC.	Ongoing	Two written progress reports are required from recipients under their contributions agreement before final deliverables are submitted to the OPC at year end. (At least four written progress reports are required for recipients who organize the Pathways to Privacy Symposia.) A template for progress reports has been developed and must be used by Recipients in reporting to the OPC. The "Recipient's Resources Page" on the OPC web site makes this form available to recipients, as well as an FAQ which allows them to more clearly understand their contribution agreement obligations. However, performance information is not always comprehensive (e.g., reach figures) or provided in a manner to allow for Program roll-up.
9. The OPC should implement the performance measurement strategy for the Program so that its overall results and impacts are tracked and summarized annually.	Partially implemented	The Program indicated it has been collecting performance measurement data but not in the specific format provided for in the performance measurement strategy (PMS). For instance, the Program's "Master Tracking Chart," had been used to track Program management outputs, while memos and written assessments in individual project files have been used to assess and track individual project outcomes and outputs. Performance information was not available at the start of the evaluation and had to be compiled during the evaluation process. The Program did this using the tabular format provided for in the PMS. All projects going back to 2010-11 were put in this format, for the purposes of the evaluation.
10. The OPC should draft an annual report of program accomplishments and results that is provided to Senior Management, OPC's Research, Public Education and Policy Branches, the media, and the privacy community (e.g., via the channels used to launch the Program).	Ongoing	Program accomplishments are regularly reported upon by senior managers at the Senior Management Committee meetings. Furthermore, the PIPEDA Annual Report to Parliament contains a section on the Program's accomplishments in the year under review. However, this is focused on announcing new projects, outputs and initiatives (not outcomes achieved). Also, the OPC has published "Real Results", a magazine style publication that reports and celebrates some of the projects funded under the Program. This publication has been widely distributed across Canada (in paper and online), and plans are in place to publish a second Real Results in 12 to 18 months, once distribution of the first issue is complete. In addition, a Research Symposium series has been launched, "Pathways to Privacy", where OPC-funded researchers, together with SSHRC and IC-funded researchers, exchange their findings and discuss results among relevant end-users.
11. The OPC should create formal linkages with IC's Contributions Program for Non-profit Consumer and Voluntary Organizations and SSHRC.	Implemented	Industry Canada and SSHRC have become key partners of the OPC Contributions Program. They have been invited to participate in the external peer review process for proposals; they assist the OPC in publicizing and marketing our call for proposals; they have been invited as speakers to the Pathways to Privacy Symposium of 2012; they have participated in the 2014 summative evaluation; and they provide advice to the OPC on an ongoing basis regarding best practices for managing contributions. Partnerships with other research agencies - e.g., Genome Canada and Canadian Institutes of Health Research - are also being developed. Both these organizations assist the OPC in advertising our annual call for research proposals on their respective websites targeted to different research communities, including social science and health researchers interested in certain aspects of privacy protection.
12. The OPC should consider assigning an additional 0.5 to 1 FTE to the administration of the program, under the direction of the Program Manager	Implemented	A new determinate (term) employee has been hired to assist the Program Manager (30-50% of her time), which has significantly contributed to the successful operation of the Program. Efforts are underway to have this position filled on an indeterminate (full time) basis.
13. Move towards an online application process to streamline requirements for applicants, and increase administrative efficiencies.	Implemented and ongoing	Applicants currently can submit their applications to OPC by email, and all relevant forms are available online on the Office's web page. The Program has also developed a SharePoint based application that has significantly increased efficiency and streamlined the process for receiving and reviewing applications. OPC reviewed options for putting in place an online application system, similar to that used by other funding agencies (e.g. SSHRC and Trillium Foundation of Ontario). These options - consisting of special software/electronic application forms - were not implemented as they would have been cost prohibitive (i.e. too expensive given the relatively modest size of the OPC's Program). Also, it was not clear to OPC that these options could ensure security/privacy of data submitted online, a top priority for our Office. OPC continues to explore other, more cost effective alternatives in this area - e.g. creating a project description form that applicants would be required to fill out and submit to us by email to standardize the presentation of proposals. (The current approach involves applicants following project description guidelines provided for in the Applicants Guide). This new form may be implemented in 2015-16.

5. Evaluation Findings

The findings of the current evaluation are presented below according to the evaluation issues and questions.

Issue #1: Continued Need for Program

1. Does the Program continue to address a demonstrated need?

Need in Canada to fund the production of privacy research and privacy-related knowledge translation activities

In 2013, a survey of Canadians^{Footnote 12} found two-thirds of them were concerned about the protection of their privacy, with a growing sense amongst Canadians that their ability to protect their personal information is diminishing. As well, the majority (56%) are not confident that they have enough information to know how new technologies affect their personal privacy. Looking ahead, 71% think that protecting the personal information of Canadians will be one of the most important issues facing our country in the next ten years. This finding reflects a steady increase since 2009. The Contributions Program's increasing focus on knowledge translation and communications seeks to address these public concerns and communicate program results.

Environmental scans conducted annually by the OPC note the trends related to privacy from 2010 to 2013. Consistently, the scans have indicated the following trends: the rapid pace of technology change with privacy impacts, the horizontal and collaborative nature of privacy issues that require collaboration with stakeholders, and the increasing demand for and use of personal information. The research and knowledge translation projects funded by the Contributions Program have been examining these trends and thus supporting the OPC in its ability to address these needs. As well, the OPC refers to the outputs from funded projects frequently (i.e., weekly) to respond to media requests - indicating that funded areas are of interest and relevance to Canadians broadly.

This Contributions Program is unique in Canada and was the only one identified that specifically funds privacy related research and knowledge translation. While the SSHRC's granting program and Industry Canada's Contributions Program for Non-profit Consumer and Voluntary Organizations (see Issue #3 for more details) both can fund some privacy related research, this is not the focus or priority of these funds.

All those interviewed indicated the importance of this fund in advancing privacy issues. Its current focus on both researching privacy areas and also disseminating this new knowledge was seen as important to contribute advice to Parliament on privacy related matters and to help Canadians better protect their privacy.

Demand for the Program

The demand for the program has been fairly consistent over the last 5 years. The annual average number of proposals received over the five years has been 51^{Footnote 13} and the Program has been able to fund an average of 12 per year. Therefore, the Program is only able to fund about 23% of the demand. In comparison to other programs, this is low indicating a high demand.^{Footnote 14}

Text version

Figure 2: Number of proposals submitted to and funded by OPC Contributions Program

This figure is a bar graph illustrating the number of proposals submitted to and funded by the OPC Contributions Program. The graph offers a comparison of the number of proposals submitted and the number of proposals funded for each of the following fiscal years: 2009-10, 2010-11, 2011-12, 2012-13 and 2013-14. The following table summarizes the data presented in the graph:

Fiscal Year	Number of proposals submitted	Number of proposals funded
2009-10	49	13
2010-11	62	16
2011-12	46	7
2012-13	45	11
2013-14	51	10

OPC managers have indicated that they rely on this program to explore complex areas of privacy research, advance privacy issues and contribute to the societal debate on privacy. It helps the Office fulfill its role in implementing PIPEDA, and internal interviews indicated that the OPC does not have the internal capacity to carry out all the research and knowledge translation needed itself. As well, having an externally funded program allows for a broader scope in what can be examined and recommended to complement the OPC's internal research and communications which focuses on OPC positions and directions.

External stakeholders noted that the OPC Program is critical and essential funding to support privacy related work outside of government. In a number of cases, funds from the OPC kick-started longer lasting privacy research initiatives and collaborations. It was also noted that it was important to have funding agencies like the OPC that not only support projects but are also in touch with the academic and technical communities and issues to provide some guidance on potential projects.

Program responsiveness to Canadian's needs / privacy priorities

The Contributions Program's priorities for the last 5 years are noted below. These have remained the same over the last 5 years (in fact since 2008-09) and are aligned with the OPC's overall priorities.

Research priorities include:

1. Public safety and national security
2. Identity integrity & protection
3. Information technology
4. Genetic privacy and bio-banking

All interviewees agreed that the Program funds projects directly aligned within the OPC priority areas. As well, the Program puts a priority on innovative public education, outreach and awareness raising initiatives to assist with knowledge translation. This new focus on knowledge dissemination and outreach is widely supported by stakeholders.

To provide additional guidance to applicants within these priority areas, and to ensure applications are targeting timely and strategic issues relevant to OPC's mandate, the Program annually puts out key research questions in each of these priority areas. This information is put into the Applicant's Guide which is updated annually and informs the call for proposals.

Priorities are currently set for the program through strategic planning for the Office as a whole (not specific to just the Contributions Program). The priorities are based on whether they were national or international in scope and their relative urgency and relevance to Canadians, as well as OPC's role, federal jurisdiction, alignment with mandate and both public and private sector, what type of leadership was needed, and OPC value added^{Footnote 15}. They are set to be both broad enough to cover the key issues areas (e.g., information technology) with specific priorities to target new frontier areas (e.g., genetic privacy). The regular environmental scans help to inform the priorities. According to public opinion polls conducted by the OPC in 2013, the current priorities most relevant to Canadian's needs relate to identity integrity and information technology (see Text box 2).

Those interviewed indicated that they believe the program is responsive to Canadian's needs and that the OPC's priorities for the program are relevant. The topic areas funded by the Program have evolved as the Program has matured; for example, moving away from examining how PIPEDA is implemented to the invasiveness of privacy technologies (e.g., impact of social networking, new tracking technologies).

Emerging and rapidly evolving issues noted in the document review and by those interviewed included:

• More advanced technologies with lower cost of retaining information and of deploying
• Value and use of personal information such as the use of personal information to make predictions; and, the online tracking, profiling and targeting of consumers (e.g., by business and authorities)
• Big data
• Consent and what it really means
• Global nature of privacy issues which are interconnected with worldwide impacts
• Updated legislation to address a rapidly evolving privacy issues based on clear and common definitions
• Privacy impact assessments of new initiatives
• Cybercrime and cyber espionage posing new threats
• Cloud computing
• Youth and social networking/ use of technology
• Biometrics

Issue #2: Alignment with Government Priorities

2. Does the program continue to be aligned with OPC priorities?

Program meets the Government's intent with the PIPEDA

The Program continues to meet the intent in PIPEDA for the OPC to 'undertake and publish research that is related to the protection of personal information' to contribute to the Act's overall goal of protecting personal information in the course of a commercial activity.

The PIPEDA sets out ground rules for how organizations may collect, use or disclose personal information in the course of commercial activities. The Act has come into force in phases since 2001. PIPEDA now applies to personal information collected, used or disclosed in the course of commercial activities by all organizations, except in those provinces that have enacted legislation that is deemed to be substantially similar to PIPEDA. PIPEDA continues to apply throughout Canada to federally regulated organizations engaged in commercial activities and to personal information personal information collected, used and disclosed across borders.

The PIPEDA Contributions Program is intended to further the goals of privacy protection by encouraging research into, and contributing to raising awareness of, the protection of personal information - the Program's objectives. This evaluation found that all projects funded related to the intent of PIPEDA. The increasing focus on knowledge translation since the last evaluation further increases the Program's relevance to the intent of PIPEDA.

Program corresponds with recent/current priorities of the OPC

The Program helps to contribute to the OPC's strategic outcome which is: 'the privacy rights of individuals are protected'. This is the goal of the Program and is contributed to by working towards the ultimate Program outcomes of:

• Enhance privacy legislation, regulations, policies and practices; and
• Improve individual (public) capacity to guard against threats to personal information.

The program's objectives, noted in Section 2, are directly relevant to PIPEDA and the Program's priorities are directly aligned with the OPC's priorities. However, it should be noted that the Program's objectives have evolved (i.e., those included in the Applicant's Guide further detail the objectives noted in the 2010 Terms and Conditions and the 2009 Performance Measurement Strategy), particularly noting the increased emphasis on knowledge translation.

To ensure projects funded did in fact meet the stated OPC priorities, this evaluation matched projects to priority areas. While it was difficult to definitively match projects to just one priority, all projects funded did fit within the broad priorities established. The overall assessment is that the vast majority of projects relate the top priorities for Canadians - either to the' identify integrity and protection' or 'information technology' priorities (about 80%) - with a fewer projects relating more broadly to 'public safety and national security' or 'genetic privacy' (see Figure 3).

Text version

Figure 3: Comparison of planned spending by priority (2009-10 to 2013-14) 5-year total

This figure is a pie graph representing the average planned spending by priority over a five- year period from 2009 to 2014. The following table summarizes the data presented in the graph:

Priority	Percentage of planned spending
Identity integrity and protection	57%
Information technology	23%
Genetic privacy and biobanking	11%
Public safety and national security	9%

Issue #3: Alignment with Federal Roles and Responsibilities

3. Does the Program continue to be aligned with the roles and responsibilities of the OPC?

Program's objectives aligned to the mandate and role of OPC

The program continues to be relevant to the OPC's mandate. The mandate of the Office of the Privacy Commissioner of Canada is overseeing compliance with both the Privacy Act, which covers the personal information-handling practices of federal government departments and agencies, and the Personal Information Protection and Electronic Documents Act (PIPEDA), Canada's private sector privacy law. As such, the Program's funding goes to projects that address privacy issues in the private sector or at the interface between the private and public sectors.

The Commissioner is an advocate for the privacy rights of Canadians and the powers include:

• Investigating complaints, conducting audits and pursuing court action under two federal laws;
• Publicly reporting on the personal information-handling practices of public and private sector organizations;
• Supporting, undertaking and publishing research into privacy issues; and
• Promoting public awareness and understanding of privacy issues.

All stakeholders agreed that the OPC is the most appropriate organization for this Program since it is directly related to the mandate and authorities under PIPEDA.

Presence/absence of other programs that complement or duplicate the objectives of the Program

The OPC's Program was the only noted Canada-wide and privacy-specific funding program, and most interviewees indicated that they would not have been able to undertake their project in a timely manner without OPC funding.

Other granting councils fund research that may broadly relate to privacy issues, but these do not specifically target or prioritize privacy research. For example, the SSHRC supports university-based research and training in the humanities and social sciences. A search of their online awards search engine for the term 'privacy' indicated that 14 projects had been funded from 2009-10 to 2012-13 ranging in value from $18,000 to $105,000 (annual average funding to the area was just under $200,000). While a few of the funded studies were for one year (as is required by the OPC Program), most studies were multi-year. Another difference is that the SSHRC provides grants not contributions, which involves fewer performance conditions. Privacy is a small proportion of what the SSHRC funds and it does not have 'privacy' as one of its priority funding areas. The SSHRC indicated that the privacy area does not have sufficient breadth or impact to generate a lot of scholarly research without sufficient support available to and specific to it, and that the OPC's Contributions Program is needed to support and foster this research area.

Since the last evaluation, the OPC Program has established a collaborative/networking relationship with the SSHRC. SSHRC representatives have been part of the application assessment process, and they share, in an ongoing manner, information about processes and best practices, as well about their funding communities/researchers to foster greater connections.

The other relevant program identified was Industry Canada's Contributions Program for Non-profit Consumer and Voluntary Organizations (see Text box 3). This program does fund research into privacy related issues; however this is only a small part of the overall program's projects in the broader area of consumer protection. Over the last 5 years, they have funded projects in areas such as cybersecurity, data breaches, and cloud computing but generally only fund one 'privacy related' project each year (values ranging from $28,000 to $75,000). Where relevant, linkages with the OPC and PIPEDA are noted in funded projects.

As this Industry Canada program has a specific focus and target in terms of funding, like the OPC, it funds a relatively small set of specific groups which tend to be funded year after year (e.g., Option Consommateurs, Public Interest Advocacy Centre, Union des Consommateurs, Consumer Council of Canada).

This Industry Canada contributions program does not target privacy-related research and outreach projects are ineligible - its intent is more around consumer protection and improving the capacity of consumer organizations, and it is limited to the non-profit sector (academics and industry associations are excluded).

It is interesting to note that the Consumer Policy Research Database maintained for this program does include project results funded under the OPC's Contributions Program, as well as OPC related papers/research.

Since the last evaluation, the OPC Program has established a stronger collaborative/networking relationship with this Industry Canada program. The Manager of Industry Canada's program has been involved in the OPC application assessment process, and vice-versa, and both share, in an ongoing manner, information about policies, processes and projects (to enhance communications, look for stacking levels and eliminate duplication).

Issue #4: Achievement of Expected Outcomes

4. Is the Program achieving its intended outcomes?

Overview of Projects

The OPC's Contributions Program has funded the following projects over its second 5 years. The projects highlighted in green (and prefaced with an exclamation mark "!") were chosen for a more in-depth analysis as part of this evaluation:

Organization	Project Name	Amount
2013-14 - Total of 10 projects
! MediaSmarts	Young Canadians in a Wired World - Phase III Quantitative Research Project (Year Two)	$50,000
Queen's University, Faculty of Law	The Privacy Implications of the Foreign Account Tax Compliance Act (FATCA)	$10,000
Queen's University, Surveillance Studies Centre	Privacy Implications of the Spread of Unmanned Aerial Vehicles (UAVs) in Canada	$50,000
Canadian Civil Liberties Association	Police Background Checks and the Private Sector	$50,000
Carleton University, School of Computer Science	Improving Mental Models of Security and Privacy Through Visualizations	$50,000
Association sur l'accès et la protection de l'information (AAPI)	Adaptation and release, in French and English, of an educational kit: "Développement de saines pratiques dans la diffusion de son image et de ses renseignements dans le Net"	$50,000
Université de Sherbrooke and Ryerson University	Fraud and Privacy Violation Risks in the Financial Aggregation Industry	$41,515
Option Consommateurs	New services provided by credit agencies	$45,570
University of Western Ontario, Faculty of Information and Media Studies	Hidden Surveillance by Consumer Health Websites	$49,910
University of Guelph, Department of Psychology	Privacy risks of direct to consumer genetic testing: How do consumers interpret the privacy risks associated with sharing their genetic material with private companies?	$49,680
2012-13 - Total of 11 projects
MediaSmarts	Young Canadians in a Wired World Phase III: Quantitative Research Project	$50,000
University of Victoria	The Canadian Access to Social Media Information (CATSMI) Project	$25,337
La Coopérative radiophonique de Toronto (CHOQ FM)	Privacy: Better integrated knowledge for better informed communities	$50,000
! The British Columbia Society of Transition Houses (BCSTH)	Safety Net Canada: Technology, Privacy, Safety and Violence Against Women, Youth and Children	$50,000
Tekdesk (a division of Community Opportunity and Innovation Network)	An Investigation of the Role of Smartphone Application Permissions in Risks to End User Privacy	$44,160
University of Ontario Institute of Technology	Privacy and Civic Duty: The Legitimate Scope of Voluntary Information Sharing by Private Enterprises in Law Enforcement Investigations into Cybercrime	$16,100
University of Alberta (Health Law Institute)	Designing a Model Policy Framework for Privacy Challenges in Cell Therapy Research	$50,000
University of Toronto	IXmaps: Mapping Canadian Privacy Risks in the Internet Cloud	$49,920
University of Toronto	What Do Canadians Know About Their Video/Visual Privacy?	$42,320
! British Columbia Civil Liberties Association (BCCLA)	A National ID Card by Stealth?	$50,000
Option Consommateurs	Privacy: How can we reconcile the interests of consumers with those of businesses, licensees and professionals?	$43,078
2011-12 - Total of 7 projects
! Atmosphere Industries	Gaming Privacy: Creating a Privacy Game with Canadian Children	$49,265
Canadian Association of the Deaf	Understanding Your Privacy Rights: PIPEDA in Signs	$48,815
University of Ottawa	Electronic Communications Interception and Privacy: Can the Imperatives of Privacy and National Security be Reconciled?	$25,000
! Association sur l'accès et la protection de l'information (AAPI)	Educational kit for developing sound practices for posting pictures and personal information on the Net	$50,000
Centre international de recherches d'experts consultants juridiques indépendants (CIRECJI)	Training for volunteers on privacy education and outreach	$40,564
University of Victoria, Sociology Department	Public Safety, Private Security, and Temporary Re-deployable Video Surveillance Cameras at Outdoors Public Events	$47,450
University of Western Ontario, Faculty of Information and Media Studies	The View from Here: User-Centered Perspectives on the Privacy Expectations of Digital Citizens	$49,851
2010-11 - Total of 16 projects
University of Toronto	'Smart' Private Eyes in Public Places? Video Surveillance Analytics, New Privacy Threats and Protective Alternatives	$50,000
Union des consommateurs	Seminar on new technologies and consumer protection	$ 5,000
Queen's University	The Private Sector, National Security and Personal Data: An Exploratory Assessment of Private Sector Involvement in Airport and Border Security in Canada	$50,000
Media Awareness Network (now referred to as MediaSmarts)	Young Canadians in a Wired World Phase III Qualitative Research Project	$ 21,599
University of Victoria	First Nations Privacy and Electronic Health Record Systems	$46,250
University of Victoria	Privacy Risks and Mitigation in Consumer Health Informatics Products	$46,250
! McGill University, Department of Genomics and Policy	Privacy in Canadian Paediatric Biobanks: A Changing Landscape	$50,000
Option Consommateurs	Guide and Workshop on One's Credit File	$27,550
Public Interest Advocacy Centre	Consumers Anonymous? The Privacy Risks of De-Identified and Aggregated Consumer Data	$50,000
York Centre for Public Policy & Law	Privacy Rights Mobilization Among Marginal Groups in Canada: Fulfilling the Mandate of PIPEDA	$49,330
University of Toronto	A Privacy Protective 'Proportionate ID Digital Wallet' for Canadians	$48,000
Ryerson University	Privacy as a Risk Management Challenge for Corporate Practice	$34,787
Ryerson University	Targeted Online Advertising	$16,008
Canadian Internet Policy and Public Interest Clinic (CIPPIC)	Agents of the State? The Evolving Role of Internet Intermediaries in Public Sector Surveillance	$40,500
! CHEO Research Institute: Electronic Health Information Laboratory	Managing the Risk of Re-identification for Public Use Files	$ 36,000
Ryerson University	Applying PIPEDA to the Smart Grid	$11,615
2009-10 - Total of 13 projects
Canadian Association of the Deaf	Deaf Perspectives on Identity Theft and Privacy project	$49,855
International Association of Privacy Professionals Canada (IAPP Canada)	The IAPP Canada KnowledgeNet Expansion and Promotion Initiative	$48,440
Coopérative radiophonique de Toronto (CHOQ-FM)	Awareness Campaign on the Protection of Personal Information and Privacy	$48,750
Option consommateurs	Awareness Workshop on Identity Theft and Seniors – Prevention is Better Than Cure	$25,000
Association sur l'accès et la protection de l'information (AAPI)	Internet portal on privacy protection	$15,000
University of Guelph	Privacy and Disclosure on Facebook: Youth & Adults' Information Disclosure and Perceptions of Privacy Risks	$49,991
University of Alberta, Health Law Institute	Analysis of Privacy Policies and Practices of Direct-to-Consumer Genetic Testing Companies: Private Sector Databanks and Privacy Protection Norms	$49,450
University of Victoria	Deep-Packet Inspection: Resources for the Analysis of Privacy Implications in Canada	$28,211
Memorial University of Newfoundland	Privacy Protection and Biobanks: A Conjoint Analysis of Priorities and Preferences of Stakeholder Groups	$50,000
Union des consommateurs	Electronic Health Records: Controlling Personal Health Data in the Context of Medical Records Information	$40,000
! Queen's University	Camera Surveillance in Canada: A Research Workshop	$50,000
University of Ottawa, Centre for Law, Technology and Society	Doing Girl Online: How Social Networking is Transforming Gender, Equality and Privacy	7,958.02
Institute of Public Administration of Canada (IPAC)	Social Media and Public Sector Policy Dilemmas	$ 10,000.00

Most organizations have received close to the $50,000 maximum available and no organization has received over the maximum of $100,000 per year. While some groups have been funded multiple times over the 5 years (Queens University , University of Victoria, University of Toronto, Option Consommateurs), there is greater variety in the range of funded applicants than in the last evaluation. The majority of recipients continue to be academic institutions, however this is significantly reduced from the last evaluation (universities received 54% of the total projects compared to 68% in the last evaluation). Non-profit privacy/consumer advocacy groups and industry associations represent 46% of the funded projects. This change reflects the Program's efforts in reaching out further to other potential applicants and further encouraging knowledge translation initiatives.

Overview of Outputs

The following table summarizes performance data against the outputs, outcomes and indicators noted in the Program’s Performance Measurement Strategy (2009):

Table: Performance Measurement Strategy - Performance Information from the OPC Contributions Program (2009-10 to 2013-14)

Performance Element	Performance Indicators	OPC Target	Performance Data Collected
Outputs
Program Management Outputs - Data collected by Evaluator (June 2014) covering 57 projects funded 2009-10 to 2013-14
Projects approved	% projects funded versus applications received for funding	10% to 25% (not to exceed Program limits)	23% of applications submitted are funded
Funds allocated	% funds allocated to projects versus total funding available	85%	92% of total funding available was allocated to projects
	% funds allocated by OPC priority area	Subject to the eligibility of projects submitted	Estimated percentage of projects funded by priority area: Identity Integrity & Protection - 57% Information Technology - 23% Genetic Privacy & Biobanking - 11% Public Safety & National Security - 9%
	% funds allocated spent by recipient group	Subject to the eligibility of projects submitted	Percentage of projects funded by recipient group: Universities - 54% Non-profit privacy/consumer advocacy groups or industry Associations - 46%
Project Outputs - Data collected by Program (June 2014) covering 39 projects funded 2010-11 to 2013-14
Project Reports	Reports that meet the conditions of the Contributions Agreement	100%	95% of projects met all conditions of the Contribution Agreement
Academic papers	Peer-reviewed papers on privacy-related research and knowledge	Research projects to meet targets noted in Contributions Agreement	77% of projects produced research papers, including about 50 academic/research reports with 20 specifically noting results to be published in peer review journals (note: not all projects are expected to produce research papers)
Workshops, Conferences	Engagement of individuals in privacy-related research or information events	Projects to meet targets noted in Contributions Agreement	64% of projects held workshops, sessions or conferences to disseminate project information (comprehensive information on the number of participants was not available) (note: not all projects are expected to hold workshops, sessions or conferences)
Informational Materials (toolkits, web pages, etc.)	Reach of privacy-related materials to individuals	Outreach projects to meet reach targets noted in Contributions Agreement	79% of projects produced informational materials for dissemination (e.g., guides, fact sheets, final reports, videos, pamphlets, games, toolkits, DVDs, websites, blogs, educational materials, etc.). Generally, these were disseminated electronically and reach figures were not available (note: not all projects are expected to produce informational materials)
Recommendations or Advice	Provision of useful guidance and advice on how to enhance privacy rights and obligations	At least 1 useful recommendation per project funded	79% of projects provided recommendations, advice, best practices or guidance to improve privacy protection practices in project deliverables (note: not all projects were required to produce recommendations; a number of projects, for instance, were public education and awareness raising in nature, and thus not conducive to generating analysis and recommendations)
Immediate outcomes - Data collected by Program (June 2014) covering 39 Projects funded 2010-11 to 2013-14
Increased and enhanced information, knowledge and best practices	Amount of new knowledge, information and best practices developed by project recipients	All research projects to make a contribution to privacy-related knowledge	85% of projects provided new knowledge, information or best practices related to privacy related fields (e.g., nature and/or awareness of new privacy threats or risks, ways to protect privacy, changes required to policy/legislation, etc.) (note: some public education/awareness projects were not intended to create new knowledge)
	Capacity of recipients for privacy-related work	None (indirect benefit)	Not measured; anecdotal evidence provided by interviewees to indicate the Program makes a key contribution to building privacy-related capacity through funding students and not-for-profit staff
Increased and enhanced sharing and dissemination of information, knowledge and best practices	Reach of all project outputs to target audiences	All projects funded to have stated information dissemination or outreach component	90% of projects reported they reached their target audience through knowledge translation activities (actual numbers reached were not comprehensively tracked)

At the output level, it is clear that the Program is achieving or close to achieving its targets in all areas:

• At the management level, the target funding allocation (% of project funded versus applications) is being met and in fact, since the last evaluation, this data indicates a stronger demand for the program. Virtually all of the total funding was allocated to projects which has also improved since the last evaluation, ensuring available funds are spent on privacy-related projects. As well, there is a diversity of projects funded across priority areas and recipient groups; this diversity has increased since the last evaluation.
• At the project level, all but a few (5%) of the funding projects met all conditions of their Contribution Agreement. Different projects had different aims, outputs and methodologies - the majority (77-79%) of projects produced research papers and/or informational materials (depending on the type of project) and many (64%) used workshops or conferences to disseminate project results (note: about 20% of projects are public education/awareness). About 20 peer-reviewed publications were estimated to have resulted from the 39 projects assessed. However, while a high percentage (79%), not all projects provided recommendations or advice to improve privacy protection practices as was targeted. In some cases, particularly for public educations/awareness projects, recommendations were not relevant.

Overview of Case Studies

To further assess the performance of the Program, this evaluation selected 8 funded projects from the past 5 years to examine the range and type of outcomes emerging from the Contributions Program. Key results from these projects are noted below and have been used to inform the evaluation issues and questions related to the achievement of outcomes.

Organisation	Project	Key Results
2009–10
1. Queen's University, Surveillance Studies Centre and Surveillance Camera Awareness Network (SCAN) Kingston, Ontario	Camera Surveillance in Canada $50,000	Outputs: Two-day workshop to discuss camera surveillance in Canada. The aim of the workshop was to build upon A Report on Camera Surveillance in Canada, (part one released in January 2009 and part two released in December 2009) prepared by SCAN and funded by the OPC by generating fresh, clear, independent findings on camera surveillance in Canada and to have an open and public discussion of issues related to privacy and camera surveillance Outcomes: With release of 2009 report, launched web page for SCAN, added frequently asked questions and Resource page (with links to OPC and others), and forum Offered resources to researchers, policy makers and media Network of researchers on camera surveillance established Report was highlighted in Ottawa Citizen (online and print), and media outlets coast to coast picked it up Book on Camera Surveillance - Eyes Everywhere: The Global Growth of Camera Surveillance - first international perspective on the development of camera surveillance Launched Surveillance Studies Centre at Queens, a leading global hub for research on expanding surveillance practices, funded by SSHRC with OPC support/partnership
2. McGill University, Department of Genomics and Policy, Montreal, Quebec	Privacy in Canadian Paediatric Biobanks: A Changing Landscape $50,000	Outputs: Review of the ethical issues involved in privacy and confidentiality of paediatric biobanks; comparative review of related policies between a few countries; and review of provincial legislation Report 'Privacy in Canadian Paediatric Biobanks: A Changing Landscape' and Guidelines for paediatric biobanks and their handling of personal information Easy- to-read pamphlet outlining findings Outcomes: Distribution of pamphlets to Research Ethics Boards and paediatric researchers and online (relevant websites), and presentation at 3 conferences Preparation for paper to peer-reviewed journals (2) Recommendations to the OPC regarding privacy issues in paediatric biobanks Provided better understanding of current situation in paediatric biobanks regarding privacy and confidentiality and raised awareness with stakeholders In some contexts, made recommendations as to how to maximize the protection of the child's privacy and confidentiality in the unique context of the tri-partite relationship without compromising their participation in biobanks to provide better protection of children's personal, health and genetic information (e.g., recent publications on newborn screening)
2010–11
3. CHEO Research Institute: Electronic Health Information Laboratory Ottawa, Ontario	Managing the Risk of Re-identification for Public Use Files $36,000	Outputs: Report on Managing the Risk of Re-identification for Public Use Files: The report discusses in detail the principles, metrics, and methods that can be used to manage the privacy risks associated with disclosing data, and to ensure that the probability of re-identifying individuals in publicly disclosed files is low and that the probability of discovering sensitive information about them is low Outcomes: Report provided useful, specific guidance for people who are interested in de-identifying data sets and gave data custodians the tools to make decisions about the best way to disclose this data, but also ensure that the privacy of individuals is protected Manual grew into a 400 page book (with other research) published in 2013 by CRC Press: Guide to the De-Identification of Personal Health Information - uptake in first 6 months has been quite high (above 1000) Companion series of training modules now being offered in cities across North America - 130 people trained, with professional 2-day course in Canada and now in the US Private spin-off company - Privacy Analytics - which develops specialized software to help government, health care providers and commercial enterprises to minimize the probability that anonymous personal data can be re-identified at a later date (40 people employed;$3.5 million US seed financing)
2011–12
4. Association sur l'accès et la protection de l'information (AAPI) Quebec, Quebec	Educational kit for developing sound practices for posting pictures and personal information on the Net $50,000	Outputs: Education kit for tweens and teens to foster sound practices for posting pictures and disclosing personal information online (aligned to school curriculum) Interactive materials for junior high school teachers to spring board discussions about online privacy and personal information protection (2 volumes - 1 for teachers and 1 exercise book for students) Website for online availability Outcomes: Widely distributed in Quebec secondary school system - 40 of 72 school board requested copies (bilingual version to be developed for rest of Canada) Through online registration by teachers, 80 schools reached (history and ethics teachers) - estimated 5000 students reached Distribution to Éducaloi, Commission scolaire, Fédération des enseignants, CPVP du Canada, CAI
5. Atmosphere Industries, Toronto, Ontario	Gaming Privacy: Creating a Privacy Game with Canadian Children $49,265	Outputs: Augmented board game (iPad + board game hybrid) created to help kids develop autonomous privacy decision-making skills - the 'Watchers'; can be downloaded for free or purchased as a board game together with web app Downloadable game kit Website for distribution Research and design blog Companion research which provides an overview of what children know about privacy and what is important to teach children Research report detailing development process and evaluating performance Smaller papers for submission to journals and conferences Outline of how to support children as co-designers in game development process Outcomes: Provided ground-breaking knowledge - children understand 'stranger danger' but not how personal information is used Efficacy of the game results were positive but preliminary (project did not include post launch test phase) - need to pilot the game to test efficacy Feedback from game co-creators noted that, through the game, they understood the basics of the privacy concepts, particularly around negative concepts, however they were still unclear about some details; a proposed teachers' or parents' discussion kit could help Feedback suggested using experiential learning, narrative and scaffolding were useful techniques to convey the concepts Likely increased privacy, autonomy and critical thinking skills amongst the children and youth who co-developed and played the game Provided insight into best practices when engaging youth in online privacy protection
2012–13
6. MediaSmarts, Ottawa, Ontario	Young Canadians in a Wired World III - Quantitative Research Project $50,000	Outputs: Comprehensive and wide-ranging study of children's and teens' Internet use in Canada - literature review on how children navigate online space, surveillance by parents and corporations online, etc. and national survey for Grades 4-6, 7-11, Parent info sheet and consent form, teacher instruction sheet Survey sent to 13,000 students, and completed by 3,800 (as of final reporting) Outcomes: Deeper understanding of how Canadian youth use the internet and the lived experiences of Canadian youth who are far more resilient than parents and school administrations believe; common myths dispelled (e.g., boys are equally likely to cyber bully) Best practices for teachers to contribute to learning in today's networked schools and how parents can help young people gain digital literacy skills Information used by researchers and government agencies Articles for educational publications and a wide range of knowledge translation activities in both online and offline environments
7. British Columbia Society of Transition Houses Vancouver, British Columbia	Safety Net Canada 2013: Technology, Privacy, Safety and Violence Against Women, Youth and Children $50,000	Outputs: Reports and resources that focus on technology and how it impacts the safety and privacy of women and children fleeing or living with the effects of domestic violence and survivors of sexualized violence, stalking and harassment (e.g., 2 national surveys) Study to understand how technology practices increase or decrease safety risk for women and their loved ones, with guidance on: how to help those who collect information maintain the safety and security of women; and, how to help program determine their capacity to use social media and examine the potential risks that social media has for Violence against Women Agencies, women, youth and children Canadian legal remedies for technology enabled violence against women Organizational technology practices for Anti-violence programs, protecting the safety and privacy of women, youth and children Educational materials: safety tip postcard, Infographic, Webpage, Social media (twitter), Video resources Training: 8 technology safety trainings to law enforcement and anti-violence workers in BC, ON, QC Outcomes: Recommendations for practices and policies around technology, data and social media use that protect the privacy, safety and confidentiality of women, youth and children (e.g., legal gaps and remedies) Media interest (CBC) that indicated this work was key source of information for technology invasions story Subsequent training of various audiences (anti-violence sector, justice system workers, law enforcement , legal aid) and supported education in communities and schools In some provinces, Privacy Tort remedies based on civil law piece Collaborative partnership between NGO (violence against women) and Canadian Internet Policy and Public Interest Clinic (legal) Input to policy on cyber bullying legislation
8. British Columbia Civil Liberties Association Vancouver, British Columbia	National ID Card by Stealth (The BC Services Card: Privacy Risks, Opportunities& Alternatives) $50,000	Outputs: Survey National ID card vendors and technologies in NA and UK Literature review of national identity management programs and ID card schemes Legal analysis of associated privacy and security implications PIPEDA & FOIPPA compliant framework for assessing privacy enhancing ID card technologies Technical review of specific solutions for ID card vendors Public forum and associated media Final report on private sector contributions to government ID card schemes Outcomes: Aim to better assess the impact of ID cards and vendors on data protection and the identify integrity of Canadians Call for OPC to work with provincial privacy commissioners to issue a joint resolution on the applicable privacy and security standards for the provincial systems on the basis that they will ultimately compass the national federated system Recommendations for designing an identity system that is secure, privacy-protective, trusted and fit for purpose (focus on BC)

Achievement of Expected Immediate Outcome - Increased and enhanced information, knowledge and best practices

Based on performance data, 85% of all projects funded over the last 5 years provided new knowledge, information or best practices related to privacy related fields (others related to public education and awareness building). The Case Studies illustrated the ways in which the Program is increasing information or knowledge. While unique to each project, it is clear that projects are providing new information on the nature and/or awareness of privacy threats or risks, better understanding on ways to protect privacy, and suggestions for what changes are required to policy/legislation.

The 2012 Science-Metrix study (examining the years from 2005 to 2012) found half of the survey respondents indicated they produced scientific publications as a results of OPC funding, with most publications from 2008 to 2012. Since 2008, OPC funded projects have generated 3-6 publications per year (as opposed to 1 from 2005-2007). This indicates increased level and dissemination of new knowledge over the last few years of the Program.

Achievement of Expected Immediate Outcome - Increased and enhanced sharing and dissemination of information, knowledge and best practices

Based on performance data, 90% of projects reported they reached their target audience through knowledge translation activities (actual numbers reached was not comprehensively tracked). The Case Studies illustrated the ways in which the Program is enhancing the dissemination of information. While unique to each project, it is clear that projects are using primarily electronic dissemination mechanisms as well as presenting findings at workshops and conferences. In some cases, projects are partnering with others who further disseminate the project outputs, and in other cases they are directly training the target audience. Following the completion of the project, a number of cases exist where books or follow-on publications have resulted and been widely disseminated.

The 2012 Science-Metrix study found, in addition to scientific publications, the majority of survey respondents indicated they produced at least one output due to OPC funding. These included primarily conference presentations, but also internet publications/posts, and media. Uptake of research outside of the immediate research group focussed on conference or information sessions, publication of articles or research results being used in mainstream media. A cluster of other methods centred on sensitization or training of organizations, professionals, volunteers, industry, associates and the general public on privacy issues. Another cluster involved the use of research findings for policy-making purposes.

In terms of social media impact, Science-Metrix found most of the reports resulting from OPC funded projects were detected on the web. About half were present and shared (liked, shared, tweeted) on social media platforms. The study indicated that web users have the opportunity to find these publications on the internet. One publication in particular was detected in the most number of web pages and shared the most: Hey mom, what's on your Facebook? Comparing Facebook Disclosure and Privacy in Adolescents and Adults (2010).

In terms of scientific impact, the Science-Metrix study indicated few citations to OPC funded documents were made in the scientific literature (13 OPC supported documents with 29 total citations in years 2006 to 2009, increasing over time). Those that were cited were only featured in a limited number of publications. These cited publications did not have much impact in the social media.

As well as the projects themselves disseminating their results, the OPC Program has enhanced its role in knowledge translation over the last few years (see also Question 5). This includes:

• Five year strategy for the Program (2011), which in part focused on promoting the Program more broadly to the appropriate public education and outreach community.
• Communications strategy to better reach out to key stakeholders, to promote the Program's existence as a source of funding, and to promote the new knowledge generated under the Program.
• The Pathways to Privacy Symposiums (May 2012 & March 2014) to enable uptake and application of research results by stakeholders and promote the Program more broadly. Evaluation results from the events were positive with most agreeing that the symposium met its stated objectives.
• "Real Results," a magazine style publication that celebrates some of the projects funded under the Program.
• Networking with other funding agencies, university grants and research offices, and external peer reviewers to promote the new knowledge generated under the Program.

OPC Senior management and other stakeholders interviewed indicated the Program has improved the knowledge translation component of the Program, both through internal communications activities and by requiring each funded project to have an explicit outreach plan.

Achievement of Expected Intermediate Outcome - Increased and enhanced policy development

Based on performance data, 79% of projects provided recommendations, advice, best practices or guidance to improve privacy protection practices in project deliverables. The Case Studies illustrated a number of areas where recommendations were made to influence policy or legislation and, in one case, input was made both to developing legislation (cyberbullying) and to provinces that have come out with new privacy tort remedies. However, while interviews confirmed that OPC-funded research helps inform internal policy positions, this is not tracked/documented on a Program wide-basis by the OPC and the Science Metrix study concluded that there is limited evidence that OPC-funded research outputs had an impact on policy-making overall.

Interviews with OPC Senior Managers indicated that the Program contributes advice to Parliament (OPC positions are informed by the Program and also funded parties contribute their thoughts to Parliament directly and with their stakeholders). It was noted by the OPC that the Office is not a "pure" funding agency, but rather an Agent of Parliament, providing informed advice to parliamentarians and other policy makers in government. The OPC is also a regulator, charged with ensuring compliance with PIPEDA and the Privacy Act. As such, the Office needs to extract value from the projects funded under the Program to inform and support its mandate. Likewise, the OPC needs to extract value from projects to supplement and buttress its public education and knowledge translation mandate. Interviews confirmed that the Program has a role in this regard and, as well, the Program strengthens the credibility of the OPC as an intellectual leader in the privacy area.

External stakeholders all indicated that funded projects are used to inform OPC policy or positions (i.e., evidence-based policy), and that the Program has a positive impact on the credibility and profile of the OPC overall.

Achievement of Expected Intermediate Outcome - Increased and enhanced public awareness

Based on performance data, 79% of projects produced informational materials for dissemination (e.g., guides, fact sheets, final reports, videos, pamphlets, games, toolkits, DVDs, websites, blogs, educational materials, etc.). Generally, these were disseminated electronically and reach figures were not available. The Case Studies also illustrated a wide variety of ways in which project results were disseminated; however, the actual change in public awareness overall is difficult to assess and quantify.

As context, Public Opinion Research (POR) completed by the OPC in 2013 is provided below. It should be noted however that the Program would not be expected to have a measurable impact on general public awareness levels given its size and objectives.

• Canadians' knowledge about their privacy rights under Canada's privacy laws is limited, although improving. While one-third (35%) rated their knowledge relatively highly (scores of 5-7 on a 7-point scale), a clear majority (63%) rated their knowledge low on the scale or in the neutral range. Compared to previous years, Canadians' understanding of their privacy rights is higher than it has been in the past - 35% vs. 28-30% in 2009-2011 (and lower before that).
• Despite placing heavy importance on privacy issues, only 21% have ever sought out information about their privacy rights.
• Canadians' awareness of federal privacy institutions is relatively low, but rising. Seven in ten Canadians are not aware of any federal institutions that help them with privacy and the protection of personal information from inappropriate collection, use and disclosure. However, since 2005, Canadians' level of awareness has risen overall. Of those who claimed to be aware of a federal privacy institution, 15% were able to identify the OPC by name.

The Science Metrix study concluded that, overall, and considering the types of research outputs produced and the channels through which they were disseminated, the OPC Contributions Program had a greater impact in sensitizing public opinion (i.e., general population reached via social media) to privacy issues than it had on the academic community (i.e., scientific publications cited in the academic literature).

Unintended Outcomes

The key unintended outcomes related to capacity building. While not measured, anecdotal evidence provided by interviewees indicated that the Program makes a key contribution to building privacy-related capacity through funding students and not-for-profit staff. The symposiums have provided a networking function to build each other's capacity and learn more broadly about a range of privacy issues. Some funded projects and the Symposiums have also enhanced the capacity of individual Canadians to protect their privacy (e.g., educational resources provided to schools).

5. Is the Program implemented as planned to achieve its intended outcomes?

Implementation in line with Terms and Conditions

The review of several funded and unfunded projects as case studies indicated that projects are generally being implemented in line with the Program's Terms and Conditions. For example, the areas funded reflect program objectives and reflect studies into privacy related research and public education/awareness; eligible recipients are funded (not-for-profit consumer, voluntary and advocacy org or educational institutions and industry and trade organizations); and, the maximum amount payable of $100,000 per year to a single organization is respected.

The Program has a Basic Eligibility Checklist to ensure that all selected projects meet the eligibility requirements of the Program's Terms and Conditions, specifically the eligibility of applicants, activities and expenses. As well, there is a comprehensive process to review and document if project deliverables are provided in line with the contribution agreement. One Case Study involved a project that initially did not produce results in line with documented expectations. In this case, the OPC withheld payment until the study was revised and the OPC was satisfied. Final project assessment documents noted these deficiencies and, subsequently, the Program further improved monitoring of projects so that now (in 2014-15) recipients are required to provide an annotated table of contents for review a few months before they provide their final product in order to ensure they are still on track for delivering what they set out to do in their agreements.

The two areas in which the Terms and Conditions are not being met include:

• Quarterly progress reports are not being provided by all projects given the small size of the grants and the low risk involved to the organization. Usually, projects provide just a mid-point and final progress report. Most funded projects interviewed indicated that quarterly reporting is too frequent for the amount of funds provided and is difficult for smaller organizations to manage this efficiently, with little value added.
• While significant monitoring does take place to ensure compliance with program requirements, internal performance reports in line with the performance measurement strategy are not being regularly produced by the Program. A bibliometric study was conducted in 2012 by Science Metrix to assess some aspects of performance; and, performance information was compiled at the time of the evaluation.

Application Review Process Appropriate and Documentation Complete

The application review process has been improved since the last evaluation. It was deemed to be rigorous and comprehensive. The process now involves external reviewers (e.g., from universities) and other government departments (Industry Canada, SSHRC) to broaden the range of expertise brought to the assessment process, as well as learn from other funding agencies. The process involves the following:

• Program staff perform an initial eligibility review (basic checklist)
• If eligible, internal reviewers do an assessment to produce a shortlist (top half) of proposals across all 4 priority areas (note: considerations in the application assessment process include items such as the quality of the proposal, likely success, OPC need, past work done to indicate success and need, level of detail provided, overall need/relevance to others, appropriateness of methods, and feasibility)
• The top half of proposals is then assessed by external peer reviewers
• A meeting takes place among both internal and external reviewers to discuss the relative relevance, timeliness, quality and feasibility of applications and to reach group consensus on a final shortlist
• Recommendations for funding are made to the Commissioner

The document review provided evidence that projects generally had complete and detailed records of key transactions including:

• Checklist completed by Program with all key actions required/taken for project oversight (e.g., proposal, assessment, notifications, progress reporting, payments, web posting)
• Project proposal
• Application review scores and comments
• Peer review meeting minutes
• Contribution agreement
• Project reports
• Final deliverables
• Expenditure reports
• Correspondence
• Confirmation that deliverables reviewed and approved before payments provided

The review of two unfunded application files indicated appropriate rationale for why projects were not funded, and appropriate documentation and correspondence to validate the decision.

Priorities and Guidance to Focus Proposals and Result in Intended Projects

The Program's priorities are clearly articulated in Program's Application Guide and Calls for Proposals. All interviews with funded projects noted that the Program provides adequate guidance to focus proposals, the Applicant's Guide is very clear and straightforward, and priorities are understandable and helpful to indicate what research results are intended.

To provide additional guidance to applicants within these priority areas, and to ensure applications are targeting current needs, the Program annually solicits key research questions internally that the Program should fund. This information is put into the Applicant's Guide which is updated annually and informs the call for proposals.

Intended Recipients Targeted and Diversity in Applicants

The Program is promoted through the formal annual launch process (e.g., press release, distribution to OPC database, and the use of networks [Canadian Association of University Research Administrators; Industry Canada's consumer groups, Genome Canada, SSHRC, Canadian Institutes of Health Research, VP's of Research] to further advertise and the Program). Over the last 5 years, the Program has worked to try to reach out to new target applicants to expand the diversity of applicants. The Communications Strategy was developed and implemented to better promote funding opportunities available and help potential applicants learn about the program's existence. In the last few years, the Program has been promoted in a more organized and concerted manner (e.g., promotion to about 500 stakeholders in the privacy community, use of other OPC events (Research's Privacy Conversations) to market Program, use of Twitter and Blogs for Call for Proposals, use of speaking events to promote Program). There have also been recent efforts to try to increase applications from Francophonie applicants.

The success of these efforts is evident from the document review and interviews which show that the quality, diversity and number of proposals (particularly outreach projects) has improved. As well, applicants in the non-profit sector have built some capacity and are more informed and sophisticated about privacy issues and what can be funded by the OPC. External stakeholders believed that the Program is successful in attracting and funding the most appropriate projects, and that that there is a high level of awareness of the Program within the privacy community.

Most interviewees knew about the Program through their university administrators and/or through their general knowledge of the OPC and privacy research in Canada. Of those not in a university setting or dedicated to just privacy research, they heard about the Program from colleague who worked or used to work in the OPC or through a partnerships with other groups who knew about the Program. A number commented that some groups not in the privacy field, but concerned with privacy issues (e.g., social justice groups, health groups or internet researchers), are not aware of the Program and that partnerships with engaged universities may encourage more participation from the not-for-profit sector. For example, universities could be encouraged through the Program to partner with not-for-profit groups on relevant projects to increase the reach of the Program, build not-for-profit's capacity in terms of privacy, and bring different skills and capacities to a project (e.g., not-for-profit's knowledge of and expertise in raising awareness with certain populations).

Project results shared and disseminated with appropriate target groups

The result from each funded project is a final research report or, in the case of outreach projects, the outreach tools and report. Links to all reports/tools are put on the OPC's website but are actually housed on the recipients' websites. It was noted that project outputs may not be put up on the researchers' websites in timely manner, or left on these sites long enough, which negatively impacts the Program's intent in terms of active dissemination.

In addition, all funded projects are encouraged to integrate knowledge-translation plans into their projects. The objective is to facilitate the adoption of research results by end-users. These activities include a wide range of dissemination techniques depending on the type of project (e.g., blogs, presentations at academic conferences, publications, media coverage, policy recommendations, best practices, educational tools, videos, kits, materials, etc.). It was suggested by some stakeholders that the Program could help recipients with their ability to disseminate their results by providing a 'communications toolkit' like the one provided by Industry Canada's Contributions Program. Many funded projects in the past have not focused their outreach efforts or clearly articulated their top specific target audiences, but rather, have just used passive distribution methods like posting reports on websites. The Science Metrix study in 2012 found that some OPC-funded research was shared, liked or tweeted on social media but there was limited scientific impact (which may be attributable to the type of research rather than its quality).

The Program itself implemented additional mechanisms to disseminate project results. The first is the development and distribution of the magazine type publication "Real Results" (see Text box 4). This publication highlights a sample of the innovative and socially relevant independent research projects funded by the Program since 2004. While a few key informants felt the "Real Results" publication was too expensive and splashy for such a small program and was not needed on a frequent basis, it was praised by many of those interviewed.

The second mechanism is the "Pathways to Privacy Research" Symposium series. The symposium series is intended to showcase privacy-related research funded by the Program and partner organizations. It also aims to promote dialogue between the researchers and stakeholders (see Text box 5).

Finally, the Program, as directed in its Communication Strategy, enhanced its web presence making the website more interesting and the individual research reports more accessible (e.g., by topic index or by year, in a searchable database).

While most of the initiatives proposed in the Communications Strategy have been implemented, a number of approaches were not implemented due to other priorities or lack of resources (e.g., e-newsletter, postcard, and annual report have not been implemented; however, it was never intended that all suggested or potential activities be implemented). A revised and updated Communications Strategy is planned for the renewal of the Program.

Moving forward, external stakeholders emphasized the importance of disseminating research results and putting 'research into action'. All indicated that additional effort could be put towards 'getting the results out'. A number of external researchers suggested developing short (5-page) policy briefs that are collaborative efforts with the OPC to foster 'evidence-based policy' and showcase how research results are informing policy (see Genome Canada example in Text box 6). However, not all Program research lends itself to this format and, while OPC policy can be informed by research projects, these don't direct OPC policy.

While there is a wide range of actions in place to disseminate project results, no statistics are compiled for the Program on the reach achieved overall (note: individual project statistics are included in a number of specific project files, and the OPC is currently collecting web hits to the OPC research pages).

Program objectives and priorities evolve and improve to meet emerging needs and respond to risks and challenges

The Program produced its "Next Generation" Strategy (December 2011) to provide a new research strategy going forward. This strategy built from the previous evaluation and from lessons learned during the first 5 years of Program implementation. The proposed strategies included:

1. Leveraging impact through partnership (achieved with SSHRCC (Digital Economy), Genome Canada (knowledge translation), Industry Canada (knowledge dissemination))
2. Enabling knowledge translation and application (revised Applicant's Guide to ensure applicants articulate their plans for knowledge transfer, Real Results, knowledge translation symposiums)
3. Strengthening peer review (external reviewers engaged)
4. Facilitating access through technical enhancement (searchable database implemented; enhanced webpage on Research)
5. Evaluating the success of the program (Science-Metrix bibliometric study)
6. Renewing public communication strategy (Communications Strategy updated 2012-13)

In addition to the evidence found regarding achieving progress against the new Strategy, the Science-Metrix study (2012) and interviews indicated that respondents have a positive opinion of the Program and the OPC team, and that the Program responds to the key privacy risks and challenges that exist.

Issue #5: Demonstration of Efficiency and Economy

6. Is the Program producing its intended outputs in an efficient manner?

Management and financial structures and processes balance accountability and flexibility to promote efficient delivery

As noted previously, the Program has a number of processes to ensure accountability (rigorous application review, checklist of deliverables before payment, etc.) which have evolved and matured over the second five years of implementation. Interviewees agreed that the Program has improved in terms of its internal controls and processes (e.g., more regular cycle, strong focus and alignment with priorities, improved assessment process, enhanced controls, greater dissemination of results). As well, there have been internal process changes to enhance efficiency (e.g., a SharePoint site to receive and review applications, new and improved program and project tracking charts and tools, new project progress report forms that must be filled out by recipients, new project assessment forms to assess quality and completeness of deliverables).

To assess controls at the project-level, for the first time, four recipient audits were completed (see Text box 7). Minor issues were identified in one project and minor issues have also been identified in a few projects by internal Finance (e.g., submission of deliverables within fiscal year, frequency of progress reporting).

From an external perspective, recipients see the administrative requirements of the program as overly burdensome. For example, quarterly reporting is seen as too frequent for the amount of funds provided, with little new information provided as opposed to biannual reporting. As well, some recipients indicated that the amount of documentation required (financial expenditures, etc.) is unreasonable for the funds provided and that a better balance is needed between justifying expenses and efficient management.

Greater flexibility is also desired in terms of payments: for smaller organizations, advance payments are desired to better enable them to manage funded projects; and, for the OPC, more flexibility for hold-backs is desired.

The Program is currently working with Finance to map all business processes to ensure appropriate and clear roles, responsibilities and controls. There is a difference in opinion internally between the Program (who promote the need for greater flexibility) and Finance (who accentuate the need for internal controls) about the appropriate degree of control and oversight required for this Program, and by whom, considering its small size and relatively low risk. To address these challenges, a "Contributions Program Working Group" has been established with representatives from Communications, Legal, Program, and Finance sections in the OPC.

Project results and performance information being collected, reported and used for Program decision-making

The Performance Measurement Strategy (Dec. 2009) sets out the indicators and targets for the Program. A performance report was to be done annually and reported in OPC's Annual Report, with a focus on outputs and immediate outcomes. As part of the evaluation, performance information was collected to document project outputs and outcomes against the PMS indicators. No evidence was provided that performance information was collected on a regular ongoing basis before the evaluation and reported annually. The Program does have much of the performance data available in individual project files, does provide a summary of project reports, and does report annual spending to Management. The project report summaries are also provided on the OPC web site with links to the full report. In addition, the PIPEDA Annual Report contains a section on the Program which includes what projects have been funded and new initiatives. The Program also assessed Program performance as part of the 2012 Science Metrix study which examined the impact of funded research since 2004; and, select program achievements were profiled in the "Real Results" publication.

Recently, an ongoing performance indicator related to the "uptake of OPC research" has been included in the Departmental Performance Report to indicate how knowledge about privacy issues is being advanced. This will be reported on annually (starting in 2014-15) and it can track internal research and Program-funded research separately through web site visits.

Percentage and comparison of administrative costs to total Program costs

To try to assess internal Program efficiency, the percentage of Program overhead/administrative costs compared to total Program costs was determined. For the OPC, the overhead costs were estimated considering Program staff salaries and contributions from Communications and Finance staff. It is estimated that administration costs are about 25% of program costs. This is high compared to other programs used for comparison and is due to the small size of the total fund - a base level of capacity is needed for any Contributions Program and thus a small fund would normally have a relatively higher overhead percentage. It should also be recognized that the Program contributes to the general mission of the OPC and therefore much of what is done (e.g., promotion of privacy research, knowledge translation, creating synergies with Program stakeholders) is not strictly 'overhead' per se. Rates for other comparison programs are noted below:

• The 2009 Evaluation of Industry Canada's Contributions Program indicated the overhead or administration costs of the program, including salary and O&M (operations and maintenance) averaged 8% over three years, which was within their target rate of 10%. This fund is over three times the size of the OPC's ($1.69M) and they fund about three times as many projects (~30/year).
• Communications with SSHRC indicate that operating costs are about 8% of the total budget for their granting program. This program is much larger than the OPC's and their multi-year grants require less administration.

If the OPC Contributions Program were to try to achieve a 10% overhead rate, it would have to drastically scale-back both its activities and internal processes and controls (e.g., assessment, oversight, communications, networking, promotion, etc.). While efficiencies may be able to be achieved through some streamlining and reducing efforts in some areas, a 10% target is not deemed reasonable for the OPC Program as it cannot benefit from the economies of scale that are present with larger funds. However, the level of effort currently made by the Program to market itself, extend its reach, communicate results, and ensure due diligence is high for such a modestly-sized program, but it is also contributing to the significant Program improvements, helping to further the mission of the OPC and resulting in the achievement of intended results documented in this evaluation.

Resources leveraged/contributed by partners

While not tracked or reported at the Program level, the Case Studies illustrated the type of leverage attained through funded projects. All projects achieved some kind of leverage, though mostly this was via in-kind administration, infrastructure or staff time. Additional amounts levered ranged from $10,000 to $100,000 (i.e., doubling the OPC contribution).

Views on Project Efficiency

The Program's projects were seen to be efficient by both internal and external stakeholders. A high level of output for the funds invested was produced from the funded projects due to the following:

• Extensive use of graduate students in research projects under the supervision of expert privacy researchers;
• Extensive use of volunteers and/or students for knowledge translation projects;
• Creative and efficient approaches utilized by projects to maximize the funds and time available;
• Results that continue beyond the funded timeframe (further research, capacity built, further outreach, etc.); and
• Levered resources contributed by recipients or other organizations.

7. Is the Program achieving its intended outcomes in an economical manner?

To examine economy, spending patterns of the Program were compared. The graphic below illustrates and compares the contributions available, planned and actually spent over the last 5 years by the Program.

Text version

Figure 4: Comparison of spending for OPC Contributions Program

This figure is a bar graph of the comparison of spending for the OPC contributions program. For each fiscal year between 2009 and 2014, it offers a comparison of funds available, planned funds and actual spending. The following table summarizes the data presented in the graph:

Fiscal year	Available spending	Planned spending	Actual spending
2009-10	$500,000	$439,542.00	$435,266.00
2010-11	$500,000	$446,385.00	$446,385.00
2011-12	$500,000	$438,080.00	$441,355.00
2012-13	$500,000	$499,240.00	$499,240.00
2013-14	$500,000	$480,998.00	$480,998.00

In summary, the Program spent the vast majority of its contribution funding each year:

Year	Percentage of total funds spent
2009-10	87%
2010-11	89%
2011-12	88%
2012-13	100%
2013-14	96%
5- year Average	92%

This level of spending is much improved since the last evaluation when only 63% of the available funds had been spent over the first 5 years of the program.

When examining spending at the regional level, it is clear that the majority of funds have been spent in the Ontario Region over the last 5 years, with no spending in Manitoba or Saskatchewan and little spent in the Atlantic Region or Alberta. The capacity for privacy research may be lower in these areas or the Program may not be effectively reaching these provinces (this was not determined in the evaluation).

Text version

Figure 5: Comparison of planned spending by Region (2009-10 to 2013-14) 5-Year Total

This figure is a pie graph representing of the percentage of total planned spending (2009-14) by region. The following table summarizes the data presented in the graph:

Region	Percentage of spending over five fiscal years
Atlantic region	2%
Quebec	18%
Ontario	62%
Manitoba and Saskatchewan	0%
Alberta	4%
British Columbia	13%

Appropriateness of inputs to achieve outcomes

External stakeholders indicated that the $50,000 amount available per project seemed to be about the right amount of funds for a one year project. Some indicated that they would prefer larger and longer projects - i.e., $100,000 for a 2 year timeframe. If more funds were available, and more time, proponents indicated they would be able to conduct more in-depth projects and better disseminate and promote the results.

Views on whether value for money

External stakeholders agreed that the Program is producing high value for money, with high leverage rates and high quality research being produced. In a number of cases, projects could be shown to have produced results that were making a difference and achieving the stated outcomes of the Program. However, it should be recognized that these views all came from funded 'successful' projects, and there was some recognition that not every funded project is a success (as with all research).

One commented that global investments into how to exploit privacy are huge and very few resources are spent on how to best protect privacy. Therefore, Programs like the one in the OPC are important to fund ground-breaking research and disseminate the results to counter-balance investments made in the areas of privacy invasion.

Alternative Approaches

The last evaluation examined several alternative approaches to the Program, which are noted below. The assessment of these approaches has not changed, although some updates have been provided and one additional alternative has been added:

Alternatives	Implications
Building the capacity of the OPC's internal research and communications groups	Would be more expensive than providing contributions, but would be under direct control of the OPC and may link more directly to policy development. Would not stimulate external research and critical thinking on privacy issues by academics. Would not allow the OPC to build on privacy expertise and reach/networks in the academic, industry and non-profit sectors - could not offer the breadth and depth of research or outreach available externally. Would not develop external capacity on privacy issues. Would not allow for free range of positions and debates which provides a breadth and depth of knowledge that can't be done internally.
Contracting out specific research and outreach projects	Would be more expensive than providing contributions, but would offer more control and direction from the OPC. Would not stimulate research and critical thinking on new privacy issues by academics. Would favour established institutions and not build capacity with not-for-profit groups.
Delivery of Contributions Program through a non-profit group	Would be more expensive than providing contributions, as capacity would need to be built, overhead funded, etc. Would provide independence and arm's length from the OPC, but result in less OPC control in terms of directing research and outreach priorities. Would reduce administrative burden in the OPC
Delivery of Contributions Program by SSHRC or Industry Canada	Would be more economical by benefitting from economies of scale and existing processes/resources in established larger programs. May have access to a larger pool of potential applicants, not necessarily specialized in privacy areas. The OPC would lose some control, influence and credibility. Funding may focus less on privacy issues over time - may lose its niche as a specialized program devoted to privacy research (as it may have to compete with many other social/consumer issues). Would lose synergy with internal Research and Policy functions. Would not benefit from the strategic orientation and leadership provided by OPC or from the experience/expertise of OPC staff.

6. Overall Conclusions

This evaluation concluded the following, in line with the evaluation issues and questions:

1. Does the Program continue to address a demonstrated need?
   
   The Program continues to address a demonstrated need based on public survey findings, the demand for the program, its unique niche in promoting privacy issues, and universal support from those consulted.
2. Does the program continue to be aligned with OPC priorities?
   
   The Program is aligned with the OPC priorities as it meets the intent of PIPEDA, contributes to the OPC's outcomes and funds projects directly aligned with the OPC's priorities.
3. Does the Program continue to be aligned with the roles and responsibilities of the OPC?
   
   The Program is aligned with the roles and responsibilities of the OPC as it falls within the Office's mandate and is the only Canadian program with a sole focus on privacy research and related knowledge translation.
4. Is the Program achieving its intended outcomes?
   
   The Program is achieving its immediate outcomes to increase and enhance both the production and sharing of privacy information, knowledge and best practices. Since the last evaluation, there has been a greater diversity of projects funded across priority areas and recipient groups as well as more knowledge translation activities. While making an impact, continued work is needed to measure and document the degree to which the Program is enhancing policy development and contributing to raising public awareness.
5. Is the Program implemented as planned to achieve its intended outcomes?
   
   The Program is generally being implemented as planned, with improvements made to the application review process and internal controls. However, the Terms and Conditions need to be updated to reflect current practices. Potential applicants who are not privacy specialists but have an interest in privacy issues are generally not aware of the Program and continued efforts to translate research into results are desired by all.
6. Is the Program producing its intended outputs in an efficient manner?
   
   The Program is seen to be efficient overall with a high level of project output for the funds invested. The Program has a relatively high overhead percentage due to the requirements and initiatives put in place to achieve the intended outcomes, contribute to the OPC's larger mandate, and to strengthen internal controls.
7. Is the Program achieving its intended outcomes in an economical manner?
   
   The Program is seen as economical, producing good value for money. The Program spends the vast majority of its contribution funding each year, with most of the funds being spent in the Ontario Region. While alternative delivery options exist that may be more economical, these options would likely reduce the benefits accrued to the OPC by managing the program.

7. Recommendations

The following recommendations are provided for the OPC's consideration in renewing the Contributions Program:

1. Continue knowledge translation efforts to translate research into results

As part of the new Program Plan and Communications Plan, the focus on knowledge translation should continue, along with the funding of research projects, to further promote project results and facilitate their use.

The Program should continue its current efforts to encourage all projects to have a knowledge translation plan as part of their proposal; and, applicants should be encouraged to identify more specific target audiences and specific dissemination methods to reach those target audiences (rather than just posting reports on a web site). A guide or toolkit to facilitate these efforts by funded recipients could be referenced or developed (e.g., Industry Canada's communications toolkit).

These efforts should also continue to include Program initiatives to share project results. Options for cost effective outreach (commensurate with the size and funding base of the Program) should be assessed as part of the communications planning process.

2. Encourage new innovative partnerships to extend the reach of the Program

While the Program has made substantial gains in funding a greater diversity of applicants, new mechanisms continue to be required to engage not-for-profit groups who are currently not aware of the Program and, through them, to achieve greater public awareness. One of the best mechanisms to achieve this may be to encourage universities and other groups who are currently aware of or engaged in the Program to develop new partnerships as part of their proposals. For example, universities could partner with public education groups or advocacy groups could partner with internet research groups. Multidisciplinary projects could be encouraged in the Application Guide and through proposal assessment criteria. Partner projects could be funded separately (e.g., $50,000 each), if warranted, or could be part of one proposal.

3. Balance Program requirements and efforts to maximize efficiency and economy

The Program has had much success in improving its internal processes and further achieving its intended outcomes; however, since it is a modestly-sized fund, it has a relatively high overhead percentage which has enabled it to accomplish these goals. If efficiency and economy are to be enhanced, the Program will have to reduce either its administrative processes or its knowledge translation efforts, or both. Being a 'small' program, the OPC Contributions Program cannot be expected to do as much as a larger fund and reasonable administrative and financial controls should be applied to the Program considering it low risk and low materiality.

4. Revise the Program Terms and Conditions to reflect current and desired requirements

At Program renewal, the Terms and Conditions should be updated to reflect the current and ongoing requirements of the Program, including:

• Ensuring the formulation of the Program objectives align in all key documents (Terms and Conditions, Performance Measurement Strategy, Applicants Guide).
• Specifying the frequency of reporting required from projects.
• Noting any new financial provisions (e.g., charges for 'other related costs', potential for advance payment, holdback requirements).
• Updating the Performance Measurement Strategy to be feasible to implement (e.g., easy and cost-effective) and useful for decision making (e.g., include the Departmental Performance Report measure already being collected; and, discuss with management other measures that should be tracked to facilitate decision-making such as 'mentions' of OPC Program / Funded Projects in Parliament, news releases, media, etc. to indicate relevance/use; or, stated impacts of projects one-year after completion).

Appendices

OPC Documents/References^{Footnote 16}:

• Annual Reports to Parliament
• Contributions Program
• Contributions Program Applicants Guide
• Contributions Program Communications Strategy. 2012-13.
• Contributions Program Draft Business Process Maps. 2014.
• Contributions Program Implementation Status of 2009 Summative Evaluation Recommendations. July 11, 2014.
• Contributions Program Performance Information Sheets. 2010-11, 2011-12, 2012-13, 2013-14.
• Contributions Program PIPEDA Performance Measurement Strategy. December 1, 2009
• Contributions Program Summary and Minutes of External Peer Review Meetings. March 8, 2013; March 19, 2014, Special Call October 10, 2013.
• Contributions Program Project Files (8 funded and 2 unfunded projects).
• Environmental Scans. 2010, 2011, 2012, 2013.
• FreeBalance Financial Management System Print outs. 2009-10, 2010-11, 2011-12, 2012-13, 2013-14.
• Mandate and Mission
• Pathways to Privacy Symposium - Privacy for Everyone. 2012.
• Pathways to Privacy Symposium - Helping Canadians Find Pathways to Privacy. 2014.
• Personal Information Protection and Electronic Documents Act (PIPEDA)
• Privacy Priorities. Reflections on the Office of the Privacy Commissioner of Canada's Strategic Priority Issues. 2013.
• Real Results. Protecting Privacy Rights through Innovative Research. April 2013.
• Survey of Canadians on Privacy-Related Issues. Phoenix Strategic Perspectives Inc. January 2013.
• Terms and Conditions: The Personal Information Protection and Electronic Documents Act Class Contribution Program. Jan. 5, 2010.
• The "Next Generation" OPC Contributions Program - a New Research Strategy Going Forward. December 9, 2011.
• Material on the OPC Website

Other Documents/References^{Footnote 17}:

• Alison Kerry, 1270665 Ontario Inc. Independent Summative Evaluation of the OPC Class Contributions Program PIPEDA. Final Report, Oct. 30, 2009.
• Genomics Canada. GPS: Where Genomics, Public Policy and Society Meet
• Industry Canada. Contributions Program for Non-profit Consumer and Voluntary Organizations
• Industry Canada. Final Evaluation of Industry Canada's Contributions Program for Nonprofit Consumer and Voluntary Organizations. 2009.
• Grant Space - Knowledge Base
• Science-Metrix. Impact Analysis of Research Supported by the OPC Contributions Program. 2012.
• Samson & Associates, Auditor's Report on OPC Contributions Program Projects. January 2014 and March 2014.
• Social Sciences and Humanities Research Council of Canada. Awards Search Engine

Management Response and Action Plan

September 26, 2014

Recommendation 1: Knowledge Translation
Evaluation Recommendation (text taken integrally from evaluation report)	Management Response (agree or disagree with rationale)	Action Plan (specific actions to implement recommendation)	Deadline for implementation	Responsibility (Branch)
1. Continue knowledge translation efforts to translate research into results. As part of the new Program Plan and Communications Plan, the focus on knowledge translation should continue, along with the funding of research projects, to further promote project results and facilitate their use. The Program should continue its current efforts to encourage all projects to have a knowledge translation plan as part of their proposal; and, applicants should be encouraged to identify more specific target audiences and specific dissemination methods to reach those target audiences (rather than just posting reports on a web site). A guide or toolkit to facilitate these efforts by funded recipients could be referenced or developed (e.g., Industry Canada's communications toolkit). These efforts should also continue to include Program initiatives to share project results. Options for cost effective outreach (commensurate with the size and funding base of the Program) should be assessed as part of the communications planning process.	Agree with rationale.	1. Continue to encourage knowledge translation activities in recipient proposals/projects: The Program will continue its efforts to encourage all applicants to have a knowledge translation plan as part of their proposals. Recipients will be required to implement their knowledge translation plans in order to receive full payment for their deliverables.	2014-15	Contributions Program.
		2. Update Program Communications Strategy:Update the communications strategy to reflect program focus on "knowledge translation" and support for P2P Symposium call for proposals. Undertake cost-effective outreach activities that promote dissemination and uptake of project results, commensurate with the size of the Program.	2014-15	Communications Branch & Contributions Program.
		3. Continue to issue special calls for proposals to hold the Pathways to Privacy Symposium as a major dissemination vehicle.Frequency to be determined according to available resources and response to events.	2015-16	Contributions Program.
		4. Require applicants to more clearly specify target audiences for their projects, and outline in more detail the intended strategy to reach them: The Applicant's Guide will be revised to emphasize the requirement for a more detailed dissemination strategy than is currently the case. E.g. applicants can be asked to identify specific target audiences and describe the different knowledge translation activities to be used to reach them.	2015-16	Contributions Program
		5. Create tools to help recipients communicate and disseminate the results of their projects: Continue existing tools (e.g. Real Results) or explore new tools to help recipients identify opportunities and approaches for disseminating knowledge created by their project. These tools could be made available on the OPC website under "Resources for Program Recipients", as appropriate.	2015-16	Communications Branch & Contributions Program

Recommendation 2: Developing Partnerships
Evaluation Recommendation (text taken integrally from evaluation report)	Management Response (agree or disagree with rationale)	Action Plan (specific actions to implement recommendation)	Deadline for implementation	Responsibility (Branch)
2. Encourage new innovative partnerships to extend the reach of the Program While the Program has made substantial gains in funding a greater diversity of applicants, new mechanisms continue to be required to engage not-for-profit groups who are currently not aware of the Program and, through them, to achieve greater public awareness. One of the best mechanisms to achieve this may be to encourage universities and other groups who are currently aware of or engaged in the Program to develop new partnerships as part of their proposals. For ex., universities could partner with public education groups or advocacy groups could partner with Internet research groups. Multidisciplinary projects could be encouraged in the Application Guide and through proposal assessment criteria. Partner projects could be funded separately (e.g., $50,000 each), if warranted, or could be part of one proposal.	Agree with rationale.	6. Revise Applicant's Guide to explicitly encourage universities to partner with the not-for-profit sector and develop a more multidisciplinary approach to funded research. (N.B. This is already done in the symposium call).	2016-17	Contributions Program.
		7. Revise proposal assessment criteriato credit applicants who have partnered or collaborated with non-profit organizations.	2016-17	Contributions Program.
		8. Initiate dialogue (e.g. letter writing) with not-for-profit groups, inviting them to submit funding proposals and promote the Program to their network of stakeholders.	2016-17	Contributions Program.

Recommendation 3: Efficiency and Economy
Evaluation Recommendation (text taken integrally from evaluation report)	Management Response (agree or disagree with rationale)	Action Plan (specific actions to implement recommendation)	Deadline for implementation	Responsibility (Branch)
3. Balance Program requirements and efforts to maximize efficiency and economy The Program has had much success in improving its internal processes and further achieving its intended outcomes; however, since it is a modestly-sized fund, it has a relatively high overhead percentage which has enabled it to accomplish these goals. If efficiency and economy are to be enhanced, the Program will have to reduce either its administrative processes or its knowledge translation efforts, or both. Being a 'small' program, the OPC Contributions Program cannot be expected to do as much as a larger fund and reasonable administrative and financial controls should be applied to the Program considering it low risk and low materiality.	Agree with rationale.	9. Revise procedures associated with the administration of the Program to ensure that reasonable administrative and financial controls are applied: Revise payments process and financial reporting requirements imposed on recipients, with a view to simplifying them while maintaining appropriate expenditures reporting. Simplify the peer review process with a view to initiating new efficiencies and increasing cost-effectiveness. Develop a project description form to standardize proposal presentations upon application. Simplify the project reporting requirements to alleviate administrative burden for applicants and OPC staff, while still ensuring necessary due diligence.	2016-17	Contributions Program & OPC Finance.
		10. Complete review and streamlining of Internal controls as per the Treasury Board Policy on Internal Controls.	2014-15	Contributions Program and OPC Corporate Services/Finance.

Recommendation 4: Program Terms and Conditions
Evaluation Recommendation (text taken integrally from evaluation report)	Management Response (agree or disagree with rationale)	Action Plan (specific actions to implement recommendation)	Deadline for implementation	Responsibility (Branch)
4. Revise the Program Terms and Conditions to reflect current and desired requirements At Program renewal, the Terms and Conditions should be updated to reflect the current and ongoing requirements of the Program, including: Ensuring the formulation of the Program objectives align in all key documents (Terms and Conditions, Performance Measurement Strategy, Applicants Guide). Specifying the frequency of reporting required from projects. Noting any new financial provisions (e.g., charges for 'other related costs', potential for advance payment, holdback requirements). Updating the Performance Measurement Strategy to be feasible to implement (e.g., easy and cost-effective) and useful for decision making (e.g., include the Departmental Performance Report measure already being collected; and, discuss with management other measures that should be tracked to facilitate decision-making such as 'mentions' of OPC Program / Funded Projects in Parliament, news releases, media, etc. to indicate relevance/use; or, stated impacts of projects one-year after completion).	Agree with rationale.	11. Revise the Terms and Conditions of the Contributions Program. In particular: Ensure that the language describing Program objectives alignwith language found in other key Program documents. Clarify frequency of reporting required for projects. Note/integrate new and improved provisions (e.g. provide for 20% advance).	2014-15	Contributions Program.
		12. Revise the Performance Measurement Strategyso that it is more feasible and cost-effective to implement, and more useful for decision-making.	2014-15	Contributions Program & Corporate Services.

Notes