Review Of‭ ‬12‭ ‬Budget Scorecards,‭ ‬Indexes or‭ ‬Ranking Systems

Arthur Gron
September, 2017.

The purpose of this is to see what type of scorecard,‭ ‬index,‭ ‬or‭ ‬ranking works well.‭ ‬Budgetpedia's Toronto Annual‭ ‬Budget‭ ‬Scorecard‭ ‬(TABS‭) ‬hopes to‭ ‬evaluate the city of the Toronto's budget by comparing it with cities of a‭ ‬similar size.‭ ‬Other scorecards exist but few are specific to Toronto or to a municipal budget.‭ ‬This‭ ‬paper‭ ‬examines some that exist.‭ ‬They were found using‭ ‬a‭ ‬simple Internet search.‭ ‬The first‭ ‬12‭ ‬that were relevant‭ (‬municipal oriented,‭ ‬reflecting policy items that would impact a budget‭) ‬were selected.‭ ‬ An attempt was made to mix scorecards created by non-profit agencies as well as for profit business and media organizations.‭ ‬This was to get a wide variety of examples.‭

After reading through‭ ‬12‭ ‬existing systems I noted their methodology,‭ ‬evaluated what I felt were positive attributes for each,‭ ‬and record criticisms I think were warranted.‭ ‬Next I tried to see if I could recognize any trends in the methodologies.‭ ‬Below I begin with the overall findings and then follow up with each of the‭ ‬12‭ ‬different scorecards.‭

Examples of‭ ‬indexes,‭ ‬scorecards‭ ‬or ranking systems‭ ‬that worked well.‭

The main theme that emerged from‭ ‬examining the‭ ‬12‭ ‬different types of ranking systems is that complexity should be avoided,‭ ‬and that background qualitative information needs to be provided at the expense of volume of data.‭ ‬It would also be beneficial to create original data and not recompile existing information.‭

1.‭ ‬A good background‭ ‬explaining‭ ‬what is meant by‭ ‬a metro's‭ ‬political vs.‭ ‬geographical boundary,‭ ‬and‭ ‬basic‭ ‬demographics

Toronto's Vital Signs Report and Toronto Region Board of Trade's Scorecard on Prosperity both give good background regarding what is meant by the city boundary that is being studied.‭ ‬The political boundary of a city may be different from the economic boundary.‭ ‬Toronto's border ends at Steels Ave.,‭ ‬but many people consider Markham and Richmond Hill by what is meant by Toronto.‭ ‬When comparing cities‭ ‬it‭ ‬is important to state what is the exact area being compared and why.‭ ‬Also,‭ ‬including the basic demographics of a city helps to visualize other issues that are being discusses,‭ ‬such as total size of the budget.‭

2.‭ ‬Use a simple score‭ ‬(so that‭ ‬it can be adapted by others‭ ‬/‭ ‬sum up neatly‭ ‬any meta-data analysis‭)

A simple score makes a comparison between several cities easier to digest.‭ ‬Perhaps the best example of this would be the‭ ‬Globalization and World Cities Research Network‭ ‬(GaWC‭)‬,‭ ‬this uses the designation of Alpha,‭ ‬Beta and Gamma to rank cities.‭ ‬These groups are easy to remember.‭ ‬Though GaWC has a complex networking analysis to rank each city the resultant rankings summarize the network analysis impressively.‭ ‬A simple ranking also means that the rankings can be easily used by others.‭ ‬The‭ ‬Mercer's‭ ‬Quality of Living‭ ‬Ranking,‭ ‬The‭ ‬Global Financial Centres‭ ‬Index‭ ‬21,‭ ‬and the‭ ‬World Justice Project‭ ‬all have simple rankings for cities.‭ ‬However by merely ranking cities in order the studies make the mistake of projection too much accuracy for the research tool used.‭ ‬Is there a real difference between a city ranked‭ ‬271‭ ‬and‭ ‬272‭? ‬GaWC ranked ordered categories is ideal in that its categories are wide enough to include several cities.‭ ‬ Cities like Sydney and Chicago are listed in the same category.‭

3.‭ ‬Create some‭ ‬original data‭ ‬(through surveys‭)

The Global Financial Centres Index‭ ‬21‭ ‬uses on online questionnaire as part of its methodology as does the World Justice Project,‭ ‬Rule of Law Index.‭ ‬This gives these two ranking systems data not‭ ‬available from other sources.‭ ‬Many of the other reports compile their indexes from mainly published governmental statistics.‭ ‬Original data can also be created through the building of a formula that transforms available data‭; ‬if the formula has significant enough meaning in it and is reflective of a critical theory.‭ ‬The GaWC used sophisticated network analysis and social networking theory to do just this.‭

4.‭ ‬Having the report examined by an independent organization

The report issued by the Toronto Region Board of Trade stated that work was reviews by Chartered Professional Accountants.‭ ‬Using an independent body helps to counter any label of bias that may be attached to a report that is forthcoming from a group with vested interest in shaping legislation.‭

5.‭ ‬Comparing‭ ‬a set of‭ ‬diverse cities from around the world

More than half of the index examined compared cities internationally.‭ ‬Including a diverse set of cities works well in some context such as the Rule of Law Index and the Global Financial Centres Index.‭ ‬Such a comparison works because the reader is left trying to understand the root underlying causes of the differences between cities.‭ ‬Why some cities are more inclined to be financial centres than others.‭ ‬However in the case of the Safe Cities Index and the Quality of Living Ranking,‭ ‬the action of comparing the cities moves from trying to understand if there are underlying social phenomenon to a more competitive reflexivity‭; ‬an unproductive‭ '‬is my city better‭' ‬stance.‭

6.‭ ‬Providing qualitative background to results

Reports that added qualitative information to their rankings where more of interest.‭ ‬Providing a brief historical explanation of a city context helped to flush out and solidify any conclusions a report may have made.‭

Pitfalls to avoid.‭

1.‭ ‬Comparing‭ ‬dissimilar cities

Sometimes cities are too dissimilar to compare.‭ ‬For example‭ ‬Visualizing Ontario’s Municipal Finance Data by the Institute on Municipal Finance‭ & ‬Governance‭ ‬compares property taxes between Toronto and some of its neighbouring municipalities.‭ ‬The analysis leaves out mention of the large structural difference between the jurisdictions.

2.‭ ‬Presentation is too complex

Some reports are too long,‭ ‬having pdf files approaching a hundred pages.‭ ‬This makes the report almost unreadable.‭ ‬Others rely too much on‭ ‬digital technology,‭ ‬the World Council on City Data only loads if Java is working properly.‭ ‬Other have graphs that are not easy to interpret because too much information is presented in them,‭ ‬such as‭ ‬The‭ ‬Safe Cities Index.‭

3.‭ ‬Too‭ ‬'meta‭'‬ of a data analysis

Overly meta analysis can take three forms.‭ ‬The first being a fire-hose of information as‭ ‬the World Council on City Data approach takes.‭ ‬Almost every statistic is presented at once to a user.‭ ‬The‭ ‬second problem is having data that is too esoteric.‭ ‬The‭ ‬2016‭ ‬Open Cities Index‭ ‬asks very specific question about a city's open data,‭ ‬such as‭ '‬does the data have an API‭' ‬and is it available in bulk.‭ ‬This leaves the index with data cells that have no information,‭ ‬and the overall scores with little value.‭ ‬The third is the reuse of already gathered public information.‭ ‬Several indexes were repackaging‭ ‬already gathered statistics and adding little to no value.

4.‭ ‬No‭ ‬overall principle‭ ‬underlying the index

As it becomes easier to scrap data from various sources it becomes easier to compile and present the information back to a user.‭ ‬However the author of an index is an editor and as such their editorial decisions need to have some justification behind them.‭ ‬Not editing a stream of data is akin to not editing the minutes of a board meeting for a newspaper story‭; ‬it will be tediously long and not have its point come across.‭

5.‭ ‬Opaque methodology

Many of the studies mentioned do publish their methodologies and some‭ ‬do publish their‭ ‬raw data.‭ ‬However a few did not publish specifics of their methodology taking away the weight that their endeavour may have had.‭

Evaluated Ranking Systems

1‭)‬Toronto's Vital Signs
Toronto's Vital Signs Report‭ ‬2016‭
Toronto Foundation

The Toronto Foundation is a pool of‭ ‬191‭ ‬community foundations‭ ‬administering‭ ‬$400‭ ‬million is‭ ‬assets.‭ The foundation engages in‭ '‬city building,‭ ‬mobilizing people and resources to increase the quality of life in Toronto‭'‬.‭ ‬The aim of the report is to‭ '‬inspire civic‭ ‬engagement and provide focus for public debate...‭'‬.‭ The Toronto's Vital Signs report is an annual report,‭ ‬265‭ ‬pages in length,‭ ‬and addresses ten issues such as economic health,‭ ‬health and wellness,‭ ‬safety,‭ ‬work,‭ ‬housing and transportation.‭

The report is‭ '‬compiled from current‭ ‬statistics and studies‭'‬.‭ ‬Each section is researched and the findings are drawn from governmental statistics and academic sources.‭ ‬The presentation is not as formal as an academic paper but it's final form is‭ ‬similar in style.‭

Positive Attributes
The report is seemingly well written and sourced.‭ ‬This may be due to many‭ ‬contributors helping to research each section.‭ ‬The final report produces a good snapshot of the city,‭ ‬and being an annual report,‭ ‬multi-year‭ ‬comparison can be done. The introductory chapter‭ ‬explaining the‭ ‬demographics and census areas‭ ‬/‭ ‬municipal‭ ‬boundaries helps to ground the reader as to the size and‭ ‬complicity of the city of Toronto.‭ There are many simple graphs and short facts that can easily be used for press releases to gain publicity.‭

The report draws from other studies,‭ ‬and does not contribute original data.‭ There are only ten areas of study.‭ ‬Each area that the report focused on,‭ ‬such as work or transportation,‭ ‬was a mini-paper within itself.‭ ‬There is no overall principle as to how to systematically study Toronto.‭

2‭) ‬Visualizing Ontario’s Municipal Finance Data
University of Toronto's Munk School's Institute on Municipal Finance‭ & ‬Governance with the Institute without Boundaries at George Brown College‭

University of Toronto's Munk School's Institute on Municipal Finance and Governance partnered with the‭ ‬Institute without‭ ‬Boundaries at George Brown College to‭ ‬analyze the Ontario government's Financial Information Returns‭ (‬FIR‭)‬.‭

A series of visualizations of Ontario's cities based around specific research questions,‭ ‬using the Ontario government's Financial Information Returns‭ (‬FIR‭) ‬that each municipality files.‭

Positive Attributes
This data visualization project tries to‭ ‬analyze very specific questions regarding municipal‭ ‬budgets,‭ ‬though only two are so far‭ ‬are‭ ‬explored on the website.‭

The project is still in development with only limited data.‭ ‬The graph design seems to designed for expounding a thesis rather than for an‭ ‬in-depth‭ ‬analysis,‭ ‬less an academic style more of a newspaper style.‭ ‬Little context as to the meanings and‭ ‬interpretations presented.‭

3‭)‬World Council on City Data‭
World Council on City Data‭ (‬WCCD,‭ ‬ISO‭ ‬37120‭)
170‭ ‬Bloor St.‭ ‬W.,‭ ‬Suite‭ ‬1103,‭ ‬Toronto,‭ ‬Ontario,‭ ‬Canada's

Describes itself as a‭ '‬global leader on standardized metrics‭'‬.‭ ‬The‭ ‬organization has members from around the world,‭ ‬and an ISO standard.‭

The World Council on City Data seems to‭ ‬amalgamate statistics regarding different cities from governmental sources.‭ ‬Though,‭ ‬it is difficult to find where they are sourcing their information.‭ ‬Dozens of statistical facts are presented about each city,‭ ‬from total population,‭ ‬to percentage of women employed in the city government workforce.‭

Positive Attributes
The website is very‭ ‬dynamic,‭ ‬impressive‭ ‬visually‭ ‬with a good user interface.‭

It seems to just‭ '‬fire hose‭' ‬data and stats at you from a database that might be scrapping the‭ ‬Internet.‭ ‬Lots of factors/data but none of them‭ ‬properly explained as the what they are.‭ ‬Only works using‭ ‬Java.‭

4‭) ‬Globalization and World Cities Research Network‭ ‬(GaWC‭)
The World According to GaWC‭ ‬2016

A project created by the geography‭ ‬department at Loughborough‭ ‬University,‭ ‬its purpose is to study the relations between cities.‭ ‬The primary output of the GaWC is a world cities ranking dividing cities in to Alpha,‭ ‬Beta,‭ ‬and Gamma cities.‭ ‬They also have a series of academic papers.‭

GaWC uses a‭ ‬network‭ ‬analysis of multi-national firms that provide business services.‭ (‬Specification of the World City Network,‭ ‬P.J.Taylor,‭ ‬GaWC Research Bulletin‭ ‬23,‭ ‬‭)

Positive‭ ‬Attributes‭
It has wide‭ ‬recognition with a self-reported‭ ‬35,000‭ ‬hits across‭ ‬150‭ ‬countries.‭(‬What GaWC is About,‭ ‬‭)‬ The ranking is simple‭; ‬Apha,‭ ‬Beta,‭ ‬Gamma and therefore easy to use‭ ‬with only a few categories.‭

The‭ ‬website is hard to navigate for details regarding the project,‭ ‬as a results its methodology seems opaque.‭

5‭) ‬Mercer's‭ ‬Quality of Living‭ ‬Ranking
2017‭ ‬Quality of Living Rankings‭

Mercer is a large,‭ ‬worldwide,‭ ‬human resources consulting firm.‭ ‬Looks at quality of living.‭ ‬The quality of living report is part of their mandate,‭ ‬to encourage‭ “‬employment mobility‭”‬.‭ ‬

The quality of living ranking is based on‭ ‬39‭ ‬factors across‭ ‬450‭ ‬cities.‭

Positive‭ ‬Attributes‭
Simple ranking‭ ‬from‭ ‬1‭ ‬to‭ ‬231‭ (‬the number of cities ranked‭)‬.‭ ‬Very brief summation of findings.‭

The final report is not easily available,‭ ‬only a summation,‭ ‬likewise the methodology.‭

6‭) ‬The‭ ‬Global Financial Centres‭ ‬Index‭ ‬21
Z/Yen Group Limited‭_‬05‭_‬04‭_‬17.pdf
March‭ ‬2017‭

This index is the result of‭ ‬cooperation between several organization,‭ ‬including the Z/Yen Group‭ (‬a commercial think-tank in London,‭ ‬About Z/Yen,‭ ‬‭)‬,‭ ‬the China Development Institute‭ (‬based in Shenzhe‭)‬,‭ ‬and the Financial Centre Futures‭ (‬Qatar Financial Centre Authority.‭ (‬The Global Financial Centres Index‭ ‬17,‭ ‬‭_‬23march2015.pdf‭))‬.‭

The ranking is based on a data from other ranking‭ ‬indexes such as the World Economic Readiness Index and the‭ ‬Telecommunication Infrastructure Index.‭ ‬An online questionnaire is also used.‭

Positive‭ ‬Attributes‭
It is a simple ranking to understand.‭ ‬Cities from other than N.‭ ‬America and Europe are represented on the index.‭ ‬An online questionnaire means that the index is built from data other than just other indexes.‭ ‬Because they have‭ ‬21‭ ‬of these reports a‭ ‬longitudinal‭ ‬comparison is done on how regions such as Asia/Pacific and Western Europe change in ranking over time.‭

The report is business centred,‭ ‬because it is trying to assess what are the main financial capitals of the world.‭

7‭)‬Toronto‭ ‬Region Board of Trade‭ ‬,‭ ‬Toronto as a Global City:‭ ‬Scorecard on Prosperity‭ – ‬2015‭‭_‬2015.pdf

This is a Toronto Region Board of Trade report that is sponsored by the Chartered Professional Accountants of Ontario with research support through The Conference Board of Canada.‭ ‬The purpose of the report is to‭ '‬measures‭ ‬and assesses the economy and labour attractiveness of the Toronto Census Metropolitan Area‭ (‬CMA‭) ‬against‭ ‬23‭ ‬other metropolitan areas around the globe‭'‬.

Since‭ ‬2010‭ ‬the Scorecard on Prosperity has measured the same‭ (‬mostly economic‭) ‬indicators of‭ ‬24‭ ‬metropolitan areas.‭

Positive Attributes
The report looks at‭ ‬24‭ ‬metro areas.‭ ‬First a definition of the metro area is given.‭ ‬Because it was done over‭ ‬7‭ ‬years,‭ ‬a year to year comparison can be made.‭ ‬To keep the report interesting and relevant every year a special focus section is added,‭ ‬in‭ ‬2009‭ ‬the focus was on‭ ‬regional governance and‭ ‬2010‭ ‬it‭ ‬was‭ ‬access to capital.‭ ‬The‭ ‬study uses a report card-style ranking of A,B,C,D.‭ ‬There is also an effort of look at factors outside of business such as diversity and the‭ ‬environment.‭

The report is somewhat cumbersome to read at‭ ‬90‭ ‬pages.‭ ‬It is unclear in their methodology where the numbers came from,‭ ‬they were also unclear on how cities were weighed for differences.‭ ‬Did not attempt show what made cities structural different from one another‭ (‬comparing Toronto and Halifax seems odd‭)‬.The report tends to repeat statements,‭ ‬a typical statement from the report would be‭ “‬However,‭ ‬Toronto region’s initiatives must reach beyond‭ ‬North America.‭ ‬Ontario’s heavy reliance on the U.S.‭ ‬as a‭ ‬trade partner and the failure to expand to other fast-growing‭ ‬export markets and exploit them,‭ ‬particularly those located‭ ‬in Asia,‭ ‬helps explain why the‭ ‬2000s were largely a lost‭ ‬decade for provincial trade.‭” ‬It reads as a policy paper,‭ ‬with an aim to shape trade regulation in the Greater Toronto Area.‭

8‭)‬Competitive Alternatives,‭ ‬KPMG's guide to international business locations costs,‭ ‬2016‭ ‬edition.‭‭_‬report_vol1‭_‬en.pdf

KPMG has been producing Competitive Alternatives since‭ ‬1996.‭ ‬The report attempts to determine business costs in several cities.

The study examines‭ “‬26‭ ‬individual cost factors that are likely to vary significantly‭ ‬by location‭”‬,‭ ‬such as‭ ‬labour costs and facility costs.

Positive‭ ‬Attributes‭
Each country gets its own scorecard to understand what is happening per country.‭ ‬Clear rankings and visuals.‭ ‬Only a‭ ‬handful of factors studied‭ ‬in depth‭ (‬such as labour costs,‭ ‬facility costs,‭ ‬transportation costs‭)‬,‭ ‬this keeps the report clearly focused.‭

The report is wells researched and written but it seems to be part of a KPMG package to sell its Global Location and Expansion Services.‭

9‭) ‬World Justice Project,‭ ‬Rule of Law Index,‭ ‬2016‭_‬0.pdf

The organization describes itself as‭ “ ‬is an independent,‭ ‬multidisciplinary organization working to advance the rule of law worldwide.‭” (‬About Us,‭ ‬‭)

The results are based on a survey of‭ ‬110,000‭ ‬households and‭ ‬2,700‭ ‬experts in‭ ‬113‭ ‬countries and jurisdictions.‭

Positive Attributes‭
The study is based on survey data,‭ ‬so doesn't depend on any other compiled data.‭ ‬The ranking is simple to understand,‭ ‬the report is‭ ‬succinct.‭ ‬Each country gets its own mini report.‭

Survey data can be biased if the sample per jurisdiction is small.‭

10‭)‬ US City Open Data Census

This is a report of how well municipalities are at publishing open data.‭ ‬It is a partnership between Open Knowledge International‭ (‬a global non-profit,‭ ‬incorporated in England‭ & ‬Wales‭)‬,‭ ‬The Sunlight Foundation‭ (‬a nonpartisan,‭ ‬non-profit organization‭) ‬and Code For America.‭

An online spreadsheet listing the type of open-data sets from each municipality is‭ ‬crowd-sourced.‭

Positive Attributes‭
Because it is‭ ‬crowd-sourced‭ ‬anybody can participate in the report,‭ ‬but there are editors involved.‭ ‬Each city has one score‭ (‬an aggregate scored based on how many‭ ‬data-sets‭ ‬available‭)‬.‭

Too many subcategories looked for each city‭ (‬such as crime,‭ ‬transit,‭ ‬crime...‭) ‬too much work for people to do filling in each category,‭ ‬as a result little has been done.‭ ‬ There are‭ ‬less than a hundred cities listed for the United States.‭

11‭)‬2016‭ ‬Open Cities Index:‭ ‬Top‭ ‬20‭ ‬Results‭
Public Sector Digest,‭ ‬2017‭

The Public Sector Digest is a monthly digital and quarterly print publication written to advance the managerial capacity of Canada’s public sector.‭ ‬The report is written in‭ ‬partnership with Canada’s‭ ‬Open Data Exchange‭ (‬ODX‭)‬.‭ ‬ODX is a public-private-academic partnership based in Waterloo‭ ‬specializing in financial budgeting.‭ ‬The Open Cities Index measures cities‭’ ‬open data initiatives.‭

Municipalities are scored‭ ‬ against the availability of‭ ‬32‭ ‬identified‭ ‬data-sets.‭ ‬These data-sets are checked to see if‭;
1.‭ ‬Do not have access to this data-visualizations
2.‭ ‬The data exists
3.‭ ‬The data is‭ ‬available in any form online
4.‭ ‬Machine readable
5.‭ ‬Accessible/Permanent
6.‭ ‬Free
7.‭ ‬Available in bulk
9.‭ ‬Up to date
10.‭ ‬Automated
11.‭ ‬Linked to APIs

Positive Attributes‭
Uses‭ ‬11‭ ‬factors to assess a city,‭ ‬but gives each city one score.‭ ‬Limits cities to only Canada so results are comparable without too much differences between structure of cities.‭ ‬Quick to read and understand the information‭; ‬all the cities can be shown,‭ ‬compared on one graph.‭ ‬You can compare your city with other cities,‭ ‬plus to a national average,‭ ‬and‭ ‬ranks each city.‭

The reader losses a sense of what is in the open data set for each city,‭ ‬what type of data‭ ‬each city has,‭ ‬by limiting the examination to only‭ ‬38‭ ‬sets.‭ ‬The Index is only based on‭ ‬38‭ ‬data-sets per city,‭ ‬crucial‭ ‬data-sets,‭ ‬but only‭ ‬38‭ ‬of them.‭ ‬Are there critical data sets that are missing from a city‭? ‬There is little of background about each cities open data initiative from a qualitative prescriptive,‭ ‬what is the story of a city's success/challenge to opening data‭?

12.‭ ‬The Economist,‭ ‬Intelligence Unit.‭ (‬2015‭)‬.‭ ‬The‭ ‬Safe Cities Index‭ ‬2015.
The Safe Cities Index:‭ ‬Assessing Urban Security In The Digital Age,‭ ‬A Report By The Economist Intelligence Unit‭_‬2015‭_‬white_paper-1.pd

The Safe Cities Index‭ ‬2015‭ ‬by the‭ ‬Economist‭ ‬(A weekly magazine in London,‭ ‬England‭) ‬is‭ ‬sponsored by NEC‭ ‬(a communications company in Japan‭)‬.‭ ‬The reports attempts to find which is the safest city to live in,‭ ‬and to rank cities by how personally safe a person would be living in each city.‭ ‬The index looks at‭ ‬50‭ ‬cities.‭

50‭ ‬cities are examined,‭ ‬with‭ ‬40‭ ‬quantitative and qualitative indicators‭ ‬studied.‭ ‬ Indicators are split‭ ‬into‭ ‬four categories:‭ ‬digital security‭; ‬health security‭; ‬infrastructure safety‭; ‬and personal safety.‭

Positive Attributes.‭
Though a simple ranking,‭ ‬using only a few variables,‭ ‬it can give‭ ‬surprisingly insightful results such as‭ “‬...wealth is not a byword for safety:‭ ‬every Middle Eastern city in the Index falls in the highest income bracket,‭ ‬yet only one—Abu Dhabi makes it into the top half‭ ‬[of the index for being safe‭]‬.‭”

At times‭ ‬it just feels like a list of cities with little insight about how‭ ‬they are‭ ‬good or bad.‭ ‬The study uses personally safety for a proxy for economic development‭ ‬“[s]afety is closely linked to wealth and economic development‭”‬. ‭ ‬The multivariate comparison of means graph does not work,‭ ‬it looks like a spiderweb and is hard to decipher meaningful statistics from it.‭ The index‭ ‬compares mega-cities with populations of‭ ‬10‭ ‬million with cities that have a population of under a million.‭ Some of the database is‭ ‬available through download,‭ ‬though you have to register.‭


Globalization and World Cities Research Network‭ ‬(GaWC‭)‬.‭ ‬(2016‭)‬.‭ ‬The World According to GaWC‭ ‬2016.‭ ‬Retrieved from‭ ‬

KPMG.‭ (‬2016‭) ‬Competitive Alternatives,‭ ‬KPMG's guide to international business locations costs,‭ ‬2016‭ ‬edition.‭ ‬Retrieved from‭ ‬‭_‬report_vol1‭_‬en.pdf

Mercer.‭ ‬(2017‭)‬.‭ ‬Quality of Living‭ ‬Rankin.‭ ‬Retrieved From‭ ‬

Open Knowledge.‭ ‬(2017‭)‬.US City Open Data Census.‭ ‬Retrieved from‭ ‬

Public Sector Digest.‭ (‬2016‭)‬.‭ ‬2016‭ ‬Open Cities Index:‭ ‬Top‭ ‬20‭ ‬Results.‭ ‬Retrieved from‭ ‬

The Economist,‭ ‬Intelligence Unit.‭ (‬2015‭)‬.‭ ‬The Safe Cities Index:‭ ‬Assessing Urban Security In The Digital Age.‭ ‬Retrieved from‭ ‬‭_‬2015‭_‬white_paper-1.pd

Toronto‭ ‬Foundation.‭ ‬(2016‭)‬.‭ ‬Toronto's Vital Signs Report‭ ‬2016.‭ ‬Retrieved from‭ ‬

Toronto‭ ‬Region Board of Trade.‭ ‬(2015‭)‬.‭ ‬ Toronto as a Global City:‭ ‬Scorecard on Prosperity‭ – ‬2015.‭ ‬Retrieved from‭ ‬‭_‬2015.pdf

University of Toronto's Munk School's Institute on Municipal Finance‭ & ‬Governance with the Institute without Boundaries at George Brown College.‭ (‬2017‭)‬.‭ ‬Visualizing Ontario’s Municipal Finance Data.‭ ‬Retrieved from‭ ‬

World Council on City Data.‭ ‬(2017‭)‬.‭ ‬World Council on City Data,‭ ‬ISO‭ ‬37120.‭ ‬Toronto,‭ ‬Canada.‭ ‬Retrieved from‭ ‬

World Justice Project.‭ (‬2016‭)‬.‭ ‬Rule of Law Index,‭ ‬2016.‭ ‬Retrieved from‭ ‬‭_‬0.pdf

Z/Yen Group Limited.‭ (‬2017‭)‬.‭ ‬The‭ ‬Global Financial Centres‭ ‬Index‭ ‬21.‭ ‬Retrieved from‭ ‬‭_‬05‭_‬04‭_‬17.pdf