The purpose of this is to see what type of scorecard, index, or ranking works well. Budgetpedia's Toronto Annual Budget Scorecard (TABS) hopes to evaluate the city of the Toronto's budget by comparing it with cities of a similar size. Other scorecards exist but few are specific to Toronto or to a municipal budget. This paper examines some that exist. They were found using a simple Internet search. The first 12 that were relevant (municipal oriented, reflecting policy items that would impact a budget) were selected. An attempt was made to mix scorecards created by non-profit agencies as well as for profit business and media organizations. This was to get a wide variety of examples.
After reading through 12 existing systems I noted their methodology, evaluated what I felt were positive attributes for each, and record criticisms I think were warranted. Next I tried to see if I could recognize any trends in the methodologies. Below I begin with the overall findings and then follow up with each of the 12 different scorecards.
The main theme that emerged from examining the 12 different types of ranking systems is that complexity should be avoided, and that background qualitative information needs to be provided at the expense of volume of data. It would also be beneficial to create original data and not recompile existing information.
1. A good background explaining what is meant by a metro's political vs. geographical boundary, and basic demographics
Toronto's Vital Signs Report and Toronto Region Board of Trade's Scorecard on Prosperity both give good background regarding what is meant by the city boundary that is being studied. The political boundary of a city may be different from the economic boundary. Toronto's border ends at Steels Ave., but many people consider Markham and Richmond Hill by what is meant by Toronto. When comparing cities it is important to state what is the exact area being compared and why. Also, including the basic demographics of a city helps to visualize other issues that are being discusses, such as total size of the budget.
2. Use a simple score (so that it can be adapted by others / sum up neatly any meta-data analysis)
A simple score makes a comparison between several cities easier to digest. Perhaps the best example of this would be the Globalization and World Cities Research Network (GaWC), this uses the designation of Alpha, Beta and Gamma to rank cities. These groups are easy to remember. Though GaWC has a complex networking analysis to rank each city the resultant rankings summarize the network analysis impressively. A simple ranking also means that the rankings can be easily used by others. The Mercer's Quality of Living Ranking, The Global Financial Centres Index 21, and the World Justice Project all have simple rankings for cities. However by merely ranking cities in order the studies make the mistake of projection too much accuracy for the research tool used. Is there a real difference between a city ranked 271 and 272? GaWC ranked ordered categories is ideal in that its categories are wide enough to include several cities. Cities like Sydney and Chicago are listed in the same category.
3. Create some original data (through surveys)
The Global Financial Centres Index 21 uses on online questionnaire as part of its methodology as does the World Justice Project, Rule of Law Index. This gives these two ranking systems data not available from other sources. Many of the other reports compile their indexes from mainly published governmental statistics. Original data can also be created through the building of a formula that transforms available data; if the formula has significant enough meaning in it and is reflective of a critical theory. The GaWC used sophisticated network analysis and social networking theory to do just this.
4. Having the report examined by an independent organization
The report issued by the Toronto Region Board of Trade stated that work was reviews by Chartered Professional Accountants. Using an independent body helps to counter any label of bias that may be attached to a report that is forthcoming from a group with vested interest in shaping legislation.
5. Comparing a set of diverse cities from around the world
More than half of the index examined compared cities internationally. Including a diverse set of cities works well in some context such as the Rule of Law Index and the Global Financial Centres Index. Such a comparison works because the reader is left trying to understand the root underlying causes of the differences between cities. Why some cities are more inclined to be financial centres than others. However in the case of the Safe Cities Index and the Quality of Living Ranking, the action of comparing the cities moves from trying to understand if there are underlying social phenomenon to a more competitive reflexivity; an unproductive 'is my city better' stance.
6. Providing qualitative background to results
Reports that added qualitative information to their rankings where more of interest. Providing a brief historical explanation of a city context helped to flush out and solidify any conclusions a report may have made.1. Comparing dissimilar cities
Sometimes cities are too dissimilar to compare. For example Visualizing Ontario’s Municipal Finance Data by the Institute on Municipal Finance & Governance compares property taxes between Toronto and some of its neighbouring municipalities. The analysis leaves out mention of the large structural difference between the jurisdictions.
2. Presentation is too complex
Some reports are too long, having pdf files approaching a hundred pages. This makes the report almost unreadable. Others rely too much on digital technology, the World Council on City Data only loads if Java is working properly. Other have graphs that are not easy to interpret because too much information is presented in them, such as The Safe Cities Index.
3. Too 'meta' of a data analysis
Overly meta analysis can take three forms. The first being a fire-hose of information as the World Council on City Data approach takes. Almost every statistic is presented at once to a user. The second problem is having data that is too esoteric. The 2016 Open Cities Index asks very specific question about a city's open data, such as 'does the data have an API' and is it available in bulk. This leaves the index with data cells that have no information, and the overall scores with little value. The third is the reuse of already gathered public information. Several indexes were repackaging already gathered statistics and adding little to no value.
4. No overall principle underlying the index
As it becomes easier to scrap data from various sources it becomes easier to compile and present the information back to a user. However the author of an index is an editor and as such their editorial decisions need to have some justification behind them. Not editing a stream of data is akin to not editing the minutes of a board meeting for a newspaper story; it will be tediously long and not have its point come across.
5. Opaque methodology
Many of the studies mentioned do publish their methodologies and some do publish their raw data. However a few did not publish specifics of their methodology taking away the weight that their endeavour may have had.
1)Toronto's Vital Signs
Toronto's Vital Signs Report 2016
Toronto Foundation
http://torontosvitalsigns.ca/wp-content/uploads/2016/10/WEB-OP-TorontosVitalSignsReport2016FINAL.pdf
About
The Toronto Foundation is a pool of 191 community foundations administering $400 million is assets.
The foundation engages in 'city building, mobilizing people and resources to increase the quality of life in Toronto'. The aim of the report is to 'inspire civic engagement and provide focus for public debate...'.
The Toronto's Vital Signs report is an annual report, 265 pages in length, and addresses ten issues such as economic health, health and wellness, safety, work, housing and transportation.
Methodology
The report is 'compiled from current statistics and studies'. Each section is researched and the findings are drawn from governmental statistics and academic sources. The presentation is not as formal as an academic paper but it's final form is similar in style.
Positive Attributes
The report is seemingly well written and sourced. This may be due to many contributors helping to research each section. The final report produces a good snapshot of the city, and being an annual report, multi-year comparison can be done.
The introductory chapter explaining the demographics and census areas / municipal boundaries helps to ground the reader as to the size and complicity of the city of Toronto.
There are many simple graphs and short facts that can easily be used for press releases to gain publicity.
Criticisms
The report draws from other studies, and does not contribute original data.
There are only ten areas of study. Each area that the report focused on, such as work or transportation, was a mini-paper within itself. There is no overall principle as to how to systematically study Toronto.
2) Visualizing Ontario’s Municipal Finance Data
University of Toronto's Munk School's Institute on Municipal Finance & Governance with the Institute without Boundaries at George Brown College
http://munkschool.utoronto.ca/imfg/research/data-visualizations/
About
University of Toronto's Munk School's Institute on Municipal Finance and Governance partnered with the Institute without Boundaries at George Brown College to analyze the Ontario government's Financial Information Returns (FIR).
Methodology
A series of visualizations of Ontario's cities based around specific research questions, using the Ontario government's Financial Information Returns (FIR) that each municipality files.
Positive Attributes
This data visualization project tries to analyze very specific questions regarding municipal budgets, though only two are so far are explored on the website.
Criticisms
The project is still in development with only limited data. The graph design seems to designed for expounding a thesis rather than for an in-depth analysis, less an academic style more of a newspaper style. Little context as to the meanings and interpretations presented.
3)World Council on City Data
World Council on City Data (WCCD, ISO 37120)
http://open.dataforcities.org/
170 Bloor St. W., Suite 1103, Toronto, Ontario, Canada's
About
Describes itself as a 'global leader on standardized metrics'. The organization has members from around the world, and an ISO standard.
Methodology
The World Council on City Data seems to amalgamate statistics regarding different cities from governmental sources. Though, it is difficult to find where they are sourcing their information. Dozens of statistical facts are presented about each city, from total population, to percentage of women employed in the city government workforce.
Positive Attributes
The website is very dynamic, impressive visually with a good user interface.
Criticisms
It seems to just 'fire hose' data and stats at you from a database that might be scrapping the Internet. Lots of factors/data but none of them properly explained as the what they are. Only works using Java.
4) Globalization and World Cities Research Network (GaWC)
The World According to GaWC 2016
http://www.lboro.ac.uk/gawc/world2016t.html
About
A project created by the geography department at Loughborough University, its purpose is to study the relations between cities. The primary output of the GaWC is a world cities ranking dividing cities in to Alpha, Beta, and Gamma cities. They also have a series of academic papers.
Methodology
GaWC uses a network analysis of multi-national firms that provide business services. (Specification of the World City Network, P.J.Taylor, GaWC Research Bulletin 23, http://www.lboro.ac.uk/gawc/rb/rb23.html)
Positive Attributes
It has wide recognition with a self-reported 35,000 hits across 150 countries.(What GaWC is About, http://www.lboro.ac.uk/gawc/group.html) The ranking is simple; Apha, Beta, Gamma and therefore easy to use with only a few categories.
Criticisms
The website is hard to navigate for details regarding the project, as a results its methodology seems opaque.
5) Mercer's Quality of Living Ranking
2017 Quality of Living Rankings
https://mobilityexchange.mercer.com/quality-of-living-rankings
About
Mercer is a large, worldwide, human resources consulting firm. Looks at quality of living. The quality of living report is part of their mandate, to encourage “employment mobility”. https://mobilityexchange.mercer.com/quality-of-living-reports.
Methodology
The quality of living ranking is based on 39 factors across 450 cities.
Positive Attributes
Simple ranking from 1 to 231 (the number of cities ranked). Very brief summation of findings.
Criticisms
The final report is not easily available, only a summation, likewise the methodology.
6) The Global Financial Centres Index 21
Z/Yen Group Limited
http://www.longfinance.net/images/gfci/GFCI21_05_04_17.pdf
March 2017
About
This index is the result of cooperation between several organization, including the Z/Yen Group (a commercial think-tank in London, About Z/Yen, http://www.zyen.com/who-we-do.html), the China Development Institute (based in Shenzhe), and the Financial Centre Futures (Qatar Financial Centre Authority. (The Global Financial Centres Index 17, http://www.finance-montreal.com/sites/default/files/publications/gfci17_23march2015.pdf)).
Methodology
The ranking is based on a data from other ranking indexes such as the World Economic Readiness Index and the Telecommunication Infrastructure Index. An online questionnaire is also used.
Positive Attributes
It is a simple ranking to understand. Cities from other than N. America and Europe are represented on the index. An online questionnaire means that the index is built from data other than just other indexes. Because they have 21 of these reports a longitudinal comparison is done on how regions such as Asia/Pacific and Western Europe change in ranking over time.
Criticisms
The report is business centred, because it is trying to assess what are the main financial capitals of the world.
7)Toronto Region Board of Trade , Toronto as a Global City: Scorecard on Prosperity – 2015
https://www.bot.com/Advocacy/Reports/ScorecardonProsperity.aspx
https://www.bot.com/portals/0/unsecure/advocacy/Scorecard_2015.pdf
2015
About
This is a Toronto Region Board of Trade report that is sponsored by the Chartered Professional Accountants of Ontario with research support through The Conference Board of Canada. The purpose of the report is to 'measures and assesses the economy and labour attractiveness of the Toronto Census Metropolitan Area (CMA) against 23 other metropolitan areas around the globe'.
Methodology
Since 2010 the Scorecard on Prosperity has measured the same (mostly economic) indicators of 24 metropolitan areas.
Positive Attributes
The report looks at 24 metro areas. First a definition of the metro area is given. Because it was done over 7 years, a year to year comparison can be made. To keep the report interesting and relevant every year a special focus section is added, in 2009 the focus was on regional governance and 2010 it was access to capital. The study uses a report card-style ranking of A,B,C,D. There is also an effort of look at factors outside of business such as diversity and the environment.
Criticisms
The report is somewhat cumbersome to read at 90 pages. It is unclear in their methodology where the numbers came from, they were also unclear on how cities were weighed for differences. Did not attempt show what made cities structural different from one another (comparing Toronto and Halifax seems odd).The report tends to repeat statements, a typical statement from the report would be “However, Toronto region’s initiatives must reach beyond North America. Ontario’s heavy reliance on the U.S. as a trade partner and the failure to expand to other fast-growing export markets and exploit them, particularly those located in Asia, helps explain why the 2000s were largely a lost decade for provincial trade.” It reads as a policy paper, with an aim to shape trade regulation in the Greater Toronto Area.
8)Competitive Alternatives, KPMG's guide to international business locations costs, 2016 edition.
https://www.competitivealternatives.com/reports/compalt2016_report_vol1_en.pdf
About
KPMG has been producing Competitive Alternatives since 1996. The report attempts to determine business costs in several cities.
Methodology
The study examines “26 individual cost factors that are likely to vary significantly by location”, such as labour costs and facility costs.
Positive Attributes
Each country gets its own scorecard to understand what is happening per country. Clear rankings and visuals. Only a handful of factors studied in depth (such as labour costs, facility costs, transportation costs), this keeps the report clearly focused.
Criticisms
The report is wells researched and written but it seems to be part of a KPMG package to sell its Global Location and Expansion Services.
9) World Justice Project, Rule of Law Index, 2016
https://worldjusticeproject.org/sites/default/files/documents/RoLI_Final-Digital_0.pdf
About
The organization describes itself as “ is an independent, multidisciplinary organization working to advance the rule of law worldwide.” (About Us, https://worldjusticeproject.org/about-us)
Methodology
The results are based on a survey of 110,000 households and 2,700 experts in 113 countries and jurisdictions.
Positive Attributes
The study is based on survey data, so doesn't depend on any other compiled data. The ranking is simple to understand, the report is succinct. Each country gets its own mini report.
Criticisms
Survey data can be biased if the sample per jurisdiction is small.
10) US City Open Data Census
http://us-city.census.okfn.org/
About
This is a report of how well municipalities are at publishing open data. It is a partnership between Open Knowledge International (a global non-profit, incorporated in England & Wales), The Sunlight Foundation (a nonpartisan, non-profit organization) and Code For America.
Methodology
An online spreadsheet listing the type of open-data sets from each municipality is crowd-sourced.
Positive Attributes
Because it is crowd-sourced anybody can participate in the report, but there are editors involved. Each city has one score (an aggregate scored based on how many data-sets available).
Criticisms
Too many subcategories looked for each city (such as crime, transit, crime...) too much work for people to do filling in each category, as a result little has been done. There are less than a hundred cities listed for the United States.
11)2016 Open Cities Index: Top 20 Results
Public Sector Digest, 2017
https://publicsectordigest.com/2016-open-cities-index-top-20-results
About
The Public Sector Digest is a monthly digital and quarterly print publication written to advance the managerial capacity of Canada’s public sector. The report is written in partnership with Canada’s Open Data Exchange (ODX). ODX is a public-private-academic partnership based in Waterloo specializing in financial budgeting. The Open Cities Index measures cities’ open data initiatives.
Methodology
Municipalities are scored against the availability of 32 identified data-sets. These data-sets are checked to see if;
1. Do not have access to this data-visualizations
2. The data exists
3. The data is available in any form online
4. Machine readable
5. Accessible/Permanent
6. Free
7. Available in bulk
9. Up to date
10. Automated
11. Linked to APIs
Positive Attributes
Uses 11 factors to assess a city, but gives each city one score. Limits cities to only Canada so results are comparable without too much differences between structure of cities. Quick to read and understand the information; all the cities can be shown, compared on one graph. You can compare your city with other cities, plus to a national average, and ranks each city.
Criticisms
The reader losses a sense of what is in the open data set for each city, what type of data each city has, by limiting the examination to only 38 sets. The Index is only based on 38 data-sets per city, crucial data-sets, but only 38 of them. Are there critical data sets that are missing from a city? There is little of background about each cities open data initiative from a qualitative prescriptive, what is the story of a city's success/challenge to opening data?
12. The Economist, Intelligence Unit. (2015). The Safe Cities Index 2015.
The Safe Cities Index: Assessing Urban Security In The Digital Age, A Report By The Economist Intelligence Unit
http://safecities.economist.com/wp-content/uploads/sites/5/2015/01/EIU_Safe_Cities_Index_2015_white_paper-1.pd
About
The Safe Cities Index 2015 by the Economist (A weekly magazine in London, England) is sponsored by NEC (a communications company in Japan). The reports attempts to find which is the safest city to live in, and to rank cities by how personally safe a person would be living in each city. The index looks at 50 cities.
Methodology
50 cities are examined, with 40 quantitative and qualitative indicators studied. Indicators are split into four categories: digital security; health security; infrastructure safety; and personal safety.
Positive Attributes.
Though a simple ranking, using only a few variables, it can give surprisingly insightful results such as “...wealth is not a byword for safety: every Middle Eastern city in the Index falls in the
highest income bracket, yet only one—Abu Dhabi makes it into the top half [of the index for being safe].”
Criticisms
At times it just feels like a list of cities with little insight about how they are good or bad. The study uses personally safety for a proxy for economic development “[s]afety is closely linked to wealth and economic development”.
The multivariate comparison of means graph does not work, it looks like a spiderweb and is hard to decipher meaningful statistics from it.
The index compares mega-cities with populations of 10 million with cities that have a population of under a million.
Some of the database is available through download, though you have to register.
Globalization and World Cities Research Network (GaWC). (2016). The World According to GaWC 2016. Retrieved from http://www.lboro.ac.uk/gawc/world2016t.html
KPMG. (2016) Competitive Alternatives, KPMG's guide to international business locations costs, 2016 edition. Retrieved from https://www.competitivealternatives.com/reports/compalt2016_report_vol1_en.pdf
Mercer. (2017). Quality of Living Rankin. Retrieved From https://mobilityexchange.mercer.com/quality-of-living-rankings
Open Knowledge. (2017).US City Open Data Census. Retrieved from http://us-city.census.okfn.org/
Public Sector Digest. (2016). 2016 Open Cities Index: Top 20 Results. Retrieved from https://publicsectordigest.com/2016-open-cities-index-top-20-results
The Economist, Intelligence Unit. (2015). The Safe Cities Index: Assessing Urban Security In The Digital Age. Retrieved from http://safecities.economist.com/wp-content/uploads/sites/5/2015/01/EIU_Safe_Cities_Index_2015_white_paper-1.pd
Toronto Foundation. (2016). Toronto's Vital Signs Report 2016. Retrieved from http://torontosvitalsigns.ca/wp-content/uploads/2016/10/WEB-OP-TorontosVitalSignsReport2016FINAL.pdf
Toronto Region Board of Trade. (2015). Toronto as a Global City: Scorecard on Prosperity – 2015. Retrieved from https://www.bot.com/portals/0/unsecure/advocacy/Scorecard_2015.pdf
University of Toronto's Munk School's Institute on Municipal Finance & Governance with the Institute without Boundaries at George Brown College. (2017). Visualizing Ontario’s Municipal Finance Data. Retrieved from http://munkschool.utoronto.ca/imfg/research/data-visualizations/
World Council on City Data. (2017). World Council on City Data, ISO 37120. Toronto, Canada. Retrieved from http://open.dataforcities.org
World Justice Project. (2016). Rule of Law Index, 2016. Retrieved from https://worldjusticeproject.org/sites/default/files/documents/RoLI_Final-Digital_0.pdf
Z/Yen Group Limited. (2017). The Global Financial Centres Index 21. Retrieved from http://www.longfinance.net/images/gfci/GFCI21_05_04_17.pdf