Fifty Years of Geospatial Data Analysis and Technology

Mariya Shcheglovitova, Social Science Analyst, Program Monitoring and Research Division, PD&R (2022-present)

Alex Din, Social Science Analyst, Program Monitoring and Research Division, PD&R (2019-present)

Many aspects of HUD's work include a locational component. Location data is central to HUD's mission to create strong, sustainable, inclusive communities and quality affordable homes for all.

As analysts in HUD's Office of Policy Development and Research (PD&R), we both work with spatial data and analysis tools, but our institutional knowledge is relatively recent. To explore the history of HUD's use of spatial data and tools — often called Geographic Information Systems/Sciences (GIS) — we interviewed five people who offered firsthand accounts of HUD's implementation of GIS technology. We supplemented these interviews with information from HUD publications dating back to 1972 as well as one written account from a longtime HUD employee summarizing their GIS experience.

History of GIS

1. Data: “Location is baked into the DNA of HUD”

Early location-based programs at HUD focused on helping tenants relocate to higher-opportunity neighborhoods with low poverty rates. PD&R has been critical in documenting the impacts of location-based programs by linking HUD's locational data with demographic and economic data from the U.S. Census Bureau (Census). A 1972 HUD Challenge article on housing allowances highlighted these linkages in a map overlaying Housing Allowance program participants with demographic data from the Census to show key neighborhoods to receive targeted investments in Kansas City, Missouri (figure 1). This map and the underlying program represent many of the historical and ongoing applications and challenges of GIS in PD&R and HUD in general. The Final Report on the Housing Allowance Demand Experiment in 1980 outlined some of these challenges, stating that:

[T]urning raw data into a useable analytic data base requires that data from different sources be linked together, transformed into variables, cleaned of unresolved inconsistencies and anomalies, and placed within a well-documented system that provides easy-access at reasonable cost.

The challenges of developing user-friendly databases that integrate locational data continued to define the development of GIS at HUD.

Figure 1. Map of HUD Housing Allowances program participants included in a 1972 HUD Challenge article, “Housing Allowances: A New Way to House the Poor,” by Malcolm E. Peabody, Jr.

2. The Geocoding Service Center

HUD began appending geographic information, or "geocodes," to HUD tenant data in the mid-1990s using its Geocode Service Center (GSC). The GSC allowed users to attach geographic identifiers — from standardized addresses to census tract IDs — to nearly every data element that passed through HUD. This advancement in locational intelligence generated data that HUD offices could leverage to understand spatial patterns.

When the GSC began, HUD was experiencing a proliferation of computer technology and an overall increase in large-scale data collection efforts. However, because computers were still expensive, HUD conducted its computing work at service bureaus — localized facilities that provided specific computing services. The GSC was initially a service bureau for aggregating and processing address information. HUD would mail the service bureau "a tape of addresses and 2 weeks later get it back with an address validation run." Eventually, to reduce costs and strengthen quality control, HUD relocated the GSC in house. The first interactive query response form on HUD's intranet was the submission form for the geocoder. Today, the GSC is readily available to all HUD employees. This agency-wide user base directs large volumes of traffic to the GSC, which processes nearly 500 million addresses annually.

3. The Crayon Brigade to Enterprise GIS

Early mapping applications: The Crayon Brigade

Like the Housing Allowance Demand Experiment in the 1970s, HUD's mapping applications during the 1990s drew on insights into local demographic and economic trends from Census data to inform housing and community investment policies. Kurt Usowski, deputy assistant secretary for economic affairs at PD&R, recalls an effort spearheaded by HUD Secretary Jack Kemp to create Enterprise Zones (EZs) — areas where businesses could earn federal tax exemptions for locating in census tracts meeting specific economic criteria. At the time, PD&R could not produce maps directly with computers. PD&R's master FORTRAN programmer, the late Ray Kahn, wrote a program that identified census tracts matching EZ criteria and produced a stack of printouts listing eligible tract IDs. "Nearly the entire staff of the Office of Economic Affairs in PD&R were handed copies of printed census tract maps, the associated EZ printout for their map, a pack of highlighters, and instructions to 'color by numbers.' The 'Crayon Brigade' produced dozens of maps depicting the proposed EZs in major cities."

Early mapping applications: AtlasGIS and C2020 

While the Crayon Brigade was producing hand-drawn maps, Usowski and fellow Office of Economic Affairs economist Alan Fox were determining how to make EZ maps using the program AtlasGIS. Usowski recalls that producing maps for individual cities was a straightforward process, but the computer would often crash when creating maps for areas with a very high number of eligible census tracts. Usowski arrived at a solution through a hardware fix he found in a computer manual. Fox and Usowski reconfigured the PC, allowing AtlasGIS to produce even the most complex EZ maps in seconds. Although Congress ultimately did not pass HUD's effort to create EZs, PD&R gained experience with computer mapping software that expanded its GIS skills.

As personal computers became more available and accessible, all HUD staff eventually were able to use computer software to create maps. In the late 1990s, HUD distributed a graphical GIS program called Community 2020 (C2020) to Field Offices and HUD grantees. Developed by Caliper (now known as Maptitude), C2020 was built specifically for the Office of Community Planning and Development (CPD) and allowed users to create maps using preloaded data from HUD and the U.S. Census Bureau. C2020 also was used outside HUD, including copies sent to all offices of the members of Congress.

According to Robert Renner, a GIS expert at HUD from 2001 to 2019, C2020 was advanced when it was developed but quickly became a "dinosaur because it was disconnected." Mark Mitchell, currently the community planning and development director for HUD’s Portland Field Office, joined HUD as a consolidated plan specialist in 1999 to train HUD staff and grantees in C2020. Before this position, Mark worked for a recipient of a community development block grant, and he recalled how, during this era, he had to "go to the library to look up Census data, [photocopy] pages, and then go put it into a spreadsheet." C2020 significantly improved HUD’s ability to link and map HUD and Census data because it allowed users to load these data from CDs that were mailed to them. However, the rise of the internet quickly rendered this practice obsolete.

The move to web-based GIS

By the early 2000s, it was clear that interactive online web mapping was the future of GIS. HUD's initial attempt at an online GIS used Esri's Arc Internet Map Server (ArcIMS). As part of this effort, HUD staff created a centralized spatial-enabled database that gathered HUD business and Census data in one location. This initial geodatabase allowed all HUD offices to use GIS and derive insights from HUD's locational data.

The modern Enterprise Geographic Information System

The modernization of GIS at HUD occurred during the Obama administration, when Congress passed two laws funding information technology initiatives to facilitate data-driven decision-making. The Homeless Emergency Assistance and Rapid Transition to Housing (HEARTH) Act of 2009 provided funding to CPD for multiple spatial tools, including an economic planning suite, which marked the beginning of CPD maps, and the Integrated Disbursement and Information System (IDIS), a database for HUD grantees. Renner recalled how CPD maps and IDIS finally brought the functionality of C2020 online, allowing grantees to target HUD investment to areas where it was most needed and automatically populate planning documents with the data they were required to produce for their consolidated plans.

The Transformation Initiative of 2012 was launched shortly after the HEARTH Act in response to a 2011 White House memorandum instructing federal agencies to adopt digital recordkeeping systems. The Transformation Initiative allocated 1 percent of HUD's annual budget to Enterprise Data Management Services, including HUD's Enterprise GIS (eGIS) platform. The Transformation Initiative was funded by the Office of the Chief Information Officer (OCIO) and led jointly by OCIO and PD&R. Under this initiative, HUD built the Community Assessment Reporting Tool, the HUD Resource Locator (HUD's first mobile application), the Affirmatively Furthering Fair Housing Tool, and the Tribal Directory Assessment Tool. HUD used the Transformation Initiative budget to enter the world of cloud computing through the ArcGIS Online subscription and to create the eGIS Open Data Storefront, which hosts HUD's spatially enabled datasets. Today, these technological improvements have made HUD's geospatial data and applications more accessible and widely used. The eGIS Open Data Storefront receives more than 5 million requests for data per month, and the various web mapping applications receive thousands of information requests per month (table 1).

Table 1: Peak number of monthly users for each HUD eGIS application in 2022.

Application	Peak Monthly Requests for Service
eGIS Open Data Storefront	5,445,594
Affirmatively Furthering Fair Housing Tool	602
Community Assessment Reporting Tool	259
Community Planning and Development maps	1,071
HUD Resource Locator	177,976
Tribal Directory Assessment Tool	2,440
Total	5,627,942

 

The future of GIS at PD&R

HUD's recent Geospatial Strategic Plan formalizes what many of the people we interviewed identified as the goal of advancing GIS technology at HUD: to "democratize geospatial data and services" by making HUD's locational data available to communities and stakeholders and ensuring that all HUD offices can visualize, analyze, and inform policies through spatial data. This vision has remained the same since the days of C2020. In 1997, HUD Secretary Andrew Cuomo described HUD's desktop mapping software as part of "a future where no one is left behind and everyone has access to technology which allows them to reach as high as their talents and hard work will take them."

HUD's growing technical capacity and eGIS Open Data Storefront have been vital in its efforts to democratize GIS by increasing the availability of data and analytical tools. More staff across HUD's program offices use spatial data and GIS technologies for innovative analyses and research that support data-driven policy development. HUD's GIS capacities and programs are outlined in the Geospatial Strategic Plan and are guided by federal acts, including the Open Government Data Act of 2018, the Foundations for Evidence-Based Policymaking Act of 2018, and the Geospatial Data Act of 2018.

In the broader world of GIS, the availability of free and open-source software such as QGIS has facilitated mapping efforts and applications produced by and for communities facing housing insecurity. Community-based mapping projects with a deep grounding in the places and histories the maps represent have revealed that democratizing GIS is not just about making data and technology open and available but also about interrogating the social processes through which locational data are collected and maps and mapping technologies are produced. These questions are particularly relevant for maps representing urban neighborhoods that have famously been linked to processes of racial, ethnic, and economic segregation, such as redlining. The history of GIS at HUD has focused mainly on the development of systems that implement geospatial technologies and data storage strategies that have allowed HUD offices to integrate spatial data into their everyday work. The future of GIS at HUD presents opportunities to continue building technological tools and expertise while extending internal efforts to democratize GIS and learning from local community-based mapping efforts that ask hard questions about how maps and mapping technologies can remediate the effects of unjust histories.