Dear reader,

Of all my many adventures while plying my trade as a humble government employee, all else paled in comparison to seeking to tame the beast of inter-agency data sharing.  Actually, that’s a lie. It was just another incredibly important and difficult thing to do, alongside challenges like hiring through civil service, and navigating contracts through procurement. There are dedicated, smart, talented people trying to solve these challenges, but they are still deeply troubled systems whose opportunity costs are incredibly steep. Believe me, we pay for them in millions of small ways.

The reasons one might wish to share data with another city agency partner are as myriad as they are obvious: improved access to public services and benefits, reduced data input and management, increased communication and collaboration, happier constituents and service users, more productive staff, and the list goes on.  

The reasons one might wish to share data with another city agency partner are as myriad as they are obvious.

One would think that inter-agency data sharing would have been solved long ago.  But when one stops to consider that much of the data involves highly vulnerable populations, including children, women fleeing domestic violence, recently arrived immigrants, people affected by homelessness, or those dealing with mental health or substance abuse issues, just to name a few of the issues, then the question of being able to share data responsibly, thoughtfully, safely and appropriately with only those who need to see it among hundreds of thousands of employees becomes a bit more, well, complicated.  

I knew very little of this when, around 2018, my colleagues and I hit on the brilliant idea of creating a data-sharing application for shelter partners, so that they could have real-time access to the same attendance, academic and behavioral incident information as schools.  After all, much work had already been done within the DOE to dramatically improve the capturing and presentation of this data, led by pioneers like New Visions for Public Schools and implemented through groundbreaking initiatives like Community Schools (where I worked at the time).  We had learned some years before that low student attendance and chronic absenteeism (missing just two days of school per month), were highly predictive of poor academic performance, reduced grade promotion, increased likelihood of dropping out of school, and failing to graduate on time (among other issues).  

Teaching schools to use these new information tools and build data-driven approaches to monitoring and addressing chronic absence was a powerful pathway to understanding underlying student challenges (illness, homelessness, learning disabilities, depression, etc.).  As a friend of mine with a Divinity degree likes to say, you have to know the demon’s name in order to cast them out.  Data gave schools the ability to name unseen challenges, track interventions, and pursue better results for their students and families.  And it worked.  

Surely, we reasoned, we could offer shelters the same benefits?  After all, shelter staff repeatedly asked for it.  They knew that they would be better able to speak with schools, and partner to align supports and resources between shelters and schools to make those more seamless for families (and thus more likely to be effective).  They also wanted to better understand their students so they could collaborate more effectively with parents to keep families stable, and position them to move into permanent housing.

So, the two agencies jointly submitted a proposal for a special innovation programs hosted through the Mayor’s Office of Economic Opportunity, and voila we had $1 million and three years to make it all happen: adapt the current data system for shelter-based users, build inter-agency training approaches, and support new users with hands on technical assistance.  We were confident that timeline would not pose major constraints.  

I normally saw such [data sharing legal] agreements take 18 months or more to be promulgated, and a number of colleagues have seen agreements take several years.

There’s a lot more to this story, because inter-agency collaborations and scaled implementations of information management systems and coordinating external vendors are all complicated logistically, culturally, and technically.  Our focus here is on the narrower set of issues we found in simply moving data back and forth between willing partners with significant knowledge, clear intention, plenty of goodwill, and better results for their students and families firmly centered.  

So, what did we discover?  Or rather, learn the hard way?  Below are some of my key takeaways, enumerated for your pleasure.  

1. First off, sharing data between agencies means navigating complex and sometimes contradictory Federal, State and City data privacy regulations and requirements, greatly complicating the process of negotiating the necessary legal agreements between agencies. Little guidance or legal framing exists to provide templates.  Legal teams themselves are overwhelmed and often lack subject matter expertise in data privacy, meaning that these legal agreements become a time-consuming research project and are often “over-built,” with highly restrictive terms and conditions that place a lot of boundaries on just what data can be shared and with whom.  I normally saw such agreements take 18 months or more to be promulgated, and a number of colleagues have seen agreements take several years.  Imagine what happens if the lawyer in charge moves to a new role, or project management leadership changes, or new regulatory requirements come into being, or the design for the system changes based on user input.  You can wind up back at square one, or, if you’re lucky, square two.  
2. Most improvements to inter-agency data sharing are conceived of as sweeping overhauls, requiring high levels of legal, technical and administrative support over long periods of time, and are highly costly and extremely difficult to implement. Overwhelmed IT teams have little bandwidth for solving granular data system problems, and as a result common workflows (like registering users for programs and services, or conducting routine oversight of scripted processes) that could be automated languish for lack of attention, leaving frontline workers without important tools to improve efficiency and accountability, let alone build their digital literacy in the process.  In the Students in Temporary Housing team, for instance, we conducted some 20,000 intakes a year on paper.  In the process, we often made families provide to us information they had shared already with colleagues in other agencies, as well as offices in the DOE itself.  But without a simple database (let alone one that could send and receive data from other databases), we were stuck with paper.  Oh, and we weren’t allowed to purchase and develop our own database.  We were told in no uncertain terms that was IT’s job, and strictly off limits for us.
3. Because of this tendency to implement only large-scale and sweeping change, well-intentioned systems designers often failed to develop specific enough use cases defining what minimum viable data collection is required, who will collect it, and how it will be applied in ways that improve interactions with clients and consumers, setting new systems up for failure from the start.  There’s a tendency to over-collect data, or to build data systems without much needed user input, resulting in systems that are cumbersome, repetitive of other data sources, and not focused on providing the captured information in a way that drives results for priority needs.  In other words, it’s really easy to get bad design, and wind up with a system that doesn’t work well, and that therefore no one really wants to use.  
4. As I noted above, many internal operational procedures within city agencies are still paper-based or have very nominal data automations, so that the workforce itself has low data literacy and familiarity in working with data systems, including entering and maintaining high quality data.  You’ve probably heard the phrase, “garbage in, garbage out.”  As someone who’s worked with crappy data, it’s arguably not very worthwhile spending a lot of time cleaning it to make it useful (often involving manual review).  Good data starts with people who do good data entry, and we have built a workforce that doesn’t do that well.  
5. Other existing data environments are sometimes decades out of date (like using command prompts to navigate and code entries instead of a mouse and GUI), so that new system upgrades are difficult to integrate and operate for users.  There are data systems that govern critical business operations in use right now that can only exist in a special server environment that runs a 20-year-old operating system, because that’s when the data system was built.  Try getting your API to speak to that.  Good luck.  
6. So, low data literacy and low system utility mean that every new implementation requires a huge investment in staff training and support in order to create a culture of understanding and using data to drive continuous improvement.  As the saying goes, “culture eats strategy for breakfast.”  Change is hard, because change is loss.  To really drive change, especially adaptive change that is aimed at system transformation, people need to feel hopeful, supported and confident.  That takes time, care, and intention.  Nothing will kill innovation faster than rolling out a new system and not providing proper support to use it.
7. Many data systems will require multiple access levels by hundreds and sometimes thousands of users, but most city agencies do not have clear protocols or significant enough personnel support for identifying and authenticating users.  New users can struggle to get system access and former users can mistakenly maintain access long after their departure.  I mean, in the shelter system we were designing for, there were over a hundred nonprofit service providers, each with dozens or hundreds of staff, and anyone who was going to access highly sensitive, personally identifiable student data needed to prove that they were who they said they were.  Bear in mind that there’s also high turnover in lower-level case management personnel, and not everyone had the same level of access (managers and supervisors would often have access to more sensitive data, like reports of behavioral issues or service interventions).  Basically it’s going to become someone’s full time job to track logins and update permissions just to maintain the most basic level of data user security.  Try telling that overworked IT team they have to give up a head count just to count heads.  

Looking back, I’m surprised we managed to pull the thing off.  We did, and I hope the effort is still going strong, but it required constant problem-solving, cajoling and wheedling, re-negotiating, and revisioning.  I think, in a way, that’s part of what makes for good project management, and it was certainly a learning experience.  But who wants to test a diamond upon the anvil?  I admit it’s not everyone’s idea of a good time.  

I’ve now worked with three mayoral administrations, and each one wants desperately to crack the code on inter-agency data sharing because each one knows that doing it well could mean really significant improvements in how the city runs and serves its citizens.

I’ve now worked with three mayoral administrations, and each one wants desperately to crack the code on inter-agency data sharing because each one knows that doing it well could mean really significant improvements in how the city runs and serves its citizens.  It’s a worthy effort, but it takes time to learn these lessons, to anticipate the challenges, to invest in a wide array of solution components, and build approaches that turn the great ship of bureaucracy towards the sunrise. It’s nice to know that this is possible, because we’ve seen it done. It would be even nicer to know that it was getting progressively easier and more impactful, but I don’t think we’re there yet.

Thanks for continuing on this journey with me, and until next time stay strong, stay dapper.