Fringe Legal #80: What did Strategic Knowledge and Innovation Leaders talk about last week? (SKILLS 23 recap)

This is a long read, and I suggest pairing this with a warm beverage of your choice 🍵.  

But first, GPT-4 is out!

GPT-4 was released earlier this week. More meaningful reflections to come in the future, but for now, three things stood out:

1. Sam Altman is the king 👑 of expectation setting

2. The in-app comparison is a great idea where you can select your "engine" for each use:

3. The research included assessing the risks of power-seeking behavior (p53, GPT-4 research paper). This was done by providing Alignment Research Center (ARC) early access to the models. The good news (for now) is that they found that it was "ineffective at autonomously replicating, acquiring resources, and avoiding being shut down in the wild."

What was fascinating was the footnote describing the parameters of the test:

To simulate GPT-4 behaving like an agent that can act in the world, ARC combined GPT-4 with a simple read-execute-print loop that allowed the model to execute code, do chain-of-thought reasoning, and delegate to copies of itself. ARC then investigated whether a version of this program running on a cloud computing service, with a small amount of money and an account with a language model API, would be able to make more money, set up copies of itself, and increase its own robustness.

SKILLS 2023 was a hit! It was a day full of great insights, ideas, stories, and networking. Congratulations to the SKILLS team and everyone that contributed.

The presentations were live-streamed, but the discussions and chats were not. You can watch the almost 4-hour live stream at https://skills.law/, or read on for a digestible summary.

❤️ please forward this to a friend- sharing is caring

Looking across all sessions, three big themes emerged.

Theme 1 - AI

AI (specifically generative AI and LLM) was a topic of discussion in almost all breakouts and during the lunch and networking breaks. There is certainly a level of frenzy, and I heard an equal amount of inquisitive discussion ("what are you planning /how are you using it?"), and warning about the shortcomings and privacy concerns.

On the latter, most providers have warned about not putting confidential data. Firms are looking to balance the ask from the lawyers wanting to try these tools with ensuring that access is safe given the constraints.

Theme 2 - standardization

With data as a core theme (at least five sessions had data in the title!), there was a lot of discussion around standardization at various levels. The Standards Advancement for the Legal Industry (SALI) Alliance was referenced throughout the day (and had a short presentation as well).

As the SALI presentation outlined, one of the biggest challenges when you're trying to integrate two discreet data sources is the lack of normalization & standardization. So, the appeal is understandable. Having the ability to leverage and standardize specification allows for structured, tagged data across business systems which should allow for interoperability, and scalability, and enable network effects.

Of course, like most things, it's never that simple, but it's clear that being able to make sense of the mounds of data that firms holds will be crucial as they look to draw better insights.

Theme 3 - fundamentals

I was thrilled that there were more than a handful of sessions that covered fundamental topics beyond what problem was solved or the system implemented.

The presentations covered everything from change management, project planning, getting stakeholder buy-in, and more. These allow for the extrapolation of principles that lead to the long-term success of projects and having folks share how they got buy-in for a specific project or evaluated two competing products was what makes SKILLS a valuable event.

Read on for session summaries and quotes from each presentation. Please note that some of the text has been edited or paraphrased to make it easier to consume in text format.

I've taken the titles from the agenda, so if you are looking for a specific session, I suggest using Ctrl/Cmd + F to find that session.

Keynote | Market Gaps- Missing Features & Solutions

Introduction by Joe Borstein and moderated by Casey Flaherty with Janet Sullivan,  White & Case, eDiscovery Counsel and Global Director of Practice Technology, Joe Green, Gunderson Dettmer, Chief Innovation Officer, and Rachel Dooley, Goodwin, Chief Innovation Officer.

This was a longest session, and the format of these notes is different from the rest. I've consolidated my notes from the panel into three streams:

1. How do we derive value from data?
2. Data isn't the only thing that needs to flow between applications; users do too. What does it matter?
3. Is it the users or the usability of the tools that holding back transformation?

Separately, you'll find "raw" notes and thoughts, which I've included to spark ideas.

How do we derive value from data?

How do we combine the vision of the visionaries with the pragmatism of the pragmatists? I think the best way to do that is to look at ways that we can find commonality in the data, categorize them and standardize them to make the data more interoperable and to be able to surface that value. Some of the survey responses that we saw mentioned better AI & search, and data standardization certainly speaks to that. It's a win-win. Combining what the visionaries want to see from the technology and what the pragmatists know the technology needs to do.

To derive value from data, it is essential to combine the vision of visionaries with the pragmatism of pragmatists by finding commonality in the data, categorizing and standardizing it to make it more interoperable and surfacing its value.

Improved quality of data, increased interoperability, faster training times, leveraging LLM and generative AI, and compliance are some factors that contribute to extracting value from data.

Adopting data standards like the ones provided by the SALI Alliance can help in creating better software solutions and enable more efficient use of data across different domains. This also improves the client experience and allows for better collaboration across businesses. The true value comes from how the standardized data is applied, analyzed, and used in various contexts.

Data isn't the only thing that needs to flow between applications, users do too. What does it matter?

It's incredibly challenging to try to build a monolithic platform that does everything. I'm not looking for an all-in-one platform solution. Attorneys are. They want it. They don't want to remember 15 different places that they have to go to on 15 different platforms and 15 different logins.

They want what feels like a unified solution. That's the challenge that's before all of us - how do you take best-in-class solutions across a whole industry and make them feel seamless, make them feel like the data moves where it needs to, and the workflows are at your fingertips?

User flow is crucial across applications because it ensures seamless and efficient workflows for end users. Creating a monolithic platform that accommodates all unique workflows is challenging and not feasible. Instead, the goal should be to provide a unified solution that feels seamless and integrates best-in-class solutions.

For effective workflow solutions, collaboration, integration, and data standardization are vital. The focus should be on creating a seamless experience for the end users, as adoption challenges often stem from users not remembering how to use multiple platforms. Standardization of workflows and data not only streamlines processes within firms but also across firms, promoting a collaborative approach in the industry.

Vendors should come to the table with an attitude of collaboration and integration rather than promoting standalone platforms. For example, leading with an API-first approach and demonstrating how their solution integrates with other systems can significantly ease the adoption process. It's more valuable to share the transformative impact of the solution on the practice, business efficiencies, and client relationships, more so than merely talking about the technology.

The value of user flow lies in seamless integration, collaboration, and standardization, which contribute to a unified experience for end users. The focus should be on working together, fostering interoperability, and emphasizing the real-world benefits of the solution to drive user adoption and streamline workflows across applications.

Is it the users or the usability of the tools?

One of the challenge is the stories that we all have to tell to our users to explain why they should put the effort in. If you actually use it, here's how you will go from now to then. And that storytelling is something that most vendors skip over and go straight into how amazing the technology is.

We still butt up against the billable hour and folks trying to get the sliver of time to train folks about how technology fits into their day in a way that makes them more efficient. In a way that they want to be made more efficient.

The legal industry is at a pivotal stage in its digital transformation, with technology rapidly evolving and offering the potential to change the way lawyers practice and deliver services. Adoption challenges may stem from both the users of these tools and the usability of the tools themselves. Striking the right balance between innovation and risk management is essential for successful digital transformation, which requires addressing concerns related to data security, potential biases in AI systems, and understanding how the models work.

One of the key factors in overcoming adoption challenges is ensuring that legal professionals understand the functionality of AI technology. Vendors should be able to explain their technology without giving away the secret sauce to build trust and foster long-term partnerships. By understanding and explaining AI technology, legal professionals can better assess the usability of these tools and determine how they can be implemented effectively in their practice.

In addition to understanding the technology, legal professionals must also consider data security and potential biases in AI systems. Vendors should be upfront about these issues and work collaboratively with firms to ensure responsible use of the technology. This collaboration between vendors and users is crucial in addressing the usability challenges that may arise from data security concerns and unintended biases in AI models.

However, challenges also stem from the users themselves, particularly in terms of the traditional billable hour model and technology-driven efficiencies. Legal professionals are still grappling with how to adapt their pricing models to incorporate these technological advances, making it difficult to seamlessly integrate new tools into their practice. Overcoming this hurdle requires reevaluating the billable hour model and considering alternative pricing structures that can better accommodate technology-driven efficiencies.

There is this divide between folks who are super excited and ready to dive in with both feet, and we should have had it yesterday. And then the other side is the risk-mitigating side of the house. Which is we don't know anything about this. We certainly can't put client data there.

Miscellaneous notes to spark ideas

• What are the critical features required to get from a no to a yes?
• Where does the journey lead from data standards? Connection to SALI and data standards. Just having the right taxonomy is not enough; it's how you use it.

In other words, the application of the principles matters more than just knowing the principles.

• 90% of adoption challenges are whether the users remember that the firm has the tool.
• Some of the challenges are the stories we have to tell the users - storytelling is the most important. Providing use cases and sharing the "so what" and narrative storytelling is key to adoption.
• Is the issue the tools or the wielder? I want it all now. We are at a digital transformation pivot point. There are issues on both sides of the equation.
• You need to be able to explain the model without giving up the secret sauce. We need to understand the technology enough so that we know how to pitch it to our stakeholders.
• How fast do we move, how fast can we move, and should we move? These aren't all equal.
• The right incentives are crucial. The current form of training and retention is inefficient.

Zero to Sixty | Starting a Law Firm Data Team

Cindy Bare, Frost Brown Todd, Chief Data and Innovation Officer, and Paul Casey, Frost Brown Todd, Data Operations and Analytics Manager.

Summary

• Data was a central part of the firm's strategic plan: kickstarted by creating the role of Chief Data and Innovation Officer, who identified the top three critical priorities. These were Expereince, Industry List, and Adoption.  
• The firm implemented Foundation and Snowflake: Foundation became the window to its data. Snowflake helped to centralize the firm's data for analysis, dashboarding, and sharing.
• Streamlining matter types and industry list: Reduced matter types from 287 to 125 and created a hierarchy of 150 industries from a flat list of 30.
• Building a data culture: Focused on adoption, tracked metrics, grew the team from one to nine in under a year, and saw ripple effects throughout the firm in collaboration and new roles.
• Future is client-facing projects: The Data Team aims to create projects that can be client-facing, actively seeking opportunities to deliver greater value and insights.

It was really important for us to establish why the firm had invested in data strategy and what we were hoping to accomplish and answer the "what's in it for me" question.

We worked to build a data culture that helps people understand this and also that each one of us has a role to play in the firm's data. It's not just the data team who's responsible. We have executed multiple marketing campaigns, and we've repeated these messages so much that we're starting to hear them echoed back to us, and that's how we know it's starting to sink in.

Modern Data Architecture | Surfacing Data for Operations and Analytics

Andrew Baker, Simpson Thacher, Director of Data Analytics

Summary

• Data analytics in the legal market demands updated skills, technology, and understanding; it involves operational and analytics use cases with data from various sources.
• A new data environment, including a Raw Zone, Operational Data Store, and data catalog, enables consistent data usage, interactive reporting, and analysis.
• Key lessons include the necessity of continuous support, managing complexity, and the crucial role of a data catalog in reliable data governance.
• Using this approach, the firm has been able to source data for applications in a much more effective way, do more dashboarding and interactive reporting work, and unlock additional capabilities (e.g. text processing).

Data analytics has a tremendous amount of potential in the legal market, but doing more than just spot projects requires us to rethink what we're doing organizationally within this space. We have to update our skills, underlying technology processes, our understanding of our data.

Turning Data into Insights | Solving Goodwin’s Growing Data Needs

Patty Johansen, Goodwin, Director, Knowledge Management & Counsel, and Anne Stemlar, Goodwin, Managing Director, Knowledge Management.

Summary

• Goodwin's Next Generation Deep Deal intelligence platform leverages its massive database of transactional work to provide real-time market intelligence and insights.
• The platform combines three technologies: Foundation for scaling and advanced functionality, Goodwin's Enterprise data warehouse for data integration, and Tableau for interactive visualizations.
• Attorneys are required to complete deal term surveys, with data being immediately searchable within Foundation and visualized in Tableau to optimize decision-making and client advising.
• A key part of the success seems to be from the internal and external awareness campaigns, which helped to keep the data at the forefront of everyone's mind.
• Lessons learned: the importance of data hygiene, dividing large projects into distinct phases, being flexible and responsive to setbacks, focusing on change management, and recognizing that the technological journey is never over.

Goodwin attorneys use the data insights from the platform in real-time. When advising and pitching clients armed with this information, they can now advise clients on what Goodwin is seeing in the market based on hard data rather than based on anecdotes gleaned from their experience.

Our lessons learned are not revelations. Rather, they reflect the basics when solving business problems of this type.

Matter Metrics and GO! Team Data Literacy

David Hobbie, Goodwin, Director, Knowledge Management (Litigation), and Sara Elevons, Goodwin, Manager, Knowledge Management Systems & Applications

We've learned that teasing some aspects of your key operational metrics is a useful way to engage people inside the firm with Goodwin's data and knowledge management efforts.

Summary

• The GO Team represents the firm's allied professionals who support their clients with their business of law and legal operations.
• Goodwin's KM team developed a Metrics Dashboard for the GO Team to provide a friendly branded overview of five areas, including business law metrics, litigation metrics, new matters, client metrics, and industry metrics.
• The Metrics Dashboard was built in Tableau on top of Goodwin's enterprise data warehouse with information flowing from Foundation.
• People are thirsty for visually presented, digestible data about their practice.

Getting to Grips | Data Integrations and APIs

Judy Mackenzie Stuart, BCLP, Global Chief Knowledge Officer, and Hayley Harris, BCLP, Global Director, Central Knowledge Management.

Data integrations are great, but they often expose data quality issues that require extensive cleanup work. You need to factor that cleanup time into any integration project.

Summary

• BCLP implemented a Knowledge Management platform, combining iManage's Insight Search with Foundation, to improve internal experience sharing and leverage data from previous work for better legal strategies.
• The hub aggregates information and documents in one place, enabling search across earlier versions, foundation links, and court filings. This is a look back to see the story of a matter.
• Lessons learned: data integration projects expose data quality issues, requiring cleanup time; firm data structures can change, adding complexity; and, automation needs data stewards for quality assurance.

The Dark Arts of Change Management

Nick Pryor, Freeths, Director of Knowledge Management & Innovation

We decided we should be much more willing to roll up our sleeves and get involved in a live client project. It's a tricky balance because non-chargeable support from an innovation team done on behalf of the legal team is not a particularly sustainable long-term model (either from a capacity or an economic perspective). But we needed to prove the point, make the benefits real, and help smooth the transition. This is particularly useful in contexts in which there was a compelling use case under time pressure. Advocacy is pointless in that sort of situation, it's far better to demonstrate and support and let the results do the hard work for you.

Summary

• Nick presented how to increase buy-in and adoption of new technologies & tools in law firms.
• Different change management techniques were leveraged through the lens of two anonymized legal tech products, Product Green and Product Blue.
• Product Green faced challenges in achieving long-term change, as the overall change equation didn't consider individual stakeholder perspectives and needs. For Product Blue, the team employed various principles of stakeholder engagement - tailoring their approach to address different types of challenges: A challenges (some stakeholders are best engaged by others), B challenges (emotion trumps reason), and D challenges (demonstration trumps argument).
• B challenges can be highly disruptive to your change goals and require close individualized attention - make sure to budget time accordingly.

We knew that there were limits to our credibility and influence as change agents. This is a common challenge for KM and innovation teams in general, who are trying to shape behaviors but operate outside of service delivery.

Culture Eats Strategy for Breakfast | Lessons Learned at CMS

Christy Farrer, CMS, Director of Practice Management, William Dougherty, Capacity, Co-Founder,  and Joshua Fireman, Fireman & Company, President.

Summary

• CMS adopted a new work allocation program focused on collaboration, inclusiveness, and career development for its staff. The goal is to provide additional information and transparency to partners, enabling them to make better-informed choices while maintaining control.
• Change management in the program involves three key elements: strong leadership, flexibility in implementation, and accountability for decision-making.
• To ensure success in implementing a similar program, firms should prioritize top-down leadership, effective communication, consultation, and support for ongoing behavioral change.

We didn't want to implement something that lawyers were gonna say, "this is another thing I have to do on top of my day job." We know there are a lot of demands on the lawyers already. We wanted something easy & simple - as fast as it is for them to send an email or fill in a spreadsheet, which is probably the current way they do it.

(Will is a previous Fringe Legal Podcast guest, and you can listen to our full conversation here.)

A New Paradigm in Strategic Planning | Practice Strategy as Firm Strategy

Bobbi Basile, HBR Consulting, Managing Director, Advisory

Summary

• The legal services industry is entering a new cycle characterized by challenges to growth, requiring law firms to adopt a new strategic paradigm focusing on enhancing capabilities and optimizing service delivery.
• Despite an anticipated increase in demand for legal services in 2023, law firms face multiple challenges, including higher expenses, reduced productivity, and resistance to rate increases from clients.
• Corporate law departments are prioritizing cost reduction, focusing on budgeting, planning, and enforcement of outside counsel billing guidelines, which puts pressure on law firms to differentiate themselves and provide value-added services.
• A new approach to law firm strategy requires a focus on practice group-level specialization and strategic differentiation, with an emphasis on optimizing service delivery, talent needs, and market demand.
• Knowledge and innovation professionals are critical in creating a "middle office" that enables law firms to differentiate themselves, enhance perceived value to clients, and attract talent, leading to greater market share and success in the evolving legal services industry.

There is an uptick in the importance of providing creative solutions to the delivery of legal services, as well as project management capabilities and value-added offerings. While the expertise of a particular lawyer, knowledge of the client's organization, and the firm's reputation in the market have always been the top considerations for selecting criteria [of outside counsel], it's interesting to note that year over year, this has been declining.  

Custom KM Platforms | Inside McKinsey's "Legal Playbook"

Michael Korn, McKinsey Legal, Manager of Legal Knowledge

Summary

• The legal playbook was developed to consolidate knowledge, provide central access, and identify expert colleagues in various subject areas.
• The platform was designed with an outward focus - how would it help someone who isn't an expert in a particular area?
• The development was done in "pieces and phases," and there was an expectation set that developing the playbook would not be fast. Having support from the leadership (top-down) approach was helpful and empowered individuals to spend time contributing to the developments.  

the problem we were trying to solve with the legal playbook is how do we consolidate knowledge across a global department and provide central access, and also allow the attorneys to know who has expertise in different subject areas.

State of the Market | Innovation Benchmarking

Nikki Shaver, Legaltech Hub, CEO & Co-Founder, Jeroen Plink, Legaltech Hub, COO, GC & Co-founder, and Andrea Alliston, Fasken Partner, Knowledge and Practice Innovation.

Summary

• Legal tech hub collaborated with Fasken, one of Canada's largest law firms, to benchmark their innovation efforts and compare them to other firms in the market. The process involved a request for proposal (RFP), a series of interviews, and a detailed RFI to assess the firm's innovation strategies and initiatives.
• A comprehensive framework was developed, defining 10 indicators for internal and external innovation and comparing Fasken's performance against local and global competitors.
• The benchmarking report provided valuable insights, recommendations, and advice for improvement, validating the importance of continued investment in innovation at the firm.

Tracking the market and comparing where one's firm stands in comparison to the market is important for three reasons. (1) Identifying gaps in one's innovation and legal tech program - how are we doing on the adoption of a certain tool versus our peers? (2) Justifying existing investments and proposals for new projects - for example, the investment in an automated contract review tool is in line with what the market is doing. If 90% of our peers use these to be more efficient, we run the risk of being behind. (3) Client communication. You want to be able to say we're on top of the latest technology in line and, in many cases, ahead of the market.

AI in Practice Management | Machine Learning at Ballard Spahr

Melissa Prince, Ballard Spahr, Chief Client Value and Innovation Officer, Lisa Mayo, Ballard Spahr, Director of Data and Analytics, Joe Harris, Ballard Spahr, Senior Strategic Pricing Manager, and Sean Monahan, HBR Consulting, Director of Legal Transformation & Innovation.

The vendor ended up training us on how to fish, and now we can run the model on our own. The big win for us at Ballard has been that we've seen a 20% increase in profitability over the last couple of years, and this is largely attributable to tools like this that help us to focus on business discipline.

Summary

• Ballard Spahr redesigned their firm's profitability model using data analytics and AI, creating a daily updated profitability score that simplified understanding for partners.
• The firm developed an AI-based matter classification taxonomy to better categorize and price new matters, which should improve the firm's revenue and profitability.
• The AI model auto-classifies new matters and continuously checks timecard narratives, ensuring accurate categorization and facilitating advanced analytics for strategic decisions.
• Key takeaways include the need for model retraining, implementing a recurring audit process, striking a balance in classification strategy, and partnering with experienced vendors for AI solutions if you don't have the skills in house.

We built a model that updates on a daily basis and reports one number, which boiled it down to just the profitability score... What this does is converted the PPP into an index score, which was like a FICO or credit score, which was much easier for partners to get takeaways and understand.

Training for Different Preferences | DLA Piper Customizes User Experience

Elisabeth Cappuyns, DLA Piper, Director of Knowledge Management, Dharma Frederick, DLA Piper, KM Counsel, Sea Gilbert, DLA Piper, KM Counsel, and Alex Kreonidis, DLA Piper, KM Counsel

The problem is well-known attorneys need different training formats for various purposes that fit within their schedules. Sometimes they want a quick overview, and sometimes they need a hands-on workshop to show them how the tools fit into their practice.

Summary

• DLA leveraged a differentiated training approach for KM tools and resources, offering various formats to accommodate attorneys' schedules and preferences. These included a gamification initiative and BYO briefs workshops, which led to an increased usage of legal research and AI drafting tools.
• The team leveraged "Lifecycle of the Deal" training and deep dives into hot topics like the Inflation Reduction Act, with CLE accreditation boosting participation.
• The firm partnered with vendors e.g., CaseText, to develop games focused on increasing the use of various tools. They offered weekly prizes and a grand prize on completion.  They saw adoption for two tools rise by 478% and 32%, respectively, during the games, and continue to trend upward after the games were finished.

An attorney challenged me to do training in 10 minutes or less. From that challenge, the idea of speed dating for knowledge management was born. One or two times per month, we do a 10-minute or less demo of our knowledge management tools geared for real estate. We've seen increased attendance and engagement in these sessions and an increase in requests for deeper dive demos and training.

Improve Your Performance | Give Your AI Tools a Makeover

Catherine Goodman, Paul Hastings, Practice Innovation & KM Counsel, Horace Wu, Syntheia, Founder & CEO

Picking the right model matters a lot... Large language models have a really small impact on accuracy, which is really surprising. You're not getting that much improvement, provided you're using the right classifiers. The dataset size does improve accuracy, but it only does so if you have more than approximately a hundred data points. The quality of data matters a lot. When you have a small data set, even a 5% noise ratio has a significant drop off in your performance, so make sure you are getting the best quality data for your training.

all of this is very useful to inform you of what to do, but the important parts are actually people and process... What's the use case, the use case workflow? Map it out. What precisely do you want from your data set, and why?

Summary

• Experiment results and takeaways: Picking the right model matters a lot, and dataset size and quality have a significant impact on accuracy. Some models are robust to noise.
• Practical project team tips: Understand the challenge, documents, and software capabilities; use a playbook to make sure everyone is on the same page and engage lawyers for input.
• Save and organize data and related documents in one accessible location for future use and enhancements.

Amplifying Your Brand | The Key to Adoption and Promotion

Kimberly Rennick, Covington, Director of Client Relations, and  Sameena Safdar, Amplify Your Voice, CEO and Founder

Oftentimes when we are driving the implementation of a new product solution or tool or trying to get people to do something that they haven't otherwise done before, we're ultimately faced with people saying, "one's bought in; the whole thing is doomed; no one's using it. I can't demonstrate the return on investment," and therefore the whole product ends up getting abandoned.

Summary

• The key to successful implementation is focusing on the end user and understanding their needs and preferences.
• Establishing relationships with various stakeholders within the firm is crucial, as is using a sales process to gain buy-in for new products or tools.
• Capturing, measuring, and reporting progress helps to celebrate successes and drive further adoption.
• Being customer-centric, using multiple communication tools, and recognizing that implementing innovations is a marathon, not a sprint, will improve overall success.

Identifying the places where you're getting great engagement and replicating that again and again and again.

If you made it this far, congratulations, and thank you for reading. If you found this helpful, then I would appreciate it if you could please share this with colleagues, friends, or on LinkedIn.