Wolters Kluwer USA -- Report that showcases AI usage and insights

Our team at WK USA is interested on benchmarking the AI usage and components around how their improving the RFX Process with AI. This report would showcase the data points on how the AI functionality is helping them showcase adoption metrics vs. their previous tool qvidian.

Find more posts tagged with

Future consideration

Status: Future consideration

Comments

Khalees V

@Ginno Bergunio Do you know what data points they are expecting? For time-being, can we pull a report from our interntal tools?

Khalees V

@Ginno Bergunio It would be great if we can get more insights on what data point Customer is expecting.

Ginno Bergunio

Thanks for your help here @Khalees V — After our meeting here's the overview on what the Leadership team is expecting below:

AI Reporting:
- Discussions centered on maintaining audit trails and proceeding with key changes pending security approvals.
- Rhonda and Glen Tamm discussed the shift to a centralized hub for policy management.
- Bethany Spriggs emphasized the need for leadership reports to track AI tool usage.
  - Ideally Time spent using the AI tools
    - How much time is saved by using the AI tools
  - Number of questions addressed in AI Draft
  - AI Draft (%) Completion rates and how that can equate to a number on time range.
  - Content Library impact — how may questions were modified using AIA
  - Total number of questions addressed in ASK
    - Break this down based on filtering via tags/collections/languages
  - Overall time saved on projects using AI vs. nonAI answering

Erica Nelson

We have multiple customers in the Major segment that are requesting reporting for ASK/AI Draft/AIA in the system that they can access within their dashboards and support the ROI of the AI tools. The initial AIA reporting has been well received, but customers are looking for additional things like:

Overall response rates
Overall usage of answer rates
Data for time with tool for individual users (Application Usage Dashboard)

Tara Kolinsky

We need this in the IRI as well.

Atiq Khan

Carnegie Learning, Billtrust, BlueConic, and Gov Os are just a few names that are looking for an on-demand method to show ROI on AI assistant. Specifically efficiency gains across projects and AI assistant usage. They would like to compare it to benchmarks across similar industry and use cases. At the very least, my customers would like to easily see QoQ/ YoY changes.

Nicole Stanford

Any of my clients that leverage or are interested in AI want to be able to report on usage and show ROI to their leadership teams. These clients haven't specifically asked me yet but I know that they will eventually want this type of reporting:

Fortrea

Optum

RLDatix

MFS Investment Management

Nicole Stanford

Fortrea

Optum

RLDatix

MFS Investment Management

Gosia Rios

Fortrea, Inspirus, Qualtrics, Sodexo, Microsoft are looking for additional reporting.

Ideas I received in the past are:

How can I identify champions using AI?
What is the time saving gained since I started using AI?
How many AI responses were applied AI Draft, AI Assistant and ASK?

Khalees V

@Gosia Rios We do have plans to bring the ROI dashboard within Responsive Platform. Timeline is unknown at this point of time. Keep you informed once we have it added to the roadmap.

Andrew LeBrun

Adding that One Inc (a 7 year customer of ours) is asking for reporting that helps them track AI efficiency gains in time savings, accuracy of response, projects/answered supported by AI, etc.

What's the update on timeline here?

Steffi Fernandes

Akeneo, Included Health, Revinate, Payit, Ideas As A SAS, Archer, Bizagi and Sumo logic "

Adding the above customers also to the list requesting detailed reporting on AI/AI Draft/ASK
1. Reporting based on activities and dates for each module separately.
2. Widgets for each module on the Executive dashboard
3. How many times was ASK used by users?
4. Comparison to show the time savings using all three modules for completing questionaires

Nicole Stanford

Other clients that I know would benefit from enhanced AI reporting:

Cloudflare

e2open

Capstone Publishing

Kristen Lupo

FIS and Moody's are both pushing for enhanced AI reporting. In addition to Ginno's note from 2/19, they are both very interested in being able to capture (simply) AI Draft, what was applied and passed off as a successful first pass, what was further edited, and how many times, and then having better insights to if the AI Draft was ever scrapped. Ideally, they're trying to take those numbers and run them up against a time saving baseline they have established from their previous practices.

IE, this response had 600 questions, 85% was answered using AI metrics that did not have to be further edited. This is estimated at a 50% time savings increase.

Tara Kolinsky

This is becoming a critical ask for the IRI as well. We need a way to reliably report on AI usage (Ask, AI Draft, and AI Assistant) ASAP as leadership is routinely asking for this level of info.

Khalees V

@Tara Kolinsky What metrics you are looking for in the report?

Tara Kolinsky

We'd like to be able to show both the usage of AI across each solution as well as the usage of content as it relates to AI usage. So some of the things I'd like to easily report on:
- How many questions were answered solely using AI (across AI Draft, Ask, and AI Assistant)
- How many questions were answered with the assistance of AI (either AI created a draft that was refined or AI helped refine an existing answer) (across AI Draft, Ask, and AI Assistant)
- How many queries have been made of Ask over a set time period
- How many Ask queries were answered by content in the content library (even better if I can see metadata on that content…such as how many Ask queries were answered by content in the Premium Response collection) - this helps me correlate the value of the content we maintain
- What Ask queries were NOT answered by available content (helps me identify new content we need to add)
- What users are using AI and how much time are they spending using AI

That's the start of what I'd like to see, but I see these needs growing as we use it more. At the heart of it is being able to prove the value of the AI itself as well as the content that is feeding the AI