Facebook
X
LinkedIn
ForwardAI Evaluation in the Era of Large Language Models and Natural Language Generation - Guanhua Chen, PhD; Frank Liao, PhD, ACHIP
Series Overview
This bi-monthly one-hour educational series will connect frontline clinicians, champions in artificial intelligence (AI) applications in healthcare, and researchers to learn about and discuss leveraging AI technologies in healthcare settings within UW Health and beyond, rapidly evolving AI solutions, and AI governance.
Intended Audience
The primary audience for this educational series is physicians, nurse practitioners, physician assistants, registered nurses, and researchers affiliated with UW Health. Other members of the clinical team will benefit from participation.
Learning Objectives
As a result of participation in this educational activity, learners as members of healthcare or research team, will improve their ability to
- Explain the pros and cons of human-proposed, pre-specified evaluation metrics versus using large language model (LLM) as a judge.
- Discuss the use of evaluation metrics and its impact on adoption of AI-generated clinical summaries.
- Describe the Provider Documentation Summarization Quality Instrument (PDSQI-9) as a framework for use with clinical summary evaluation.
FACULTY DISCLOSURE
It is the policy of the University of Wisconsin–Madison Interprofessional Continuing Education Partnership (ICEP) to identify, mitigate and disclose all relevant financial relationships with ineligible companies* held by the speakers/presenters, authors, planners, and other persons who may influence content of this accredited continuing education (CE). In addition, speakers, presenters and authors must disclose any planned discussion of unlabeled/unapproved uses of drugs or devices during their presentation.
* Ineligible companies are those whose primary business is producing, marketing, selling, re-selling, or distributing healthcare products used by or on, patients.
The ACCME does not consider providers of clinical services directly to patients to be ineligible companies.
For this accredited continuing educational activity all relevant financial relationships have been mitigated and detailed disclosures are listed below:
| Name | Role | Financial Relationship Disclosures | Discussion of Unlabeled/Unapproved uses of drugs/devices in presentation? |
| Guanhua Chen, PhD Associate Professor Department of Biostatistics & Medical Informatics University of Wisconsin School of Medicine and Public Health | Speaker/Author | No relevant relationships with ineligible companies to disclose | No |
| Frank Liao, PhD, ACHIP Senior Director, Digital Health and Emerging Technologies, UW Health Adjunct Clinical Assistant Professor, BerbeeWalsh Department of Emergency Medicine, University of Wisconsin-Madison | Speaker/Author | No relevant relationships with ineligible companies to disclose | No |
| Majid Afshar, MD, MS, ATSF, FAMIA Chair, RSS Planning Committee Associate Professor Director, Institute for Clinical and Translation Research (ICTR) Learning Health System Pulmonary and Critical Care, Department of Medicine Affiliate, Department of Biostatistics and Medical Informatics Affiliate, Data Science Institute University of Wisconsin School of Medicine and Public Health | Committee Member, Course Director | No relevant relationships with ineligible companies to disclose | No |
| Alaa (AL) Alsadi, MD CE AI Advisory Committee Associate Chief Medical Information Officer, Medical Director, Laboratory and Pathology Informatics, Associate Program Director, Clinical Informatics Fellowship, UW Health Assistant Professor of Pathology Department of Pathology University of Wisconsin School of Medicine and Public Health | Committee Member | No relevant relationships with ineligible companies to disclose | No |
| Barbara Anderson, MS Director, Interprofessional Continuing Education Partnership University of Wisconsin School of Medicine and Public Health | Planner | No relevant relationships with ineligible companies to disclose | No |
| Peter Kleinschmidt, MD, FACP CE AI Advisory Committee Associate Professor Department of Medicine University of Wisconsin School of Medicine and Public Health | Committee Member | No relevant relationships with ineligible companies to disclose | No |
| Becky Kohler, MPH, RN-BC Chief Nursing Information Officer UW Health Information Services | Planner | No relevant relationships with ineligible companies to disclose | No |
| Neil Munjal, MD, MS Assistant Professor Department of Pediatrics University of Wisconsin School of Medicine and Public Health | Committee Member | No relevant relationships with ineligible companies to disclose | No |
| Marianna Shershneva, MD, PhD Coordinator Interprofessional Continuing Education Partnership University of Wisconsin School of Medicine and Public Health | Planner | No relevant relationships with ineligible companies to disclose | No |
| Sue Wenker, PT, PhD, MS, FNAP CE AI Advisory Committee Department of Family Medicine and Community Health University of Wisconsin School of Medicine and Public Health | Committee Member, Peer Reviewer, Planner | No relevant relationships with ineligible companies to disclose | No |
| Jocelyn Wilke, MSN, APNP, MSCS CE AI Advisory Committee Advanced Practice Nurse Practitioner Department of Neurology University of Wisconsin School of Medicine and Public Health | Committee Member | No relevant relationships with ineligible companies to disclose | No |
Accreditation Statement
 | In support of improving patient care, the University of Wisconsin–Madison ICEP is jointly accredited by the Accreditation Council for Continuing Medical Education (ACCME), the Accreditation Council for Pharmacy Education (ACPE), and the American Nurses Credentialing Center (ANCC) to provide continuing education for the healthcare team. |
Credit Designation Statements
American Medical Association (AMA)
The University of Wisconsin–Madison ICEP designates this enduring material for a maximum of 1.0 AMA PRA Category 1 Credits™. Physicians should claim only the credit commensurate with the extent of their participation in the activity.
American Nurses Credentialing Center (ANCC)
The University of Wisconsin–Madison ICEP designates this enduring material for a maximum of 1 ANCC contact hour.
American Academy of PAs (AAPA)
 | The University of Wisconsin–Madison ICEP has been authorized by the American Academy of PAs (AAPA) to award AAPA Category 1 CME credit for activities planned in accordance with AAPA CME Criteria. This activity is designated for 1 AAPA Category 1 CME credits. Approval is valid until 2/25/2027. PAs should only claim credit commensurate with the extent of their participation. |
American Board of Surgery (ABS) CME Only
 | Successful completion of this CME activity enables the learner to earn credit toward the CME requirements of the American Board of Surgery’s Continuous Certification program. It is the CME activity provider's responsibility to submit learner completion information to ACCME for the purpose of granting ABS credit. |
Continuing Education Units
The University of Wisconsin–Madison, as a member of the University Continuing Education Association (UCEA), authorizes this program for .1 continuing education units (CEUs) or 1 hour.
Available Credit
- 1.00 AAPA Category 1 CME
- 1.00 ABS Accredited CME
- 1.00 AMA PRA Category 1 Credit™
- 1.00 ANCC Contact Hours
- 1.00 University of Wisconsin–Madison Continuing Education Hours
- 1.00 Approved for AMA PRA Category 1 Credit™
Registration for this activity can only be completed through the ICEP Learning Portal. Attendee registrations made through any other sites cannot be honored. UW-Madison ICEP is not able to refund fees paid through unaffiliated registration sites, such as eMedEvents.com, MedConfWorld.com, EventEgg.com, and 10times.com. Please report any unauthorized websites or solicitations for registrations to [email protected]
Accessibility
If you need anything to participate in this program, please contact [email protected]
Required Hardware/software
Computer, tablet, or other mobile device with sound (closed captioning also available).
Free, current version of Edge, Firefox, Safari, or Chrome. Some older browsers and Internet Explorer could produce error messages or not display the content correctly.
Free, current version of Adobe Acrobat Reader or other .pdf reader.