ࡱ> vu3( / 0DArialngsܖ0tt- 0DTahomagsܖ0tt- 0 DWingdingsܖ0tt- 0 A .  @n?" dd@  @@`` P)   !"#%$&b( 0AA@8wʚ;ʚ;g4GdGdU 0ppp@ <4dddd@k 0t- <4BdBd@l 0t80___PPT10 pp?% Program Evaluation Unit, Univerity of MelbourneO  =,!Evaluating organisational context""  .Measurement approaches: evaluation David Dunt/P/*  The contribution of realistic evaluation in studying the effects of organisational context on KU & KU s take-up at grassroots level ,D  For me, Realistic evaluation, not a new set of ideas but a name for an approach that I had progressively discovered during my career as an evaluator ( that s what it is ) My outline not so much an exposition on a realist theory of science But a description of this path of discovery As it encapsualtes the path to enlightenment for many other health practitioners interested in KU particularly after their discovery of Evidence-based Medicine & healthcareZ"  Outline the main ideas of Realistic evaluation particularly the key concepts of CMOs Context Mechanism Outcomes Outline applications to KU & organisational contextV4/    3,Q9  &Outcomes movement - point of departure"'$ The randomised control trial Treatment Program Outcome  Thing D*7 9 !Outcomes movement - first problem""$The randomised control trial Treatment Program Outcome  Thing Treatment, program content - a black boxBS72 c With a black box:-@Don t know what element of the program was responsible for success failure Was program failure due to model failure or implementation failure (of a good model)<<U<U  Distinction is fundamental to evidence-based decision making where effective and ineffective programs must be correctly identified  [Points to the importance of the study of process The importance of the use of Program logic\\  Program logic(Is a concept much loved by evaluators Similar to management flowsheets Specify successive stages (chronological, causal) in program rollout Use key performance indicators (KPIs) to assess whether a stage has been successfully reached Gather relevant information collected at each stage why stage has been reached (or not) But need to move beyond their common sense models of causation & chronologyHGPPMPGM,< f Program theory - Mechanisms 4$ $(Instead program logic based upon a program theory in order to identify optimal program content including all its components and their relationship (stages) Application of a Mechanism or causal model based on best information (literature, experts, stakeholders)0 O Program theory - Context&$Program logic also based upon knowledge of local Context (analysis of barriers, risks eg negotiating key gatekeepers, usual work patterns, usual organisational practice and barriers01},V9  In summary  (With program development There should be an analysis of the application of the proposed Mechanism in the local Context in order to ensure a program with maximum functionality in that context and with achievement of best Outcomes With program implementation There should be monitoring of the important stages of the program s rollout to ensure they are occurringi? f i DA second problem for the Outcomes movement embedded in this analysisEE  Context is assumed and Context of EBM studies typically is quite atypical Teaching hospitals Atypical patients chosen for their ease of study (no co-morbidities, English-speaking, no mental impairment, and with unrepresentative social & demographic characteristics),JJ tFundamental to evidence-based decision making where effective and ineffective programs must be correctly identified utRealistic evaluation in KU Focus on Context and Mechanism occurring to some extent already EPOC group Examples: academic detailing, use of opinion leadersX@@  "@8Realistic evaluation  in KU(Still little program theory (psychological, sociological [systems], or economic) Confirmed in a recent UK review by Grimshaw et al Reliance instead on atheoretical, tickbox questionnaire-based study of behavior change Program logic approaches notably absent>u VBRealistic evaluation - measuring organisational context & change 1CC !Challenge is first to build program theory Application of Mechanisms using existing theory and evidence to Local organisational Contexts using best local knowledge , "%, "  t'BRealistic evaluation - measuring organisational context & change 2CC !Generally, aim is to build heuristic models to assist in thinking about program development and implementation. Not to construct a causal model by eg path analysis, with each path s contribution being measured by a correlation coefficient(q|Z>A few exceptions where may be important to build causal models?? CFor very important and central dimensions of the health care systemDD>A few exceptions where may be important to build causal models?? Finish briefly with an example of this from my own work in the Australian health care system The development of a psychometric tool to measure the level of integration of the GP with the remainder of the health care system,GP integration with the health care system 1--$70 question inventory completed by GPs about their current activities relevant to integration 9 integration factors organised around 2 higher order factors patient care management and community health Plus 5 enabling factors<..t f,GP integration with the health care system 2--$rTwo key findings It is possible to map levels of GP integration across Australia  considerable variation exists Levels strongly relate to GP s knowledge of local services and resources(,GP integration with the health care system 3--$+Further details are a story for another day,,/n  0` 33PP` 13` 3333` Q_{` 333fpKNāvI` j@v۩ῑ΂H>?" dd@,?n<d@ `7 `2@`7``2 n?" dd@   @@``PR    @ ` ` p>>   @ & (    <" F 0   Tld" F 0   <L"U_ F 0   Td">& F 0   NĐ"P F 0   <"p F 0   C x?d?"bUv F 0    <l #" `   T Click to edit Master title style! !$   0 "   RClick to edit Master text styles Second level Third level Fourth level Fifth level!     S   6 "]}  `*0   6쪒 "] }   b*0   6诒 "]T}  b*0B  s *޽h ? 333380___PPT10.(г%  Blends   0 ; 3 P (  T +  "+bb P@ # "Dwoh  s *"PP  Bd" P@bb P 0  # "Nyh  s *"P    Bd"P 0 z   <" a*h   s *"    f?d?"+)   B8+ ?#" ` p + T Click to edit Master title style! !   0, " `   + W#Click to edit Master subtitle style$ $  6+ "`p   d* 0  6`+ "`p  + f*"0  6\+ "` + f*"0B  s *޽h ? 333380___PPT10.(г% 0 zr` (    09/ P   / P*    0>/    / R*  d  c $ ?  /  0A/  0 / RClick to edit Master text styles Second level Third level Fourth level Fifth level!     S  6F/ _P  / P*    6K/ _  / R*  H  0޽h ? 3380___PPT10. P P(    0H P    X*   0     Z*   6 _P   X*   6X _   Z* H  0޽h ? 3380___PPT10.1 0  *(  x  c $8/  p / r  S p/  `   / H  0޽h ? 33___PPT10i.ܿ=+D='  = @B +}  0 $$(  $r $ S ~/    / r $ S ~/    / H $ 0޽h ? 3333___PPT10i.j+D='  = @B +}  0 8$(  8r 8 S /    / r 8 S /   / H 8 0޽h ? 3333___PPT10i. +D='  = @B +}  0  $(   r  S /    / r  S /    / H  0޽h ? 3333___PPT10i.D+D='  = @B +  0 <(  <r < S D)     r < S +    X < 0F 7 H < 0޽h ? 3333___PPT10i.0sV+D='  = @B +  0 `X@(  @x @ c $Ħ/    / x @ c $/   / X @ 0- X @ 0  H @ 0޽h ? 333380___PPT10.0sV$  0 D$(  Dr D S /    / r D S /   / H D 0޽h ? 333380___PPT10.J$  0  H$(  Hr H S @/    / r H S /   / H H 0޽h ? 333380___PPT10. $  0 0L$(  Lr L S P/    / r L S (/   / H L 0޽h ? 333380___PPT10. $  0 @P$(  Pr P S /    / r P S /    / H P 0޽h ? 333380___PPT10.`P0  0 PT0(  Tx T c $$     x T c $D%     H T 0޽h ? 333380___PPT10.`P0  0 p\0(  \x \ c $1     x \ c $t2     H \ 0޽h ? 333380___PPT10.`P0  0 d0(  dx d c $|'     x d c $P!     H d 0޽h ? 333380___PPT10.`P$  0 h$(  hr h S W     r h S X 7   H h 0޽h ? 333380___PPT10.`ɳ$  0 l$(  lr l S $i     r l S i    H l 0޽h ? 333380___PPT10.B2  0 p2(  pr p S k SY    p S \t    H p 0޽h ? 333380___PPT10.;=>  0 x>(  xx x c $z SY    x c $+    H x 0޽h ? 333380___PPT10.;=>  0 |>(  |x | c $d SY    | c $`    H | 0޽h ? 333380___PPT10.;=J  0 J(  x  c $ SY     c $    &xSx yH  0޽h ? 333380___PPT10.;=J  0 J(  x  c $d SY     c $    &xSx yH  0޽h ? 333380___PPT10.;=J  0 J(  x  c $ SY     c $    &xSx yH  0޽h ? 333380___PPT10.;=J  0  J(  x  c $l S     c $    &xSx yH  0޽h ? 333380___PPT10.;=J  0 0J(  x  c $|+ SY  +   c $+   + &xSx yH  0޽h ? 333380___PPT10.;=$  0 t$(  tr t S      r t S     H t 0޽h ? 333380___PPT10.fQ  0 p(  X  C    /  S [/ 0  /  H  0޽h ? 3380___PPT10. JGr|eM3BW"zZ_\ bdgj>ljnpru>wjy{}\RgS61Oh+'03 hp    $,$Evaluating organisational contextdrduntBlends cmmiller17Microsoft Office PowerPoint@f*@ @ͣG2g  ZQ  y--$xx--'33--$,4 4 ,,--'-::--$ , 5 5 , ,--ff--$ , 5 5 , ,----$ , 55, ,----$,55,,----$,55,,----$,55,,---'--$4<<44--'---$ 3 <<3 3--X--$3<<33--݌--$3<<33----$3<<33----$3<<33----$3<<33----$3<<33---'QA   2( ~~ll[[JJ99%%rraaOO>>--xxffUUDD5678&'() ~?0123 !"4$|+,-./}?&'() *}? !"#$%~??  '--$ * = = * *--'---$89988-- --$89988--"""--$89988--$$$--$89 9 88--&&&--$ 8 9 9 8 8--)))--$ 8 9 9 8 8--+++--$ 8 9 9 8 8-------$ 8 998 8--000--$89988--222--$89988--444--$89988--666--$89988--888--$89988--:::--$89988--===--$89988--???--$89988--AAA--$89988--CCC--$89988--EEE--$89988--HHH--$89988--JJJ--$89988--LLL--$89988--NNN--$89988--QQQ--$89988--SSS--$89 9 88--UUU--$ 8 9!9!8 8--WWW--$!8!9"9"8!8--YYY--$"8"9#9#8"8--[[[--$#8#9$9$8#8--]]]--$$8$9%9%8$8--___--$%8%9&9&8%8--aaa--$&8&9'9'8&8--ddd--$'8'9(9(8'8--fff--$(8(9)9)8(8--hhh--$)8)9*9*8)8--jjj--$*8*9+9+8*8--lll--$+8+9,9,8+8--nnn--$,8,9-9-8,8--ppp--$-8-9.9.8-8--sss--$.8.9/9/8.8--uuu--$/8/90908/8--www--$0809191808--zzz--$1819292818--|||--$2829393828--~~~--$3839494838----$4849595848----$5859696858----$6869797868----$7879898878----$8889999888----$9899:9:898----$:8:9;9;8:8----$;8;9<9<8;8----$<8<9=9=8<8----$=8=9>9>8=8----$>8>9?9?8>8----$?8?9@9@8?8----$@8@9A9A8@8----$A8A9B9B8A8----$B8B9C9C8B8----$C8C9D9D8C8----$D8D9E9E8D8----$E8E9G9G8E8----$G8G9H9H8G8----$H8H9I9I8H8----$I8I9J9J8I8----$J8J9K9K8J8----$K8K9L9L8K8----$L8L9M9M8L8----$M8M9N9N8M8----$N8N9O9O8N8----$O8O9P9P8O8----$P8P9Q9Q8P8----$Q8Q9S9S8Q8----$S8S9T9T8S8----$T8T9V9V8T8----$V8V9W9W8V8----$W8W9Y9Y8W8----$Y8Y9Z9Z8Y8----$Z8Z9[9[8Z8----$[8[9]9]8[8----$]8]9^9^8]8----$^8^9`9`8^8----$`8`9a9a8`8----$a8a9c9c8a8----$c8c9d9d8c8----$d8d9f9f8d8----$f8f9h9h8f8----$h8h9j9j8h8----$j8j9l9l8j8----$l8l9n9n8l8----$n8n9p9p8n8----$p8p9r9r8p8----$r8r9t9t8r8----$t8t9w9w8t8----$w8w9y9y8w8----$y8y9{9{8y8----${8{998{8----$89988----$89988----$89988----$89988----$89988----$89988---'@Tahoma-. I2 pD,Program Evaluation Unit, ."System;-@Tahoma-. (2 tEUniverity of Melbourne.-@Tahoma-. 332 ( Evaluating n.-@Tahoma-. 332 (Jorganisational.-@Tahoma-. 332 4context.-@Tahoma-. +2 J'Measurement approaches: .-@Tahoma-. 2 P? evaluation.-@Tahoma-. 2 _>David .-@Tahoma-.  2 _RDunt.-՜.+,0     $ On-screen ShowThe University of Melbourneޒ` ArialTahoma WingdingsBlends"Evaluating organisational contextSlide 2Slide 3Slide 4'Outcomes movement - point of departure"Outcomes movement - first problemWith a black box:-Slide 8Slide 9Program logicProgram theory - Mechanisms Program theory - Context In summaryEA second problem for the Outcomes movement embedded in this analysis Slide 15Realistic evaluation in KURealistic evaluation in KUCRealistic evaluation - measuring organisational context & change 1CRealistic evaluation - measuring organisational context & change 2?A few exceptions where may be important to build causal models?A few exceptions where may be important to build causal models-GP integration with the health care system 1-GP integration with the health care system 2-GP integration with the health care system 3  Fonts UsedDesign Template Slide Titles _㺒0cmmillercmmiller  !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIKLMNOPQRSTUVWXYZ[\]^_`abcdfghijklnopqrstwRoot EntrydO)Current UsermSummaryInformation(J(4PowerPoint Document(ޒDocumentSummaryInformation8e