RT Journal Article
SR Electronic
T1 16 A friendly accessible description of the ‘L-test’ – measuring (dis)information in incomplete incident reporting
JF BMJ Health & Care Informatics
JO BMJ Health Care Inform
FD BMJ Publishing Group Ltd
SP A10
OP A10
DO 10.1136/bmjhci-2022-FCIASC.16
VO 29
IS Suppl 1
A1 Keith Reid
YR 2022
UL http://informatics.bmj.com/content/29/Suppl_1/A10.1.abstract
AB Objective Incomplete incident reporting is concerning. England’s Mental Health Units Use of Force Act 2018 (Seni’s Law), responding to deaths and incomplete reporting, will mandate central restraint reporting per-person including ethnicity. ‘L’ is a proposed test for disinformation, i.e. ‘false surprise’ regarding true reports. Information, or ‘surprise’, is measurable as H = -log(p) ‘bits’, as defined by Shannon (1948).The author explains his conjectured ‘L-test’, in a friendly accessible way. It is generalisable from incomplete restraint reports to other incomplete centralised safety reports. L is increased if complete reports seem falsely surprising consequent to noise from incomplete reports.Methods Incident registers and minimum data sets are ubiquitous. Each hospital reports diverse incidents alongside measures of size or need. Notionally then data may include a) restraints; b) detentions … m) bed days n) injuries.L postulates that each hospitals’ report of {a, b, ... m, n}, implies signals of ratios (log a/log b), (log a/log m)… which each can be received from the set of reports and combined to estimate e.g. a typical ratio of safety events per-patient per-month. Omissions are noise.Procedure:Split the ordered list of complete report estimates into alternate halves E ‘even’ and O ‘odd’.Derive a probability p(E~O) that E and O are similar using Mann-Whitney U test, approaching p(E~O) =1.0 for large similar E and O. The test tolerates non-normally distributed estimates.Calculate h(E~O) information as -log(p(E~O)), approaching zero as O and E seem unsurprisingly similar.Construct a noisy odd group ‘NO’ made of O mixed with estimates from incomplete reporters.Calculate h(E~NO) information, approaching high values as incomplete reporters make E seem falsely surprising.L is the proportional increase in h(E~O) due to noise: h(E~NO) – h(E~O)L = _____________________ h(E~O)Results Estimate signals support funnel plotting, scatter plotting, and coefficients of determination (R^2) as a measure of correlation.The author will show that omissions (allowing for size and Poisson distribution) can be obvious on visual inspection of funnel and scatter plots and aid categorisation.Where the estimates follow a normal distribution among reasonably complete reporters, this can be used to plot a typical ratio and infer incidents, with confidence intervals, even in null reporters, from measures of size and need.Funnel plots from safety reports may have interesting properties such as innate asymmetry; they may reflect institutional-social processes such as regulation and closure as much as academic processes such as purported ‘publication’ bias.H varies with the effect of incomplete reports and has other desirable features such as being zero when there are no omissions.Conclusion Omitted reports have a measurable effect upon the standing of complete reports.The author responds to this observation quantitatively, showing the roots and reasoning behind their conjectured ‘L-test’, in a friendly accessible way, with reference to papers under submission, other public data, and toy data sets.In summary, L can tell investigators which incomplete reports skew the overall picture most.In a context of restricted resource, regulatory efforts could concentrate on the omissions which have the most distorting effect – the biggest L score.