Utility of Bland-Altman Plot in the Assessment of Inter-observer Variability for Internal Quality Control in Semen Analysis: A Cross-sectional Study

1. Associate Professor, Department of Pathology, Pondicherry Institute of Medical Sciences, Pondicherry, India. 2. Associate Professor, Department of Pathology, Karpaga Vinayaga Institute of Medical Sciences and Research Center, Maduranthakam, Tamil Nadu, India. 3. Associate Professor, Department of Biochemistry, Indira Gandhi Medical College and Research Institute, Puducherry, India.

Abstract

Introduction: Assessment of inter-observer variability is an essential component of quality control in semen analysis. The authors compared Bland-Altman (BA) plot, Intraclass correlation coefficient and Student’s paired t-test to determine which was the most feasible method for statistical analysis of internal quality control in their laboratory and the reason for the same.

Aim: To assess inter-observer variability in sperm concentration and motility in fresh samples using Bland-Altman plot, Student’s paired t-test and Intraclass Correlation Coefficient (ICC), as a part of internal quality control.

Materials and Methods: A cross-sectional observational study was conducted in the South of India, Puducherry, over a period of six months from 1^st January 2020 to 30^th June 2020. As a part of internal quality control two assessors independently analysed sperm concentration, progressive and non-progressive motility and immotile spermson two aliquots of the samples tested by the manual method. Inter-observer variability was analysed using Bland-Altman plot, Students paired t-test and ICC. Data was analysed using Microsoft Excel® (2016), GraphPad Prism version 9, Mangold ICC calculator software and online ICC calculator at the website http://vassarstats.net/index.html

Results: Nineteen men were included in the study. The ICC coefficient showed good correlation for sperm concentration, progressive motility, immotile sperms with values of 0.77, 0.84, 0.95 respectively and moderate correlation for non-progressive motility with ICC=0.72. The p-value of Student’s paired t-test was above 0.05 for all parameters. There was no significant difference between the assessors for sperm concentration and motility using ICC and Student’s paired t-test, although the Student’s paired t-test does not measure agreement. Bland-Altman plot showed an occasional outlier for both parameters. The authors attributed different pockets of sampling as the reason for outlier in sperm concentration. In motility testing, the two outliers were a result of delayed reporting by one of the assessors resulting in decline in the progressive motile and an increase in the non-progressive motile and immotile sperms.

Conclusion: In a low throughput laboratory, fresh semen is a useful sample for daily quality control. Bland-Altman plot is an easy and effective tool for monitoring internal quality control when there are two assessors, obviating the need for complex statistical tests. Adopting simple yet effective methods for internal quality control, would encourage more laboratories to come into the ambit of quality assurance.

Intraclass correlation coefficient, Semen, Quality assurance, Sperm concentration, Sperm motility, Student’s paired t-test, WHO guidelines

Semen analysis is an essential test for the diagnostic evaluation of male infertility (1). It is often affected by physiological and analytical variabilities leading to imprecision in the results. In today’s world of automation, the test is still subject to manual processing and interpretation, thus, compounding the problem. The results may not be useful diagnostically or be actionable if there are significant differences in reports from laboratories (2). These challenges often make the utility of semen analysis questionable and the requirement of quality assurance inescapable (3).

Guidelines for the analysis of semen to ensure standardisation in reporting have been formulated by the WHO from 1980, when the first edition was published. Currently the 6^th edition brought out in 2021 is in use (4). Despite this, compliance to the guidelines have a wide range across the globe ranging from 7-84% (5),(6),(7). Quality Control (QC) was incorporated in the World Health Organisation (WHO) manual from the 3^rd edition and expanded significantly in the 4^th edition (4). Inter-observer variability is an important cause for imprecision, and thus, should be rigorously evaluated as a part of Internal Quality Control (IQC) (3),(8). Methodologies for the same have been provided in the WHO manual (1),(9). Only a few studies have been published specifically on inter-observer variability in the evaluation of fresh samples with observations of significant variation in the motility as against sperm concentration (3),(10),(11). Thus, in the present study, the authors aimed to assess inter-observer variability in sperm concentration and motility in fresh samples using Bland-Altman plots, Student’s paired t-test and ICC, as a part of internal quality control. The objective of the study was to analyse the utility of Bland-Altman plot in the assessment of inter-observer variability for internal quality control in semen analysis.

Material and Methods

A cross-sectional observational study was conducted in the south of India, Puducherry, over a period of six months from 1^st January 2020 to 30^th June 2020. The study population consisted of patients undergoing semen analysis for routine diagnostic purpose between 1^st January 2020 to 30^th June 2020. Procedures followed in the study were in accordance with the ethical standards of the Institutional review and ethical board and with the Helsinki Declaration of 1975 that was revised in 2013 (IEC number – MGMCRI/IRC/06/2020/47/IHEC/171).

Inclusion and Exclusion criteria: Semen samples which were analysed by two assessors on the same occasion were included in the study, and those analysed by just one assessor were excluded.

Sample size: The sample size was calculated as 92, based on an estimate of 120 semen analysis cases over six months with 95% CI, population proportion of 50% and margin of error as 5%. The basis for sample size was also on published studies on intra-laboratory inter-observer variability mentioned in (Table/Fig 1) (3),(8),(10),(11).

Study Procedure

The assessors were trained consultant pathologists with 6 years and 4 years of experience, respectively, in reporting semen samples. The semen samples included azoospermic and oligospermic samples, as internal quality control should encompass all possible ranges. Sperm concentration and sperm motility were the two parameters included in the study to assess for inter-observer variability. No patient intervention was used. Privacy and confidentiality of the patients was maintained throughout the study.

All semen samples received in the laboratory where this study was undertaken were routinely analysed for volume, liquefaction time, features on gross inspection, sperm concentration, sperm motility, morphology, round cell and pus cell count. It was the regular practice to evaluate semen analysis by either one or two consultants. The methodology for counting and reporting sperm concentration and sperm motility was based on the WHO 2010 guidelines (9).

Sperm motility: Samples were kept at room temperature. On observing liquefaction, 10 μL of the sample was pipetted on to a clean glass slide and a standard sized 22 mm × 22 mm coverslip was placed on it. A minimum of 200 sperms were counted at 400X for motility and graded as progressive, non-progressive and immotile. From each sample two aliquots were prepared by the same technician, each of which was analysed by one assessor. The WHO 2010 guideline advises the use of phase contrast microscopy (9). However, due to non-availability of this equipment in the study Institute, a regular bright field microscope was used with adjustments of condenser, diaphragm diameter and light to give the effect of a contrast.

Sperm concentration: Two aliquots were prepared from all the samples by the same technician and each aliquot was analysed by one assessor. Semen samples were diluted using a micropipette with a commercial diluent immediately upon liquefaction and the dilution required was decided based on the wet mount preparation as per the WHO 2010 guidelines (9). The preparation of the two aliquots and analysis by the assessors were done simultaneously at the same time. Improved Neubauer chamber was used for counting sperms. Appropriate number of squares were counted and the concentration was determined as N/mL.

The two assessors were blinded to the results of each other. Thus, for each parameter of a sample two values were generated, one by each assessor.

Statistical Analysis

Data was compiled and analysed for inter-observer variability in the two parameters – sperm concentration, sperm motility. Bland-Altman plot (BA plot) with 95% limit of agreement and Students paired t-test were used to assess inter-observer variability, as prescribed by the WHO 2010 for internal quality control (9). As an additional test for inter-observer variability, the ICC {(type - two-way Analysis of Variance (ANOVA)}, random effects model with consistency) was calculated. A p-value was set at <0.05 and confidence interval of 95% on either side. Data was analysed using Microsoft excel (2016), GraphPad Prism version 9, Mangold ICC calculator software and online ICC calculator at the website http://vassarstats.net/index.html. Data was checked for normality and was normalised using logarithmic scaling.

Results

Twenty-one patients underwent semen analysis for infertility during this period, out of whom 19 were included in the study on fulfilling the inclusion and exclusion criteria. The age ranged from 22 years to 47 years with a median of 29 years and 6 months. The volume of semen collected ranged from 0.5 mL to 3 mL with a mean of 1.5 mL and median of 1.25 mL. The gross analysis of all the samples was unremarkable, with normal colour, viscosity and absence of frank blood or pus. The liquefaction time for the samples ranged from 15 minutes to more than 60 minutes, with a mean of 40 minutes and a median of 30 minutes. Cases which failed to liquify at 60 minutes were subject to gentle pipetting using a 19 G needle attached to a syringe to induce liquefaction.

Two different assessors (assessor 1 and assessor 2) analysed sperm concentration and sperm motility. The descriptive statistics is given in (Table/Fig 2). WHO 2010 guidelines define the lower 5^th percentile of sperm concentration as 15 million/mL (9). In the present study 10 cases had concentration lower than this given by both assessors. Three cases had total motile sperms less than the lower 5^th percentile of 39% as defined by WHO 2010. Progressive motility values below the WHO 2010 5^th percentile of 32% were seen in 10 cases by both assessors. More than 98% of sperms showed normal morphology in all cases. Bland-Altman Plot, Students paired t-test, ICC were done to assess inter-observer variability. Results of the analysis are presented in (Table/Fig 3),(Table/Fig 4),(Table/Fig 5),(Table/Fig 6),(Table/Fig 7).

All three statistical tests showed good to moderate agreement between the two assessors for both parameters (Table/Fig 2). The ICC values of 0.77, 0.84, 0.95 for sperm concentration, progressive motility and immotility respectively signified good correlation, whereas a value of 0.72 for non-progressive motility signified moderate correlation. Students paired t test showed no significant difference between the two assessors. In the BA plots most of the values were well within the 95% limits of agreement and showed clustering around line of zero difference with very little bias (Table/Fig 4),(Table/Fig 5),(Table/Fig 6),(Table/Fig 7). However, there was an occasional outlier in each plot. In the BA plot for sperm concentration one sample (number 5) was an outlier (mean=18M, difference=24M) (Table/Fig 4). The observed discrepancy may have resulted from heterogeneous sample distribution between aliquots. BA plots for motility showed a consistent outlier in all the three motility plots (sample 18) (Table/Fig 5),(Table/Fig 6),(Table/Fig 7). Another sample (sample 15) was an outlier only in the BA plot for immotile sperms. This finding may be attributable to a delay in sample analysis by one of the assessors resulting in decline in the progressive motile and an increase in the non-progressive motile and immotile sperms. It was also noted that the outliers in all the plots had normal liquefaction time and their position in the plot were not a result of prolonged liquefaction or induction of liquefaction.

Discussion

Guidelines for quality assurance in semen analysis were incorporated in the WHO manual from the third edition onward and expanded significantly from the fourth edition (4). Precision testing is an important aspect of quality assurance, of which inter-observer variability is an essential component. The various types of QC samples prescribed for precision testing are either stored QC material (purchased, laboratory made, stored samples) or fresh samples. Both types of QC samples have advantages and disadvantages inherent to them and the testing process. The WHO guidelines also recommend fresh samples over others for IQC of concentration and motility testing (1),(9). The biggest advantage of using fresh samples for IQC is that it reflects nearly all the variabilities of testing a real sample. The various statistical methods for assessing inter-observer variability suggested in the WHO manuals are - Xbar chart, S chart, BA plot, Youden plot, two-way ANOVA, Student’s paired t test and monitoring monthly means. All methods can be used in the analysis of fresh samples except for the Xbar chart which is used only for stored QC material (1),(9).

Studies on precision testing can be grouped in two depending on the nature of QC material used – either fresh sample or stored QC. Few studies have used fresh samples for investigation on inter-observer variability in concentration and motility (Table/Fig 8) as in the current study (3),(10),(11). Discarded semen was used in one study (Table/Fig 1) (8). In contrast, many authors have used EQA samples for precision testing in the form of inter laboratory comparison (Table/Fig 8) (5),(8),(12),(13),(14). These studies also found acceptable inter-observer/laboratory variability similar to our results, attributable to standardisation of procedures of testing. It is of note that the participants in these studies hadn’t uniformly followed the WHO guidelines. Nevertheless, they found an improvement in the precision of concentration following adoption of the guidelines. This implies, following a standardised protocol for testing improves precision (5),(8),(12).

Unlike concentration or morphology, analysis of motility brings a special challenge because of its rapid decay with time. Studies on EQA programme have used video recordings of motility for rating inter-observer variability (8),(12),(13),(14). However, this is not an ideal replica of real-life scenario as recordings can be replayed or played at slower speed, and errors arising from sample preparation are circumvented (8),(12),(15). Also, this model may not be useful for daily IQC. Hence, the authors feel fresh samples are the best for precision testing of motility.

Studies can also be grouped into two based on the goal of statistical analysis: i) to test if the variability between the observers/laboratories is statistically significant (Student’s Paired t test, ANOVA, Chi-square test) (16),(17),(18), or, ii) to evaluate the extent of variability or reliability between the observers/laboratories - as plots (BA plot, S chart, X bar chart, Youden plot) (1),(19), or as numerical values (CV%, ICC) (17),(20),(21) . Statistical tests for significance of difference between groups can only be done periodically after a significant number of data is generated. The same could be said of ICC, Youden plot, Xbar and S chart (1),(20),(21). Thus, their feasibility for day-to-day assessment of IQC is questionable in laboratories with a small sample load. Periodical assessment may fail to detect random and systematic errors, thus, allowing samples to be erroneously reported in the interim period.

In studies mentioned earlier the most common measure of variability was the CV%, even though it is not prescribed by the WHO manual (3),(5),(8),(10),(11),(12),(13),(14). The authors used ICC as an alternative to CV%, (albeit this was also not prescribed in the WHO manual) as CV% may not be meaningful when there are only two observers, unless an acceptable range is available for comparison. To the best of our knowledge there are no clear guidelines on the minimum number of data required for calculation of a meaningful CV% in semen parameters.

In the present study the authors used one statistical test from each of these groups – Student’s paired t-test, BA plot and ICC. The findings were; BA plot can be analysed on a daily basis using an excel sheet. This allows for using a ‘bias’ and ‘limits of agreement’ in real time for every sample added to the chart. The BA plot demonstrated characteristics comparable to moving-average chart monitoring approaches, simulated to an extent in the Levy Jenning chart. BA plot provided us with the ease of interpretation based on visual inspection, thus obviating the need for complex statistical tests every day. BA plot is also capable of detecting different types of errors like systematic, random and proportional (19), although this wasn’t investigated. Thus, it was inferred, BA plot was the most feasible method to accommodate usage of ‘fresh sample’ as a part of ‘daily IQC’ to be analysed by ‘two observers’.

In laboratories like ours where the numbers of semen analysis are usually only up to two cases per day, financial and logistic justification may not permit usage of purchased QC or preparation of QC material in-house. In our country no EQA program is available for semen analysis, unless procured from an international organisation. The authors assume these problems in implementation of a quality assurance program for semen analysis may be faced by many around the world. It was addressed by using fresh diagnostic samples from patients as IQC in the study and use of BA plot to conveniently analyse the results. Implementation of this approach improved procedural standardisation within the laboratory, thus allowing for a robust IQC to be accepted and adopted by the entire team.

Limitation(s)

The present study had a limitation. The study period was the year COVID-19 pandemic struck, and probably conception was the least of worries for humanity. Thus, only 21 patients underwent semen analysis for infertility during this period, out of whom 19 were included in the study on fulfilling the inclusion and exclusion criteria.

Conclusion

Constraints of funding and logistics often direct the choice of IQC and EQA. The factors one may consider before choosing the process of IQC are - type of IQC sample, number of observers and type of statistical test. In a low throughput laboratory, daily IQC testing can easily be performed using fresh samples available for testing, and analysed using the BA plot. Implementation of simple internal quality-control procedures may encourage wider adoption. This may segue into improvement in reporting standards based on self-reflection, and eventually to participation in EQA programme, thus fulfilling the mandate of quality assurance by WHO guidelines.

DOI and Others

DOI: 10.7860/JCDR/2026/84714.24291

Date of Submission: Oct 16, 2025
Date of Peer Review: Jan 08, 2026
Date of Acceptance: Jun 17, 2026
Date of Publishing: Sep 01, 2026

AUTHOR DECLARATION:

• Financial or Other Competing Interests: None
• Was Ethics Committee Approval obtained for this study? Yes
• Was informed consent obtained from the subjects involved in the study? Yes
• For any images presented appropriate consent has been obtained from the subjects. Yes

PLAGIARISM CHECKING METHODS:

• Plagiarism X-checker: Nov 11, 2026
• Manual Googling: Jun 13, 2026
• iThenticate Software: Jun 15, 2026 (2%)

ETYMOLOGY: Author Origin

EMENDATIONS: 6