Get Permission Sengupta, Tej, Bansal, Saravanan, Raju, Reddy, and Reddy: Identifying early surrogate markers for blood culture positive sepsis in an urban NICU-A retrospective observational study


Introduction

Sepsis is a leading cause of morbidity and mortality in neonates and is recognised as a global health challenge.1, 2, 3 Its incidence in India ranges from 20-30/1,000 live birth of which blood culture-proven sepsis occurs in 6.2-60 %.4 Sepsis contributes to a quarter of the neonatal deaths in India.5 Therefore, it constitutes a significant health burden.6

Blood culture, the gold standard for diagnosis of neonatal sepsis is time consuming, expensive and yields positive results in limited cases due to inadequate sampling, low yield of organism, prior antibiotic exposure etc.7, 5 Thus the diagnosis of sepsis may be missed out in these neonates who are often empirically treated. Establishing alternate diagnostic markers which are easily available, give quick results and are inexpensive will enable early appropriate treatment and reduce injudicious antibiotic usage thus curbing the emergence of multidrug-resistant micro-organisms.

Need of the study

This study was undertaken to ascertain early, reliable and inexpensive surrogate markers to blood culture for early detection of neonatal sepsis which would enable early and appropriate treatment and avoid unnecessary antimicrobial therapy.

Aim and Objectives

To ascertain haematological parameters which are reliable adjuvants to blood culture for diagnosing Neonatal Sepsis.

Materials and Methods

A retrospective cohort study was undertaken in neonates hospitalized in an urban, tertiary care Neonatal Intensive Care Unit (NICU) over a period of 24 months viz May 2017 to April 2019. Data was obtained from medical records of neonates hospitalized with a diagnosis of neonatal sepsis. Clearance from Institutional Ethics Committee and Hospital Research Committee was obtained. Given retrospective nature of study, waiver of informed consent was approved for the study.

During the study period, neonates hospitalised with perinatal risk factors and those with clinical features suggestive of sepsis were enrolled in the study. Those infants hospitalised with a diagnosis of sepsis but in whom complete laboratory investigations could not be carried out were excluded.

For all enrolled neonates, a detailed maternal history including antenatal, natal & postnatal data was recorded. Details of mode of delivery, resuscitation and relevant postnatal history prior to hospitalization was obtained. The risk factors of infection i.e. maternal, natal or postnatal factors were noted. The other parameters collected included gestational age, birth weight, and gender. Detailed clinical examination including vital parameter recordings, general and systemic examination was carried out and recorded on admission and during course of hospitalization.

Laboratory investigations done included Blood Culture, Complete Blood Picture (CBP) and C Reactive Protein (CRP). Blood was collected for culture under sterile precautions at admission and during trough phase in case of prior exposure of antibiotics. It was inoculated into the BD Bactec 9050 machine. In case of growth of organism, sub-culturing was done, organism identified and antibiotic sensitivity obtained.

For haematological studies, 1.0 ml of blood was collected under sterile precautions in an EDTA vacutainer and analysed in a cell counter viz Mindray BC 5150 5 Parts Analyser. Haemoglobin, hematocrit, total & differential leucocyte counts (TLC & DLC), platelet count, immature neutrophil count, IT (immature to total neutrophil) ratio were obtained. Absolute neutrophil count was calculated. Morphology for degenerative changes viz toxic granulation, vacuolation and Dohle bodies were reported by a pathologist on examination of peripheral smear. Quantitative CRP was measured.

The course in hospital and outcome viz hospitalization duration, discharge/death was recorded along with laboratory investigation results in a pre-designed, structured proforma. These were the subjected to statistical analysis.

Statistical analysis

Data were analyzed using Stata14 (StatCorp, College Station, TX). Categorical variables were compared using the Yates-corrected chi-squared test and continuous variables using the Mann-Whitney U test. A p-value <0.05 was considered statistically significant. Variables hypothesized to be associated with culture positive neonatal sepsis were first tested in univariate analysis using logistic regression. A multivariable logistic regression model was then performed to evaluate the independent effect of significant variables using backward selection.

Results

During the study period viz between April 2017-May 2019, incidence of Neonatal Sepsis was 14.9% (212/1420). Out of the 212 patients diagnosed to have Neonatal Sepsis, 182 patients met the inclusion criteria and were enrolled in the study. Clearances including from the Institutional Ethics Committee and Hospital Research Committee were obtained. Given retrospective nature of study, waiver of informed consent was approved for the study. Data was recorded on a master sheet and analyzed.

Clinical data

Maternal data

The maternal data of the study population is shown in Table 1. Of these, 31 were born by vaginal delivery and 151 by Caesarian section. It was seen that 45.2% of the babies born vaginally developed culture positive sepsis as compared to 33.1% of those born operatively. There were only 8 cases of maternal pyrexia and all of their babies were blood culture negative. Prolonged rupture of membranes >18 hours was seen in 20 cases of whom 9 babies had culture proven sepsis.

Babies born by NVD (45.1%) were more likely to develop culture positive neonatal sepsis in our cohort than those delivered through LSCS (33.1%) but the finding was not statistically significant. Also PROM (>18 Hours) were more likely to be associated with culture positive neonatal sepsis in our cohort but the finding did not reach levels of statistical significance.

Intrauterine crowth pattern

Of the study population 155 babies were appropriate for gestation age, 22 were small for gestational age and 5 large for gestational age as shown in Table 2 below. SGA babies (40.9%) were more likely to develop culture positive neonatal sepsis in our cohort than AGA (35.4%) or LGA babies (0%).

Neonatal demographics

The clinical data of the neonates in the study are shown in Table 3 . There was an increasing risk of culture positive neonatal sepsis with decreasing gestation from term (33.3%) to late preterm (34.9%) to moderate preterm (35%) to very preterm (40%) infants in our cohort but the finding was not statistically significant. Similarly, there was an increasing risk of culture positive neonatal sepsis from >2500 grams at birth (31.7%) to 1500-2499 grams at birth (35.9%) to 1000-1499 grams at birth (37.9%) to <1000 grams at birth (50%) infants in our cohort but the finding was not statistically significant. Thus there was an increasing trend to blood culture positive sepsis with reducing gestational age as well as birth weight.

The percentage risk of neonatal sepsis (suspected/probable/culture positive) in males admitted in NICU was 12.2% and 13.8% in females. However, the risk of culture positive neonatal sepsis among those admitted for sepsis was found higher in females (42.1%) than males (30.2%) which was statistically significant. On univariate analysis, there was a trend towards increased association of culture positive neonatal sepsis with female gender (Odds ratio [OR] 1.68; 95% confidence interval [CI], 0.92-3.11, p value 0.1) as compared to males.

Clinical features

In the study cohort, it was seen that the most common symptoms were lethargy, bleeding manifestations and seizures. There occurrence is shown in Table 4 below. It was seen that while all three symptoms occurred more in babies with proven sepsis, lethargy was the commonest and reached levels of statistical significance.

Laboratory investigations

Hematological parameters

In this study, the significant hematological parameters predicting culture positive sepsis were absolute neutropenia, thrombocytopenia, IT ratio and degenerative changes in neutrophils as shown in Table 5.

All of these parameters were effective surrogate markers which reached levels of high statistical significance.

CRP positive

In our study population, 54.9% of the babies had significantly raised CRP levels. Of the culture positive babies, 65.6% had raised CRP as compared to 49.15% in the culture negative babies as shown in the Table 6.

This finding of raised CRP in babies with blood culture positive sepsis reached levels of statistical significance.

Blood culture positivity

Of the study population 64 (35.16%) of the cases had blood culture positive sepsis. The most common organism that grew in blood culture was Klebsiella species (67.2%) (n=43) followed by Pseudomonas aerogenosa (n=8), Staphylococcus aureus (n=8), Candida (n=3), E coli (n=1) and Streptococcus agalactae (n=1) as shown in Table 7.

Univariate analysis

All of the factors studied were subjected to univariate analysis to determine the significant surrogate markers of sepsis in the neonate as shown in Table 8.

On the univariate analysis,  lethargy (OR 2.33; CI 1.2-4.53),   CRP positive (OR 1.98; CI 1.05-3.71),  absolute neutrophil count <1500/mm3 (OR 4.99; CI 1.17-30.66), immature to total neutrophil ratio (ITR) >0.2 (OR 14.37; CI, 1.73-119.58), platelet count < 150,000/mm3   (OR 16.34; CI, 7.58-35.25) and  cytotoxic degenerative changes in neutrophils (OR 3.34; CI, 1.74-6.39), were associated with culture positive neonatal sepsis and were statistically significant. There was a trend towards association of female gender (OR 1.68; CI 0.91-3.11) with culture positive neonatal sepsis. 

Multivariate Analysis

All of the factors found significant in univariate analysis were subjected to multivariate regression analysis to determine the significant surrogate markers of sepsis in the neonate as shown in Table 9.

On multivariable logistic regression analysis, female gender (Odds ratio [OR] 2.59; 95% confidence interval [CI], 1.1-6.0), immature to total neutrophil ratio (ITR) >0.2 (OR 16.3; CI, 1.5-176.8), platelet count less than 150,000/mm3   (OR 15; CI, 6.4-35.2) and cytotoxic degenerative changes in neutrophils (OR 4.34; CI, 1.82-10.4), were associated with culture positive neonatal sepsis. There was a trend towards higher culture positive neonatal sepsis in babies with absolute neutrophil count <1500/mm3 but it was not statistically significant.

Table 1

Maternal Demographics-Study Population

Mode of Delivery 

Study Population

Total

Statistics

OR

CI

p value

Culture Nos

Positive %

Culture Nos

Negative%

VD 

14

45.2

17

54.8

31

1.66

0.75-3.64

0.2

LSCS 

50

33.1

101

66.9

151

Total

64

118

Maternal Pyrexia

Present

0

0

8

100

8

ID

ID

ID

Absent

64

100

110

93.2

174

PROM>18 hours

Present

9

45

11

55

20

1.59

0.62-4.07

0.33

Absent

55

33.9

107

66.1

162

Table 2

Intrauterine growth pattern in study population

Intrauterine Growth Pattern

Study population

%

Culture positive

Culture negative

Total

OR

CI

P Value

AGA

55

100

155

35.4%

Ref #

Ref #

Ref #

SGA 

9

13

22

40.9%

1.26

0.51-3.13

0.62

LGA

0

5

5

0%

ID*

ID*

ID*

TOTAL

64

118

182

35.1

Table 3

Neonatal demographics-study population

Total study cases

Study population

Statistical Analysis

Gestational Age

Suspected Sepsis

Probable Sepsis

Culture Positive Sepsis

%Culture Positive

OR

CI

P Value

>37 weeks

649

84

24

32

28

33.3

Ref#

Ref#

Ref#

34-37 weeks

481

43

16

12

15

34.9

1.07

0.49-2.32

0.86

32-34 weeks

159

20

8

5

7

35

1.08

0.39-3.0

0.89

<32 weeks

131

35

12

9

14

40

1.33

0.59-3.01

0.49

Birth Weight (grams)

2500&>

696

79

23

31

25

31.7

Ref#

Ref#

Ref#

1500-2499

548

64

25

16

23

35.9

1.21

0.61-2.43

0.59

1000-1499

126

29

9

9

11

37.9

1.32

0.54-3.2

0.54

<1000

50

10

3

2

5

50

2.16

0.57-8.14

0.26

Gender

Female

550

76

25

19

32

42.1

1.68

0.91-3.11

0.1

Male

870

106

35

39

32

30.2

Total

1420

182

60

58

64

35.1

Table 4

Clinical Featuresvs Culture Positive Sepsis

Symptom

Study Population

Statistics

Total

Cases %

Culture Positive

Culture Negative

OR

CI

P Value

Lethargy

111

60.98

47(42.3%)

64(57.7%)

2.3

1.2-4.53

0.01

Bleeding Manifestations

7

3.84

4(57.1%)

3(42.3%)

2.5

0.55-11.7

0.22

Seizures

11

6.04

4(36.4%)

7(63.6%)

1.05

0.29-3.7

0.93

Table 5

Hematological Parameters

Total

Cases %

Study population

Statistical analysis

Culture positive

Culture negative

OR

CI

p value

ANC COUNT <1500

8

4.39

6(75%)

2(25%)

6

1.17-30.66

0.03

Platelet count <150,000/mm3

58

28.6

44(75.9%)

14(24.1%)

16.3

7.58-35.3

<0.001

I:T Ratio>0.2

8

4.39

7(87.5)

1(12.5)

14.4

1.73-119.6

0.01

Degenerative Changes In Neutrophil

94

51.64

45(47.87)

49(52.13)

3.34

1.74-6.38

<0.001

Table 6

CRP positivity in study population

Study Population

Statistical Analysis

Culture positive

Culture negative

Total

% Culture Positive

OR

CI

p value

CRP Positive

Yes

42

58

100

42

1.98

1.05-3.71

0.03

No

22

60

82

26.8

Total

64

118

182

35.1

Table 7

Organisms growing in blood culture

Organism

Frequency (n)

%

Klebsiella species

43

67.2

Pseudomonas aerogenosa

8

12.5

Staphylococcus aureus

8

12.5

Candida albicans

3

4.7

E coli

1

1.6

Streptococcus agalactae

1

1.6

Table 8

Univariate analysis

Factors associated with culture positive neonatal sepsis

OR

CI

P value

aFemale gender  

1.68 

0.91-3.11

0.1

Term (>37 weeks) at birth

Ref #

Ref #

Ref #

Late preterms (>34-36 weeks) at birth

1.07

0.49-2.32

0.86

Moderate preterm (>32-34) weeks at birth

1.08

0.39-3.0

0.89

Very preterms (<32 weeks) at birth

1.33

0.59-3.01

0.49

>2500 grams at birth

Ref #

Ref #

Ref #

1500-2499 grams at birth

1.21

0.61-2.43

0.59

>1000-1499 grams at birth

1.32

0.54-3.2

0.54

<1000 grams at birth

2.16

0.57-8.14

0.26

aNVD vs LSCS

1.66

0.75-3.64

0.20

AGA

Ref #

Ref #

Ref #

SGA 

1.26

0.51-3.13

0.62

LGA

ID*

ID*

ID*

aMaternal fever

ID*

ID*

ID*

aPROM >18 Hours

1.59

0.62-4.07

0.33    

aLethargy 

2.33 

1.2- 4.53

0.01

aBleeding Manifestations

2.5

0.55-11.7

0.22

aSeizures 

1.05

0.29-3.7

0.93     

aCRP positive

1.98

1.05-3.71

0.03

aAbsolute neutrophil count <1500/mm3    

6

1.17-30.66

0.03

aImmature to total neutrophil ratio (ITR) >0.2

14.4

1.73-119.6

0.01

aPlatelet count less than 150,000/mm3  

16.3

7.58-35.3

<0.001

aCytotoxic degenerative changes in neutrophils

3.34

1.74-6.38

<0.001

Table 9

Multivariate Analysis

Factors associated with culture positive neonatal sepsis

Univariate Analysis

p value

Multivariable Analysis

p value

OR (95% Cl)

OR (95% CI)

Female gender  a

1.68(0.91, 3.11) 

0.1

2.59(1.1, 6.0)

0.03

Absolute neutrophil count <1500/mm3 a

6(1.17, 30.66)

0.03

6.83(0.67, 70.1)

0.10

Immature to total neutrophil ratio (ITR) >0.2 a

14.4(1.73, 119.6)

0.01

16.3(1.5,176.8)

0.02

Platelet count less than 150,000/mm3  a

16.3(7.58, 35.3)

<0.001

15(6.4,35.2)

<0.001

Cytotoxic degenerative changes in neutrophils a

3.34(1.74, 6.38)

<0.001

4.34(1.82, 10.4)

0.001

Abbreviations: OR, odds ratio; CI, confidence interval. a Variables in the table analyzed as categorical (Yes vs No) variables; 

Discussion

In this retrospective cohort study during the study period between April 2017-May 2019, 212 out of total of 1420 patients were admitted in the NICU with a diagnosis of suspected neonatal sepsis and were considered for the study. Among them, 30 neonates were excluded from the study due to unwillingness for investigations or discharge prematurely against medical advice. A cohort of 182 patients met the inclusion criteria and were enrolled in the study. Clearances including from the Hospital Research Committee and Institutional Ethics Committee were obtained. Data was recorded on a master sheet and analysed.

In our study cohort, out of the 182 blood cultures sent, 64(35.2%) were positive blood cultures. This is comparable to positivity rates in other studies in developing countries such as Pruthadesai et al in which out of a cohort of 120 patients, 60(50%) were positive blood cultures, D. Jeyaganguli et al in which out of a cohort of 60 patients, 26(43%) were positive blood cultures and Kajal Basavaraj Punyashetty et al in which out of a cohort of 100 patients, 42(42%) were positive blood cultures.3, 4, 6, 7, 5, 8, 9, 10

The most common organism that grew in blood culture was Klebsiella (67.2%) (n=43) followed by Pseudomonas (n=8), Staphylococcus aureus (n=8), Candida (n=3), E coli (n=1) and Streptococcus agalactae (n=1). The aetiology of culture positive neonatal sepsis is similar to other studies in developing countries like Pruthadesai et al, D. Jeyaganguli et al, Kajal Basavaraj Punyashetty et al. 8, 9, 10

Demographics

In our cohort of patients, it was seen that there was an increasing incidence of culture positive sepsis with reducing gestational age and birth weight. Further there was an increasing trend of incidence of culture positive sepsis in SGA babies. However, these trends did not reach levels of statistical significance. In our cohort of patients, more males were present than females. The percentage risk of neonatal sepsis (suspected/probable/culture positive) in males admitted in NICU was 12.2% in males and 13.8% in females. However, the risk of culture positive neonatal sepsis among those admitted for concern for sepsis was found higher in females (42.1%) than males (30.2%) which was statistically significant. This is in contrast to other studies in India such as Pruthadesai et al, D. Jeyaganguli et al, Kajal Basavaraj Punyashetty et al in which male admitted in NICU were at higher risk of sepsis than females. This may be because in our population, females are admitted when they were sicker than males perhaps because of the local ethos.8, 9, 10

Clinical factors

Maternal Factors: In our study it was observed that vaginal delivery and premature rupture of membranes (PROM) >18 hours prior to delivery had an increased association with blood culture positivity but these findings did not reach levels of statistical significance. Our observation is in concordance with other studies. 11 In our study population, none of the mothers gave a history of peripartum febrile illness. This may have been due to limited ability to report fevers during admission of baby. Also, the number of mothers with history of fever prior to delivery may have been few and given our limited sample size may not have been captured by chance alone.

Neonatal factors

Of clinical features, 42.3% of patients with lethargy developed culture positive sepsis while 23.9% of those with no lethargy developed culture positive sepsis; this difference reached levels of statistical significance (p=0.01) on univariate analysis. Our observation is in concordance with other studies. 12 Of the clinical features, 57.1% of patients with bleeding manifestations developed culture positive sepsis while 34.2% of those with no bleeding manifestation developed culture positive sepsis; this finding did not reach levels of statistical significance on univariate analysis as with occurrence of seizures.

Laboratory parameters

In our study it was seen that positive CRP, Absolute neutrophil count <1500/mm3, Platelet counts < 150,000/mm3, Immature to total neutrophil ratio (ITR) >0.2 and presence of degenerative changes in WBC had a statistically significant association with blood culture positivity on univariate analysis. Our observation is in concordance with other studies.13, 14, 15, 16 However, in the multivariable analysis adjusted for other variables, only immature to total neutrophil ratio (ITR) >0.2, platelet count less than 150,000/ mm3, and cytotoxic degenerative changes in neutrophils were associated with culture positive neonatal sepsis. There was a trend towards higher culture positive neonatal sepsis in babies with absolute neutrophil count <1500/mm3 but it was not statistically significant.

Conclusion

Neonatal sepsis is a common occurrence in NICU. Early detection and institution of definitive treatment enables more effective recovery. Blood culture is gold standard tool for diagnosis though time consuming. Hence identification of early surrogate markers would enable early institution of definite treatment. This study identified female gender, I:T ratio >0.2, cytotoxic changes in neutrophils and platelet count less than 150,000/mm3 to be effective surrogate markers for blood culture positive neonatal sepsis. These markers would enable an early inexpensive and easily available alternative to blood culture positivity in diagnosis and appropriate management of neonatal sepsis.

Limitations

Although our cohort was of reasonable size, our study was in a single center with its own distinct demographics, admission rates and practice habits. Our study may have been underpowered to detect the importance of some clinical factors that could have independently predicted culture positive neonatal sepsis. The retrospective scope of the study also limited the number of variables we could test in our study.

Conflicts of Interest

All contributing authors declare no conflicts of interest.

Source of Funding

None.

References

1 

L Liu S Oza D Hogan J Perin I Rudan JE Lawn Global, regional, and national causes of child mortality in 2000–13, with projections to inform post-2015 priorities: an updated systematic analysisLancet201538599664304010.1016/s0140-6736(14)61698-6

2 

B. J. Stoll N. Hansen A. A. Fanaroff L. L. Wright W. A. Carlo R. A. Ehrenkranz Late-Onset Sepsis in Very Low Birth Weight Neonates: The Experience of the NICHD Neonatal Research NetworkPediatr200211022859110.1542/peds.110.2.285

3 

MH Tsai SM Chu CW Lee JF Hsu HR Huang MC Chiang Recurrent late-onset sepsis in the neonatal intensive care unit: incidence, clinical characteristics and risk factorsClin Microbiol Infect20142011O9283510.1111/1469-0691.12661

4 

NNPD Network. National neonatal-perinatal database Report 2002-2003.https://www.newbornwhocc.org/pdf/nnpd_report_2002-03.PDF

5 

J P Buttery Blood cultures in newborns and children: optimising an everyday testArch Dis Child Fetal Neonatal Ed.200287125F810.1136/fn.87.1.f25

6 

Office of the Registrar General & Census Commissioner, India. SRS statistical report2013

7 

Sherrianne Ng Tobias Strunk Pingping Jiang Tik Muk Per T. Sangild Andrew Currie Precision Medicine for Neonatal SepsisFrontiers in Molecular Biosciences2018570702296-889X10.3389/fmolb.2018.00070Frontiers Media SAhttps://dx.doi.org/10.3389/fmolb.2018.00070

8 

PP Desai T Pandya P Desai Tejal Pandya. C-Reactive Protein, Immature to total Neutrophil Ratio and Micro ESR in early diagnosis of Neonatal SepsisInt J Biomed Adv Res2014583646

9 

D. Jeyaganguli G. Velvizhi G. Sucilathangam C. Revathy Diagnostic Value of C - Reactive Protein and Hematological Markers in Neonatal SepsisInt J Curr Microbiol Appl Sci2018703722710.20546/ijcmas.2018.703.084

10 

K B Punyashetty T Patil Interpretation of haematological parameters in neonates at risk for sepsisJ Evid Based Med Healthcare20163492492610.18410/jebmh/2016/547

11 

T. Schmitz L. Sentilhes E. Lorthe D. Gallot H. Madar M. Doret-Dion Rupture prématurée des membranes avant terme : recommandations pour la pratique clinique du CNGOF — Texte courtGynecol Obstet Fertil Senol20184612998100310.1016/j.gofs.2018.10.016

12 

M Mahallei M Ahangarzadeh Rezaee B Mehramuz S Beheshtirooy B Abdinia Clinical symptoms, laboratory, and microbial patterns of suspected neonatal sepsis cases in a children's referral hospital in northwestern IranMed20189725e1063010.1097/md.0000000000010630

13 

RD Christensen E Henry J Jopling SE Wiedmeier The CBC: Reference Ranges for NeonatesSemin Perinatol200933131110.1053/j.semperi.2008.10.010

14 

BL Manroe AG Weinberg CR Rosenfeld R Browne The neonatal blood count in health and disease.I. Reference values for neutrophilic cellsJ Pediatr1979951899810.1016/s0022-3476(79)80096-7

15 

TB Newman D Draper KM Puopolo S Wi GJ Escobar Combining Immature and Total Neutrophil Counts to Predict Early Onset Sepsis in Term and Late Preterm NewbornsPediatr Infect Dis J201433879880210.1097/inf.0000000000000297

16 

N Schmutz E Henry J Jopling R D Christensen Expected ranges for blood neutrophil concentrations of neonates: the Manroe and Mouzinho charts revisitedJ Perinatol: Official J California Perinat Assoc20082842758110.1038/sj.jp.7211916



jats-html.xsl


This is an Open Access (OA) journal, and articles are distributed under the terms of the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 License, which allows others to remix, tweak, and build upon the work non-commercially, as long as appropriate credit is given and the new creations are licensed under the identical terms.

  • Article highlights
  • Article tables
  • Article images

Article History

Received : 29-01-2021

Accepted : 26-02-2021


View Article

PDF File   Full Text Article


Copyright permission

Get article permission for commercial use

Downlaod

PDF File   XML File   ePub File


Digital Object Identifier (DOI)

Article DOI

https://doi.org/ 10.18231/j.ijmpo.2021.007


Article Metrics






Article Access statistics

Viewed: 1251

PDF Downloaded: 674