ISSN 2658–5782
DOI 10.21662
Electronic Scientific Journal





© Институт механики
им. Р.Р. Мавлютова
УФИЦ РАН

Яндекс.Метрика web site traffic statistics

A.A. Lonshchakov, O.V. Darintsev Comparative analysis of approaches necessary for designing a robotic module for industrial parts defect detection. Multiphase Systems. 19 (2024) 1. 18–30 (in Russian).
2024. Vol. 19. Issue 1, Pp. 18–30
URL: http://mfs.uimech.org/mfs2024.1.003,en
DOI: 10.21662/mfs2024.1.003
Comparative analysis of approaches necessary for designing a robotic module for industrial parts defect detection
A.A. Lonshchakov∗, O.V. Darintsev∗∗
∗Ufa State Petroleum Technological University, Ufa, Russia
∗∗Mavlyutov Institute of Mechanics UFRC of the RAS, Ufa, Russia

Abstract

Over the past two decades, the most frequently implemented method of oil extraction in the Russian Federation has been a mechanized method using an electric submersible pump (ESP) unit. Due to the remoteness of the fields from the base of repair enterprises, the cost of transporting spent but maintainable ESPs significantly exceeds their purchase price, which over time leads to the formation of a significant amount of decommissioned oil production equipment. To solve this problem, it is proposed to develop a mobile robotic module for sorting, defecting and storing pump parts, thanks to its use, minor repairs of equipment will be possible directly at production sites. The article deals with the problems of complex flaw detection of metal and non-metallic parts with the possibility of applying the developed methodology for a wide range of industrial products, at the same time, the main attention is paid to work with ESP parts. Based on the decomposition of the problem, the most problematic operations were identified: classification of parts, surface control (identification of defects), dimensional control. The results of a brief comparative analysis for each of the above subtasks are based on a review of the scientific literature over the past 30 years, with the largest number of sources reviewed in the last 5 years. As a result optimal methods for solving the task were derived — a machine learning technique for classifying surface defects, the use of a coordinate measuring machine with a manipulator for dimensional control. A new approach is also proposed to solve the main problem of machine learning methods (lack of training samples) in the form of using synthetic photorealistic images for classification with transfer of defect features from semantically close and publicly available training samples.

Keywords

vision system,
defect detection,
automatic visual inspection (AVI),
surface defect classification,
automatic parts measurement

Article outline

The issues of ensuring the smooth operation of downhole equipment while reducing operating costs do not lose their relevance at the moment. One of the effective ways to solve this problem is the development of a mobile small-sized robotic module for sorting and defective parts of the ESP. If gross defects and violations of the integrity of parts can be detected by weighing or simple mold control operations using a template, then the identification and classification of surface defects is quite a difficult problem.

The main purpose of this work is to analyze known methods and techniques for controlling the shape and surface of products made of various materials. Classification accuracy and processing speed are recognized as the most significant characteristics for evaluating the effectiveness of the considered methods.

Various approaches and solutions were considered for the task of detecting surface defects, as well as ways to carry out dimensional control of ESP parts. The advantages and disadvantages of the following methods are analyzed: statistical, spectral, complex and machine learning-based. For the selected GAN method, a solution to the problem with the lack of a training sample is shown, which consists in using a synthetic photorealistic sample supplemented with images of defects transferred from the well-known databases DAGM, Kolector SSD 2, Severstal, MVTec AD.

Dimensional control requires increased accuracy from the measuring equipment used, therefore, options for using contact and non-contact measurement methods were considered. The choice was made in favor of two options: laser triangulation and a control and measuring machine in combination with a complete manipulation robot. The final choice can be made only after agreement with the customer.

The main results. It is shown that in order to determine the composition of the developed robotic defect module, it is necessary to first solve the issues of methodological and technological support for the operations performed. Thus, the use of a specialized approach to the identification of surface effects and the modification of training methods made it possible to increase the accuracy of the operation and implement the adjustment of algorithms.

Conclusion. A brief overview of the techniques and approaches used for automated part flaw detection confirms not only the high complexity, but also the great demand for the task for various industries and other fields of application. The considered examples make it possible to exclude doubts about the possibility of practical implementation of an automated module, the effectiveness of which is determined by the correct use of a manipulating robot with peripheral equipment. A plan for further research has been generated, including the construction of a laboratory stand and conducting tests.

References

  1. Исполнение федерального бюджета и бюджетов системы Российской Федерации за 2022 год. Минфин России.
    Execution of the federal budget and the budgets of the system of the Russian Federation for 2022. The Ministry of Finance of Russia (in Russian).
    https://minfin.gov.ru/common/upload/library/2023/04/main/ispolnenie_federalnogo_budzheta_predvaritelnye_itogi.pdf (дата обращения (accessed): 05.10.2023 г.)
  2. Рудницкий С.В., Зацепин А.Ю., Демин Е.В., Ющенко Т.С. Перспективное скважинное оборудование для добычи сланцевой нефти баженовской свиты // PROНЕФТЬ. Профессионально о нефти. 2021. Т. 6, № 1. C. 64–75.
    Rudnitsky S.V., Zatsepin A.Yu., Demin E.V., Yushchenko T.S. [Promising downhole equipment for the extraction of shale oil of the Bazhenov formation] Perspektivnoye skvazhinnoye oborudovaniye dlya dobychi slantsevoy nefti bazhenovskoy svity. [Professionally about oil] Professional’no o nefti. Vol. 6, No. 1. 2021. Pp. 64–75 (in Russian).
    DOI: 10.51890/2587-7399-2021-6-1-64-75
  3. Конторович А.Э., Эдер Л.В., Филимонова И.В., Мишенин М.В. Роль уникальных и крупных месторождений в нефтяной промышленности России: ретроспектива, современное состояние, прогноз // Энергетическая политика. 2016. № 2. С. 34–43.
    Kontorovich A.E., Eder L.V., Filimonova I.V., Mishenin M.V. [The role of unique and large deposits in the Russian oil industry: retrospective, current state, forecast] Rol’ unikalnykh i krupnykh mestorozhdeniy v neftyanoy promyshlennosti Rossii: retrospektiva, sovremennoye sostoyaniye, prognoz // EP. No. 2. 2016 (in Russian).
    EDN: wxmmyv
  4. Утверждение стратегического направления в области цифровой трансформации топливно-энергетического комплекса: распоряжение Правительства РФ от 28 декабря 2021 г. № 3924-р.
    [Approval of the strategic direction in the field of digital transformation of the fuel and energy complex: Decree of the Government of theRussian Federation No. 3924-r dated December 28, 2021] Utverzhdeniye strategicheskogo napravleniya v oblasti tsifrovoy transformatsii toplivno- energeticheskogo kompleksa: rasporyazheniye Pravitel’stva ot 28 decabrya 2021. No. 3924-р (in Russian).
    http://publication.pravo.gov.ru/Document/View/0001202112300029
  5. Гиндуллин И.И, Лонщаков А.А., Салихов Л.С. Структура робототехнического модуля сортировки деталей // Завалишинские чтения: XVII Международная конференция по электромеханике и робототехнике (СПб., 18–19 апреля 2023 г.). Молодёжная секция: сб. докл.: в 2 ч. Ч. 1. СПб.: ГУАП, 2023. С. 41–44.
    Gindullin I.I., Lonshakov A.A., Salikhov L.S. [The structure of the robotic module for sorting parts]. Struktura robototekhnicheskogo modulya sortirovki detaley. Zavalishinsky readings: XVII International Conference on Electromechanics and Robotics (St. Petersburg, April 18–19, 2023). Youth section. Vol. 2, No. 1. St. Petersburg: GUAP. 2023. Pp. 41-44 (in Russian).
  6. Morbid M., Saber S., Ali R.G. Industrial Parts Change Recognition Model Using Machine Vision, Image Processing in the Framework of Industrial Information Integration // Journal of Industrial Information Integration. 2022. V. 26. 100277.
    DOI: 10.1016/j.jii.2021.100277
  7. Kanopoulos N., Vasanthavada N., Baker R.L., Design of an image edge detection filter using the Sobel operator // IEEE Journal of Solid-State Circuits. 1988. V. 23(2). Pp. 358–367.
    DOI: 10.1109/4.996
  8. Seif A., Salut M.M., Marsono M.N. A hardware architecture of Prewitt edge detection // 2010 IEEE Conference on Sustainable Utilization and Development in Engineering and Technology. 2010. Pp. 99-101.
    DOI: 10.1109/STUDENT.2010.5686999
  9. Nguyen C.H., Young C. Comparison of point cloud data and 3D CAD data for on-site dimensional inspection of industrial plant piping systems // Automation in Construction. 2018. V. 91. Pp. 44–52.
    DOI: 10.1016/j.autcon.2018.03.008
  10. Rabbani T., van den Heuvel F.A., Vosselmann G. Segmentation of Point Clouds Using Smoothness Constraint // ISPRS Commission V Symposium ‘Image Engineering and Vision Metrology’. 2006. Pp. 248–253.
    ResearchGate: 228340970
  11. Schnabel R., Wahl R., Klein R. Efficient Ransac for point-cloud shape detection // Graphics Forum. 2007. V. 26 (2). Pp. 214–226.
    DOI: 10.1111/j.1467-8659.2007.01016.x
  12. Huang R., Gu J., Sun X., Hou Y., Uddin S. A rapid recognition method for electronic components based on the improved YOLO-V3 network // Electronics. 2019. V. 8 (8). 825.
    DOI: 10.3390/electronics8080825
  13. Tian Y., Yang G., Wang Z., Wang H., Li E., Liang Z. Apple detection during different growth stages in orchards using the improved YOLO-V3 model // Computers and electronics in agriculture. 157. 2019. V. 157. Pp. 417–426.
    DOI: 10.1016/j.compag.2019.01.012
  14. Sinha D., El-Sharkawy M. Thin mobilenet: An enhanced mobilenet architecture // In 2019 IEEE 10th annual ubiquitous computing, electronics & mobile communication conference (UEMCON). 2019. Pp. 280–285.
    DOI: 10.1109/UEMCON47517.2019.8993089
  15. Andrei-Alexandru T., Henrietta D.E. Low-Cost Defect Detection Using a Deep Convolutional Neural Network // IEEE International Conference on Automation, Quality and Testing, Robotics (AQTR), Cluj-Napoca, Romania. 2020. Pp. 1–5.
    DOI: 10.1109/AQTR49680.2020.9130004
  16. Muhammad U., Wang W., Chattha S.P., Ali S.August. Pre-trained VGGNet architecture for remote-sensing image scene classification // In 2018 24th International Conference on Pattern Recognition (ICPR). 2018. Pp. 1622–1627.
    DOI: 10.1109/ICPR.2018.8545591
  17. Deng J., Dong W., Socher R., Li L.J., Kai L., Li Fei-Fei. ImageNet: A large-scale hierarchical image database // 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA. 2009. Pp. 248–255.
    DOI: 10.1109/CVPR.2009.5206848
  18. Feng C., Liu M.Y., Kao C.C., Lee T.Y. Deep active learning for civil infrastructure defect detection and classification // In Computing in civil engineering. 2017. Pp. 298–306.
    DOI: 10.1061/9780784480823.036
  19. He K., Zhang X., Ren S., Sun J. Deep Residual Learning for Image Recognition // 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA. 2016. Pp. 770–778.
    DOI: 10.1109/CVPR.2016.90
  20. Dawson H.L., Debrule O., John C.M. Impact of dataset size and convolutional neural network architecture on transfer learning for carbonate rock classification // Computers & Geosciences. 2023. V. 171. 105284.
    DOI: 10.1016/j.cageo.2022.105284
  21. Northcutt C.G., Athalye A., Mueller J. Pervasive label errors in test sets destabilize machine learning benchmarks // In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track. 2021.
    DOI: 10.48550/arXiv.2103.14749
  22. Cai L., Xu X., Liew J.H., Foo C.S. Revisiting superpixels for active learning in semantic segmentation with realistic annotation costs // In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021. Pp. 10988–10997.
    https://openaccess.thecvf.com/content/CVPR2021/html/Cai_Revisiting_Superpixels_for_Active_Learning_in_Semantic_Segmentation_With_Realistic_CVPR_2021_paper. html
  23. Roovere P.D, Moonen S., Michiels M., Wyffels F. Dataset of industrial metal objects // arXiv, cs.CV. 2022. 2208.04052.
    DOI: 10.48550/arXiv.2208.04052
  24. Tobin J., Fong R., Ray A., Schneider J., Zaremba W., Abbeel P. Domain randomization for transferring deep neural networks from simulation to the real world // In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). 2017. Pp. 23–30.
    DOI: 10.1109/IROS.2017.8202133[25]
  25. Reddi S., Poczos B., Smola A. Doubly Robust Covariate Shift Correction // Proceedings of the AAAI Conference on Artificial Intelligence. 2015. V. 29 (1). Pp. 2949–2955.
    DOI: 10.1609/aaai.v29i1.9576
  26. Vanherle B., Moonen S., Van Reeth F., Michiels. Analysis of training object detection models with synthetic data // arXiv, cs.CV. 2022. 2211.16066.
    DOI: 10.48550/arXiv.2211.16066
  27. Touvron H., Vedaldi A., Douze M., Jegou H. Fixing the train-test resolution discrepancy // arXiv, cs.CV. 2022. 1906.06423.
    DOI: 10.48550/arXiv.1906.06423
  28. Dosovitski A., Beyer L., Kolesnikov A., Weissenborn D., Zhai X., Unterthiner T., Dehghani M., Minderer M., Heigold G., Gelly S., Uszkoreit J. An image is worth 16x16 words: Transformers for image recognition at scale // arXiv, cs.CV. 2021. 2010.11929.
    DOI: 10.48550/arXiv.2010.11929
  29. Djukic D., Spuzic S. Statistical discriminator of surface defects on hot rolled steel // Proceedings of Image and Vision Computing New Zealand. 2007. Pp. 158–163.
    ResearchGate: 229035389
  30. Nand G.K., Noopur, Neogi N.G. Defect detection of steel surface using entropy segmentation // 2014 Annual IEEE India Conference (INDICON), Pune, India. 2014. Pp. 1–6.
    DOI: 10.1109/INDICON.2014.7030439
  31. Neogi N., Mohanta D.K., Dutta P.K. Defect Detection of Steel Surfaces with Global Adaptive Percentile Thresholding of Gradient Image // J. Inst. Eng. India Ser. 2017. Pp. 557–565.
    DOI: 10.1007/s40031-017-0296-2
  32. Bulnes F.G., Usamentiaga R., Garca D.F., Molleda J. Vision-based sensor for early detection of periodical defects in web materials // Sensors (Basel). 2012. V. 12 (8). Pp. 10788–10809.
    DOI: 10.3390/s120810788
  33. Zhao Y., Yan Y., Song K. Vision-based automatic detection of steel surface defects in the cold rolling process: considering the influence of industrial liquids and surface textures // Int. J. Adv. Manuf. Technol. 2016. V. 90 (8). Pp. 1665–1678.
    DOI: 10.1007/S00170-016-9489-0
  34. Zhiznyakov A.L., Privezentsev D.G., Zakharov A.A. Using fractal features of digital images for the detection of surface defects // Pattern Recognition and Image Analysis. 2015. V. 25. Pp. 122–131.
    DOI: 10.1134/S105466181501023X
  35. Yazdchi M., Yazdi M., Mahyari A.G. Steel surface defect detection using texture segmentation based on multifractal dimension // In 2009 International Conference on Digital Image Processing. 2009. Pp. 346–350.
    DOI: 10.1109/ICDIP.2009.68
  36. Aiger D., Talbot H. The phase only transform for unsupervised surface defect detection // In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 2010. Pp. 295–302.
    DOI: 10.1109/CVPR.2010.5540198
  37. Choi D.C., Jeon Y.J., Kim S.H., Moon S., Yun J.P., Kim S.W. Detection of pinholes in steel slabs using Gabor filter combination and morphological features // ISIJ International. 2017. V. 57 (6). Pp. 1045–1053.
    DOI: 10.2355/isijinternational.ISIJINT-2016-160
  38. Medina R., Gayubo F., González-Rodrigo L.M., Olmedo D., Gómez-Garcı́a-Bermejo J., Zalama E., Perán J.R. Automated visual classification of frequent defects in flat steel coils // The International Journal of Advanced Manufacturing Technology. 2011. V. 57. Pp. 1087–1097.
    DOI: 10.1007/s00170-011-3352-0
  39. Randen T., Husoy J.H. Texture segmentation using filters with optimized energy separation // IEEE Transactions on Image Processing. 1999. V. 8 (4). Pp. 571–582.
    DOI: 10.1109/83.753744
  40. Kumar A., Pang G.K. Defect detection in textured materials using optimized filters // IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics). 2002. V. 32 (5). Pp. 553–570.
    DOI: 10.1109/TSMCB.2002.1033176
  41. Kumar A. Inspection of surface defects using optimal FIR filters // In 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. 2003. Pp. II–241
    DOI: 10.1109/ICASSP.2003.1202339
  42. Jeon Y.J., Choi D.C., Lee S.J., Yun J.P., Kim S.W. Steel-surface defect detection using a switching-lighting scheme // Applied Optics. 2016. V. 55 (1). Pp. 47– 57.
    DOI: 10.1364/AO.55.000047
  43. Cross G.R., Jain A.K. Markov random field texture models // IEEE Transactions on pattern analysis and machine intelligence. 1983. V. PAMI-5 (1). Pp. 25–39.
    DOI: 10.1109/TPAMI.1983.4767341
  44. Gayubo F., Gonzalez J.L., de la Fuente E., Miguel F., Perán J.R. On-line machine vision system for detect split defects in sheet-metal forming processes // In 18th International Conference on Pattern Recognition. 2006. V. 1. Pp. 723–726.
    DOI: 10.1109/ICPR.2006.902
  45. Ke X., Min S., Chaolin Y., Peng Z. Application of hidden Markov tree model to on-line detection of surface defects for steel strips // Journal of mechanical engineering. 2013. V. 49(22). Pp. 34–40.
    https://qikan.cmes.org/jxgcxb/EN/Y2013/V49/I22/34
  46. Timm F., Barth E. February. Non-parametric texture defect detection using Weibull features // In Image Processing: Machine Vision Applications IV. 2011. Pp. 150–161.
    DOI: 10.1117/12.872463
  47. Liu K., Wang H., Chen H., Qu E., Tian Y., Sun H. Steel surface defect detection using a new Haar–Weibull-variance model in unsupervised manner // IEEE transactions on instrumentation and measurement. 2017. V. 66 (10). Pp. 2585–2596.
    DOI: 10.1109/TIM.2017.2712838
  48. Song K., Yan Y. Micro surface defect detection method for silicon steel strip based on saliency convex active Contour model // Math. Probl. Eng. 2013. 429094.
    DOI: 10.1155/2013/429094
  49. Yang J., Li X., Xu J., Cao Y., Zhang Y., Wang L., Jiang S. Development of an optical defect inspection algorithm based on an active contour model for large steel roller surfaces // Applied optics. 2018. V. 57 (10). Pp. 2490–2498.
    DOI: 10.1364/AO.57.002490
  50. Ghorai S., Mukherjee A., Gangadaran M., Dutta P.K. Automatic defect detection on hot-rolled flat steel products // IEEE Transactions on Instrumentation and Measurement. 2012. V. 62 (3). Pp. 612–621.
    DOI: 10.1109/TIM.2012.2218677
  51. Kang G.W., Liu H.B. Surface defects inspection of cold rolled strips based on neural network // In 2005 international conference on machine learning and cybernetics. 2005. Pp. 5034–5037.
    DOI: 10.1109/ICMLC.2005.1527830
  52. Weimer D., Scholz-Reiter B., Shpitalni M. Design of deep convolutional neural network architectures for automated feature extraction in industrial inspection // CIRP annals. 2016. V. 65 (1). Pp. 417–420.
    DOI: 10.1016/j.cirp.2016.04.072
  53. Liu Y., Xu K., Xu J. Periodic surface defect detection in steel plates based on deep learning // Applied Sciences. 2019. V. 9 (15). 3127.
    DOI: 10.3390/app9153127
  54. Li J., Su Z., Geng J., Yin Y. Real-time detection of steel strip surface defects based on improved yolo detection network // IFAC-PapersOnLine. 2018. V. 51(21). Pp. 76–81.
    DOI: 10.1016/j.ifacol.2018.09.412
  55. Zhao Z., Li B., Dong R., Zhao P. A surface defect detection method based on positive samples // In PRICAI 2018: Trends in Artificial Intelligence: 15th Pacific Rim International Conference on Artificial Intelligence, Nanjing, China, August 28–31, 2018, Proceedings. 2018. V. 2 (15). Pp. 473–481.
    DOI: 10.1007/978-3-319-97310-4_54
  56. Youkachen S., Ruchanurucks M., Phatrapomnant T., Kaneko H. Defect Segmentation of Hot-rolled Steel Strip Surface by using Convolutional Auto- Encoder and Conventional Image processing // 10th International Conference of Information and Communication Technology for Embedded Systems (IC-ICTES). 2019. Pp. 1–5.
    DOI: 10.1109/ICTEmSys.2019.8695928
  57. DAGM dataset. University of Heidelberg. 29th Annual Symposium of the German Association for Pattern Recognition.
    https://conferences.mpi-inf.mpg.de/dagm/2007/prizes.html (дата обращения 05.10.2023 г.)
  58. Kolector SSD 2 dataset. University of Ljubljana. Visual Cognitive Systems Laboratory.
    https://www.vicos.si/resources/kolektorsdd2/ (дата обращения 05.10.2023 г.)
  59. Severstal dataset. Kaggle.
    https://www.kaggle.com/c/severstal-steel-defect-detection (дата обращения 05.10.2023 г.)
  60. MVTec AD dataset.
    https://www.mvtec.com/company/research/datasets/mvtec-ad (дата обращения 05.10.2023 г.)
  61. Sun Y., Lu L., Wu F., Xiao S., Sha J., Zhang L. Error Analysis of a Coordinate Measuring Machine with a 6-DOF Industrial Robot Holding the Probe // In Actuators. 2023. V. 12 (4). 173.
    DOI: 10.3390/act12040173
  62. Brosed F.J., Santolaria J., Aguilar J.J, Guillom?a D. Laser triangulation sensor and six axes anthropomorphic robot manipulator modelling for the measurement of complex geometry products // Robotics and Computer-Integrated Manufacturing. 2012. V. 28 (6). Pp. 660–671.
    DOI: 10.1016/j.rcim.2012.04.002
  63. Lemeš S., Šabaravdić M., Zaimović-Uzunović N. Robotic manipulation in dimensional measurement // In 2013 XXIV International Conference on Information, Communication and Automation Technologies (ICAT). 2013. Pp. 1–7.
    DOI: 10.1109/ICAT.2013.6684085