A survey of deep learning solutions for multimedia visual content analysis.

Journal article


Nadeem, Muhammad Shahroz, Franqueira, Virginia N. L., Zhai, Xiaojun and Kurugollu, Fatih 2019. A survey of deep learning solutions for multimedia visual content analysis. IEEE Access. https://doi.org/10.1109/ACCESS.2019.DOI
AuthorsNadeem, Muhammad Shahroz, Franqueira, Virginia N. L., Zhai, Xiaojun and Kurugollu, Fatih
Abstract

The increasing use of social media networks on handheld devices, especially smartphones
with powerful built-in cameras, and the widespread availability of fast and high bandwidth broadband
connections, added to the popularity of cloud storage, is enabling the generation and distribution of massive
volumes of digital media, including images and videos. Such media is full of visual information and holds
immense value in today’s world. The volume of data involved calls for automated visual content analysis
systems able to meet the demands of practice in terms of efficiency and effectiveness. Deep Learning (DL)
has recently emerged as a prominent technique for visual content analysis. It is data-driven in nature
and provides automatic end-to-end learning solutions without the need to rely explicitly on predefined
handcrafted feature extractors. Another appealing characteristic of DL solutions is the performance they
can achieve, once the network is trained, under practical constraints. This paper identifies eight problem
domains which require analysis of visual artefacts in multimedia. It surveys the recent, authoritative, and
best performing DL solutions and lists the datasets used in the development of these deep methods for the
identified types of visual analysis problems. The paper also discusses the challenges that DL solutions face
which can compromise their reliability, robustness, and accuracy for visual content analysis.

KeywordsMachine Learning; Deep Learning; Visual Content Analysis; Dataset
Year2019
JournalIEEE Access
PublisherIEEE
ISSN2169-3536
Digital Object Identifier (DOI)https://doi.org/10.1109/ACCESS.2019.DOI
Web address (URL)http://hdl.handle.net/10545/623941
hdl:10545/623941
Publication dates24 Jun 2019
Publication process dates
Deposited28 Jun 2019, 12:14
Accepted10 Jun 2019
ContributorsUniversity of Derby and University of Essex
File
File Access Level
Open
File
File Access Level
Open
Permalink -

https://repository.derby.ac.uk/item/94yw2/a-survey-of-deep-learning-solutions-for-multimedia-visual-content-analysis

Download files

  • 25
    total views
  • 1
    total downloads
  • 1
    views this month
  • 0
    downloads this month

Export as

Related outputs

Explaining deep neural networks: A survey on the global interpretation methods
Saleem, R., Yuan, B., Kurugollu, F., Anjum, A. and Liu, L. 2022. Explaining deep neural networks: A survey on the global interpretation methods. Neurocomputing. 513, pp. 165-180. https://doi.org/10.1016/j.neucom.2022.09.129
Severity Estimation of Plant Leaf Diseases Using Segmentation Method
Entuni, Chyntia Jaby, Afendi Zulcaffle, Tengku Mohd, Kipli, Kuryati and Kurugollu, Fatih 2020. Severity Estimation of Plant Leaf Diseases Using Segmentation Method. Applied Science and Engineering Progress. 14 (1), pp. 108-119. https://doi.org/10.14416/j.asep.2020.11.004
Explaining probabilistic Artificial Intelligence (AI) models by discretizing Deep Neural Networks
Saleem, Rabia, Yuan, Bo, Kurugollu, Fatih and Anjum, Ashiq 2020. Explaining probabilistic Artificial Intelligence (AI) models by discretizing Deep Neural Networks. IEEE. https://doi.org/10.1109/ucc48980.2020.00070
NOTRINO: a NOvel hybrid TRust management scheme for INternet-Of-vehicles
Ahmad, F., Kurugollu, Fatih, Kerrache, Chaker Abdelaziz, Sezer, Sakir and Liu, Lu 2021. NOTRINO: a NOvel hybrid TRust management scheme for INternet-Of-vehicles. IEEE Transactions on Vehicular Technology. https://doi.org/10.1109/TVT.2021.3049189
A Novel Security Methodology for Smart Grids: A Case Study of Microcomputer-Based Encryption for PMU Devices
Varan, Metin, Akgul, Akif, Kurugollu, Fatih, Sansli, Ahmet and Smith, K. 2021. A Novel Security Methodology for Smart Grids: A Case Study of Microcomputer-Based Encryption for PMU Devices. Complexity. 2021, pp. 1-15. https://doi.org/10.1155/2021/2798534
Persation: an IoT based personal safety prediction model aided solution
Alofe, Olasunkanmi Matthew, Fatema, Kaniz, Azad, Muhammad Ajmal and Kurugollu, Fatih 2020. Persation: an IoT based personal safety prediction model aided solution. International Journal of Computing and Digital Systems.
MARINE: Man-in-the-middle attack resistant trust model IN connEcted vehicles
Ahmad, F., Kurugollu, Fatih, Adnane, Asma, Hussain, Rasheed and Hussain, Fatima 2020. MARINE: Man-in-the-middle attack resistant trust model IN connEcted vehicles. IEEE Internet of Things. https://doi.org/10.1109/JIOT.2020.2967568
CRT-BIoV: A cognitive radio technique for blockchain-enabled internet of vehicles
Rathee, Geetanjali, Ahmad, F., Kurugollu, Fatih, Azad, Muhammad, Iqbal, Razi and Imran, Muhammad 2020. CRT-BIoV: A cognitive radio technique for blockchain-enabled internet of vehicles. IEEE Transactions on Intelligent Transportation Systems. https://doi.org/10.1109/TITS.2020.3004718
Vehicular sensor networks: Applications, advances and challenges
Kurugollu, Fatih, Ahmed, Syed Hassan, Hussain, Rasheed, Ahmad, F. and Kerrache, Chaker Abdelaziz 2020. Vehicular sensor networks: Applications, advances and challenges. Sensors. https://doi.org/10.3390/s20133686
Integration and evaluation of QUIC and TCP-BBR in longhaul science data transfers
Lopes, Raul H. C., Franqueira, Virginia N. L. and Duncan, Rand 2019. Integration and evaluation of QUIC and TCP-BBR in longhaul science data transfers. EPJ Web of Conferences. 214, p. 08026. https://doi.org/10.1051/epjconf/201921408026
Privacy verification of photoDNA based on machine learning
Nadeem, Muhammad Shahroz, Franqueira, Virginia N. L. and Zhai, Xiaojun 2019. Privacy verification of photoDNA based on machine learning. in: The Institution of Engineering and Technology (IET).
Spatio-temporal rich model-based video steganalysis on cross sections of motion vector planes
Tasdemir, Kasim, Kurugollu, Fatih and Sezer, Sakir 2016. Spatio-temporal rich model-based video steganalysis on cross sections of motion vector planes. IEEE Transactions on Image Processing. https://doi.org/10.1109/TIP.2016.2567073
Cascaded multimodal biometric recognition framework
Albesher, Badr, Kurugollu, Fatih, Bouridane, Ahmed and Baig, Asim 2013. Cascaded multimodal biometric recognition framework. IET Biometrics. https://doi.org/10.1049/iet-bmt.2012.0043
Privacy region protection for H.264/AVC with enhanced scrambling effect and a low bitrate overhead
Wang, Yongsheng, O׳Neill, Máire, Kurugollu, Fatih and O׳Sullivan, Elizabeth 2015. Privacy region protection for H.264/AVC with enhanced scrambling effect and a low bitrate overhead. Signal Processing: Image Communication. https://doi.org/10.1016/j.image.2015.04.013
Blind image watermark detection algorithm based on discrete shearlet transform using statistical decision theory
Ahmaderaghi, Baharak, Kurugollu, Fatih, Rincon, Jesus Martinez Del and Bouridane, Ahmed 2018. Blind image watermark detection algorithm based on discrete shearlet transform using statistical decision theory. IEEE Transactions on Computational Imaging. https://doi.org/10.1109/TCI.2018.2794065
Frontal view gait recognition with fusion of depth features from a time of flight camera
Afendi Tengku Mohd, Kurugollu, Fatih, Crookes, Danny, Bouridane, Ahmed and Farid, Mohsen 2018. Frontal view gait recognition with fusion of depth features from a time of flight camera. IEEE Transactions on Information Forensics and Security. https://doi.org/10.1109/TIFS.2018.2870594
Behavioural Digital Forensics Model: Embedding Behavioural Evidence Analysis into the Investigation of Digital Crimes
Al Mutawa, Noora, Bryce, Joanne, Franqueira, Virginia N.L., Marrington, Andrew and Read, Janet C. 2018. Behavioural Digital Forensics Model: Embedding Behavioural Evidence Analysis into the Investigation of Digital Crimes. Digital Investigation. https://doi.org/10.1016/j.diin.2018.12.003
Analytical tools for blockchain: review, taxonomy and open challenges.
Balaskas, Anastasios and Franqueira, Virginia N. L. 2018. Analytical tools for blockchain: review, taxonomy and open challenges. IEE Explore. https://doi.org/10.1109/CyberSecPODS.2018.8560672
Blockchain-Based Distributed Marketplace.
Kabi, Oliver R. and Franqueira, Virginia N. L. 2019. Blockchain-Based Distributed Marketplace. in: Springer Nature.
Towards a trusted unmanned aerial system using blockchain (BUAS) for the protection of critical infrastructure
Barka, Ezedin, Kerrache, Chaker Abdelaziz, Benkraouda, Hadjer, Shuaib, Khaled, Ahmad, F. and Kurugollu, Fatih 2019. Towards a trusted unmanned aerial system using blockchain (BUAS) for the protection of critical infrastructure. Transactions on Emerging Telecommunications Technologies. https://doi.org/10.1002/ett.3706
A comparative analysis of trust models for safety applications in IoT-enabled vehicular networks
Ahmad, F., Adnane, Asma, Hussain, Rasheed and Kurugollu, Fatih 2019. A comparative analysis of trust models for safety applications in IoT-enabled vehicular networks. IEEE.
Realization of blockchain in named data networking-based internet-of-vehicles
Ahmad, F., Kerrache, Chaker Abdelaziz, Kurugollu, Fatih and Hussain, Rasheed 2019. Realization of blockchain in named data networking-based internet-of-vehicles. IT Professional. https://doi.org/10.1109/MITP.2019.2912142
HD number plate localization and character segmentation on the Zynq heterogeneous SoC.
Al-Zawqari, Ali, Hommos, Omar, Al-Qahtani, Abdulhadi, Farhat, Ali, Bensaali, Faycal, Zhai, Xiaojun and Amira, Abbes 2018. HD number plate localization and character segmentation on the Zynq heterogeneous SoC. Journal of Real-Time Image Processing. https://doi.org/10.1007/s11554-017-0747-7
A tutorial introduction to IoT design and prototyping with examples.
Meruje, Manuel, Sumaila, Musa, Franqueira, Virginia N. L., Freire, Mario M. and Inacio, Pedro R. M. 2018. A tutorial introduction to IoT design and prototyping with examples. in: Wiley-IEEE Press.
Real-time automated image segmentation technique for cerebral aneurysm on reconfigurable system-on-chip.
Zhai, Xiaojun, Eslami, Mohammad, Hussein, Ealaf Sayed, Filali, Maroua Salem, Shalaby, Salma Tarek, Amira, Abbes, Bensaali, Faycal, Dakua, Sarada, Abinahed, Julien, Al-Ansari, Abdulla and Ahmed, Ayman Z. 2018. Real-time automated image segmentation technique for cerebral aneurysm on reconfigurable system-on-chip. Journal of Computational Science. https://doi.org/10.1016/j.jocs.2018.05.002
Spatio-temporal rich model-based video steganalysis on cross sections of motion vector planes.
Tasdemir, Kasim, Kurugollu, Fatih and Sezer, Sakir 2016. Spatio-temporal rich model-based video steganalysis on cross sections of motion vector planes. IEEE Transactions on Image Processing. https://doi.org/10.1109/TIP.2016.2567073
Service-oriented system engineering
Bessis, Nik, Zhai, Xiaojun and Sotiriadis, Stelios 2017. Service-oriented system engineering. Future Generation Computer Systems. https://doi.org/10.1016/j.future.2017.11.025
Cloud forensics and security
Franqueira, Virginia N. L., Lopes, Raul, Jones, Andrew and Storer, Tim 2017. Cloud forensics and security. SpringerOpen.
On Locky ransomware, Al Capone and Brexit
MacRae, John and Franqueira, Virginia N. L. 2018. On Locky ransomware, Al Capone and Brexit. in: Springer.
Inequality indexes as sparsity measures applied to ventricular ectopic beats detection and its efficient hardware implementation.
Baali, Hamza, Zhai, Xiaojun, Djelouat, Hamza, Amira, Abbes and Bensaali, Faycal 2017. Inequality indexes as sparsity measures applied to ventricular ectopic beats detection and its efficient hardware implementation. IEEE Access. https://doi.org/10.1109/ACCESS.2017.2780190
Man-In-The-Middle attacks in Vehicular Ad-Hoc Networks: Evaluating the impact of attackers’ strategies.
Ahmad, F., Adnane, Asma, Franqueira, Virginia N. L., Kurugollu, Fatih and Liu, Lu 2018. Man-In-The-Middle attacks in Vehicular Ad-Hoc Networks: Evaluating the impact of attackers’ strategies. Sensors. 18 (11), p. 4040. https://doi.org/10.3390/s18114040
TEAM: A trust evaluation and management framework in context-enabled vehicular ad-hoc networks.
Ahmad, F., Franqueira, Virginia N. L. and Adnane, Asma 2018. TEAM: A trust evaluation and management framework in context-enabled vehicular ad-hoc networks. IEEE Access. https://doi.org/10.1109/ACCESS.2018.2837887
Investigation of indecent images of children cases: Challenges and suggestions collected from the trenches.
Franqueira, Virginia N. L., Bryce, Joanne, Al Mutawa, Noora and Marrington, Andrew 2017. Investigation of indecent images of children cases: Challenges and suggestions collected from the trenches. Digital Investigation. https://doi.org/10.1016/j.diin.2017.11.002
Parallel Monte Carlo search for Hough Transform.
Lopes, Raul, Franqueira, Virginia N. L., Reid, Ivan D. and Hobson, Peter 2017. Parallel Monte Carlo search for Hough Transform. Journal of Physics: Conference Series. https://doi.org/10.1088/1742-6596/898/7/072052
ECG encryption and identification based security solution on the Zynq SoC for connected health systems
Zhai, Xiaojun, Ait Si Ali, Amine, Amira, Abbes and Bensaali, Faycal 2017. ECG encryption and identification based security solution on the Zynq SoC for connected health systems. Journal of Parallel and Distributed Computing. https://doi.org/10.1016/j.jpdc.2016.12.016
Verifiable public key encryption scheme with equality test in 5G networks
Xu, Yan, Wang, Ming, Zhong, Hong, Cui, Jie, Liu, Lu and Franqueira, Virginia N. L. 2017. Verifiable public key encryption scheme with equality test in 5G networks. IEEE Access. https://doi.org/10.1109/ACCESS.2017.2716971
Special issue on cyberharassment investigation: Advances and trends
Bryce, Joanne, Franqueira, Virginia N. L. and Marrington, Andrew 2016. Special issue on cyberharassment investigation: Advances and trends. Journal of Digital Forensics, Security and Law (JDFSL).
Faith in vehicles: A set of evaluation criteria for trust management in vehicular ad-hoc network
Ahmad, F., Hall, Jordan, Adnane, Asma and Franqueira, Virginia N. L. 2017. Faith in vehicles: A set of evaluation criteria for trust management in vehicular ad-hoc network. IEEE.
Digital video source identification based on green-channel photo response non-uniformity (G-PRNU)
Al-Athamneh, Mohammad, Kurugollu, Fatih, Crookes, Danny and Farid, Mohsen 2016. Digital video source identification based on green-channel photo response non-uniformity (G-PRNU). https://doi.org/10.5121/csit.2016.61105
Detecting compromised programs for embedded system applications
Zhai, Xiaojun, Appiah, Kofi, Ehsan, Shoaib, Cheung, Wah M, Howells, Gareth, Hu, Huosheng, Gu, Dongbing and McDonald-Maier, Klaus 2014. Detecting compromised programs for embedded system applications. in: Springer.
Space Wireless Sensor Networks for planetary exploration: Node and network architectures
Rodrigues, Pedro, Oliveira, Andre, Oddi, Guido, Liberati, Francesco, Alvarez, Francisco, Cabas, Ramiro, Vladimirova, Tanya, Zhai, Xiaojun, Jing, Hongyuan and Crosnier, Michael 2014. Space Wireless Sensor Networks for planetary exploration: Node and network architectures. IEEE. https://doi.org/10.1109/AHS.2014.6880175
Multi-sensor data fusion in wireless sensor networks for planetary exploration
Zhai, Xiaojun, Jing, Hongyuan and Vladimirova, Tanya 2014. Multi-sensor data fusion in wireless sensor networks for planetary exploration. IEEE. https://doi.org/10.1109/AHS.2014.6880176
Hardware acceleration of an image processing system for dielectrophoretic loading of single neurons inside micro-wells of microelectrode arrays
Zhai, Xiaojun, Jaber, Fadi, Bensaali, Faycal and Mishra, Arti 2015. Hardware acceleration of an image processing system for dielectrophoretic loading of single neurons inside micro-wells of microelectrode arrays. IEEE. https://doi.org/10.1109/UKSim.2015.28
Data aggregation in wireless sensor networks for lunar exploration
Zhai, Xiaojun and Vladimirova, Tanya 2015. Data aggregation in wireless sensor networks for lunar exploration. IEEE. https://doi.org/10.1109/EST.2015.9
ECG security identification system on the Zynq SoC Platform
Zhai, Xiaojun, Amira, Abbes and Bensaali, Faycal 2015. ECG security identification system on the Zynq SoC Platform. IEEE. https://doi.org/10.1109/CIT/IUCC/DASC/PICOM.2015.174
OCR based feature extraction and template matching algorithms for Qatari number plate
Farhat, Ali, Al-Zawqari, Ali, Al-Qahtani, Abdulhadi, Hommos, Omar, Bensaali, Faycal, Amira, Abbes and Zhai, Xiaojun 2016. OCR based feature extraction and template matching algorithms for Qatari number plate. IEEE. https://doi.org/10.1109/ICCSII.2016.7462419
HD Qatari ANPR system
Hommos, Omar, Al-Qahtani, Abdulhadi, Al-Zawqari, Ali, Bensaali, Faycal, Amira, Abbes and Zhai, Xiaojun 2016. HD Qatari ANPR system. IEEE. https://doi.org/10.1109/ICCSII.2016.7462420
Heterogeneous implementation of ECG encryption and identification on the Zynq SoC
Ait Si Ali, Amine, Zhai, Xiaojun, Amira, Abbes, Bensaali, Faycal and Ramzan, Naeem 2016. Heterogeneous implementation of ECG encryption and identification on the Zynq SoC. IEEE. https://doi.org/10.1109/FCCM.2016.44
Efficient data-processing algorithms for wireless-sensor-networks-based planetary exploration
Zhai, Xiaojun and Vladimirova, Tanya 2016. Efficient data-processing algorithms for wireless-sensor-networks-based planetary exploration. Journal of Aerospace Information Systems. https://doi.org/10.2514/1.I010373
MLP neural network based gas classification system on Zynq SoC
Zhai, Xiaojun, Ait Si Ali, Amine, Amira, Abbes and Bensaali, Faycal 2016. MLP neural network based gas classification system on Zynq SoC. IEEE Access. https://doi.org/10.1109/ACCESS.2016.2619181
Forensically-sound analysis of security risks of using local password managers
Gray, Joshua, Franqueira, Virginia N. L. and Yu, Yijun 2016. Forensically-sound analysis of security risks of using local password managers. IEEE Computer Society. https://doi.org/10.1109/REW.2016.034
Forensic investigation of cyberstalking cases using Behavioural Evidence Analysis
Al Mutawa, Noora, Bryce, Joanne, Marrington, Andrew and Franqueira, Virginia N. L. 2016. Forensic investigation of cyberstalking cases using Behavioural Evidence Analysis. Digital Investigation. https://doi.org/10.1016/j.diin.2016.01.012
To flip or not to flip: a critical interpretive synthesis of flipped teaching
Franqueira, Virginia N. L. and Tunnicliffe, Peter 2015. To flip or not to flip: a critical interpretive synthesis of flipped teaching. in: Springer International Publishing.
Factors influencing digital forensic investigations: Empirical evaluation of 12 years of Dubai police cases
Al Awadhi, Ibtesam, Read, Janet C., Marrington, Andrew and Franqueira, Virginia N. L. 2015. Factors influencing digital forensic investigations: Empirical evaluation of 12 years of Dubai police cases. Journal of Digital Forensics, Security and Law (JDFSL).
A systematic approach for cyber security in vehicular networks
Ahmad, F., Adnane, Asma and Franqueira, Virginia N. L. 2016. A systematic approach for cyber security in vehicular networks. Journal of Computer and Communications. https://doi.org/10.4236/jcc.2016.416004
Introduction to special issue on risk and trust in embedded critical systems
Rossebø, Judith E. Y., Houmb, Siv H., Georg, Geri, Franqueira, Virginia N. L. and Serpanos, Dimitrios 2014. Introduction to special issue on risk and trust in embedded critical systems. ACM Transactions on Embedded Computing Systems. https://doi.org/10.1145/2659008
Real-time optical character recognition on field programmable gate array for automatic number plate recognition system
Sotudeh, Reza, Zhai, Xiaojun and Bensaali, Faycal 2013. Real-time optical character recognition on field programmable gate array for automatic number plate recognition system. IET Circuits, Devices & Systems. https://doi.org/10.1049/iet-cds.2012.0339
Exploring ICMetrics to detect abnormal program behaviour on embedded devices
Zhai, Xiaojun, Ehsan, Shoaib, Howells, Gareth, Gu, Dongbing, McDonald-Maier, Klaus, Appiah, Kofi and Hu, Huosheng 2015. Exploring ICMetrics to detect abnormal program behaviour on embedded devices. Journal of Systems Architecture. https://doi.org/10.1016/j.sysarc.2015.07.007
Efficient computation of hashes
Lopes, Raul, Franqueira, Virginia N. L. and Hobson, Peter 2014. Efficient computation of hashes. Journal of Physics: Conference Series. https://doi.org/10.1088/1742-6596/513/3/032042
Field programmable gate arrays-based number plate binarization and adjustment for automatic number plate recognition systems
Zhai, Xiaojun, Bensaali, Faycal and Sotudeh, Reza 2013. Field programmable gate arrays-based number plate binarization and adjustment for automatic number plate recognition systems. Journal of Electronic Imaging. https://doi.org/10.1117/1.JEI.22.1.013009
Improved number plate localisation algorithm and its efficient field programmable gate arrays implementation
Zhai, Xiaojun, Ramalingam, Soodamani and Bensaali, Faycal 2013. Improved number plate localisation algorithm and its efficient field programmable gate arrays implementation. IET Circuits, Devices & Systems. https://doi.org/10.1049/iet-cds.2012.0064
Improved number plate character segmentation algorithm and its efficient FPGA implementation
Zhai, Xiaojun and Bensaali, Faycal 2012. Improved number plate character segmentation algorithm and its efficient FPGA implementation. Journal of Real-Time Image Processing. https://doi.org/10.1007/s11554-012-0258-5
Behavioural evidence analysis applied to digital forensics: An empirical analysis of child pornography cases using P2P networks
Mutawa, Noora Al, Bryce, Joanne, Franqueira, Virginia N. L. and Marrington, Andrew 2015. Behavioural evidence analysis applied to digital forensics: An empirical analysis of child pornography cases using P2P networks. IEEE Computer Society. https://doi.org/10.1109/ARES.2015.49
Challenges of data provenance for cloud forensic investigations
Katilu, Victoria M., Franqueira, Virginia N. L. and Angelopoulou, Olga 2015. Challenges of data provenance for cloud forensic investigations. IEEE Computer Society. https://doi.org/10.1109/ARES.2015.54
A method for detecting abnormal program behavior on embedded devices
Zhai, Xiaojun, Ehsan, Shoaib, Howells, Gareth, Dongbing, Gu, McDonald-Maier, Klaus, Appiah, Kofi and Hu, Huosheng 2015. A method for detecting abnormal program behavior on embedded devices. IEEE Transactions on Information Forensics and Security. https://doi.org/10.1109/TIFS.2015.2422674
Automated analysis of security requirements through risk-based argumentation
Yu, Yijun, Franqueira, Virginia N. L., Than Tun, Thein, Wieringa, Roel J. and Nuseibeh, Bashar 2015. Automated analysis of security requirements through risk-based argumentation. Journal of Systems and Software. https://doi.org/10.1016/j.jss.2015.04.065