key: cord-0130247-s1ycgkty
authors: Bird, Jordan J.
title: Robotic and Generative Adversarial Attacks in Offline Writer-independent Signature Verification
date: 2022-04-14
journal: nan
DOI: nan
sha: 0cece7c4e943c46f1687cb49b150e9dd623c8560
doc_id: 130247
cord_uid: s1ycgkty

This study explores how robots and generative approaches can be used to mount successful false-acceptance adversarial attacks on signature verification systems. Initially, a convolutional neural network topology and data augmentation strategy are explored and tuned, producing an 87.12% accurate model for the verification of 2,640 human signatures. Two robots are then tasked with forging 50 signatures, where 25 are used for the verification attack, and the remaining 25 are used for tuning of the model to defend against them. Adversarial attacks on the system show that there exists an information security risk; the Line-us robotic arm can fool the system 24% of the time and the iDraw 2.0 robot 32% of the time. A conditional GAN finds similar success, with around 30% forged signatures misclassified as genuine. Following fine-tune transfer learning of robotic and generative data, adversarial attacks are reduced below the model threshold by both robots and the GAN. It is observed that tuning the model reduces the risk of attack by robots to 8% and 12%, and that conditional generative adversarial attacks can be reduced to 4% when 25 images are presented and 5% when 1000 images are presented.

To forge a signature with the aim of deceiving is a serious crime throughout the world [1, 2] with financial implications and risks to personal identity. There are countless examples of signature forgery and the issues it causes such as fake historical documents fraudulently signed with Abraham Lincoln's signature [3] , the forgery and subsequent sale of celebrity signatures [4] , as well as the forging and cashing of cheques [5] . To have a signature verified by a human expert is an expensive endeavour, and is thus oftentimes another step in the process which bares even further financial implications for an individual or a business. In the UK alone, cheque fraud resulted in losses of £12.3 million in 2020 following £53.6 million losses in 2019 [6] . The COVID-19 lockdown introduced a level of fraud that had not been seen for several years. In this study, losses of £558.8 million were also shown to be prevented from cheque fraud in 2019. According to the American Bankers Association, cheque fraud amounted to $15.1 billion in 2018 and affected around half a million individuals [7] .

Modern computing proposes several solutions to the detection of forged signatures through autonomous verification as an added layer of protection against such attempts. Much state-of-the-art work in signature verification is to detect when a human being has forged another's signature. Given the rapid growth of consumer robotics due to their ease of use and low cost, the level of detail now possible via robotic signature forgery is a growing concern. Rapid analysis and near-perfect replication of a signature is now possible by machines that cost a fraction of the price of the average smartphone. In terms of generative approaches, it is possible to train models such as GANs on home computers; that is, with hardware already found in the home, signature verification systems can be succesfully attacked and fooled by data generated by a neural network. In this study, we explore how robots and generative adversarial methods can be used to fool offline writer-independent signature verification systems. Following this, efforts are made to tune and improve such systems to provide a preliminary line of defence against such adversarial attacks.

The multiple scientific contributions presented by this work are as follows: (i) tuning of a vision-based system inspired by the current State-of-the-Art for accurate signature verification. (ii) a pipeline to analyse signatures (raster images), produce vectors, and then G-code for execution of two pen-holding robotic arms (Line-us and iDraw 2.0). (iii) a Conditional Generative Adversarial Network to discern and generate real and forged signatures. (iv) Successful adversarial attacks on the verification system by both robots and generative approaches. (v) Successful defence of the verification system by fine-tune transfer learning from examples produced during adversarial attack. To the author's knowledge, this article proposes the first case of an attack on a signature verification system by using robots to physically copy and write signatures.

The remainder of this article is as follows: Section 2 reviews the background of the field, including the state of the art in signature verification and adversarial attacks. Following this, Section 3 outlines the method followed by the experiments in this work, and the results are presented and discussed in Section 3. Finally, concluding remarks and future work are discussed in Section 5.

This section explores the current state of the art related to this work. This includes biometrics, signature verification with visual and deep learning approaches, and methods of attacking verification methods.

Biometrics are systems that recognise an individual based on a given input. For example, recognition of an individual based on their fingerprint [8] , speech patterns [9, 10] , EEG [11] and ECG [12] signals, or the iris of their eye [13] to name a few. Signature verification is a biometric for the recognition and verification of an individual based on the way that they sign their name [14] . Given the ability that a genuine signature has, successful forgeries can therefore have major implications related to personal identity and finances. Online signature verification deals with smart devices, such as tablets and pens, that record the resultant signature along with features such as pressure, azimuth, velocity, and inclination [15] . Offline signature verification is based on the resultant signature alone, and is more common given that signed paper documents are often signed.

Earlier works such as Kalera et al. [16] proposed methods such as distance metrics between examples of real and forged signatures. The identification of two data sets was found to reach an accuracy of around 93%, while verification was possible for around 78% of the data objects in one dataset and around 68% in the second set. In [17] , the authors proposed a machine learning approach for least-squares support vector machines with respect to the grey-level features observed. Consideration of grey levels within pixel values provides pseudo-online insight into pen velocity and pressure. Such features are noted to be important given that a human will sign their own signature quickly, while a forgery is oftentimes more thought-out and therefore takes longer to replicate.

State-of-the-art research on signature verification relies largely on deep learning-based approaches. In 2019, Sam et al. [18] proposed an offline verification technique with the Inception-v3 Convolutional Neural Network (CNN) that achieved 88% validation accuracy on 170 images after training on 370 signatures. A similar approach was explored in [19] , with a CNN model achieving 62.5% writer-independent and 75% writer-dependent signature verification. In this study, the writer-independent dataset contained 300 images for training and 240 for validation, and the writer-dependent dataset contained 30 images for training and 24 for validation. In 2018, Souza et al. [20] proposed a feature extraction technique with a CNN prior to classification by Support Vector Machines; achieving an equal error rate of 1.48% on the Brazilian PUC-PR dataset.

Given that there is a growing reliance on automated systems for verification, therein lies the risk of attack. An adversarial attack is a method of fooling a machine learning model by engineering input data to force a chosen output prediction. In signature verification, a successful attack would allow for an individual to pass another's signature without their participation or knowledge. Work by the Autonomous University of Madrid's Biometric Recognition Group showed that a Bayesian hill-climbing attack could overcome a signature verification system 95% of the time [21] . Similarly, Li et al. [22] proposed generating invisible perturbations on signatures which led to a verification system being succesfully attacked in 92.1% of cases. In [23] the authors proposed to generate synthetic signatures by spectral analysis of trajectory functions, presenting results of up to 0.04% brute-force attack successes. Scheidat et al. [24] proposed a strategy of distance-level fusion to overcome brute-force attacks on verification systems, noting a lower error rate when combining online verification experts into a unified system. In [25] , an attack is described which can implement digital signatures into documents, by embedding PDF and TIFF files within a file which goes largely undetected. In another related study, Alonso-Fernandez et al. [26] described how forgeries do not only simply attack a system while remaining static, rather, forger skills are improved over time. This is similar to a generative adversarial learning framework, which inspired the added approach to this study alongside the two robots.

In this section, the method followed by the proposed approach is described. Initially, data pre-processing and augmentation are considered, followed by topology engineering and learning for signature verification models. Following this, adversarial attacks and defences are then outlined.

The data used in this study is the CEDAR Signature Verification Dataset 2 from [27] . In the study, 55 individuals contributed 24 of their own signatures, and some of the participants forged the signatures of others. The dataset comprises 1,320 genuine and 1,320 forged signatures. The dataset was found to contain images that varied in size and aspect ratio. Given the nature of the input topologies of convolutional neural networks, data pre-processing was performed. All images were resized to 256px and binarized; An example of an image before and after processing can be found in Figure 2 . Some signatures were also observed to be signed at a rotation of up to approximately 20%. Random rotation is therefore explored as a potential method of data augmentation along with a vanilla convolutional neural network. Convolutional Neural Networks were the main learning method within the experiments. During an 80/20 split, topology tuning was performed via CNN layers of {1, 2, 3} in number, each containing {16, 32, 64} filters. Dense interpretation multilayer perceptrons were then tuned, involving 1 or 2 layers with {64, 128, 256} rectified linear units. Since some methods would likely converge sooner than others, the CNNs are therefore given an infinite number of epochs to train, instead stopping when there were no observed improvements on the validation metrics within a 25 In addition to the generative adversarial attack, this study also explores whether robotic arms can forge signatures. Two robots are used, the Line-us robotic arm and the iDraw 2.0 handwriting plotter. These two robots can be seen in Figure  3 . Each robot was tasked with replicating the same 25 signatures from the dataset, and these images provide robotic attacks on the verification model. Figure 1 shows the general approach to enabling physical replication of the signature;

The raster scan is vectorised through a centerline trace, which is then converted to robot-compatible G-Code. The G-Code is executed on the robots, which then write the signature on paper.

For defence, a further 25 real signatures are signed by the two robots (and generated by the GAN). The model is fine-tuned for one epoch, with the resultant signatures included within the dataset labelled as forgeries. Therefore, this strategy exposes the verification model to robot-forged signatures with the aim of providing further enhanced awareness. The aim thus is to classify behaviours that the GAN and robots exhibit which humans necessarily do not.

In this section, the results of the proposed approach are presented. The verification model is first tuned and forgery methods are discussed. Following this, the robotic and generative adversarial attacks are performed and defended against with fine-tune transfer learning. Initially, Tables 1 and 2 present results for topology tuning with and without data augmentation, respectively. On average, augmentation led to a mean increase in accuracy by 5.25%, but required a mean of 50.55 additional epochs to meet the early stopping criteria. The best convolutional models for signature verification in terms of classification accuracy (87.12%) were found to be those with 2 and 3 convolutional layers, both with 32 filters. In terms of metrics, 32 filters within 2 layers were slightly more stable. As can be observed from Table  3 , other dense interpretation networks were also tested; leading to an overall best score of 87.12%. Figure 4 shows a real signature and the two counterparts forged by the iDraw 2.0 and Line-us robots. It can be observed that the iDraw 2.0 arguably creates the most realistic forgery, but features such as line intersections can differentiate it from the genuine image. Both robots seem to lose some detail, seen here via the minute squiggles in the line within the middle name; though this is not necessarily indicative of forgery given that human beings will also change the level of detail given due to reasons such as situationally dependent writing speed. Figures 5 and 6 show the generator and discriminator losses for the Conditional GAN, respectively. Mode collapse did not occur with the chosen hyperparameters, and it was observed that relatively little change in the outputs occurred beyond epoch 1000. Realistic images reminiscent of signatures were also observed to form following the first 300 epochs, which can be seen in Figure 7 . Figure 8 shows examples of how the GAN generalises "Real" signatures within the latter stages of training.

In some examples letters can be observed (e.g. the capital B in the first output), but it seems that the generator produces generalised clouds of pixels which can fool the discriminator. Figure 9 shows a comparison of the first GAN example and capital B within a signature in the data set. Table 4 shows the results of attacks on the system through robotic and GAN-based forgeries. All successful attacks are outside of the model baseline (87.12%) and show the dangers of signature forgeries using such methods. Following tuning, a successful defence is then mounted against these approaches as shown by Table 3 , wherein signature verification ability is improved by a minimum of 12% and maximum of 24.7%. Fine-tuning with examples brings the success of attack within the expected margin of error of the model. 

This work has shown the dangers surrounding two adversarial attack methods on vision-based writer-independent signature verification models. Although the base model experienced over 87% classification accuracy, successful attacks by robotic and GAN-based approaches were far above this margin of error. The Line-us and iDraw 2.0 robots were observed to fool the system 24% and 32% of the time, respectively. The Conditional GAN fooled the system 40% of the time with a set of 25 signatures, and 29.7% of the time with a set of 1000 signatures. In conclusion of this part of the study, the results have brought to the forefront information security issues when human versus human signatures forgeries are focused upon while a robot can achieve much more similar forgeries to the real human. Given this, biometric security could therefore be overcome by using these robots, which are low in cost and easily accessible.

In the second part of the study, the verification model was fine-tuned with some examples generated by the approaches. Given exposure to robotic and GAN behaviours, the model could then prevent far more of these attacks; the accuracy of the attack was reduced by 12 and 24 percentage points for the Line-us (12%) and iDraw 2.0 robots (8%), respectively. The generative approach was successful only 4% of the time for 25 images and 5% of the time for a set of 1000 images. Given that these results were below the margin of error of the model, an acceptable defence has been mounted.

In future, the study could be extended by allowing robots to generate a larger set of signatures. This is argued for by the two sets of results from the GAN (25 and 1000 images), which have a noticeable difference. In addition, further image classification approaches could be explored to form a stronger base model, such as attention modelling and residual information.

To finally conclude, this study has shown that it is relatively easy to attack a signature verification model by generating forgeries physically with robots and digitally with generative approaches. This study has also shown a method to defend against such attacks through fine-tune transfer learning. The results after fine-tuning are effective, showing that such attacks can be prevented now, rather than after the first consumer robot-based forgery crime has been committed.

The forgery and counterfeiting act 1981

The crime of forgery

Detecting forgery: forensic investigation of documents

Sports celebrity photographs and copyright law in the united states

Insurance against check forgery

UK finance

Rise in check fraud could motivate treasurers to switch to other payment tools

Secure fingerprint authentication using deep learning and minutiae verification

Voice biometrics: Deep learning-based voiceprint authentication system

Overcoming data scarcity in speaker identification: Dataset augmentation with synthetic mfccs via character-level rnn

Learning deep features for task-independent eeg-based biometric verification

Edith: Ecg biometrics aided by deep learning for reliable individual authentication

A multi-biometric iris recognition system based on a deep learning approach

Biosecure signature evaluation campaign (bsec'2009): Evaluating online signature algorithms depending on the quality of signatures

A review of signature recognition using machine learning

Offline signature verification and identification using distance statistics

Robustness of offline signature verification based on gray level features

Offline signature verification using deep learning convolutional neural network (cnn) architectures googlenet inception-v1 and inception-v3

Convolutional neural network based offline signature verification application

A writer-independent approach for offline signature verification using deep convolutional neural networks features

Bayesian hill-climbing attack and its application to signature verification

Black-box attack against handwritten signature verification with regionrestricted adversarial perturbations

Evaluation of brute-force attack to dynamic signature verification using synthetic samples

Distance-level fusion strategies for online signature verification

Fortifying the dalì attack on digital signature

Robustness of signature verification systems to imitators with increasing skills

Machine learning for signature verification