key: cord-0864341-vccq39ql
authors: Guan, Yu; Fang, Tingting; Zhang, Diankun; Jin, Congming
title: Solving Fredholm Integral Equations Using Deep Learning
date: 2022-03-29
journal: Int J Appl Comput Math
DOI: 10.1007/s40819-022-01288-3
sha: 2a0721e363d4980c72f3a2212258b26e8738b44b
doc_id: 864341
cord_uid: vccq39ql

The aim of this paper is to provide a deep learning based method that can solve high-dimensional Fredholm integral equations. A deep residual neural network is constructed at a fixed number of collocation points selected randomly in the integration domain. The loss function of the deep residual neural network is defined as a linear least-square problem using the integral equation at the collocation points in the training set. The training iteration is done for the same set of parameters for different training sets. The numerical experiments show that the deep learning method is efficient with a moderate generalization error at all points. And the computational cost does not suffer from “curse of dimensionality” problem.

Integral equations have wide applications in electrical engineering [1] , optics [2] , mathematical biology [3] and other fields. The most popular integral equations are the Fredhom integral equations and the Volterra integral equations. The Fredholm integral equation can be considered as a reformulation of the elliptic partial differential equation and the Volterra integral equation is a reformulation of the fractional-order differential equation, which has wide applications in modeling the real problems, for instance, the chaotic system [4] , the dynamics of COVID-19 [5] , the motion of beam on nanowire [6] , the capacitor microphone dynamical system [7] , etc. Since these integral equations usually can not be solved explicitly, numerical methods are necessary to be considered.

We consider the linear Fredholm integral equation of the second kind

where x, y ∈ ⊂ R m , the function g(x) and the kernel k(x, y) are given, and f (x) is the unknown that we want to find. So far, many numerical methods have been proposed to solve the Fredholm integral equations, for example, the Nyström method [3, 8, 9] , the Galerkin method [10] , the wavelet analysis method [11] , the neural network [12, 13] , the collocation method [14] , the maximum entropy method [15] , etc. However, most of these traditional methods can only solve low-dimensional Fredholm integral equations and suffer from "curse of dimensionality".

The neural network has been successful in solving partial differential equations in mathematical modelling and the applied science, such as medical smoking model [16] , nonlinear high order singular models [17] , food chain system [18] [19] [20] , Liénard differential model [21] , etc. The neural network was also used to solve the Fredholm integral equations in [12, 13] , where the authors only evaluated the approximation at some fixed points without generalization. And the integral was evaluated using numerical integral method whose cost depends on the dimension exponentially.

In recent years, deep learning method has been successfully used in artificial intelligence solving high-dimensional problems, such as image recognition [22, 23] , speech recognition [24, 25] , natural language processing [26] , and also in mathematical problems [27] [28] [29] and physical problems [30] .

E and his collaborators have done a series of works on solving high-dimensional differential equations based on deep learning method. In [28] , a deep learning-based algorithm was proposed for solving high-dimensional semilinear parabolic partial differential equations and reverse stochastic differential equations from a relation between BSDE (backward stochastic differential equations) and reinforcement learning. In [29] , the deep Ritz method for elliptic differential equations was given by numerically solving variational problems. In [27] , a machine learning approximation algorithm was raised to solve high-dimensional fully nonlinear second-order partial differential equations. These works show that deep learning method provides a new idea to solve high-dimensional mathematical problems.

In this paper, a deep residual neural network method is proposed to approximate the solution of the high-dimensional linear Fredholm integral equations of the second kind. Few novel highlights of this deep learning method are briefly provided as follows:

• A deep residual neural network is constructed to solve numerically the linear Fredholm integral equations of the second kind. • The proposed method can solve high-dimensional Fredholm integral equations and does not suffer from "curse of dimensionality" problem, that is the cost depends on the dimension linearly. • The reasonable absolute error values validate the reliability of the deep learning method.

• The proposed method has a small generalization error in the domain. This paper is organized as follows. In Sect. 2 we construct a deep residual neutral network for solving the Fredholm integral equations. In Sect. 3, some numerical experiments are given to show the efficiency of the numerical method. The conclusion is given in Sect. 4.

The output F(x, θ ) of the neural network is a composite function of the input x, where θ denotes the parameters of the neural network including the weighs and bias. Let x be any point in the domain . Now we want to train a deep neural network whose output F(x, θ ) is 

To learn the parameters θ , and so the function F(x, θ ), take randomly n points {x 1 , x 2 , · · · , x n } with a uniform distribution as the training set. Initializing the parameter vector θ , the prediction values F(x i , θ ) for i = 1, 2, · · · , n, can be obtained by forward propagation neural network.

Define the loss function as

The training of the neural network is to minimize the loss function (2) by the backward propagation neural network, which is a least-square problem

In Eq. (2) the integral term k(x i , y)F(y, θ)dy can be evaluated using the Monte Carlo method, leading to

where β = dx is the volume of .

The training can be done repeatedly for different training set until we get a stationary loss function.

As the network deepens, minimizing the loss function has great difficulties, such as vanishing gradient problem, gradient explosion, and degradation problem. The residual neural network can avoid the vanishing gradient problem and may greatly improve the solution. It also can reduce the risk of over-adapting the parameters to a specific dataset [22] . A residual block is shown in Fig. 1 , where an identity shortcut connection is added to a shallow neural network, whose output is H (x) = ϕ(x) + x, where ϕ(x) is the output of the shallow neural network. Then the output of the residual block is taken as the input of the next residual block.

Our algorithm of deep residual neural network for solving Fredholm integral equations is shown in algorithm 1.

Input: The number of training points n and the number of training iterations M; Output: The parameters θ of the residual neural network; 1: Initialize θ randomly; 2: for k = 1; k ≤ M; k + + do 3:

Sample the region with a uniform distribution to generate the training set {x 1 , x 2 , · · · , x n }; 4:

Minimize the loss function in equation (2) by the following iteration. 5:

while not converge do 6:

Forward propagate the neural network to get F(x i , θ) , i = 1, 2, · · · , n; 7:

Back propagate the neural network to update θ; 8:

Sample with a uniform distribution to generate the test set and evaluate the generalization error on the test set; 9: end while 10: end for 11: Output θ and obtain the approximate solution of the Fredholm integral equation;

In this section, several Fredholm integral equations are numerically solved using algorithm 1. In the numerical experiments, n = 1000 points in are randomly sampled uniformly as the training set to train the deep residual neural network, and the number of training iterations is M = 2000. The neural network consists of one input layer, two blocks of residual neural network shown in Fig. 1 , and one output layer. There are 30 neurons in the second layer and the forth layer and 10 neurons in the other layers. The ReLU function is used as the active function in the neural network. Minimization is realized by "AdamOptimizer" [31] built in TensorFlow (version 1.13.1 ) with a learning rate 0.001.

To measure the efficiency of the deep learning method for solving the Fredholm integral equations, we consider several examples whose exact solutions are known. Denote f * (x 1 ), f * (x 2 ), · · · , f * (x n ) as the exact solutions at n points x 1 , x 2 , · · · , x n in the test set. Define the generalization error between the exact solution f * (x) of the integral equation and the approximate solution F(x, θ ) obtained by using the deep residual neural network as

The generalization error is evaluated for each example in the following numerical experiments. For Example 1, the convergence of the loss function and the generalization error are shown in Fig. 2 . Some typical iteration data of the loss function (loss) and the generalization error (err or ) are given in Table 1 . The loss function converges to 10 −5 , and the generalization error converges to 10 −3 .

f (x, y, z) + 2 1 2 1 2 1 k(x, y, z, s, t, v) f (s, t, v)dsdtdv = g(x, y, z),(5)

Eq. (5) , that is

The exact solution is f * (x) = 1.

For Example 2, when the dimension m = 100, the convergence of the loss function and the generalization error are shown in Fig. 3 . Some typical iteration data of the loss function (loss) and the generalization error (err or ) are given in Table 2 . The loss function converges to 10 −4 , and the generalization error converges to 10 −3 . The exact solution is f * (x, y, z, w) = x yzw.

For Example 3, the convergence of the loss function and the generalization error are shown in Fig. 4 , and some typical iteration data of the loss function (loss) and the generalization error (err or ) are given in Table 3 . The loss function and the generalized error function synchronously converge very fast to a stable state. The loss function converges to 10 −3 , and the generalization error converges to 10 −2 .

Example 4 Consider a high-dimensional version of the four-dimensional Fredholm integral Eq. (6) , that is 

The exact solution of the equation is f * (x) = x 1 x 2 · · · x m .

For Example 4, when the dimension m = 100, the convergence of the loss function and the generalization error are shown in Fig. 5 , and some typical iteration data of the loss function (loss) and the generalization error (err or ) are given in Table 4 . The loss function converges to 10 −7 , and the generalization error converges to 10 −4 .

In this paper, we propose a deep learning method based on the residual neural network to solve numerically the linear Fredholm integral equations of the second kind. The output of the deep residual network is used as the numerical solution. The loss function is defined using the Fredholm integral equation. The loss function is optimized by Adam method built in TensorFlow. Then the numerical results, including high-dimensional problems, confirm the efficiency of the method. The main advantage of this method is that it can solve highdimensional Fredholm integral equations with a cost less sensitive to the dimensionality of the problem. The accuracy of the residual neural network is not as good as that of the traditional method, such as the Galerkin method. Some error analysis of the neural network has been discussed in [32] [33] [34] . But so far rigorous error analysis for neural network can not be given yet. The error of the neural network consists of three parts, that is the error between the space of the output of the neural network and the exact solution of the Fredholm integral equation, the optimization error in Eq. (3), and the approximation error in Eq. (4). The error in our numerical experiments has a good accuracy compared to the error of the Monte Carlo method 1/ √ n in Eq. (4). In the future we will explore more techniques or theory to improve the convergent accuracy. Additionally, we will try to construct a deep residual neural network to solve the Volterra integral equations.

Multilayered media Green's functions in integral equation formulations

Geometric-optics-integral-equation method for light scattering by nonspherical ice crystals

Nystörm methods for approximating the solutions of an integral equation arising from a problem in mathematical biology

A nonstandard finite difference scheme for the modelling and nonidentical synchronization of a novel fractional chaotic system

A new comparative study on the general fractional model of COVID-19 with isolation and quarantine effects

Novel fractional-order Lagrangian to describe motion of beam on nanowire

A new and general fractional Lagrangian approach: a capacitor microphone case study

Iterative variants of the Nyström method for the numerical solution of integral equations

Nyström method for solution of Fredholm integral equations of the second kind under interval data

Richardson extrapolation of iterated discrete Galerkin solution for two-dimensional Fredholm integral equations

Numerical solution of nonlinear Fredholm integral equations of the second kind using Haar wavelets

Utilizing artificial neural network approach for solving twodimensional integral equations

A neural network approach for solving Fredholm integral equations of the second kind

Taylor collocation method and convergence analysis for the Volterra-Fredholm integral equations

Solving Fredholm integral equations via a piecewise linear maximum entropy method

An advanced heuristic approach for a nonlinear mathematical based medical smoking model

An efficient stochastic numerical computing framework for the nonlinear higher order singular models

Stochastic numerical investigations for nonlinear three-species food chain system

Evolutionary heuristic with Gudermannian neural networks for the nonlinear singular models of third kind

Gudermannian neural networks using the optimization procedures of genetic algorithm and active set approach for the three-species food chain nonlinear model

Gudermannian neural networks to investigate the Liénard differential model

Deep residual learning for image recognition

Deep learning face representation by joint identification-verification

Speech recognition with deep recurrent neural networks

Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups

Recent trends in deep learning based natural language processing

Machine learning approximation algorithms for high-dimensional fully nonlinear partial differential equations and second-order backward stochastic differential equations

Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations

The deep Ritz method: a deep learning-based numerical algorithm for solving variational problems

Deep potential molecular dynamics: a scalable model with the accuracy of quantum mechanics

Adam: A Method for stochastic Optimization

Universal approximation bounds for superpositions of a sigmoidal function

Approximation and estimation for high-dimensional deep learning networks

Towards a mathematical understanding of neural network-based machine learning: what we know and what we don

Publisher's Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations

The research of Congming Jin was supported by the National Natural Science Foundation