Causal LTI Systems Described by Differential and Difference Equations
This is a continuation from the previous tutorial - properties of linear time-invariant (LTI) systems.
An extremely important class of continuous-time systems is that for which the input and output are related through a linear constant-coefficient differential equation.
Equations of this type arise in the description of a wide variety of systems and physical phenomena. For example, as we illustrated in the continuous-time and discrete-time signals tutorial, the response of the RC circuit in Figure 1.1 and the motion of a vehicle subject to acceleration inputs and frictional forces, as depicted in Figure 1.2, can both be described through linear constant-coefficient differential equations.
Similar differential equations arise in the description of mechanical systems containing restoring and damping forces, in the kinetics of chemical reactions, and in many other contexts as well.
Correspondingly, an important class of discrete-time systems is that for which the input and output are related through a linear constant-coefficient difference equation.
Equations of this type are used to describe the sequential behavior of many different processes. For instance, in Example 1.10 [refer to the continuous-time and discrete-time systems tutorial] we saw how difference equations arise in describing the accumulation of savings in a bank account, and in Example 1.11 [refer to the continuous-time and discrete-time systems tutorial] we saw how they can be used to describe a digital simulation of a continuous-time system described by a differential equation.
Difference equations also arise quite frequently in the specification of discrete-time systems designed to perform particular operations on the input signal. For example, the system that calculates the difference between successive input values, as in eq. (1.99) [refer to the continuous-time and discrete-time systems tutorial], and the system described by eq. (1.104) [refer to the continuous-time and discrete-time systems tutorial] that computes the average value of the input over an interval are described by difference equations.
In this tutorial we introduce some of the basic ideas involved in solving differential and difference equations and uncover and explore some of the properties of systems described by such equations.
In later tutorials, we develop additional tools for the analysis of signals and systems that will add considerably both to our ability to analyze systems described by such equations and to our understanding of their characteristics and behavior.
1. Linear Constant-Coefficient Differential Equations
To introduce some of the important ideas concerning systems specified by linear constant-coefficient differential equations, let us consider a first-order differential equation as in eq.(1.85) [refer to the continuous-time and discrete-time systems tutorial], viz.,
\[\tag{2.95}\frac{\text{d}y(t)}{\text{d}t}+2y(t)=x(t)\]
where \(y(t)\) denotes the output of the system and \(x(t)\) is the input.
For example, comparing eq. (2.95) to the differential equation (1.84) [refer to the continuous-time and discrete-time systems tutorial] for the velocity of a vehicle subject to applied and frictional forces, we see that eq. (2.95) would correspond exactly to this system if \(y(t)\) were identified with the vehicle's velocity \(v(t)\), if \(x(t)\) were taken as the applied force \(f(t)\), and if the parameters in eq. (1.84) were normalized in units such that \(\rho/m=2\) and \(1/m=1\).
A very important point about differential equations such as eq. (2.95) is that they
provide an implicit specification of the system. That is, they describe a relationship between the input and the output, rather than an explicit expression for the system output as a function of the input.
In order to obtain an explicit expression, we must solve the differential equation. To find a solution, we need more information than that provided by the differential equation alone.
For example, to determine the speed of an automobile at the end of a 10-second interval when it has been subjected to a constant acceleration of 1 m/sec2 for 10 seconds, we would also need to know how fast the vehicle was moving at the start of the interval.
Similarly, if we are told that a constant source voltage of 1 volt is applied to the RC circuit in Figure 1.1 [refer to the continuous-time and discrete-time signals tutorial] for 10 seconds, we cannot determine what the capacitor voltage is at the end of that interval without also knowing what the initial capacitor voltage is.
More generally, to solve a differential equation, we must specify one or more auxiliary conditions, and once these are specified, we can then, in principle, obtain an explicit expression for the output in terms of the input.
In other words, a differential equation such as eq. (2.95) describes a constraint between the input and the output of a system, but to characterize the system completely, we must also specify auxiliary conditions. Different choices for these auxiliary conditions then lead to different relationships between the input and the output.
For the most part, we will focus on the use of differential equations to describe causal LTI systems, and for such systems the auxiliary conditions take a particular, simple form. To illustrate this and to uncover some of the basic properties of the solutions to differential equations, let us take a look at the solution of eq. (2.95) for a specific input signal \(x(t)\).
Example 2.14
Consider the solution of eq. (2.95) when the input signal is
\[\tag{2.96}x(t)=Ke^{3t}u(t)\]
where \(K\) is a real number.
The complete solution to eq. (2.96) consists of the sum of a particular solution, \(y_p(t)\), and a homogenous solution, \(y_h(t)\), i.e.,
\[\tag{2.97}y(t)=y_p(t)+y_h(t)\]
where the particular solution satisfies eq. (2.95) and \(y_h(t)\) is a solution of the homogenous differential equation
\[\tag{2.98}\frac{\text{d}y(t)}{\text{d}t}+2y(t)=0\]
A common method for finding the particular solution for an exponential input signal as in eq. (2.96) is to look for so-called forced response—i.e., a signal of the same form as the input.
With regard to eq. (2.95), since \(x(t)=Ke^{3t}\) for \(t\gt0\), we hypothesize a solution for \(t\gt0\) of the form
\[\tag{2.99}y_p(t)=Ye^{3t}\]
where \(Y\) is a number that we must determine.
Substituting eqs. (2.96) and (2.99) into eq. (2.95) for \(t\gt0\) yields
\[\tag{2.100}3Ye^{3t}+2Ye^{3t}=Ke^{3t}\]
Canceling the factor \(e^{3t}\) from both sides of eq. (2.100), we obtain
\[\tag{2.101}3Y+2Y=K\]
or
\[\tag{2.102}Y=\frac{K}{5}\]
so that
\[\tag{2.103}y_p(t)=\frac{K}{5}e^{3t},\qquad\text{for }t\gt0\]
In order to determine \(y_h(t)\), we hypothesize a solution of the form
\[\tag{2.104}y_h(t)=Ae^{st}\]
Substituting this into eq. (2.98) gives
\[\tag{2.105}Ase^{st}+2Ae^{st}=Ae^{st}(s+2)=0\]
From this equation, we see that we must take \(s=-2\) and that \(Ae^{-2t}\) is a solution to eq. (2.98) for any choice of \(A\).
Utilizing this fact and eq. (2.103) in eq. (2.97), we find that the solution of the differential equation for \(t\gt0\) is
\[\tag{2.106}y(t)=Ae^{-2t}+\frac{K}{5}e^{3t},\qquad{t}\gt0\]
As noted earlier, the differential equation (2.95) by itself does not specify uniquely the response \(y(t)\) to the input \(x(t)\) in eq. (2.96). In particular, the constant \(A\) in eq. (2.106) has not yet been determined.
In order for the value of \(A\) to be determined, we need to specify an auxiliary condition in addition to the differential equation (2.95). Different choices for this auxiliary condition lead to different solutions \(y(t)\) and, consequently, to different relationships between the input and the output.
As we have indicated, we focus on differential and difference equations used to describe systems that are LTI and causal, and in this case the auxiliary condition takes the form of the condition of initial rest. That is, for a causal LTI system, if \(x(t)=0\) for \(t\lt{t_0}\), then \(y(t)\) must also equal 0 for \(t\lt{t_0}\).
From eq. (2.96), we see that for our example \(x(t)=0\) for \(t\lt0\), and thus, the condition of initial rest implies that \(y(t)=0\) for \(t\lt0\).
Evaluating eq. (2.106) at \(t=0\) and setting \(y(0)=0\) yields
\[0=A+\frac{K}{5}\]
or
\[A=-\frac{K}{5}\]
Thus, for \(t\gt0\),
\[\tag{2.107}y(t)=\frac{K}{5}\left[e^{3t}-e^{-2t}\right]\]
while for \(t\lt0\), \(y(t)=0\), because of the condition of initial rest.
Combining these two cases, we obtain the full solution
\[\tag{2.08}y(t)=\frac{K}{5}\left[e^{3t}-e^{-2t}\right]u(t)\]
Example 2.14 illustrates several very important points concerning linear constant-coefficient differential equations and the systems they represent.
First, the response to an input \(x(t)\) will generally consist of the sum of a particular solution to the differential equation and a homogeneous solution—i.e., a solution to the differential equation with the input set to zero. The homogeneous solution is often referred to as the natural response of the system.
In Example 2.14 we also saw that, in order to determine completely the relationship between the input and the output of a system described by a differential equation such as eq. (2.95), we must specify auxiliary conditions.
An implication of this fact is that different choices of auxiliary conditions lead to different relationships between the input and the output.
As we illustrated in the example, for the most part we will use the condition of initial rest for systems described by differential equations. In the example, since the input was \(0\) for \(t\lt0\), the condition of initial rest implied the initial condition \(y(0)=0\).
As we have stated, under the condition of initial rest the system described by eq. (2.95) is LTI and causal. For example, if we multiply the input in eq. (2.96) by 2, the resulting output would be twice the output in eq. (2.108).
It is important to emphasize that the condition of initial rest does not specify a zero initial condition at a fixed point in time, but rather adjusts this point in time so that the response is zero until the input becomes nonzero.
Thus, if \(x(t)=0\) for \(t\le{t_0}\) for the causal LTI system described by eq. (2.95), then \(y(t)=0\) for \(t\le{t_0}\), and we would use the initial condition \(y(t_0)=0\) to solve for the output for \(t\gt{t_0}\).
As a physical example, consider again the circuit in Figure 1.1 [refer to the continuous-time and discrete-time signals tutorial]. Initial rest for this example corresponds to the statement that, until we connect a nonzero voltage source to the circuit, the capacitor voltage is zero. Thus, if we begin to use the circuit at noon today, the initial capacitor voltage as we connect the voltage source at noon today is zero. Similarly, if we begin to use the circuit at noon tomorrow instead, the initial capacitor voltage as we connect the voltage source at noon tomorrow is zero.
This example also provides us with some intuition as to why the condition of initial rest makes a system described by a linear constant-coefficient differential equation time invariant. For example, if we perform an experiment on the circuit, starting from initial rest, then, assuming that the coefficients \(R\) and \(C\) don't change over time, we would expect to get the same results whether we ran the experiment today or tomorrow. That is, if we perform identical experiments on the two days, where the circuit starts from initial rest at noon on each day, then we would expect to see identical responses—i.e., responses that are simply time-shifted by one day with respect to each other.
While we have used the first-order differential equation (2.95) as the vehicle for the discussion of these issues, the same ideas extend directly to systems described by higher order differential equations.
A general \(N\)th-order linear constant-coefficient differential equation is given by
\[\tag{2.109}\sum_{k=0}^Na_k\frac{\text{d}^ky(t)}{\text{d}t^k}=\sum_{k=0}^Mb_k\frac{\text{d}^kx(t)}{\text{d}t^k}\]
The order refers to the highest derivative of the output \(y(t)\) appearing in the equation.
In the case when \(N=0\), eq. (2.109) reduces to
\[\tag{2.110}y(t)=\frac{1}{a_0}\sum_{k=0}^Mb_k\frac{\text{d}^kx(t)}{\text{d}t^k}\]
In this case, \(y(t)\) is an explicit function of the input \(x(t)\) and its derivatives.
For \(N\ge1\), eq. (2.109) specifies the output implicitly in terms of the input. In this case, the analysis of the equation proceeds just as in our discussion of the first-order differential equation in Example 2.14.
The solution \(y(t)\) consists of two parts—a particular solution to eq. (2.109) plus a solution to the homogenous differential equation
\[\tag{2.111}\sum_{k=0}^Na_k\frac{\text{d}^ky(t)}{\text{d}t^k}=0\]
The solutions to this equation are referred to as the natural responses of the system.
As in the first-order case, the differential equation (2.109) does not completely specify the output in terms of the input, and we need to identify auxiliary conditions to determine completely the input-output relationship for the system.
Once again, different choices for these auxiliary conditions result in different input-output relationships, but for the most part, we will use the condition of initial rest when dealing with systems described by differential equations. That is, if \(x(t)=0\) for \(t\le{t_0}\), we assume that \(y(t)=0\) for \(t\le{t_0}\), and therefore, the response for \(t\gt{t_0}\) can be calculated from the differential equation (2.109) with the initial conditions
\[\tag{2.112}y(t_0)=\frac{\text{d}y(t_0)}{\text{d}t}=\ldots=\frac{\text{d}^{N-1}y(t_0)}{\text{d}t^{N-1}}=0\]
Under the condition of initial rest, the system described by eq. (2.109) is causal and LTI. Given the initial conditions in eq. (2.112), the output \(y(t)\) can, in principle, be determined by solving the differential equation in the manner used in Example 2.14.
However, in later tutorials we will develop some tools for the analysis of continuous-time LTI systems that greatly facilitate the solution of differential equations and, in particular, provide us with powerful methods for analyzing and characterizing the properties of systems described by such equations.
2. Linear Constant-Coefficient Difference Equations
The discrete-time counterpart of eq. (2.109) is the \(N\)th-order linear constant-coefficient difference equation
\[\tag{2.113}\sum_{k=0}^Na_ky[n-k]=\sum_{k=0}^Mb_kx[n-k]\]
An equation of this type can be solved in a manner exactly analogous to that for differential equations. Specifically, the solution y[n] can be written as the sum of a particular solution to eq. (2.113) and a solution to the homogeneous equation
\[\tag{2.114}\sum_{k=0}^Na_ky[n-k]=0\]
The solutions to this homogeneous equation are often referred to as the natural responses of the system described by eq. (2.113).
As in the continuous-time case, eq. (2.113) does not completely specify the output
in terms of the input. To do this, we must also specify some auxiliary conditions.
While there are many possible choices for auxiliary conditions, leading to different input-output relationships, we will focus for the most part on the condition of initial rest—i.e., if \(x[n]=0\) for \(n\lt{n_0}\), then \(y[n]=0\) for \(n\lt{n_0}\) as well. With initial rest, the system described by eq. (2.113) is LTI and causal.
Although all of these properties can be developed following an approach that directly parallels our discussion for differential equations, the discrete-time case offers an alternative path. This stems from the observation that eq. (2.113) can be rearranged in the form
\[\tag{2.115}y[n]=\frac{1}{a_0}\left\{\sum_{k=0}^Mb_kx[n-k]-\sum_{k=1}^Na_ky[n-k]\right\}\]
Equation (2.115) directly expresses the output at time \(n\) in terms of previous values of the input and output. From this, we can immediately see the need for auxiliary conditions.
In order to calculate \(y[n]\), we need to know \(y[n-1],\ldots,y[n-N]\). Therefore, if we are given the input for all \(n\) and a set of auxiliary conditions such as \(y[-N],y[-N+1],\ldots,y[-1]\), eq. (2.115) can be solved for successive values of \(y[n]\).
An equation of the form of eq. (2.113) or eq. (2.115) is called a recursive equation, since it specifies a recursive procedure for determining the output in terms of the input and previous outputs.
In the special case when \(N=0\), eq. (2.115) reduces to
\[\tag{2.116}y[n]=\sum_{k=0}^M\left(\frac{b_k}{a_0}\right)x[n-k]\]
This is the discrete-time counterpart of the continuous-time system given in eq. (2.110). Here, \(y[n]\) is an explicit function of the present and previous values of the input. For this reason, eq. (2.116) is often called a nonrecursive equation, since we do not recursively use previously computed values of the output to compute the present value of the output.
Therefore, just as in the case of the system given in eq. (2.110), we do not need auxiliary conditions in order to determine \(y[n]\) . Furthermore, eq. (2.116) describes an LTI system, and by direct computation, the impulse response of this system is found to be
\[\tag{2.117}h[n]=\begin{cases}\frac{b_n}{a_0},\qquad0\le{n}\le{M}\\0,\quad\qquad\text{otherwise}\end{cases}\]
That is, eq. (2.116) is nothing more than the convolution sum.
Note that the impulse response for it has finite duration; that is, it is nonzero only over a finite time interval. Because of this property, the system specified by eq. (2.116) is often called a finite impulse response (FIR) system.
Although we do not require auxiliary conditions for the case of \(N=0\), such conditions are needed for the recursive case when \(N\ge1\).
To illustrate the solution of such an equation, and to gain some insight into the behavior and properties of recursive difference equations, let us examine the following simple example:
Example 2.15
Consider the difference equation
\[\tag{2.118}y[n]-\frac{1}{2}y[n-1]=x[n]\]
Eq. (2.118) can also be expressed in the form
\[\tag{2.119}y[n]=x[n]+\frac{1}{2}y[n-1]\]
highlighting the fact that we need the previous value of the output, \(y[n-1]\), to calculate the current value. Thus, to begin the recursion, we need an initial condition.
For example, suppose that we imposed the condition of initial rest and consider the input
\[\tag{2.120}x[n]=K\delta[n]\]
In this case, since \(x[n]=0\) for \(n\le-1\), the condition of initial rest implies that \(y[n]=0\) for \(n\le-1\), so that we have as an initial condition \(y[-1]=0\). Starting from this initial condition, we can solve for successive values of \(y[n]\) for \(n\ge0\) as follows:
\[\tag{2.121}y[0]=x[0]+\frac{1}{2}y[-1]=K\]
\[\tag{2.122}y[1]=x[1]+\frac{1}{2}y[0]=\frac{1}{2}K\]
\[\tag{2.123}y[2]=x[2]+\frac{1}{2}y[1]=\left(\frac{1}{2}\right)^2K\]
\[\vdots\]
\[\tag{2.124}y[n]=x[n]+\frac{1}{2}y[n-1]=\left(\frac{1}{2}\right)^nK\]
Since the system specified by eq. (2.118) and the condition of initial rest is LTI, its input-output behavior is completely characterized by its impulse response. Setting \(K=1\), we see that the impulse response for the system considered in this example is
\[\tag{2.125}h[n]=\left(\frac{1}{2}\right)^nu[n]\]
Note that the causal LTI system in Example 2.15 has an impulse response of infinite duration. In fact, if \(N\ge1\) in eq. (2.113), so that the difference equation is recursive, it is usually the case that the LTI system corresponding to this equation together with the condition of initial rest will have an impulse response of infinite duration. Such systems are commonly referred to as infinite impulse response (IIR) systems.
As we have indicated, for the most part we will use recursive difference equations in the context of describing and analyzing systems that are linear, time-invariant, and causal, and consequently, we will usually make the assumption of initial rest.
In later tutorials we will develop tools for the analysis of discrete-time systems that will provide us with very useful and efficient methods for solving linear constant-coefficient difference equations and for analyzing the properties of the systems that they describe.
3. Block Diagram Representations of First-Order Systems Described by Differential and Difference Equations
An important property of systems described by linear constant-coefficient difference and differential equations is that they can be represented in very simple and natural ways in terms of block diagram interconnections of elementary operations.
This is significant for a number of reasons. One is that it provides a pictorial representation which can add to our understanding of the behavior and properties of these systems.
In addition, such representations can be of considerable value for the simulation or implementation of the systems. For example, the block diagram representation to be introduced in this tutorial for continuous-time systems is the basis for early analog computer simulations of systems described by differential equations, and it can also be directly translated into a program for the simulation of such a system on a digital computer.
In addition, the corresponding representation for discrete-time difference equations suggests simple and efficient ways in which the systems that the equations describe can be implemented in digital hardware.
In this tutorial, we illustrate the basic ideas behind these block diagram representations by constructing them for the causal first-order systems introduced in Examples 1.8-1.11.
We begin with the discrete-time case and, in particular, the causal system described by the first-order difference equation
\[\tag{2.126}y[n]+ay[n-1]=bx[n]\]
To develop a block diagram representation of this system, note that the evaluation of eq. (2.126) requires three basic operations: addition, multiplication by a coefficient, and delay (to capture the relationship between \(y[n]\) and \(y[n-1]\).
Thus, let us define three basic network elements, as indicated in Figure 2.27. To see how these basic elements can be used to represent the causal system described by eq. (2.126), we rewrite this equation in the form that directly suggests a recursive algorithm for computing successive values of the output \(y[n]\):
\[\tag{2.127}y[n]=-ay[n-1]+bx[n]\]
This algorithm is represented pictorially in Figure 2.28, which is an example of a feedback system, since the output is fed back through a delay and a multiplication by a coefficient and is then added to \(bx[n]\). The presence of feedback is a direct consequence of the recursive nature of eq. (2.127).
The block diagram in Figure 2.28 makes clear the required memory in this system and the consequent need for initial conditions.
In particular, a delay corresponds to a memory element, as the element must retain the previous value of its input. Thus, the initial value of this memory element serves as a necessary initial condition for the recursive calculation specified pictorially in Figure 2.28 and mathematically in eq. (2.127).
Of course, if the system described by eq. (2.126) is initially at rest, the initial value stored in the memory element is zero.
Consider next the causal continuous-time system described by a first-order differential equation:
\[\tag{2.128}\frac{\text{d}y(t)}{\text{d}t}+ay(t)=bx(t)\]
As a first attempt at defining a block diagram representation for this system, let us rewrite it as
\[\tag{2.129}y(t)=-\frac{1}{a}\frac{\text{d}y(t)}{\text{d}t}+\frac{b}{a}x(t)\]
The right-hand side of this equation involves three basic operations: addition, multiplication by a coefficient, and differentiation.
Therefore, if we define the three basic network elements indicated in Figure 2.29, we can consider representing eq. (2.129) as an interconnection of these basic elements in a manner analogous to that used for the discrete-time system described previously, resulting in the block diagram of Figure 2.30.
While the latter figure is a valid representation of the causal system described by eq. (2.128), it is not the representation that is most frequently used or the representation that leads directly to practical implementations, since differentiators are both difficult to implement and extremely sensitive to errors and noise.
An alternative implementation that is much more widely used can be obtained by first rewriting eq. (2.128) as
\[\tag{2.130}\frac{\text{d}y(t)}{\text{d}t}=bx(t)-ay(t)\]
and then integrating from \(-\infty\) to \(t\).
Specifically, if we assume that in the system described by eq. (2.130) the value of \(y(-\infty)\) is zero, then the integral of \(\text{d}y(t)/\text{d}t\) from \(-\infty\) to \(t\) is precisely \(y(t)\).
Consequently, we obtain the equation
\[\tag{2.131}y(t)=\displaystyle\int\limits_{-\infty}^t[bx(\tau)-ay(\tau)]\text{d}\tau\]
In this form, our system can be implemented using the adder and coefficient multiplier indicated in Figure 2.29, together with an integrator, as defined in Figure 2.31.
Figure 2.32 is a block diagram representation for this system using these elements.
Since integrators can be readily implemented using operational amplifiers, representations such as that in Figure 2.32 lead directly to analog implementations, and indeed, this is the basis for both early analog computers and modern analog computation systems.
Note that in the continuous-time case it is the integrator that represents the memory storage element of the system. This is perhaps more readily seen if we consider integrating eq. (2.130) from a finite point in time \(t_0\), resulting in the expression
\[\tag{2.132}y(t)=y(t_0)+\displaystyle\int\limits_{t_0}^t[bx(\tau)-ay(\tau)]\text{d}\tau\]
Equation (2.132) makes clear the fact that the specification of \(y(t)\) requires an initial condition, namely, the value of \(y(t_0)\). It is precisely this value that the integrator stores at time \(t_0\).
While we have illustrated block diagram constructions only for the simplest first-order differential and difference equations, such block diagrams can also be developed for higher order systems, providing both valuable intuition for and possible implementations of these systems.
The next tutorial introduces singularity functions.