Optimisation methods

Due to their large variety of applications, complex optimisation problems induced a great effort to develop efficient solution techniques, dealing with both continuous and discrete variables involved in non-linear functions. But among the diversity of those optimisation methods, the choice of the relevant technique for the treatment of a given problem keeps being a thorny issue. Within the Process Engineering context, batch plant design problems provide a good framework to test the performances of various optimisation methods : on the one hand, two Mathematical Programming techniques-DICOPT++ and SBB, implemented in the GAMS environment-and, on the other hand, one stochastic method, i.e. a genetic algorithm. Seven examples, showing an increasing complexity, were solved with these three techniques. The result comparison enables to evaluate their efficiency in order to highlight the most appropriate method for a given problem instance. It was proved that the best performing method is SBB, even if the GA also provides interesting solutions, in terms of quality as well as of computational time.

Due to their large variety of applications, complex optimisation problems induced a great effort to develop efficient solution techniques, dealing with both continuous and discrete variables involved in non-linear functions. But among the diversity of those optimisation methods, the choice of the relevant technique for the treatment of a given problem keeps being a thorny issue. Within the Process Engineering context, batch plant design problems provide a good framework to test the performances of various optimisation methods : on the one hand, two Mathematical Programming techniques-DICOPT++ and SBB, implemented in the GAMS environment-and, on the other hand, one stochastic method, i.e. a genetic algorithm. Seven examples, showing an increasing complexity, were solved with these three techniques. The result comparison enables to evaluate their efficiency in order to highlight the most appropriate method for a given problem instance. It was proved that the best performing method is SBB, even if the GA also provides interesting solutions, in terms of quality as well as of computational time.

We discuss the use of preconditioned conjugate gradients method for solving the reduced KKT systems arising in interior point algorithms for linear programming. The (indefinite) augmented system form of this linear system has a number of advantages, notably a higher degree of sparsity than the (positive definite) normal equations form. Therefore we use the conjugate gradients method to solve the augmented system and look for a suitable preconditioner. An explicit null space representation of linear constraints is constructed by using a nonsingular basis matrix identified from an estimate of the optimal partition in the linear program. This is achieved by means of recently developed efficient basis matrix factorisation techniques which exploit hyper-sparsity and are used in implementations of the revised simplex method. The approach has been implemented within the HOPDM interior point solver and applied to medium and large-scale problems from public domain test collections. Computational experience is encouraging.

This paper develops a novel particle swarm optimiser algorithm. The focus of this study is how to improve the performance of the classical particle swarm optimisation approach, i.e., how to enhance its convergence speed and capacity to solve complex problems while reducing the computational load. The proposed approach is based on an improvement of Particle Swarm Optimisation using Evolutionary Game Theory. This method maintains the capability of the particle swarm optimiser to diversify the particles' exploration in the solution space. Moreover, the proposed approach provides an important ability to the optimisation algorithm, that is adaptation of the search direction which improves the quality of the particles based on their experience. The proposed algorithm is tested on a representative set of continuous benchmark optimisation problems and compared with some other classical optimisation approaches. Based on the test results of each benchmark problem, its performance is analysed and discussed.

Preconditioned iterative methods provide an effective alternative to direct methods for the solution of the KKT linear systems arising in Interior Point algorithms, especially when large-scale problems are considered. We analyze the behaviour of a Constraint Preconditioner coupled with Krylov solvers, in a Potential Reduction (PR) framework. We present also adaptive stopping criteria for the inner iterations that relate the accuracy in the solution of the KKT system to the quality of the current PR iterate, to increase the overall computational efficiency. Numerical experiments on a set of large-scale problems show the effectiveness of this approach. [ DOI : 10.1685 / CSC06031] About DOI

In February 1979 a note by L. G. Khachiyan indicated how an ellipsoid method for linear programming can be implemented in polynomial time. This result has caused great excitement and stimulated a flood of technical papers. Ordinarily there would be no need for a survey of work so recent, but the current circumstances are obviously exceptional. Word of Khachiyan's result has spread extraordinarily fast, much faster than comprehension of its significance. A variety of issues have, in general, not been well understood, including the exact character of the ellipsoid method and of Khachiyans result on polynomiality, its practical significance in linear programming, its implementation, its potential applicability to problems outside of the domain of linear programming, and its relationship to earlier work. Our aim is to help clarify these important issues in the context of a survey of the ellipsoid method, its historical antecedents, recent developments, and current research.

Responding to the international calls for high energy performance buildings like nearly-zero energy buildings (nZEB), recent years have seen significant growth in energy-saving and energy-supply measures in the building sector. A detailed look at the possible combinations of measures indicates that there could be a huge number (possibly millions) of candidate designs. In exploring this number of designs, looking for optimal ones is an arduous multi-objective design task. Buildings are required to be not only energy-efficient but also economically feasible and environmentally sound while adhering to an ever-increasing demand for better indoor comfort levels. The current thesis introduces suitable methods and techniques that attempt to carry out time-efficient multivariate explorations and transparent multi-objective analysis for optimizing such complex building design problems. The thesis's experiences can be considered as seeds for developing a generic simulation-based optimisation design tool for high-energyperformance buildings. Case studies are made to illustrate the effectiveness of the introduced methods and techniques. In all the studies, IDA-ICE is used for simulation and MATLAB is implemented for optimisation as well as supplementary calculations. A new program (IDA-ESBO) is used to simulate renewable energy source systems (RESs). Using detailed simulation programs was important to investigate the impact of the energy-saving measures (ESMs) and the RESs as well as their effects on the thermal and/or energy performance of the studied buildings. The case studies yielded many optimal design concepts (e.g., the type of heating/cooling (H/C) system is a key element to achieve environmentally friendly buildings with minimum life cycle cost. The cost-optimal implementations of ESMs and RESs depend significantly on the installed H/C system). On building regulations, comments are taken. For instance, in line with the cost-optimal methodology framework of the European Energy Performance of Buildings Directive (EPBD-recast 2010), our study showed that the Finnish building regulation D3-2012 specifies minimum energy performance requirements for dwellings, lower than the estimated cost-optimal level by more than 15%. The adaptive thermal comfort criteria of the Finnish Society of Indoor Air Quality (FiSIAQ-2008) are strict and do not allow for energy-efficient solutions in standard office buildings. The thesis shows that it is technically possible to speed up the optimisation resolution of the building and HVAC design problems and to reach an optimal or close-to-optimal solution set. A simulation-based optimisation approach with a suitable problem setup and resolution algorithm can efficiently explore the possible combinations of design options and support informative, optimal results for decision-makers.

In this paper, we address the preconditioned iterative solution of the saddle-point linear systems arising from the (regularized) Interior Point method applied to linear and quadratic convex programming problems, typically of large scale. Starting from the well-studied Constraint Preconditioner, we review a number of inexact variants with the aim to reduce the computational cost of the preconditioner application within the Krylov subspace solver of choice. In all cases we illustrate a spectral analysis showing the conditions under which a good clustering of the eigenvalues of the preconditioned matrix can be obtained, which foreshadows (at least in case PCG/MINRES Krylov solvers are used), a fast convergence of the iterative method. Results on a set of large size optimization problems confirm that the Inexact variants of the Constraint Preconditioner can yield efficient solution methods.

Preconditioned iterative methods provide an effective alternative to direct methods for the solution of the KKT linear systems arising in Interior Point algorithms, especially when large-scale problems are considered. We analyze the behaviour of a Constraint Preconditioner coupled with Krylov solvers, in a Potential Reduction (PR) framework. We present also adaptive stopping criteria for the inner iterations that relate the accuracy in the solution of the KKT system to the quality of the current PR iterate, to increase the overall computational efficiency. Numerical experiments on a set of large-scale problems show the effectiveness of this approach. [ DOI : 10.1685 / CSC06031] About DOI

Momentum Iterative Hessian Sketch (M-IHS) techniques, a group of solvers for large scale regularized linear Least Squares (LS) problems, are proposed and analyzed in detail. Proposed M-IHS techniques are obtained by incorporating the Heavy Ball Acceleration into the Iterative Hessian Sketch algorithm and they provide significant improvements over the randomized preconditioning techniques. By using approximate solvers along with the iterations, the proposed techniques are capable of avoiding all matrix decompositions and inversions, which is one of the main advantages over the alternative solvers such as the Blendenpik and the LSRN. Similar to the Chebyshev semi-iterations, the M-IHS variants do not use any inner products and eliminate the corresponding synchronization steps in hierarchical or distributed memory systems, yet the M-IHS converges faster than the Chebyshev Semi-iteration based solvers. Lower bounds on the required sketch size for various randomized distributions are est...

We review the use of block diagonal and block lower/upper triangular splittings for constructing iterative methods and preconditioners for solving stabilized saddle point problems. We introduce new variants of these splittings and obtain new results on the convergence of the associated stationary iterations and new bounds on the eigenvalues of the corresponding preconditioned matrices. We further consider inexact versions as preconditioners for flexible Krylov subspace methods, and show experimentally that our techniques can be highly effective for solving linear systems of saddle point type arising from stabilized finite element discretizations of two model problems, one from incompressible fluid mechanics and the other from magnetostatics.

Given A := {a 1 ,. .. , a m } ⊂ R d whose affine hull is R d , we study the problems of computing an approximate rounding of the convex hull of A and an approximation to the minimum volume enclosing ellipsoid of A. In the case of centrally symmetric sets, we first establish that Khachiyan's barycentric coordinate descent (BCD) method is exactly the polar of the deepest cut ellipsoid method using two-sided symmetric cuts. This observation gives further insight into the efficient implementation of the BCD method. We then propose a new algorithm which computes an approximate rounding of the convex hull of A, and which can also be used to compute an approximation to the minimum volume enclosing ellipsoid of A. Our algorithm is a modification of the algorithm of Kumar and Yıldırım, which combines Khachiyan's BCD method with a simple initialization scheme to achieve a slightly improved polynomial complexity result, and which returns a small "core set." We establish that our algorithm computes an approximate solution to the dual optimization formulation of the minimum volume enclosing ellipsoid problem that satisfies a more complete set of approximate optimality conditions than either of the two previous algorithms. Furthermore, this added benefit is achieved without any increase in the improved asymptotical complexity bound of the algorithm of Kumar and Yıldırım or any increase in the bound on the size of the computed core set. In addition, the "dropping idea" used in our algorithm has the potential of computing smaller core sets in practice. We also discuss several possible variants of this dropping technique.

We study the problem of computing a (1 +)-approximation to the minimum volume enclosing ellipsoid of a given point set S = {p 1 , p 2 ,. .. , p n } ⊆ R d. Based on a simple, initial volume approximation method, we propose a modification of Khachiyan's first-order algorithm. Our analysis leads to a slightly improved complexity bound of O(nd 3 /) operations for ∈ (0, 1). As a byproduct, our algorithm returns a core set X ⊆ S with the property that the minimum volume enclosing ellipsoid of X provides a good approximation to that of S. Furthermore, the size of X depends only on the dimension d and , but not on the number of points n. In particular, our results imply that |X | = O(d 2 /) for ∈ (0, 1).

Presented here is a fast method that combines curve matching techniques with a surface matching algorithm to estimate the positioning and respective matching error for the joining of three-dimensional fragmented objects. Furthermore, this paper describes how multiple joints are evaluated and how the broken artefacts are clustered and transformed to form potential solutions of the assemblage problem.

Interior point methods (IPMs) have proven to be an efficient way of solving quadratic programming problems in predictive control. A linear system of equations needs to be solved in each iteration of an IPM. The ill-conditioning of this linear system in the later iterations of the IPM prevents the use of an iterative method in solving the linear system due to a very slow rate of convergence; in some cases the solution never reaches the desired accuracy. In this paper we propose the use of a well-conditioned, approximate linear system, which increases the rate of convergence of the iterative method. The computational advantage is obtained by the use of an inexact Newton method along with the use of novel preconditioners. Numerical results indicate that the computational complexity of our proposed method scales quadratically with the number of states and linearly with the horizon length.

We propose a class of deflated preconditioners to address the problem of efficiently solving the systems of the normal equations (NE) arising at each step of the interior point method in constrained optimization. As well known, the condition number of these SPD linear systems grows asymptotically as the sequence of the Newton iterates approaches the solution. Iterative solution of such linear systems, which is mandatory for large scale problems, requires the design of ad hoc preconditioners [4]. To this aim we will apply to a given preconditioner P0, computed for the NE matrix at the initial Newton point, the BFGS-like low rank update as analyzed and tested in [1] for a general nonlinear system of equations as well as in [2, 3] for solving the sequence of nearly-singular linear systems to compute the leftmost eigenpairs of large SPD matrices by the Inexact Newton’s method. The bounded deterioration property of the sequence of preconditioned matrices, as proved in [3] for singular or...

In this article, Momentum Iterative Hessian Sketch (M-IHS) techniques, a group of solvers for large scale linear Least Squares (LS) problems, are proposed and analyzed in detail. The proposed techniques are obtained by incorporating the Heavy Ball Acceleration into the Iterative Hessian Sketch algorithm and they provide significant improvements over the randomized preconditioning techniques. Through the error analyses of the M-IHS variants, lower bounds on the sketch size for various randomized distributions to converge at a pre-determined rate with a constant probability are established. The bounds present the best results in the current literature for obtaining a solution approximation and they suggest that the sketch size can be chosen proportional to the statistical dimension of the regularized problem regardless of the size of the coefficient matrix. The statistical dimension is always smaller than the rank and it gets smaller as the regularization parameter increases. By using...

The implementation of a linear programming interior point solver is described that is based on iterative linear algebra. The linear systems are preconditioned by a basis matrix, which is updated from one interior point iteration to the next to bound the entries in a certain tableau matrix. The update scheme is based on simplex-type pivot operations and is implemented using linear algebra techniques from the revised simplex method. An initial basis is constructed by a crash procedure after a few interior point iterations. The basis at the end of the interior point solve provides the starting basis for a crossover method which recovers a basic solution to the linear program. Results of a computational study on a diverse set of medium to large-scale problems are discussed.

A dual logarithmic barrier method for solving large, sparse semidefinite programs is proposed in this paper. The method avoids any explicit use of the primal variable X and therefore is well-suited to problems with a sparse dual matrix S. It relies on inexact Newton steps in dual space which are computed by the conjugate gradient method applied to the Schur complement of the reduced KKT system. The method may take advantage of low-rank representations of matrices A i to perform implicit matrix-vector products with the Schur complement matrix and to compute only specific parts of this matrix. This allows the construction of the partial Cholesky factorization of the Schur complement matrix which serves as a good preconditioner for it and permits the method to be run in a matrix-free scheme. Convergence properties of the method are studied and a polynomial complexity result is extended to the case when inexact Newton steps are employed.

Recently, Resende and Veiga [31] have proposed an efficient implementation of the Dual Affine (DA) interior-point algorithm for the solution of linear transportation models with integer costs and right-hand side coefficients. This procedure incorporates a Preconditioned Conjugate Gradient (PCG) method for solving the linear system that is required in each iteration of the DA algorithm. In this paper, we introduce an Incomplete QR Decomposition (IQRD) preconditioning for the PCG algorithm. Computational experience shows that the IQRD preconditioning is quite appropriate in this instance and is more efficient than the preconditioning introduced by Resende and Veiga. We also show that the Primal Dual (PD) and the Predictor Corrector (PC) interior point algorithms can also be implemented by using the same type of technique. A comparison among these three algorithms is also included and indicates that the PD an PC algorithms are more appropriate for the solution of transportation problem...

Systems of linear equations with "normal" matrices of the form AD 2 A T is a key ingredient in the computation of search directions for interior-point algorithms. In this article, we establish that a well-known basis preconditioner for such systems of linear equations produces scaled matrices with uniformly bounded condition numbers as D varies over the set of all positive diagonal matrices. In particular, we show that when A is the node-arc incidence matrix of a connected directed graph with one of its rows deleted, then the condition number of the corrsponding preconditioned normal matrix is bounded above by m(n − m + 1), where m and n are the number of nodes and arcs of the network.

We propose a method for solving a Hermitian positive definite linear system Ax = b, where A is an explicit sparse matrix (real or complex). A sparse approximate right inverse M is computed and replaced by M̃ = (M + MH)/2, which is used as a left-right preconditioner in a modified version of the preconditioned conjugate gradient (PCG) method. M is formed column by column and can therefore be computed in parallel. PCG requires only matrix-vector multiplications with A and M̃ (not solving a linear system with the preconditioner), and so too can be carried out in parallel. We compare it with incomplete Cholesky factorization (the gold standard for PCG) and with MATLAB’s backslash operator (sparse Cholesky) on matrices from various applications.

We propose a framework for building preconditioners for sequences of linear systems of the form (A + ∆ k)x k = b k , where A is symmetric positive semidefinite and ∆ k is diagonal positive semidefinite. Such sequences arise in several optimization methods, e.g., in affine-scaling methods for bound-constrained convex quadratic programming and bound-constrained linear least squares, as well as in trust-region and overestimation methods for convex unconstrained optimization problems and nonlinear least squares. For all the matrices of a sequence, the preconditioners are obtained by updating any preconditioner for A available in the LDL T form. The preconditioners in the framework satisfy the natural requirement of being effective on slowly varying sequences; furthermore, under an additional property they are also able to cluster eigenvalues of the preconditioned matrix when some entries of ∆ k are sufficiently large. We present two low-cost preconditioners sharing the abovementioned properties and evaluate them on sequences of linear systems generated by the reflective Newton method applied to bound-constrained convex quadratic programming problems, and on sequences arising in solving nonlinear least-squares problems with the Regularized Euclidean Residual method. The results of the numerical experiments show the effectiveness of these preconditioners.

Large-scale optimization problems that seek sparse solutions have become ubiquitous. They are routinely solved with various specialized first-order methods. Although such methods are often fast, they usually struggle with not-so-well conditioned problems. In this paper, specialized variants of an interior point-proximal method of multipliers are proposed and analyzed for problems of this class. Computational experience on a variety of problems, namely, multi-period portfolio optimization, classification of data coming from functional Magnetic Resonance Imaging, restoration of images corrupted by Poisson noise, and classification via regularized logistic regression, provides substantial evidence that interior point methods, equipped with suitable linear algebra, can offer a noticeable advantage over first-order approaches.

Recently, Resende and Veiga 31] have proposed an e cient implementation of the Dual A ne (DA) interior-pointalgorithm for the solution of linear transportationmodels with integer costs and right-hand side coe cients. This procedure incorporates a Preconditioned Conjugate Gradient (PCG) method for solving the linear system that is required in each iteration of the DA algorithm. In this paper, we introduce an Incomplete QR Decomposition (IQRD) preconditioning for the PCG algorithm. Computational experience shows that the IQRD preconditioning is quite appropriate in this instance and is more e cient than the preconditioning introduced by Resende and Veiga. We also show that the Primal Dual (PD) and the Predictor Corrector (PC) interior point algorithms can also be implemented by using the same type of technique. A comparison among these three algorithms is also included and indicates that the PD an PC algorithms are more appropriate for the solution of transportation problems with well scaled cost coe cients. On the other hand the DA algorithm seems to be more e cient for assignment problems with well scaled cost coe cients and transportation problems whose costs and right-hand side coe cients are both badly scaled.

One of the most efficient interior-point methods for some classes of block-angular structured problems solves the normal equations by a combination of Cholesky factorizations and preconditioned conjugate gradient for, respectively, the block and linking constraints. In this work we show that the choice of a good preconditioner depends
on geometrical properties of the constraints structure. In particular, it is seen that the principal angles between the subspaces generated by the diagonal blocks and the linking constraints can be used to estimate ex-ante the efficiency of the preconditioner. Numerical validation is provided with some generated optimization problems. An application to
the solution of multicommodity network flow problems with nodal capacities and equal flows is also presented.

The paper extends affine conjugate Newton methods from convex to nonconvex minimization, with particular emphasis on PDE problems originating from compressible hyperelasticity. Based on well-known schemes from finite dimensional nonlinear optimization, three different algorithmic variants are worked out in a function space setting, which permits an adaptive multilevel finite element implementation. These algorithms are tested on two well-known 3D test problems and a real-life example from surgical operation planning.

Several issues concerning an analysis of large and sparse linear programming problems prior to solving them with an interior point based optimizer are addressed in this paper. Three types of presolve procedures are distinguished. Routines from the rst class repeatedly analyze an LP problem formulation: eliminate empty or singleton rows and columns, look for primal and dual forcing or dominated constraints, tighten bounds for variables and shadow prices or just the opposite, relax them to nd implied free variables. The second type of analysis aims at reducing a ll-in of the Cholesky factor of the normal equations matrix used to compute orthogonal projections and includes a heuristic for increasing the sparsity of the LP constraint matrix and a technique of splitting dense columns in it. Finally, routines from the third class detect, and remove, di erent linear dependecies of rows and columns in a constraint matrix.

We discuss the use of preconditioned conjugate gradients (CG) method for solving the reduced KKT systems arising in interior point algorithms for linear programming. The (indefinite) augmented system form of this linear system has a number of advantages, notably a higher degree of ...

Systems of linear equations with "normal" matrices of the form AD 2 A T is a key ingredient in the computation of search directions for interior-point algorithms. In this article, we establish that a well-known basis preconditioner for such systems of linear equations produces scaled matrices with uniformly bounded condition numbers as D varies over the set of all positive diagonal matrices. In particular, we show that when A is the node-arc incidence matrix of a connected directed graph with one of its rows deleted, then the condition number of the corrsponding preconditioned normal matrix is bounded above by m(n − m + 1), where m and n are the number of nodes and arcs of the network.

In this paper we analyze a class of approximate constraint preconditioners in the acceleration of Krylov subspace methods fot the solution of reduced Newton systems arising in optimization with interior point methods. We propose a dynamic sparsification of the Jacobian matrix at every stage of the interior point method. Spectral analysis of the preconditioned matrix is performed and bounds on its non-unit eigenvalues are provided. Preliminary computational results are encouraging.

Every Newton step in an interior-point method for optimization requires a solution of a symmetric indefinite system of linear equations. Most of today's codes apply direct solution methods to perform this task. The use of logarithmic barriers in interior point methods causes unavoidable ill-conditioning of linear systems and, hence, iterative methods fail to provide sufficient accuracy unless appropriately preconditioned. Two types of preconditioners which use some form of incomplete Cholesky factorization for indefinite systems are proposed in this paper. Although they involve significantly sparser factorizations than those used in direct approaches they still capture most of the numerical properties of the preconditioned system. The spectral analysis of the preconditioned matrix is performed: for convex optimization problems all the eigenvalues of this matrix are strictly positive. Numerical results are given for a set of public domain large linearly constrained convex quadratic programming problems with sizes reaching tens of thousands of variables. The analysis of these results reveals that the solution times for such problems on a modern PC are measured in minutes when direct methods are used and drop to seconds when iterative methods with appropriate preconditioners are used.

Optimal process design often requires the solution of mixed integer non-linear programming problems. Optimization procedures must be robust and efficient if they are to be incorporated in automated design systems. For heat integrated separation process design, a natural hybrid evolutionary/local search method with these properties is possible. The method is based on the use of local search methods for the continuous design parameters for the units in the process and the use of an evolutionary optimization procedure for the design of the heat exchanger network. The use of a stochastic method for the heat exchanger network aspect introduces noise in the evaluation of the objective function used by the local search methods. A smoothing procedure has been designed and implemented to improve the efficacy of the hybrid approach.

Nature-inspired algorithms are proving to be very successful on complex optimisation problems. A new algorithm, inspired by the way plants, and in particular the strawberry plant, propagate is presented. The algorithm is explained, tested on standard test functions, and compared with the well known Nelder-Mead algorithm. The new approach is then applied to a complex process design problem that arises in Chlorobenzene purification, a problem that exhibits strong nonlinear behaviour and has a small feasible region.

Given an arbitrary set A ∈ I R n , we know that there exists an ellipsoid E which provides an n-rounding of the set A, i.e. n −1 E ⊆ conv(A) ⊆ E. The minimumvolume ellipsoid that encloses the set A provides such an ellipsoid and will be denoted as MVEE(A). Finding good approximations to this ellipsoid is very useful for many applications in computational geometry, computational statistics, and convex optimization. For example, given a 2-D (or 3-D) image of a convex object, the area (or the volume) of the minimum-volume enclosing ellipsoid provides a good approximation to the actual area (or the volume) of the object and can be useful in image processing applications.

We show the linear convergence of a simple first-order algorithm for the minimum-volume enclosing ellipsoid problem and its dual, the D-optimal design problem of statistics. Computational tests confirm the attractive features of this method.