Upgrade cvxopt to 1.1.6

[kiwifb]

There are two doctests related to cvxopt failing on power7

sage -t --long -force_lib "devel/sage/doc/en/numerical_sage/cvxopt.rst"
**********************************************************************
File "/hpc/scratch/frb15/sandbox/sage-5.7.beta4/devel/sage/doc/en/numerical_sage/cvxopt.rst", line 129:
    sage: print sol['x']      # ... below since can get -00 or +00 depending on architecture
Expected:
    [ 1.00e...00]
    [ 1.00e+00]
Got:
    [ 9.87e-01]
    [ 9.92e-01]
    <BLANKLINE>
**********************************************************************

and

sage -t -long "devel/sage-main/sage/numerical/optimize.py"  
**********************************************************************
File "/hpc/scratch/frb15/sandbox/sage-5.7.beta4/devel/sage-main/sage/numerical/optimize.py", line 515:
    sage: sol['x']
Expected:
    (0.999..., 1.000...)
Got nothing
**********************************************************************
File "/hpc/scratch/frb15/sandbox/sage-5.7.beta4/devel/sage-main/sage/numerical/optimize.py", line 525:
    sage: sol['x']
Expected:
    (45.000000..., 6.2499999...3, 1.00000000...)
Got nothing
GLPK Simplex Optimizer, v4.44
6 rows, 3 columns, 8 non-zeros
Preprocessing...
2 rows, 2 columns, 4 non-zeros
Scaling...
 A: min|aij| =  2.400e+01  max|aij| =  5.000e+01  ratio =  2.083e+00
GM: min|aij| =  8.128e-01  max|aij| =  1.230e+00  ratio =  1.514e+00
EQ: min|aij| =  6.606e-01  max|aij| =  1.000e+00  ratio =  1.514e+00
Constructing initial basis...
Size of triangular part = 2
*     0: obj =  -5.100000000e+01  infeas =  0.000e+00 (0)
*     1: obj =  -5.225000000e+01  infeas =  0.000e+00 (0)
OPTIMAL SOLUTION FOUND
**********************************************************************

This is now solved in cvxopt 1.1.6. The updated cvxopt spkg also fixes a blas path issue on Darwin and Cygwin.

spkg: http://boxen.math.washington.edu/home/jdemeyer/spkg/cvxopt-1.1.6.p0.spkg (spkg diff)

Upstream: Fixed upstream, in a later stable release.

Component: packages: standard

Author: Jeroen Demeyer

Reviewer: Volker Braun

Merged: sage-5.10.beta4

Issue created by migration from https://trac.sagemath.org/ticket/12832

[kiwifb]

comment:1

I think they may be both upstream problems. I have tried to run test on a stock scipy-0.10.0 on power7 and several results indicate that at least some of the problems are in scipy itself

ERROR: test_linear_1d (test_fitpack.TestUnivariateSpline)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/hpc/scratch/frb15/work/scipy-0.10.0/test/lib/python/scipy/interpolate/tests/test_fitpack.py", line 49, in test_linear_1d
    assert_array_almost_equal(lut([1,1.5,2]),[0,1,2])
  File "scipy/interpolate/fitpack2.py", line 221, in __call__
    return fitpack.splev(x, self._eval_args, der=nu)
  File "scipy/interpolate/fitpack.py", line 550, in splev
    raise ValueError("Invalid input data")
ValueError: Invalid input data

======================================================================
ERROR: test_linear_constant (test_fitpack.TestUnivariateSpline)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/hpc/scratch/frb15/work/scipy-0.10.0/test/lib/python/scipy/interpolate/tests/test_fitpack.py", line 29, in test_linear_constant
    assert_array_almost_equal(lut([1,1.5,2]),[3,3,3])
  File "scipy/interpolate/fitpack2.py", line 221, in __call__
    return fitpack.splev(x, self._eval_args, der=nu)
  File "scipy/interpolate/fitpack.py", line 550, in splev
    raise ValueError("Invalid input data")
ValueError: Invalid input data

======================================================================
ERROR: test_preserve_shape (test_fitpack.TestUnivariateSpline)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/hpc/scratch/frb15/work/scipy-0.10.0/test/lib/python/scipy/interpolate/tests/test_fitpack.py", line 36, in test_preserve_shape
    assert_equal(shape(arg), shape(lut(arg)))
  File "scipy/interpolate/fitpack2.py", line 221, in __call__
    return fitpack.splev(x, self._eval_args, der=nu)
  File "scipy/interpolate/fitpack.py", line 550, in splev
    raise ValueError("Invalid input data")
ValueError: Invalid input data

======================================================================
ERROR: Regression test for #1375.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/hpc/scratch/frb15/work/scipy-0.10.0/test/lib/python/scipy/interpolate/tests/test_fitpack.py", line 82, in test_resize_regression
    assert_allclose(spl([0.1, 0.5, 0.9, 0.99]), desired, atol=5e-4)
  File "scipy/interpolate/fitpack2.py", line 221, in __call__
    return fitpack.splev(x, self._eval_args, der=nu)
  File "scipy/interpolate/fitpack.py", line 550, in splev
    raise ValueError("Invalid input data")
ValueError: Invalid input data

======================================================================
ERROR: Failure: AttributeError ('module' object has no attribute 'qhull')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/pkg/python/version/lib/python2.7/site-packages/nose-1.1.2-py2.7.egg/nose/loader.py", line 390, in loadTestsFromName
    addr.filename, addr.module)
  File "/usr/local/pkg/python/version/lib/python2.7/site-packages/nose-1.1.2-py2.7.egg/nose/importer.py", line 39, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/usr/local/pkg/python/version/lib/python2.7/site-packages/nose-1.1.2-py2.7.egg/nose/importer.py", line 86, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/hpc/scratch/frb15/work/scipy-0.10.0/test/lib/python/scipy/interpolate/tests/test_interpnd.py", line 6, in <module>
    import scipy.spatial.qhull as qhull
AttributeError: 'module' object has no attribute 'qhull'

======================================================================
ERROR: test_interp2d (test_interpolate.TestInterp2D)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/hpc/scratch/frb15/work/scipy-0.10.0/test/lib/python/scipy/interpolate/tests/test_interpolate.py", line 14, in test_interp2d
    I = interp2d(x, y, z)
  File "scipy/interpolate/interpolate.py", line 160, in __init__
    self.tck = fitpack.bisplrep(self.x, self.y, self.z, kx=kx, ky=ky, s=0.)
  File "scipy/interpolate/fitpack.py", line 873, in bisplrep
    tx,ty,nxest,nyest,wrk,lwrk1,lwrk2)
ValueError: negative dimensions are not allowed

======================================================================
ERROR: test_interp2d_meshgrid_input (test_interpolate.TestInterp2D)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/hpc/scratch/frb15/work/scipy-0.10.0/test/lib/python/scipy/interpolate/tests/test_interpolate.py", line 25, in test_interp2d_meshgrid_input
    I = interp2d(x, y, z)
  File "scipy/interpolate/interpolate.py", line 160, in __init__
    self.tck = fitpack.bisplrep(self.x, self.y, self.z, kx=kx, ky=ky, s=0.)
  File "scipy/interpolate/fitpack.py", line 873, in bisplrep
    tx,ty,nxest,nyest,wrk,lwrk1,lwrk2)
ValueError: negative dimensions are not allowed

======================================================================
ERROR: test_construction (test_polyint.CheckPiecewise)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/hpc/scratch/frb15/work/scipy-0.10.0/test/lib/python/scipy/interpolate/tests/test_polyint.py", line 194, in setUp
    self.spline_ys = splev(self.test_xs, self.tck)
  File "scipy/interpolate/fitpack.py", line 550, in splev
    raise ValueError("Invalid input data")
ValueError: Invalid input data

======================================================================
ERROR: test_derivative (test_polyint.CheckPiecewise)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/hpc/scratch/frb15/work/scipy-0.10.0/test/lib/python/scipy/interpolate/tests/test_polyint.py", line 194, in setUp
    self.spline_ys = splev(self.test_xs, self.tck)
  File "scipy/interpolate/fitpack.py", line 550, in splev
    raise ValueError("Invalid input data")
ValueError: Invalid input data

======================================================================
ERROR: test_derivatives (test_polyint.CheckPiecewise)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/hpc/scratch/frb15/work/scipy-0.10.0/test/lib/python/scipy/interpolate/tests/test_polyint.py", line 194, in setUp
    self.spline_ys = splev(self.test_xs, self.tck)
  File "scipy/interpolate/fitpack.py", line 550, in splev
    raise ValueError("Invalid input data")
ValueError: Invalid input data

======================================================================
ERROR: test_incremental (test_polyint.CheckPiecewise)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/hpc/scratch/frb15/work/scipy-0.10.0/test/lib/python/scipy/interpolate/tests/test_polyint.py", line 194, in setUp
    self.spline_ys = splev(self.test_xs, self.tck)
  File "scipy/interpolate/fitpack.py", line 550, in splev
    raise ValueError("Invalid input data")
ValueError: Invalid input data

======================================================================
ERROR: test_scalar (test_polyint.CheckPiecewise)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/hpc/scratch/frb15/work/scipy-0.10.0/test/lib/python/scipy/interpolate/tests/test_polyint.py", line 194, in setUp
    self.spline_ys = splev(self.test_xs, self.tck)
  File "scipy/interpolate/fitpack.py", line 550, in splev
    raise ValueError("Invalid input data")
ValueError: Invalid input data

======================================================================
ERROR: test_shapes_scalarvalue (test_polyint.CheckPiecewise)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/hpc/scratch/frb15/work/scipy-0.10.0/test/lib/python/scipy/interpolate/tests/test_polyint.py", line 194, in setUp
    self.spline_ys = splev(self.test_xs, self.tck)
  File "scipy/interpolate/fitpack.py", line 550, in splev
    raise ValueError("Invalid input data")
ValueError: Invalid input data

======================================================================
ERROR: test_shapes_scalarvalue_derivative (test_polyint.CheckPiecewise)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/hpc/scratch/frb15/work/scipy-0.10.0/test/lib/python/scipy/interpolate/tests/test_polyint.py", line 194, in setUp
    self.spline_ys = splev(self.test_xs, self.tck)
  File "scipy/interpolate/fitpack.py", line 550, in splev
    raise ValueError("Invalid input data")
ValueError: Invalid input data

======================================================================
ERROR: test_shapes_vectorvalue (test_polyint.CheckPiecewise)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/hpc/scratch/frb15/work/scipy-0.10.0/test/lib/python/scipy/interpolate/tests/test_polyint.py", line 194, in setUp
    self.spline_ys = splev(self.test_xs, self.tck)
  File "scipy/interpolate/fitpack.py", line 550, in splev
    raise ValueError("Invalid input data")
ValueError: Invalid input data

======================================================================
ERROR: test_shapes_vectorvalue_1d (test_polyint.CheckPiecewise)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/hpc/scratch/frb15/work/scipy-0.10.0/test/lib/python/scipy/interpolate/tests/test_polyint.py", line 194, in setUp
    self.spline_ys = splev(self.test_xs, self.tck)
  File "scipy/interpolate/fitpack.py", line 550, in splev
    raise ValueError("Invalid input data")
ValueError: Invalid input data

======================================================================
ERROR: test_shapes_vectorvalue_derivative (test_polyint.CheckPiecewise)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/hpc/scratch/frb15/work/scipy-0.10.0/test/lib/python/scipy/interpolate/tests/test_polyint.py", line 194, in setUp
    self.spline_ys = splev(self.test_xs, self.tck)
  File "scipy/interpolate/fitpack.py", line 550, in splev
    raise ValueError("Invalid input data")
ValueError: Invalid input data

======================================================================
ERROR: test_vector (test_polyint.CheckPiecewise)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/hpc/scratch/frb15/work/scipy-0.10.0/test/lib/python/scipy/interpolate/tests/test_polyint.py", line 194, in setUp
    self.spline_ys = splev(self.test_xs, self.tck)
  File "scipy/interpolate/fitpack.py", line 550, in splev
    raise ValueError("Invalid input data")
ValueError: Invalid input data

======================================================================
ERROR: test_wrapper (test_polyint.CheckPiecewise)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/hpc/scratch/frb15/work/scipy-0.10.0/test/lib/python/scipy/interpolate/tests/test_polyint.py", line 194, in setUp
    self.spline_ys = splev(self.test_xs, self.tck)
  File "scipy/interpolate/fitpack.py", line 550, in splev
    raise ValueError("Invalid input data")
ValueError: Invalid input data

======================================================================
ERROR: Ticket #629
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/hpc/scratch/frb15/work/scipy-0.10.0/test/lib/python/scipy/interpolate/tests/test_regression.py", line 11, in test_spalde_scalar_input
    res = interp.spalde(np.float64(1), tck)
  File "scipy/interpolate/fitpack.py", line 724, in spalde
    raise TypeError("Invalid input data. t(k)<=x<=t(n-k+1) must hold.")
TypeError: Invalid input data. t(k)<=x<=t(n-k+1) must hold.

======================================================================

There are more error linked to failures to import modules which may be more related to the way test were run. As in the doctest a number of improper parameters or invalid data and negative dimensions problems.

[zimmermann6]

comment:2

I investigated the first error in optimize.py:

c=vector(RDF,[-4,-5])
G=matrix(RDF,[[2,1],[1,2],[-1,0],[0,-1]])
h=vector(RDF,[3,3,0,0])
from cvxopt.base import matrix as m
from cvxopt import solvers
solvers.options['show_progress']=True
c_=m(c.base_extend(RDF).numpy())
G_=m(G.base_extend(RDF).numpy())
h_=m(h.base_extend(RDF).numpy())
sol=solvers.lp(c_,G_,h_,solver=None)

The first numerical difference comes from the call base.gemm(spmatrix(W['di'], list(range(ml)), list(range(ml))), G, F['Gs'], partial = True) in local/lib/python2.7/site-packages/cvxopt/misc.py, where the inputs are:

W['di']= [ 3.06e+00]
[ 3.32e+00]
[ 1.29e+00]
[ 1.05e+00]
ml= 4
G= [ 2.00e+00  1.00e+00]
[ 1.00e+00  2.00e+00]
[-1.00e+00  0.00e+00]
[ 0.00e+00 -1.00e+00]

and the expected output is:

F['Gs']= [ 6.11e+00  3.06e+00]
[ 3.32e+00  6.63e+00]
[-1.29e+00  0.00e+00]
[ 0.00e+00 -1.05e+00]

whereas we get:

 F['Gs']= [ 8.11e+00  4.06e+00]
[ 4.32e+00  8.63e+00]
[-2.29e+00  0.00e+00]
[ 0.00e+00 -2.05e+00]

Does anybody know where the base.gemm function is defined?

Paul

[zimmermann6]

comment:3

I could reduce the issue to the following:

import numpy
from cvxopt.base import spmatrix
from cvxopt.base import matrix as m
from cvxopt import umfpack
RealNumber=float
W = [ 3.06e+00, 3.32e+00, 1.29e+00, 1.05e+00]
ml=4
W2 = spmatrix(W, list(range(ml)), list(range(ml)))
G= m([[2.00e+00,  1.00e+00, -1.00e+00,  0.00e+00],
      [1.00e+00,  2.00e+00,  0.00e+00, -1.00e+00]])
F = m([[0.0,0.0,0.0,0.0],[0.0,0.0,0.0,0.0]])
from cvxopt.base import gemm
import cvxopt
cvxopt.base.gemm(W2,G,F,partial = True)
print F
cvxopt.base.gemm(W2,G,F,partial = True)
print F

The expected output is twice the same matrix F:

[ 6.12e+00  3.06e+00]
[ 3.32e+00  6.64e+00]
[-1.29e+00  0.00e+00]
[ 0.00e+00 -1.05e+00]

[ 6.12e+00  3.06e+00]
[ 3.32e+00  6.64e+00]
[-1.29e+00  0.00e+00]
[ 0.00e+00 -1.05e+00]

On silius with sage-5.0.rc1 I get instead:

[ 6.12e+00  3.06e+00]
[ 3.32e+00  6.64e+00]
[-1.29e+00  0.00e+00]
[ 0.00e+00 -1.05e+00]

[ 1.22e+01  6.12e+00]
[ 6.64e+00  1.33e+01]
[-2.58e+00  0.00e+00]
[ 0.00e+00 -2.10e+00]

thus it seems the product is accumulated to F.

Paul

[kiwifb]

comment:4

That's unexpected, It could be a cvxopt problem rather than a scipy problem. I may be able to check on Monday when I go back to work for real if there are more problems with cvxopt.

[kiwifb]

comment:5

Yes I can confirm that with 5.1beta5 (and cvxopt-1.1.5 from #13160). Will look at the code.

[kiwifb]

comment:6

If after that we do (http://abel.ee.ucla.edu/cvxopt/userguide/blas.html#level-3-blas)

F = m([[0.0,0.0,0.0,0.0],[0.0,0.0,0.0,0.0]])
cvxopt.base.gemm(W2,G,F,beta=0.5,partial = True)
print F
cvxopt.base.gemm(W2,G,F,beta=0.5,partial = True
print F

we get

[ 6.12e+00  3.06e+00]
[ 3.32e+00  6.64e+00]
[-1.29e+00  0.00e+00]
[ 0.00e+00 -1.05e+00]

[ 1.22e+01  6.12e+00]
[ 6.64e+00  1.33e+01]
[-2.58e+00  0.00e+00]
[ 0.00e+00 -2.10e+00]

Clearly the beta factor is always passed as "1" rather than the default zero or the value you want. I checked that passing a value for alpha works.

[kiwifb]

comment:7

I am not completely sure of what is happening there. I need to have a control build locally to be sure. The evaluation eventually goes to sp_dgemm in C/sparse.c, more specifically this section

  else if (sp_a && !sp_b && !sp_c) {

    ccs *A = (tA == 'N' ? a : transpose(a, 0));
    double *B = b, *C = c;

    int j, l, mn_ = m*n, ib = (tB == 'N' ? k : 1);
    scal[DOUBLE](&mn_, &beta, C, (int *)&One[INT]);

    for (j=0; j<A->ncols; j++) {
      for (l=A->colptr[j]; l<A->colptr[j+1]; l++) {

        double a_ = alpha.d*((double *)A->values)[l];
        axpy[DOUBLE](&n, &a_, B + (tB == 'N' ? j : j*n), &ib,
            C + A->rowind[l], &m);
      }
    }
    if (A != a) free_ccs(A);
  }

  else if (!sp_a && sp_b && sp_c && partial){

Notice that in this particular case "partial" is ignored. The issue here is that we don't seem to go through scal[DOUBLE] (part of C/blas.c) at all. That particular call should replace our F input by beta.d*F where beta.d is 0 by default. The fact that this call seem completely bypassed explains why it is useless to give a value for beta.

[kiwifb]

comment:8

I cannot see it going in scal[DOUBLE] on my x86_64 box either. I will have to dig it a bit more to find how to crack what is happening there.

[kiwifb]

comment:9

cvxopt.base is not linked to cvxopt.blas which contains the interface to scal (it is working fine). So in the previous code this should be a direct call to dscal. I am guessing there is something about the declarations in that bit of code that I don't understand fully. But then why does it work on anything but power7?

[kiwifb]

comment:10

Finally found out it is supposed to be a direct call to blas dscal_ but the declaration are all wrong for that I think. I may have to copy the declaration from base.c and see if it works.

[kiwifb]

comment:11

I have no clue why the call to dscal_ either doesn't happen or return the wrong thing.