os x 10.6 port -- numerous mysterious errors caused by weird "abort trap" issue

[williamstein]

After testing sage-4.1.2.rc0 (and applying a tiny numerical noise fix), we have 12 files that all fail doctests with "myserious error":

wstein@bsd:~/build/sage-4.1.2.rc0$ grep mysterious testlong.log |wc -l
      12

The files with failures are:

        sage -t -long "devel/sage/sage/calculus/calculus.py"
        sage -t -long "devel/sage/sage/calculus/tests.py"
        sage -t -long "devel/sage/sage/calculus/wester.py"
        sage -t -long "devel/sage/sage/functions/hyperbolic.py"
        sage -t -long "devel/sage/sage/functions/other.py"
        sage -t -long "devel/sage/sage/functions/trig.py"
        sage -t -long "devel/sage/sage/gsl/interpolation.pyx"
        sage -t -long "devel/sage/sage/matrix/matrix_symbolic_dense.pyx"
        sage -t -long "devel/sage/sage/rings/polynomial/pbori.pyx"
        sage -t -long "devel/sage/sage/symbolic/constants.py"
        sage -t -long "devel/sage/sage/symbolic/expression.pyx"
        sage -t -long "devel/sage/sage/symbolic/function.pyx"

To fix, use the following new spkgs, plus the two patches to the Sage library (trac-7095.patch and trac_7095-ref.patch), plus the patch to sage_scripts, plus (finally) add the new sage_env also to SAGE_ROOT/spkg/base (available only in a Sage source distribution), mind to chmod the right flags for this sage-env:

• http://sage.math.washington.edu/home/craigcitro/sage/SPKGs/python-2.6.4.p4.spkg

• http://sage.math.washington.edu/home/craigcitro/sage/SPKGs/matplotlib-0.99.1.p4.spkg

Component: porting

Author: Craig Citro, John Palmieri, Francis Clarke, William Stein

Reviewer: David Kirkby, Georg S. Weber

Merged: sage-4.3.1.rc1

Issue created by migration from https://trac.sagemath.org/ticket/7095

[williamstein]

comment:2

A good example:

sage: factorial(factorial(x))

Program received signal SIGABRT, Aborted.
0x00007fff82adf096 in __kill ()
(gdb) bt
#0  0x00007fff82adf096 in __kill ()
#1  0x00007fff82b800c2 in abort ()
#2  0x0000000100204d51 in uw_init_context_1 ()
#3  0x00000001002054cc in _Unwind_Resume ()
#4  0x000000010742c4cb in __gnu_cxx::new_allocator<GiNaC::ex>::deallocate () at new_allocator.h:97
#5  0x000000010742c4cb in std::_Vector_base<GiNaC::ex, std::allocator<GiNaC::ex> >::_M_deallocate () at new_allocator.h:137
#6  0x000000010742c4cb in ~auto_ptr [inlined] () at new_allocator.h:123
#7  0x000000010742c4cb in ~vector [inlined] () at new_allocator.h:593
#8  0x000000010742c4cb in ~_Vector_base [inlined] () at /usr/include/c++/4.2.1/memory:259
#9  0x000000010742c4cb in GiNaC::container<std::vector>::subs (this=0x10b010e70, m=@0x7fff5fbfbdb0, options=0) at new_allocator.h:271
#10 0x00000001077b5923 in GiNaC::ex::subs () at /Users/was/build/sage-4.1.2.rc0/local/include/pynac/ex.h:827
#11 0x00000001077b5923 in __pyx_pf_4sage_8symbolic_10expression_10Expression_substitute (__pyx_v_self=0x10b1a9638, __pyx_args=<value temporarily unavailable, due to optimizations>, __pyx_kwds=0x10b059480) at ex.h:13728
#12 0x0000000100006fd2 in PyObject_Call (func=0x10b18ca28, arg=0x10b1b0c50, kw=0x0) at Objects/abstract.c:2492
#13 0x0000000107799f2d in __pyx_pf_4sage_8symbolic_10expression_10Expression_test_relation (__pyx_v_self=0x109bea7a0, __pyx_args=<value temporarily unavailable, due to optimizations>, __pyx_kwds=<value temporarily unavailable, due to optimizations>) at sage/symbolic/expression.cpp:9041
#14 0x0000000100006fd2 in PyObject_Call (func=0x1006c7200, arg=0x100230050, kw=0x0) at Objects/abstract.c:2492
#15 0x00000001077a6d84 in __pyx_pf_4sage_8symbolic_10expression_10Expression___nonzero__ (__pyx_v_self=0x109bea7a0) at sage/symbolic/expression.cpp:7677
#16 0x000000010004d336 in PyObject_IsTrue (v=<value temporarily unavailable, due to optimizations>) at Objects/object.c:1547
#17 0x00000001000b1ec3 in PyEval_EvalFrameEx (f=0x10b066620, throwflag=<value temporarily unavailable, due to optimizations>) at Python/ceval.c:2185
#18 0x00000001000b7650 in PyEval_EvalCodeEx (co=0x103c33300, globals=<value temporarily unavailable, due to optimizations>, locals=<value temporarily unavailable, due to optimizations>, args=0x10b1b0c28, argcount=1, kws=0x10b00b4f0, kwcount=0, defs=0x103c46d68, defcount=1, closure=0x0) at Python/ceval.c:2968
#19 0x0000000100035f0d in function_call (func=0x103c47848, arg=0x10b1b0c10, kw=0x10b062680) at Objects/funcobject.c:524
#20 0x0000000100006fd2 in PyObject_Call (func=0x103c47848, arg=0x10b1b0c10, kw=0x10b062680) at Objects/abstract.c:2492
#21 0x00000001000b1a92 in PyEval_EvalFrameEx (f=0x10b066410, throwflag=<value temporarily unavailable, due to optimizations>) at Python/ceval.c:4019

[sagetrac-fwclarke]

comment:3

Some of these errors seems to arise from test_relation in sage/symbolic/expression.pyx. Checking out

bool(diff(csch(x), x) == diff(Integer(1)/sinh(x), x))

which gives rise to the "mysterious error" in sage/functions/hyperbolic.py, I find the following phenomena (with Mac OS X 10.61, 2 GHz Intel Core 2 Duo, 2 GB):

----------------------------------------------------------------------
| Sage Version 4.1.2.rc0, Release Date: 2009-09-30                   |
| Type notebook() for the GUI, and license() for information.        |
----------------------------------------------------------------------
**********************************************************************
*                                                                    *
* Warning: this is a prerelease version, and it may be unstable.     *
*                                                                    *
**********************************************************************
sage: sinh(x).subs({x: 2})
sinh(2)
sage: sinh(x).subs({x: 2.2})
4.45710517053589
sage: sinh(x).subs({x: RIF(2.2)})
4.457105170535895?
sage: sinh(x).subs({x: CIF(2.2)})
/Users/mafwc/sage-4.1.2.rc0/local/bin/sage-sage: line 202:  2260 Abort trap              sage-ipython "$@" -i

and similarly for cosh, tanh and several other hyperbolic functions. But

----------------------------------------------------------------------
| Sage Version 4.1.2.rc0, Release Date: 2009-09-30                   |
| Type notebook() for the GUI, and license() for information.        |
----------------------------------------------------------------------
**********************************************************************
*                                                                    *
* Warning: this is a prerelease version, and it may be unstable.     *
*                                                                    *
**********************************************************************
sage: coth(x).subs({x: 2})
coth(2)
sage: coth(x).subs({x: 2.2})
coth(2.20000000000000)
sage: coth(x).subs({x: RIF(2.2)})
coth(2.2000000000000002?)
sage: coth(x).subs({x: CIF(2.2)})
coth(2.2000000000000002?)
sage: 

There's a serious inconsistency here, and clearly a problems with CIF.

[williamstein]

comment:4

I was reading this blog post and it mentions that Apple made some massive changes to GCC-4.2.x in XCode:

http://dobbscodetalk.com/index.php?option=com_myblog&show=Snow-Leopard-Changes-to-C.html&Itemid=29

In particular, they added closures to C!

This page talks about weird issues with the new Xcode and GCC-4.2.x on OS X and Python: http://bugs.python.org/issue6957
But it also points out that XCode for 10.6 comes with GCC-4.0.1. The default is 32-bit for GCC-4.0.1, unlike GCC-4.2.x, which on OS X defaults to 643-bit. So... I'm now trying fresh 32 and 64-bit builds of Sage using GCC-4.0.1. I'm doing this by changing my path to point to versions of gcc and g++ that are the 4.0.1 versions, instead of trusting environment variables, which are bound to fail. I'll report more here later.

Anyway, if this works, we'll be in good shape -- we could just revisit 4.2.x support later as components of Sage stabilize and people iron out the bugs in the huge changes Apple may have made to C.

[williamstein]

comment:5

Major progress -- a lot of problems appear to just "go away" when using GCC-4.0.x! In particular, the above issue with "Abort Trap" goes away. The main problem is that some of the fixes I put in for OS X 10.6 with GCC-4.2.x actually break the 4.0.x build.... anyway, I'm very optimistic for a complete resolution to the problem of building Sage on OS X 10.6 using GCC 4.0.x.

[sagetrac-drkirkby]

comment:6

Two things

• sage -t "devel/sage/sage/gsl/interpolation.pyx fails on Solaris too.
• If gcc 4.2.x is an issue on OS X 10.6, I could create a prereq-0.5 which gives a warning with this combination of operating system version and compiler, so people don't get caught out. Just let me know
  • The exact OS version
  • The exact compiler version
  • The exact wording you want.

I was intending updating prereq anyway, after getting some feedback. If nothing else, I can fix #7156 at some point, but that is not an important fix. But this seems a bit more important.

Dave

[williamstein]

comment:7

Major progress -- a lot of problems appear to just "go away" when
using GCC-4.0.x! In particular, the above issue with "Abort Trap" goes away.

NOTE: Actually, it is only in 32-bit mode that this Abort Trap issue goes away. In 64-bit mode with gcc-4.0.x, the problem remains. Unfortunately a full build in 32-bit mode has 97 files that have failing doctests. In 64-bit mode the one remaining problem above doesn't go away, and we still have tons of failures. So using GCC-4.0.x is no solution at all, unfortunately... not even close.

[williamstein]

comment:8

The factorial(factorial(x)) error above is really from factorial(CIF(1)) which leads to the same error. Combining this with fwclarke's remark above, suggests complex interval fields are somehow the one place where things are broken somehow.

[williamstein]

comment:9

No, the problem isn't CIF, since:

sage: factorial(CC(1.0))
^P/Users/was/build/sage-4.1.2.rc1.alpha3/local/bin/sage-sage: line 202: 67490 Abort trap              sage-ipython "$@" -i
sage: factorial(CDF(1.0))
/Users/was/build/sage-4.1.2.rc1.alpha3/local/bin/sage-sage: line 202: 67569 Abort trap              sage-ipython "$@" -i

[sagetrac-fwclarke]

comment:10

I wonder about lines 123--124 of local/lib/python/config/Makefile (which have been there since at least 3.1.4):

MACOSX_DEPLOYMENT_TARGET=10.3
export MACOSX_DEPLOYMENT_TARGET

It seems to be these lines that are giving rise (in 10.6 but not in 10.5) to many warnings of the form

#warning Building for Intel with Mac OS X Deployment Target < 10.4 is invalid.

Whether this is causing the problems, I don't know, but it does need to be changed (presumably to 10.4).