assumptions and fs tests fail on macOS PPC

[barracuda156]

I am bringing clingcon to Macports, where we support all range of macOS including PowerPC systems. I ran test suite on 10.6.8 in Rosetta, and two tests failed:

[100%] Built target test_clingcon
make[1]: Leaving directory `/opt/local/var/macports/build/_opt_PPCRosettaPorts_math_clingcon/clingcon/work/build'
/opt/local/bin/cmake -E cmake_progress_start /opt/local/var/macports/build/_opt_PPCRosettaPorts_math_clingcon/clingcon/work/build/CMakeFiles 0
make: Leaving directory `/opt/local/var/macports/build/_opt_PPCRosettaPorts_math_clingcon/clingcon/work/build'
--->  Testing clingcon
Executing:  cd "/opt/local/var/macports/build/_opt_PPCRosettaPorts_math_clingcon/clingcon/work/build/bin" && ./test_clingcon 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
test_clingcon is a Catch v2.11.3 host application.
Run with -? for options

-------------------------------------------------------------------------------
assumptions
-------------------------------------------------------------------------------
/opt/local/var/macports/build/_opt_PPCRosettaPorts_math_clingcon/clingcon/work/clingcon-5.2.0/libclingcon/tests/assumptions.cc:81
...............................................................................

/opt/local/var/macports/build/_opt_PPCRosettaPorts_math_clingcon/clingcon/work/clingcon-5.2.0/libclingcon/tests/assumptions.cc:95: FAILED:
  REQUIRE( bound(ctl) == 2 )
with expansion:
  1 == 2

-------------------------------------------------------------------------------
fs
  fse
-------------------------------------------------------------------------------
/opt/local/var/macports/build/_opt_PPCRosettaPorts_math_clingcon/clingcon/work/clingcon-5.2.0/libclingcon/tests/fs.cc:123
...............................................................................

/opt/local/var/macports/build/_opt_PPCRosettaPorts_math_clingcon/clingcon/work/clingcon-5.2.0/libclingcon/tests/fs.cc:125: FAILED:
  REQUIRE( solve(FSB + FSE + FSI, 0, 11) == SOL11 )
with expansion:
  {  }
  ==
  { "permutation(a,c) permutation(b,a) (a,1)=1 (a,2)=7 (b,1)=0 (b,2)=1 (c,1)=4
  (c,2)=11", "permutation(a,c) permutation(b,a) (a,1)=1 (a,2)=7 (b,1)=0 (b,2)=1
  (c,1)=5 (c,2)=11", "permutation(a,c) permutation(b,a) (a,1)=1 (a,2)=7 (b,1)=0
  (b,2)=1 (c,1)=6 (c,2)=11", "permutation(a,c) permutation(b,a) (a,1)=2 (a,2)=7
  (b,1)=0 (b,2)=1 (c,1)=5 (c,2)=11", "permutation(a,c) permutation(b,a) (a,1)=2
  (a,2)=7 (b,1)=0 (b,2)=1 (c,1)=6 (c,2)=11", "permutation(a,c) permutation(b,a)
  (a,1)=3 (a,2)=7 (b,1)=0 (b,2)=1 (c,1)=6 (c,2)=11" }

-------------------------------------------------------------------------------
fs
  fsd
-------------------------------------------------------------------------------
/opt/local/var/macports/build/_opt_PPCRosettaPorts_math_clingcon/clingcon/work/clingcon-5.2.0/libclingcon/tests/fs.cc:128
...............................................................................

/opt/local/var/macports/build/_opt_PPCRosettaPorts_math_clingcon/clingcon/work/clingcon-5.2.0/libclingcon/tests/fs.cc:130: FAILED:
  REQUIRE( solve(FSB + FSD + FSI, 0, 11) == SOL11 )
with expansion:
  {  }
  ==
  { "permutation(a,c) permutation(b,a) (a,1)=1 (a,2)=7 (b,1)=0 (b,2)=1 (c,1)=4
  (c,2)=11", "permutation(a,c) permutation(b,a) (a,1)=1 (a,2)=7 (b,1)=0 (b,2)=1
  (c,1)=5 (c,2)=11", "permutation(a,c) permutation(b,a) (a,1)=1 (a,2)=7 (b,1)=0
  (b,2)=1 (c,1)=6 (c,2)=11", "permutation(a,c) permutation(b,a) (a,1)=2 (a,2)=7
  (b,1)=0 (b,2)=1 (c,1)=5 (c,2)=11", "permutation(a,c) permutation(b,a) (a,1)=2
  (a,2)=7 (b,1)=0 (b,2)=1 (c,1)=6 (c,2)=11", "permutation(a,c) permutation(b,a)
  (a,1)=3 (a,2)=7 (b,1)=0 (b,2)=1 (c,1)=6 (c,2)=11" }

-------------------------------------------------------------------------------
fs
  fso
-------------------------------------------------------------------------------
/opt/local/var/macports/build/_opt_PPCRosettaPorts_math_clingcon/clingcon/work/clingcon-5.2.0/libclingcon/tests/fs.cc:133
...............................................................................

/opt/local/var/macports/build/_opt_PPCRosettaPorts_math_clingcon/clingcon/work/clingcon-5.2.0/libclingcon/tests/fs.cc:134: FAILED:
  REQUIRE( remove_bound(solve(FSO + FSE + FSI, -256, 256)) == SOL16 )
with expansion:
  {  }
  ==
  { "permutation(a,c) permutation(b,a) (a,1)=1 (a,2)=7 (b,1)=0 (b,2)=1 (c,1)=4
  (c,2)=11", "permutation(a,c) permutation(b,a) (a,1)=1 (a,2)=7 (b,1)=0 (b,2)=1
  (c,1)=5 (c,2)=11", "permutation(a,c) permutation(b,a) (a,1)=1 (a,2)=7 (b,1)=0
  (b,2)=1 (c,1)=6 (c,2)=11", "permutation(a,c) permutation(b,a) (a,1)=2 (a,2)=7
  (b,1)=0 (b,2)=1 (c,1)=5 (c,2)=11", "permutation(a,c) permutation(b,a) (a,1)=2
  (a,2)=7 (b,1)=0 (b,2)=1 (c,1)=6 (c,2)=11", "permutation(a,c) permutation(b,a)
  (a,1)=3 (a,2)=7 (b,1)=0 (b,2)=1 (c,1)=6 (c,2)=11", "permutation(b,c)
  permutation(c,a) (a,1)=6 (a,2)=12 (b,1)=0 (b,2)=1 (c,1)=1 (c,2)=7",
  "permutation(b,c) permutation(c,a) (a,1)=7 (a,2)=12 (b,1)=0 (b,2)=1 (c,1)=1
  (c,2)=7", "permutation(b,c) permutation(c,a) (a,1)=7 (a,2)=12 (b,1)=0 (b,2)=1
  (c,1)=2 (c,2)=7", "permutation(b,c) permutation(c,a) (a,1)=8 (a,2)=12 (b,1)=0
  (b,2)=1 (c,1)=1 (c,2)=7", "permutation(b,c) permutation(c,a) (a,1)=8 (a,2)=12
  (b,1)=0 (b,2)=1 (c,1)=2 (c,2)=7", "permutation(b,c) permutation(c,a) (a,1)=9
  (a,2)=12 (b,1)=0 (b,2)=1 (c,1)=1 (c,2)=7", "permutation(b,c) permutation(c,a)
  (a,1)=9 (a,2)=12 (b,1)=0 (b,2)=1 (c,1)=2 (c,2)=7" }

===============================================================================
test cases:   13 |   11 passed | 2 failed
assertions: 2355 | 2351 passed | 4 failed

Any suggestions how to fix that would be greatly appreciated.

[rkaminsk]

I am not sure what could be the reason. Here are some points:

1. Do these failures consistently appear?
2. Did you make sure to use the latest clingo version? We fixed a lot of issues regarding clingcon. Calling clingcon --version should show the version.

And thanks for the efforts.

[barracuda156]

@rkaminsk Thank you for responding!

1. Do these failures consistently appear?

I can only test in Rosetta at the moment, being away from my PPC hardware. I can update on results on ppc32 and ppc64 natively in a couple of weeks.

2. Did you make sure to use the latest clingo version? We fixed a lot of issues regarding clingcon. Calling clingcon --version should show the version.

Yes, I have just built 5.6.2 and using that.

[rkaminsk]

@rkaminsk Thank you for responding!

1. Do these failures consistently appear?

I can only test in Rosetta at the moment, being away from my PPC hardware. I can update on results on ppc32 and ppc64 natively in a couple of weeks.

The tests use multi-threading. This might leave to non-deterministic failures. Is it always the same tests that fail in the same way?

I did not notice any test failures using the MacOS image with github's actions.

[barracuda156]

@rkaminsk I will try a couple of times to make sure.

P. S. I don’t think GitHub gonna have any PPC machines, not for macOS for sure.

[rkaminsk]

P. S. I don’t think GitHub gonna have any PPC machines, not for macOS for sure.

Yes, I know. Since you are using Rosetta an abstraction layer is involved. This can lead to "uncommon" scheduling of multi-threaded programs and can uncover bugs. That's why I was asking how the errors appear. Unfortunately, I only have an Intel Mac in office. Maybe it could still be used to reproduce the bug?

[barracuda156]

@rkaminsk If you happen to have 10.5.8 or 10.6.8 installed in a VM, that should be reproducible, provided Rosetta component is installed. This is what I used myself at the moment.
I will be back to PPC machines by mid Jan, then I can do testing there.
Generally speaking, Rosetta reproduces native ppc32 fairly well, but it does indeed have its own bugs (and performance in Rosetta as such is not a huge concern, what we want is native PPC).