[PATCH 08/25] Fix co-array allocation

[Toon Moene]

-------- Forwarded Message --------
Subject: [PATCH 08/25] Fix co-array allocation
Date: Wed, 5 Sep 2018 12:49:40 +0100
From: ***@codesourcery.com
To: gcc-***@gcc.gnu.org


The Fortran front-end has a bug in which it uses "int" values for "size_t"
parameters. I don't know why this isn't problem for all 64-bit
architectures,
but GCN ends up with the data in the wrong argument register and/or
stack slot,
and bad things happen.

This patch corrects the issue by setting the correct type.

2018-09-05 Kwok Cheung Yeung <***@codesourcery.com>

gcc/fortran/
* trans-expr.c (gfc_trans_structure_assign): Ensure that
integer_zero_node is of sizetype when used as the first
argument of a call to _gfortran_caf_register.
* trans-intrinsic.c (conv_intrinsic_event_query): Convert computed
index to a size_t type.
* trans-stmt.c (gfc_trans_event_post_wait): Likewise.
---
gcc/fortran/trans-expr.c | 2 +-
gcc/fortran/trans-intrinsic.c | 3 ++-
gcc/fortran/trans-stmt.c | 3 ++-
3 files changed, 5 insertions(+), 3 deletions(-)

[Bernhard Reutner-Fischer]

On Wed, 5 Sep 2018 at 18:54, Toon Moene <***@moene.org> wrote:
mhm, but why don't these use size_zero_node ?

thanks,

Post by Toon Moene
-------- Forwarded Message --------
Subject: [PATCH 08/25] Fix co-array allocation
Date: Wed, 5 Sep 2018 12:49:40 +0100
The Fortran front-end has a bug in which it uses "int" values for "size_t"
parameters. I don't know why this isn't problem for all 64-bit
architectures,
but GCN ends up with the data in the wrong argument register and/or
stack slot,
and bad things happen.
This patch corrects the issue by setting the correct type.
gcc/fortran/
* trans-expr.c (gfc_trans_structure_assign): Ensure that
integer_zero_node is of sizetype when used as the first
argument of a call to _gfortran_caf_register.
* trans-intrinsic.c (conv_intrinsic_event_query): Convert computed
index to a size_t type.
* trans-stmt.c (gfc_trans_event_post_wait): Likewise.
---
gcc/fortran/trans-expr.c | 2 +-
gcc/fortran/trans-intrinsic.c | 3 ++-
gcc/fortran/trans-stmt.c | 3 ++-
3 files changed, 5 insertions(+), 3 deletions(-)

[Janne Blomqvist]

Please send fortran patches to the fortran list as well!

Post by Toon Moene
-------- Forwarded Message --------
Subject: [PATCH 08/25] Fix co-array allocation
Date: Wed, 5 Sep 2018 12:49:40 +0100
The Fortran front-end has a bug in which it uses "int" values for "size_t"
parameters. I don't know why this isn't problem for all 64-bit
architectures,
but GCN ends up with the data in the wrong argument register and/or
stack slot,
and bad things happen.
This patch corrects the issue by setting the correct type.
gcc/fortran/
* trans-expr.c (gfc_trans_structure_assign): Ensure that
integer_zero_node is of sizetype when used as the first
argument of a call to _gfortran_caf_register.

The argument must be of type size_type_node, not sizetype. Please instead
use

size = build_zero_cst (size_type_node);

Post by Toon Moene
* trans-intrinsic.c (conv_intrinsic_event_query): Convert computed
index to a size_t type.

Using integer_type_node is wrong, but the correct type for calculating
array indices (lbound, ubound, etc.) is not size_type_node but rather
gfc_array_index_type (which in practice maps to ptrdiff_t). So please use
that, and then fold_convert index to size_type_node just before generating
the call to event_query.

Post by Toon Moene
* trans-stmt.c (gfc_trans_event_post_wait): Likewise.

Same here as above.

Thanks,

[Andrew Stubbs]

Post by Janne Blomqvist
The argument must be of type size_type_node, not sizetype. Please instead
use
size = build_zero_cst (size_type_node);

Post by Toon Moene
* trans-intrinsic.c (conv_intrinsic_event_query): Convert computed
index to a size_t type.

Using integer_type_node is wrong, but the correct type for calculating
array indices (lbound, ubound, etc.) is not size_type_node but rather
gfc_array_index_type (which in practice maps to ptrdiff_t). So please use
that, and then fold_convert index to size_type_node just before generating
the call to event_query.

Post by Toon Moene
* trans-stmt.c (gfc_trans_event_post_wait): Likewise.

Same here as above.

How is the attached? I retested and found no regressions.

Andrew

[Damian Rouson]

Has this been tested in multi-image execution using OpenCoarrays? If not,
I would be glad to assist with installing OpenCoarrays so that it can be
part of the testing process.

On a related note, two Sourcery Institute developers have attempted to edit
the GCC build system to make the downloading and building of OpenCoarrays
automatically part of the gfortran build process. Neither developer
succeeded. If anyone has any interest in figuring out how to do this, it
will prevent a lot of potential regressions when single-image testing
doesn't expose issues that only arise with multi-image execution.

Damian

Post by Toon Moene

Post by Janne Blomqvist
The argument must be of type size_type_node, not sizetype. Please instead
use
size = build_zero_cst (size_type_node);

Post by Toon Moene
* trans-intrinsic.c (conv_intrinsic_event_query): Convert

computed

Post by Janne Blomqvist

Post by Toon Moene
index to a size_t type.

Using integer_type_node is wrong, but the correct type for calculating
array indices (lbound, ubound, etc.) is not size_type_node but rather
gfc_array_index_type (which in practice maps to ptrdiff_t). So please use
that, and then fold_convert index to size_type_node just before

generating

Post by Janne Blomqvist
the call to event_query.

Post by Toon Moene
* trans-stmt.c (gfc_trans_event_post_wait): Likewise.

Same here as above.

How is the attached? I retested and found no regressions.
Andrew

[Andrew Stubbs]

Has this been tested in multi-image execution using OpenCoarrays?   If
not, I would be glad to assist with installing OpenCoarrays so that it
can be part of the testing process.

It's been tested with the GCC testsuite -- the same suite that found the
issue in the first place.

If you want to port your tool to GCN that would be cool, but I suspect
non-trivial.

On a related note, two Sourcery Institute developers have attempted to
edit the GCC build system to make the downloading and building of
OpenCoarrays automatically part of the gfortran build process.  Neither
developer succeeded.  If anyone has any interest in figuring out how to
do this, it will prevent a lot of potential regressions when
single-image testing doesn't expose issues that only arise with
multi-image execution.

I suggest you post this question in a fresh thread.

Andrew

[Damian Rouson]

Post by Andrew Stubbs
If you want to port your tool to GCN that would be cool, but I suspect
non-trivial.

To clarify, OpenCoarrays is not a tool. It is the parallel ABI required to
create executable programs capable of executing in multiple images as
required by the Fortran 2008 standard. Multi-image execution is the reason
coarray features exist so I hope the maintainers won't approve a patch that
impacts coarray features but has not been tested against OpenCoarrays,
which has its own test suite. Again, I would be glad to assist with
installing OpenCoarrays on your system. Whether it's trivial or not, it's
essential to protect against breaking a large feature set that is part of
Fortran 2008 and 2018.

Damian

[Thomas Koenig]

Hi Damian,

Post by Damian Rouson
On a related note, two Sourcery Institute developers have attempted to edit
the GCC build system to make the downloading and building of OpenCoarrays
automatically part of the gfortran build process. Neither developer
succeeded.

We addressed integrating OpenCoarray into the gcc source tree at the
recent Gcc summit during the gfortran BoF session.

Feedback from people working for big Linux distributions was that they
would prefer to package OpenCoarrays as a separate library.
(They also mentioned it was quite hard to build.)

Maybe these people could use some help from you.

Regards

Thomas

[Damian Rouson]

Post by Thomas Koenig
We addressed integrating OpenCoarray into the gcc source tree at the
recent Gcc summit during the gfortran BoF session.

I agree with keeping it as a separate code base, but comments from some
gfortran developers on the gfortran mailing list suggest that they liked
the idea of integrating the building of OpenCoarrays into the GCC build
system to simplify multi-image testing.

Post by Thomas Koenig
Feedback from people working for big Linux distributions was that they
would prefer to package OpenCoarrays as a separate library.
(They also mentioned it was quite hard to build.)
Maybe these people could use some help from you.

Thanks for the feedback. Please feel free to put me in touch with them or
suggest that they submit issues on the OpenCoarrays repository. We would
be glad to help. We've put a lot of time into addressing installation
issues that have been submitted to us and we'll continue to do so if we
receive reports.

Damian

[Toon Moene]

Post by Thomas Koenig
Hi Damian,

Post by Damian Rouson
On a related note, two Sourcery Institute developers have attempted to edit
the GCC build system to make the downloading and building of OpenCoarrays
automatically part of the gfortran build process.  Neither developer
succeeded.

We addressed integrating OpenCoarray into the gcc source tree at the
recent Gcc summit during the gfortran BoF session.
Feedback from people working for big Linux distributions was that they
would prefer to package OpenCoarrays as a separate library.
(They also mentioned it was quite hard to build.)

Well, Linux distributors have to fit the build of OpenCoarrays into
*their* build system, which might be just as complicated as we trying it
to force it into *gcc's* build system ...

For an individual, OpenCoarrays is not hard to build, and the web page
www.opencoarrays.org offers multiple solutions:

"Installation via package management is generally the easiest and most
reliable option. See below for the package-management installation
options for Linux, macOS, and FreeBSD. Alternatively, download and
build the latest OpenCoarrays release via the contained installation
scripts or with CMake."

I choose the cmake based one, because I already had cmake installed to
be able to build ECMWF's (ecmwf.int) eccodes package. It probably helped
that I also already had openmpi installed. From my command history:

1754 tar zxvf ~/Downloads/OpenCoarrays-2.2.0.tar.gz
1755 cd OpenCoarrays-2.2.0/
1756 ls
1757 less README.md
1758 cd ..
1759 mkdir opencoarrays-build
1760 cd opencoarrays-build
1761 (export FC=gfortran; export CC=gcc; cmake ../OpenCoarrays-2.2.0/
-DCMAKE_INSTALL_PREFIX=$HOME/opencoarrays)
1762 make
1763 make test
1764 make install

After that, it was a breeze to test my mock weather program
(moene.org/~toon/random-weather.f90), that I had built until then only
with -fcoarray=single.

[Janne Blomqvist]

Post by Damian Rouson

Post by Thomas Koenig
Hi Damian,

Post by Damian Rouson
On a related note, two Sourcery Institute developers have attempted to edit
the GCC build system to make the downloading and building of

OpenCoarrays

Post by Thomas Koenig

Post by Damian Rouson
automatically part of the gfortran build process. Neither developer
succeeded.

We addressed integrating OpenCoarray into the gcc source tree at the
recent Gcc summit during the gfortran BoF session.
Feedback from people working for big Linux distributions was that they
would prefer to package OpenCoarrays as a separate library.
(They also mentioned it was quite hard to build.)

Well, Linux distributors have to fit the build of OpenCoarrays into
*their* build system, which might be just as complicated as we trying it
to force it into *gcc's* build system ...
For an individual, OpenCoarrays is not hard to build, and the web page
"Installation via package management is generally the easiest and most
reliable option. See below for the package-management installation
options for Linux, macOS, and FreeBSD. Alternatively, download and
build the latest OpenCoarrays release via the contained installation
scripts or with CMake."
I choose the cmake based one, because I already had cmake installed to
be able to build ECMWF's (ecmwf.int) eccodes package. It probably helped
1754 tar zxvf ~/Downloads/OpenCoarrays-2.2.0.tar.gz
1755 cd OpenCoarrays-2.2.0/
1756 ls
1757 less README.md
1758 cd ..
1759 mkdir opencoarrays-build
1760 cd opencoarrays-build
1761 (export FC=gfortran; export CC=gcc; cmake ../OpenCoarrays-2.2.0/
-DCMAKE_INSTALL_PREFIX=$HOME/opencoarrays)
1762 make
1763 make test
1764 make install

FWIW, this didn't work for me, as I want to use my own build of gfortran
trunk. It did correctly use the correct gfortran binary as specified by the
FC env. variable, but it still insists on linking against libgfortran.so.4
(installed by the system package manager) and not the libgfortran.so.5 from
my own gfortran installation (found both on LD_RUN_PATH and
LD_LIBRARY_PATH). I tried -DCMAKE_PREFIX_PATH=... but that didn't work any
better. Gah, I hate cmake..

Any ideas?

[Jerry DeLisle]

Post by Thomas Koenig
Hi Damian,

Post by Damian Rouson
On a related note, two Sourcery Institute developers have attempted to edit
the GCC build system to make the downloading and building of OpenCoarrays
automatically part of the gfortran build process.  Neither developer
succeeded.

We addressed integrating OpenCoarray into the gcc source tree at the
recent Gcc summit during the gfortran BoF session.
Feedback from people working for big Linux distributions was that they
would prefer to package OpenCoarrays as a separate library.
(They also mentioned it was quite hard to build.)

I would like to put in my humble 2 cents worth here.

OpenCoarrays was/is intended for a very broad audience, various large
systems such as Cray, etc. I think this influenced heavily the path of
its development, which is certainly OK.

It was/is intended to interface libraries such as OpenMPI or MPICH to
gfortran as well as other Fortran compilers.

The actual library source code is contained mostly in one source file.
After all the attempts to integrate into the GNU build systems without
much success my thinking has shifted. Keep in mind that the OpenCoarrays
implementation is quite dependent on gfortran and in fact has to do
special things in the build dependent on the version of gcc/gfortran a
user happens to use. I dont think this is a good situation.

So I see two realistic strategies. The first is already talked about a
lot and is the cleanest approach for gfortran:

1) Focus on distribution packages such as Fedora, Debian, Ubuntu,
Windows, etc. Building of these packages needs to be automated into the
distributions. I think mostly this is what is happening and relies on
the various distribution maintainers to do so. Their support is greatly
appreciated and this really is the cleanest approach.

The second option is not discussed as much because it leaves
OpenCoarrays behind in a sense and requires an editing cycle in two
places to fix bugs or add features.

2) Take the one source file, edit out all the macros that define
prefixes to function calls, hard code the gfortran prefixes etc and fork
it directly into the libgfortran library under GPL with attributions to
the original developers as appropriate.

Strategy 2 would lock into specific current standard versions of the MPI
interface and would support less bleeding edge changes. It would also
require either OpenMPI or MPICH as a new gfortran dependency for
building, which not all users may need. So we would need some
configuration magic to enable or disable this portion of the build.
Something like --with-MPI-support would do the trick.

Strategy 2 does add burden to gfortran maintainers who are already
overloaded. But, as the code matures the burden would decrease,
particularly once TEAMS are finished.

Strategy 2 does have some advantages. For example, eliminating the need
for separate CAF and CAFRUN scripts which are a wrapper on gfortran.
The coarray features are part of the Fortran language and gfortran
should just "handle it" transparently using an environment variable to
define the number of images at run time. It would also actually
eliminate the need to manage all of the separate distribution packages.
So from a global point of view the overall maintanance effort would be
reduced.

Strategy 2 would enable a set of users who are not focused so much on
distributions and loading packages, etc etc and those who are dependent
on getting through bureaucratic administrations who already are loading
gfortran on systems and would not have to also get another package
approved. People would just have to stop thinking about it and just use it.

So I think there are real advantages to Strategy 2 as well as Strategy 1
and think it should be at least included in discussions. I would even
suggest there is likely a combination of 1 and 2 that may hit the mark.
For example, keeping OpenCoarrays as a separate package for bleeding
edge development and migrating the stable features into libgfortran on a
less frequent cycle.

As I said, my 2 cents worth.

Regards to all,

Jerry

[Janne Blomqvist]

Post by Damian Rouson

Post by Thomas Koenig
Hi Damian,

Post by Damian Rouson
On a related note, two Sourcery Institute developers have attempted to edit
the GCC build system to make the downloading and building of

OpenCoarrays

Post by Thomas Koenig

Post by Damian Rouson
automatically part of the gfortran build process. Neither developer
succeeded.

We addressed integrating OpenCoarray into the gcc source tree at the
recent Gcc summit during the gfortran BoF session.
Feedback from people working for big Linux distributions was that they
would prefer to package OpenCoarrays as a separate library.
(They also mentioned it was quite hard to build.)

I would like to put in my humble 2 cents worth here.
OpenCoarrays was/is intended for a very broad audience, various large
systems such as Cray, etc. I think this influenced heavily the path of
its development, which is certainly OK.
It was/is intended to interface libraries such as OpenMPI or MPICH to
gfortran as well as other Fortran compilers.
The actual library source code is contained mostly in one source file.
After all the attempts to integrate into the GNU build systems without
much success my thinking has shifted. Keep in mind that the OpenCoarrays
implementation is quite dependent on gfortran and in fact has to do
special things in the build dependent on the version of gcc/gfortran a
user happens to use. I dont think this is a good situation.
So I see two realistic strategies. The first is already talked about a
1) Focus on distribution packages such as Fedora, Debian, Ubuntu,
Windows, etc. Building of these packages needs to be automated into the
distributions. I think mostly this is what is happening and relies on
the various distribution maintainers to do so. Their support is greatly
appreciated and this really is the cleanest approach.
The second option is not discussed as much because it leaves
OpenCoarrays behind in a sense and requires an editing cycle in two
places to fix bugs or add features.
2) Take the one source file, edit out all the macros that define
prefixes to function calls, hard code the gfortran prefixes etc and fork
it directly into the libgfortran library under GPL with attributions to
the original developers as appropriate.
Strategy 2 would lock into specific current standard versions of the MPI
interface and would support less bleeding edge changes. It would also
require either OpenMPI or MPICH as a new gfortran dependency for
building, which not all users may need. So we would need some
configuration magic to enable or disable this portion of the build.
Something like --with-MPI-support would do the trick.
Strategy 2 does add burden to gfortran maintainers who are already
overloaded. But, as the code matures the burden would decrease,
particularly once TEAMS are finished.
Strategy 2 does have some advantages. For example, eliminating the need
for separate CAF and CAFRUN scripts which are a wrapper on gfortran.
The coarray features are part of the Fortran language and gfortran
should just "handle it" transparently using an environment variable to
define the number of images at run time. It would also actually
eliminate the need to manage all of the separate distribution packages.
So from a global point of view the overall maintanance effort would be
reduced.
Strategy 2 would enable a set of users who are not focused so much on
distributions and loading packages, etc etc and those who are dependent
on getting through bureaucratic administrations who already are loading
gfortran on systems and would not have to also get another package
approved. People would just have to stop thinking about it and just use it.
So I think there are real advantages to Strategy 2 as well as Strategy 1
and think it should be at least included in discussions. I would even
suggest there is likely a combination of 1 and 2 that may hit the mark.
For example, keeping OpenCoarrays as a separate package for bleeding
edge development and migrating the stable features into libgfortran on a
less frequent cycle.
As I said, my 2 cents worth.
Regards to all,
Jerry

I recall one motivation for the current sort-of loose coupling between the
coarray library and gfortran was to support, at runtime, different MPI
libraries. This can be useful on cluster and supercomputers, where it's
important to use a MPI library that can use the high-performance cluster
network. If libgfortran includes the coarray library which links against a
MPI library, it means libgfortran has to be rebuilt against every MPI
library in use on a system, and most likely, one cannot use the
distro-provided gfortran. This might not be insurmountable on cluster
using some kind of module system, but still.

I guess it might be possible to use weak symbols, like we currently use for
some things in libgfortran (e.g. clock_gettime), but that would mean a
quite big diff compared to upstream OpenCoarrays. And how to handle targets
that don't support weak symbols in some sane fashion, etc.

[Richard Biener]

Post by Damian Rouson

Post by Thomas Koenig
Hi Damian,

Post by Damian Rouson
On a related note, two Sourcery Institute developers have attempted

to

Post by Thomas Koenig

Post by Damian Rouson
edit
the GCC build system to make the downloading and building of

OpenCoarrays

Post by Thomas Koenig

Post by Damian Rouson
automatically part of the gfortran build process.  Neither developer
succeeded.

We addressed integrating OpenCoarray into the gcc source tree at the
recent Gcc summit during the gfortran BoF session.
Feedback from people working for big Linux distributions was that

they

Post by Thomas Koenig
would prefer to package OpenCoarrays as a separate library.
(They also mentioned it was quite hard to build.)

I would like to put in my humble 2 cents worth here.
OpenCoarrays was/is intended for a very broad audience, various large
systems such as Cray, etc. I think this influenced heavily the path of
its development, which is certainly OK.
It was/is intended to interface libraries such as OpenMPI or MPICH to
gfortran as well as other Fortran compilers.
The actual library source code is contained mostly in one source file.
After all the attempts to integrate into the GNU build systems without
much success my thinking has shifted. Keep in mind that the
OpenCoarrays
implementation is quite dependent on gfortran and in fact has to do
special things in the build dependent on the version of gcc/gfortran a
user happens to use. I dont think this is a good situation.
So I see two realistic strategies. The first is already talked about a
1) Focus on distribution packages such as Fedora, Debian, Ubuntu,
Windows, etc. Building of these packages needs to be automated into the
distributions. I think mostly this is what is happening and relies on
the various distribution maintainers to do so. Their support is greatly
appreciated and this really is the cleanest approach.
The second option is not discussed as much because it leaves
OpenCoarrays behind in a sense and requires an editing cycle in two
places to fix bugs or add features.
2) Take the one source file, edit out all the macros that define
prefixes to function calls, hard code the gfortran prefixes etc and fork
it directly into the libgfortran library under GPL with attributions to
the original developers as appropriate.
Strategy 2 would lock into specific current standard versions of the MPI
interface and would support less bleeding edge changes. It would also
require either OpenMPI or MPICH as a new gfortran dependency for
building, which not all users may need. So we would need some
configuration magic to enable or disable this portion of the build.
Something like --with-MPI-support would do the trick.
Strategy 2 does add burden to gfortran maintainers who are already
overloaded. But, as the code matures the burden would decrease,
particularly once TEAMS are finished.
Strategy 2 does have some advantages. For example, eliminating the need
for separate CAF and CAFRUN scripts which are a wrapper on gfortran.
The coarray features are part of the Fortran language and gfortran
should just "handle it" transparently using an environment variable to
define the number of images at run time. It would also actually
eliminate the need to manage all of the separate distribution packages.
So from a global point of view the overall maintanance effort would be
reduced.
Strategy 2 would enable a set of users who are not focused so much on
distributions and loading packages, etc etc and those who are dependent
on getting through bureaucratic administrations who already are loading
gfortran on systems and would not have to also get another package
approved. People would just have to stop thinking about it and just use it.
So I think there are real advantages to Strategy 2 as well as Strategy 1
and think it should be at least included in discussions. I would even
suggest there is likely a combination of 1 and 2 that may hit the mark.
For example, keeping OpenCoarrays as a separate package for bleeding
edge development and migrating the stable features into libgfortran on a
less frequent cycle.

Sounds reasonable to me. License issues will be the most difficult here given integration with libgfortran likely requires a FSF copyright rather than just a compatible license.

Richard.

Post by Damian Rouson
As I said, my 2 cents worth.
Regards to all,
Jerry

[Damian Rouson]

Post by Jerry DeLisle
The actual library source code is contained mostly in one source file.

There are as many files as there are options for the underlying
parallel programming
model. The default is MPI, but I've co-authored conference papers last year
and this year in which the OpenCoarrays OpenSHEM option outperformed MPI.
One paper even described a platform on which OpenSHMEM was the only option
beyond a few thousand cores because the required MPI features were immature on
that platform. Early versions of OpenCoarrays also provided GASNet
and ARMCI options.
I recommend against tying gfortran to MPI only.

Post by Jerry DeLisle
After all the attempts to integrate into the GNU build systems without
much success my thinking has shifted.

Thanks for all your efforts!

Post by Jerry DeLisle
Keep in mind that the OpenCoarrays
implementation is quite dependent on gfortran and in fact has to do
special things in the build dependent on the version of gcc/gfortran a
user happens to use. I dont think this is a good situation.

I agree. Possibly OpenCoarrays could drop support for older gfortran versions
at some point to avoid maintaining code that exists solely to support compiler
versions that are several years old.

Post by Jerry DeLisle
1) Focus on distribution packages such as Fedora, Debian, Ubuntu,
Windows, etc. Building of these packages needs to be automated into the
distributions.

This is the option that the OpenCoarrays documentation recommends as easiest for
most users.

Post by Jerry DeLisle
2) Take the one source file, edit out all the macros that define
prefixes to function calls, hard code the gfortran prefixes etc and fork
it directly into the libgfortran library under GPL with attributions to
the original developers as appropriate.

See above. Also, this means that changes in the gfortran repository would not
propagate back upstream unless each gfortran developer agrees to
distribute his or her
work under both GPL and BSD. Even that is only feasible if the copied
files stay cohesive
and don't reference code outside the copied file. I think it's more
likely that copying the code
into gfortran would be a branch point, after which the relevant files
would diverge and
work on the GPL side would be harder to fund than the BSD side.

Most commercial entities are more likely to contribute to a
BSD-licensed project than a
GPL-licensed one. Over the past several months, one commercial compiler vendor
authorized one of their developers to contribute to OpenCoarrays. and
another commercial
compiler vendor invited community input on whether to use OpenCoarrays
during a public
teleconference. The prospect of commercial support is the motivation
for using BSD.

Post by Jerry DeLisle
Strategy 2 does have some advantages. For example, eliminating the need
for separate CAF and CAFRUN scripts which are a wrapper on gfortran.

Even in the case of just one underlying parallel programming model,
this is tricky. To wit, Cray uses
a compiler wrapper and a program launcher. Intel was able to
eliminate the compiler wrapper,
but still required a program launcher for distributed-memory execution
until recently. I don't
know the details, but I've heard it was not trivial for Intel to
accomplish this and I imagine it would be
even more complicated if they weren't hardwiring Intel MPI into their back-end.

Post by Jerry DeLisle
People would just have to stop thinking about it and just use it.

The same would be true if someone could coax the GCC build system to
build OpenCoarrays
just as it builds other prerequisites. The big difference is that
OpenCoarrays is a prerequisite
for using gfortran rather than for building gfortran so it needs to be
built after gfortran rather
than before like other prerequisites. The real problem is finding
anyone who can work the
proper magic in the GCC build system.

Thanks for your input. I hope my response is helpful.

Damian

[Jerry DeLisle]

On 9/21/18 1:16 PM, Damian Rouson wrote:> On Fri, Sep 21, 2018 at 9:25

Post by Damian Rouson

Post by Jerry DeLisle
The actual library source code is contained mostly in one source file.

There are as many files as there are options for the underlying
parallel programming
model. The default is MPI, but I've co-authored conference papers last year
and this year in which the OpenCoarrays OpenSHEM option outperformed MPI.
One paper even described a platform on which OpenSHMEM was the only option
beyond a few thousand cores because the required MPI features were immature on
that platform. Early versions of OpenCoarrays also provided GASNet
and ARMCI options.
I recommend against tying gfortran to MPI only.

I agree with you on this point. Perhaps the Opencoarrays implementation
should somehow do some runtime introspection to allow the library to
sync to whatever is desired on a given system. The gfortran interface
was designed to be generic. Implementation should be more dynamic in run
time linking and abstracted in such a way that OpenCoarrays could be
compiled stand alone and use something like "plugins" to allow post
build the determination of what which interface to use.

I am by no means a software expert in these techniques, but they are
becoming common practice in other areas, for example linux/Gnu kernel
modules

Post by Damian Rouson

Post by Jerry DeLisle
After all the attempts to integrate into the GNU build systems without
much success my thinking has shifted.

Thanks for all your efforts!

Post by Jerry DeLisle
Keep in mind that the OpenCoarrays
implementation is quite dependent on gfortran and in fact has to do
special things in the build dependent on the version of gcc/gfortran a
user happens to use. I dont think this is a good situation.

I agree. Possibly OpenCoarrays could drop support for older gfortran versions
at some point to avoid maintaining code that exists solely to support compiler
versions that are several years old.

See my comments above about pluggable modules. Maybe libgfortran should
have this pluggable interface and Opencoarrays provide the plugins.
Think how useful it would be to be able to choose the backend at time of
execution based on a simple envoronment variable set by the user.

Post by Damian Rouson

Post by Jerry DeLisle
1) Focus on distribution packages such as Fedora, Debian, Ubuntu,
Windows, etc. Building of these packages needs to be automated into the
distributions.

This is the option that the OpenCoarrays documentation recommends as easiest for
most users.

Agree.

Post by Damian Rouson

Post by Jerry DeLisle
2) Take the one source file, edit out all the macros that define
prefixes to function calls, hard code the gfortran prefixes etc and fork
it directly into the libgfortran library under GPL with attributions to
the original developers as appropriate.

See above. Also, this means that changes in the gfortran repository would not
propagate back upstream unless each gfortran developer agrees to
distribute his or her
work under both GPL and BSD. Even that is only feasible if the copied
files stay cohesive

The flip of this would be to have the OpenCorrays developers to agree to
the GPL and release under both. The libgfortran license says:

"Under Section 7 of GPL version 3, you are granted additional
permissions described in the GCC Runtime Library Exception, version
3.1, as published by the Free Software Foundation."

Probably worth a fresh look.

Post by Damian Rouson
and don't reference code outside the copied file. I think it's more
likely that copying the code
into gfortran would be a branch point, after which the relevant files
would diverge and
work on the GPL side would be harder to fund than the BSD side.
Most commercial entities are more likely to contribute to a
BSD-licensed project than a
GPL-licensed one. Over the past several months, one commercial compiler vendor
authorized one of their developers to contribute to OpenCoarrays. and
another commercial
compiler vendor invited community input on whether to use OpenCoarrays
during a public
teleconference. The prospect of commercial support is the motivation
for using BSD.

I really have no commercial interest. So I will not comment on GPL vs
BSD other than referring to the multitude of FSF recommendations about
why one should choose one of the FSF flavors rather than BSD.

Post by Damian Rouson

Post by Jerry DeLisle
Strategy 2 does have some advantages. For example, eliminating the need
for separate CAF and CAFRUN scripts which are a wrapper on gfortran.

Even in the case of just one underlying parallel programming model,
this is tricky. To wit, Cray uses
a compiler wrapper and a program launcher. Intel was able to
eliminate the compiler wrapper,
but still required a program launcher for distributed-memory execution
until recently. I don't
know the details, but I've heard it was not trivial for Intel to
accomplish this and I imagine it would be
even more complicated if they weren't hardwiring Intel MPI into their back-end.

Well here is one commercial entity that did not shy away from
'hardwiring' MPI, Regardless, using plugins would resolve concerns about
which MPI to use or whether to use shared memory or some other model.

Post by Damian Rouson

Post by Jerry DeLisle
People would just have to stop thinking about it and just use it.

The same would be true if someone could coax the GCC build system to
build OpenCoarrays
just as it builds other prerequisites. The big difference is that
OpenCoarrays is a prerequisite
for using gfortran rather than for building gfortran so it needs to be
built after gfortran rather
than before like other prerequisites. The real problem is finding
anyone who can work the
proper magic in the GCC build system.

I dont see this as the real problem. The forking idea would resolve this
fairly easily. Above, you mentioned concern about locking into MPI. Do
the packaged versions of OpenCoarrays not lock into MPI, either OpenMPI
or MPICH? I have not tried one yet since I am waiting for the Fedora one
to hit the release.

If the tight coupling is needed maybe there ought to be a set of
libraries or modules, one for each "backend". (Back to my pluggable
modules concept) The more I think about it the more I think this is the
fundamental design issue.

Post by Damian Rouson
Thanks for your input. I hope my response is helpful.
Damian

As always, best regards.

Jerry

[Toon Moene]

Post by Jerry DeLisle
On 9/21/18 1:16 PM, Damian Rouson wrote:> On Fri, Sep 21, 2018 at 9:25

Post by Damian Rouson

Post by Jerry DeLisle
1) Focus on distribution packages such as Fedora, Debian, Ubuntu,
Windows, etc. Building of these packages needs to be automated into the
distributions.

This is the option that the OpenCoarrays documentation recommends as

easiest for

Post by Damian Rouson
most users.

Agree.

I just installed opencoarrays on my system at home (Debian Testing):

***@moene:~# apt-get install libcoarrays-openmpi-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
libcaf-openmpi-3
The following NEW packages will be installed:
libcaf-openmpi-3 libcoarrays-openmpi-dev
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 107 kB of archives.
After this operation, 317 kB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://ftp.nl.debian.org/debian testing/main amd64
libcaf-openmpi-3 amd64 2.2.0-3 [38.2 kB]
Get:2 http://ftp.nl.debian.org/debian testing/main amd64
libcoarrays-openmpi-dev amd64 2.2.0-3 [68.9 kB]
Fetched 107 kB in 0s (634 kB/s)
Selecting previously unselected package libcaf-openmpi-3:amd64.
(Reading database ... 212249 files and directories currently installed.)
Preparing to unpack .../libcaf-openmpi-3_2.2.0-3_amd64.deb ...
Unpacking libcaf-openmpi-3:amd64 (2.2.0-3) ...
Selecting previously unselected package libcoarrays-openmpi-dev:amd64.
Preparing to unpack .../libcoarrays-openmpi-dev_2.2.0-3_amd64.deb ...
Unpacking libcoarrays-openmpi-dev:amd64 (2.2.0-3) ...
Setting up libcaf-openmpi-3:amd64 (2.2.0-3) ...
Setting up libcoarrays-openmpi-dev:amd64 (2.2.0-3) ...
Processing triggers for libc-bin (2.27-6) ...

[ previously this led to apt errors, but not now. ]

and moved my own installation of the OpenCoarrays-2.2.0.tar.gz out of
the way:

***@moene:~$ ls -ld *pen*
drwxr-xr-x 6 toon toon 4096 Aug 10 16:01 OpenCoarrays-2.2.0.opzij
drwxr-xr-x 8 toon toon 4096 Sep 15 11:26 opencoarrays-build.opzij
drwxr-xr-x 6 toon toon 4096 Sep 15 11:26 opencoarrays.opzij

and recompiled my stuff:

gfortran -g -fbacktrace -fcoarray=lib random-weather.f90
-L/usr/lib/x86_64-linux-gnu/open-coarrays/openmpi/lib -lcaf_mpi

[ Yes, the location of the libs is quite experimental, but OK for the
"Testing" variant of Debian ... ]

I couldn't find cafrun, but mpirun works just fine:

***@moene:~/src$ echo ' &config /' | mpirun --oversubscribe --bind-to
none -np 20 ./a.out
Decomposition information on image 7 is 4 * 5 slabs with 23 *
18 grid cells on this image.
Decomposition information on image 6 is 4 * 5 slabs with 23 *
18 grid cells on this image.
Decomposition information on image 11 is 4 * 5 slabs with 23 *
18 grid cells on this image.
Decomposition information on image 15 is 4 * 5 slabs with 23 *
18 grid cells on this image.
Decomposition information on image 1 is 4 * 5 slabs with 23 *
18 grid cells on this image.
Decomposition information on image 13 is 4 * 5 slabs with 23 *
18 grid cells on this image.
Decomposition information on image 12 is 4 * 5 slabs with 21 *
18 grid cells on this image.
Decomposition information on image 20 is 4 * 5 slabs with 21 *
18 grid cells on this image.
Decomposition information on image 9 is 4 * 5 slabs with 23 *
18 grid cells on this image.
Decomposition information on image 14 is 4 * 5 slabs with 23 *
18 grid cells on this image.
Decomposition information on image 16 is 4 * 5 slabs with 21 *
18 grid cells on this image.
Decomposition information on image 17 is 4 * 5 slabs with 23 *
18 grid cells on this image.
Decomposition information on image 18 is 4 * 5 slabs with 23 *
18 grid cells on this image.
Decomposition information on image 2 is 4 * 5 slabs with 23 *
18 grid cells on this image.
Decomposition information on image 4 is 4 * 5 slabs with 21 *
18 grid cells on this image.
Decomposition information on image 5 is 4 * 5 slabs with 23 *
18 grid cells on this image.
Decomposition information on image 3 is 4 * 5 slabs with 23 *
18 grid cells on this image.
Decomposition information on image 8 is 4 * 5 slabs with 21 *
18 grid cells on this image.
Decomposition information on image 10 is 4 * 5 slabs with 23 *
18 grid cells on this image.
Decomposition information on image 19 is 4 * 5 slabs with 23 *
18 grid cells on this image.

... etc. (see http://moene.org/~toon/random-weather.f90).

I presume other Linux distributors will follow shortly (this *is* Debian
Testing, which can be a bit testy at times - but I do trust my main
business at home on it for over 15 years now).

Kind regards,

[Bernhard Reutner-Fischer]

Post by Thomas Koenig

Post by Jerry DeLisle
On 9/21/18 1:16 PM, Damian Rouson wrote:> On Fri, Sep 21, 2018 at

9:25

Post by Jerry DeLisle

Post by Damian Rouson

Post by Jerry DeLisle
1) Focus on distribution packages such as Fedora, Debian, Ubuntu,
Windows, etc. Building of these packages needs to be automated

into the

Post by Jerry DeLisle

Post by Damian Rouson

Post by Jerry DeLisle
distributions.

This is the option that the OpenCoarrays documentation recommends

as

Post by Jerry DeLisle
easiest for

Post by Damian Rouson
most users.

Agree.

Reading package lists... Done
Building dependency tree
Reading state information... Done
libcaf-openmpi-3
libcaf-openmpi-3 libcoarrays-openmpi-dev
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 107 kB of archives.
After this operation, 317 kB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://ftp.nl.debian.org/debian testing/main amd64
libcaf-openmpi-3 amd64 2.2.0-3 [38.2 kB]
Get:2 http://ftp.nl.debian.org/debian testing/main amd64
libcoarrays-openmpi-dev amd64 2.2.0-3 [68.9 kB]
Fetched 107 kB in 0s (634 kB/s)
Selecting previously unselected package libcaf-openmpi-3:amd64.
(Reading database ... 212249 files and directories currently
installed.)
Preparing to unpack .../libcaf-openmpi-3_2.2.0-3_amd64.deb ...
Unpacking libcaf-openmpi-3:amd64 (2.2.0-3) ...
Selecting previously unselected package libcoarrays-openmpi-dev:amd64.
Preparing to unpack .../libcoarrays-openmpi-dev_2.2.0-3_amd64.deb ...
Unpacking libcoarrays-openmpi-dev:amd64 (2.2.0-3) ...
Setting up libcaf-openmpi-3:amd64 (2.2.0-3) ...
Setting up libcoarrays-openmpi-dev:amd64 (2.2.0-3) ...
Processing triggers for libc-bin (2.27-6) ...
[ previously this led to apt errors, but not now. ]
and moved my own installation of the OpenCoarrays-2.2.0.tar.gz out of
drwxr-xr-x 6 toon toon 4096 Aug 10 16:01 OpenCoarrays-2.2.0.opzij
drwxr-xr-x 8 toon toon 4096 Sep 15 11:26 opencoarrays-build.opzij
drwxr-xr-x 6 toon toon 4096 Sep 15 11:26 opencoarrays.opzij
gfortran -g -fbacktrace -fcoarray=lib random-weather.f90
-L/usr/lib/x86_64-linux-gnu/open-coarrays/openmpi/lib -lcaf_mpi
[ Yes, the location of the libs is quite experimental, but OK for the
"Testing" variant of Debian ... ]

Are you sure you need the -L?
For me a simple -fcoarray=lib -lcaf_mpi
links fine.
Along the same lines a simple
$ mpirun -np 4 ./a.out
runs fine as expected, like any other mpi program.

Cheers,

Post by Thomas Koenig
none -np 20 ./a.out
Decomposition information on image 7 is 4 * 5 slabs with 23 *
18 grid cells on this image.
Decomposition information on image 6 is 4 * 5 slabs with 23 *
18 grid cells on this image.
Decomposition information on image 11 is 4 * 5 slabs with 23 *
18 grid cells on this image.
Decomposition information on image 15 is 4 * 5 slabs with 23 *
18 grid cells on this image.
Decomposition information on image 1 is 4 * 5 slabs with 23 *
18 grid cells on this image.
Decomposition information on image 13 is 4 * 5 slabs with 23 *
18 grid cells on this image.
Decomposition information on image 12 is 4 * 5 slabs with 21 *
18 grid cells on this image.
Decomposition information on image 20 is 4 * 5 slabs with 21 *
18 grid cells on this image.
Decomposition information on image 9 is 4 * 5 slabs with 23 *
18 grid cells on this image.
Decomposition information on image 14 is 4 * 5 slabs with 23 *
18 grid cells on this image.
Decomposition information on image 16 is 4 * 5 slabs with 21 *
18 grid cells on this image.
Decomposition information on image 17 is 4 * 5 slabs with 23 *
18 grid cells on this image.
Decomposition information on image 18 is 4 * 5 slabs with 23 *
18 grid cells on this image.
Decomposition information on image 2 is 4 * 5 slabs with 23 *
18 grid cells on this image.
Decomposition information on image 4 is 4 * 5 slabs with 21 *
18 grid cells on this image.
Decomposition information on image 5 is 4 * 5 slabs with 23 *
18 grid cells on this image.
Decomposition information on image 3 is 4 * 5 slabs with 23 *
18 grid cells on this image.
Decomposition information on image 8 is 4 * 5 slabs with 21 *
18 grid cells on this image.
Decomposition information on image 10 is 4 * 5 slabs with 23 *
18 grid cells on this image.
Decomposition information on image 19 is 4 * 5 slabs with 23 *
18 grid cells on this image.
... etc. (see http://moene.org/~toon/random-weather.f90).
I presume other Linux distributors will follow shortly (this *is* Debian
Testing, which can be a bit testy at times - but I do trust my main
business at home on it for over 15 years now).
Kind regards,

[Toon Moene]

[ dropping mailing list gcc-patches, as it is not relevant to *this*
discussion :-) ]

Post by Bernhard Reutner-Fischer

Post by Toon Moene
gfortran -g -fbacktrace -fcoarray=lib random-weather.f90
-L/usr/lib/x86_64-linux-gnu/open-coarrays/openmpi/lib -lcaf_mpi
[ Yes, the location of the libs is quite experimental, but OK for the
"Testing" variant of Debian ... ]

Are you sure you need the -L?
For me a simple -fcoarray=lib -lcaf_mpi
links fine.

I get this:

***@moene:~/src$ gfortran -g -fbacktrace -fcoarray=lib
random-weather.f90 -lcaf_mpi
/usr/bin/ld: cannot find -lcaf_mpi
collect2: error: ld returned 1 exit status

Are you sure the linker isn't finding another libcaf_mpi.so ?

Post by Bernhard Reutner-Fischer
Along the same lines a simple
$ mpirun -np 4 ./a.out
runs fine as expected, like any other mpi program.

Yeah, sorry. I was already using the options --oversubscribe --bind-to
none (of which the --oversubscribe certainly is necessary) before I
installed Debian's coarray packages. I need it because otherwise openmpi
will complain that I only have 10 cores (which is correct).

Kind regards,

[Bernhard Reutner-Fischer]

Post by Toon Moene
[ dropping mailing list gcc-patches, as it is not relevant to *this*
discussion :-) ]

yea, sorry for that.

Post by Toon Moene

Post by Bernhard Reutner-Fischer

Post by Toon Moene
gfortran -g -fbacktrace -fcoarray=lib random-weather.f90
-L/usr/lib/x86_64-linux-gnu/open-coarrays/openmpi/lib -lcaf_mpi
[ Yes, the location of the libs is quite experimental, but OK for the
"Testing" variant of Debian ... ]

Are you sure you need the -L?
For me a simple -fcoarray=lib -lcaf_mpi
links fine.

random-weather.f90 -lcaf_mpi
/usr/bin/ld: cannot find -lcaf_mpi
collect2: error: ld returned 1 exit status
Are you sure the linker isn't finding another libcaf_mpi.so ?

Yes, i'm certain it finds the right libcaf_mpi and openmpi, but then
you're on testing and i'm on sid, maybe that's a difference.
For reference (my compiler built from trunk lives in /opt/$(uname
-m)/gcc-9.0.mine):
$ cat ~/tmp/hello.f08;echo EOF
implicit none
character(len=64) :: str, hi[*]
integer :: i, nprocs
nprocs = num_images()
do i=1, nprocs
!write(hi[this_image()], '(i0,a,i0,": ",a)') this_image(), '/',
nprocs, 'hello world' ! :) but data-ref :(
write(str, '(i0,a,i0,": ",a)') this_image(), '/', nprocs, 'hello world' ! :(
hi[this_image()] = str
end do
sync all
write(*,*) hi[this_image()]
end
EOF
$ gfortran -o hi -fcoarray=lib -lcaf_mpi ~/tmp/hello.f08 && mpirun -np 3 ./hi
1/3: hello world
2/3: hello world
3/3: hello world
$ ldd ./hi
linux-vdso.so.1 (0x00007ffd03986000)
libcaf_mpi.so.3 => /usr/lib/x86_64-linux-gnu/libcaf_mpi.so.3
(0x00007fe9e6a72000)
libgfortran.so.5 => /opt/x86_64/gcc-9.0.mine/lib64/libgfortran.so.5
(0x00007fe9e677a000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fe9e65e6000)
libgcc_s.so.1 => /opt/x86_64/gcc-9.0.mine/lib64/libgcc_s.so.1
(0x00007fe9e65c2000)
libquadmath.so.0 => /opt/x86_64/gcc-9.0.mine/lib64/libquadmath.so.0
(0x00007fe9e6576000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fe9e63b9000)
libmpi.so.40 => /usr/lib/x86_64-linux-gnu/libmpi.so.40 (0x00007fe9e62af000)
libmpi_usempif08.so.40 =>
/usr/lib/x86_64-linux-gnu/libmpi_usempif08.so.40 (0x00007fe9e6278000)
libmpi_usempi_ignore_tkr.so.40 =>
/usr/lib/x86_64-linux-gnu/libmpi_usempi_ignore_tkr.so.40
(0x00007fe9e626c000)
libmpi_mpifh.so.40 => /usr/lib/x86_64-linux-gnu/libmpi_mpifh.so.40
(0x00007fe9e620f000)
/lib64/ld-linux-x86-64.so.2 (0x00007fe9e6acb000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fe9e620a000)
libopen-rte.so.40 => /usr/lib/x86_64-linux-gnu/libopen-rte.so.40
(0x00007fe9e6150000)
libopen-pal.so.40 => /usr/lib/x86_64-linux-gnu/libopen-pal.so.40
(0x00007fe9e60a3000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fe9e6099000)
libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007fe9e6094000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fe9e5e76000)
libhwloc.so.5 => /usr/lib/x86_64-linux-gnu/libhwloc.so.5 (0x00007fe9e5e35000)
libevent-2.1.so.6 => /usr/lib/x86_64-linux-gnu/libevent-2.1.so.6
(0x00007fe9e5bdd000)
libevent_pthreads-2.1.so.6 =>
/usr/lib/x86_64-linux-gnu/libevent_pthreads-2.1.so.6
(0x00007fe9e59da000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fe9e59b9000)
libnuma.so.1 => /usr/lib/x86_64-linux-gnu/libnuma.so.1 (0x00007fe9e59ab000)
libltdl.so.7 => /usr/lib/x86_64-linux-gnu/libltdl.so.7 (0x00007fe9e59a0000)
$ dpkg -S /usr/lib/x86_64-linux-gnu/libcaf_mpi.so.3
libcaf-mpi3:amd64: /usr/lib/x86_64-linux-gnu/libcaf_mpi.so.3
$ dpkg -l libcaf-mpi3
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=====================-===============-===============-===============================================
ii libcaf-mpi3:amd64 2.2.0-1 amd64 Co-Array
Fortran libraries for gfortran

I'd check if you really got the debian files so you don't run afoul a
manual install or leftovers thereof.
If it's really unpleasant in testing, it will at least ripple down
from sid sometimes and hence should work without hazzle soonish :)

HTH and cheers,
Bernhard

[Alastair McKinstry]

Post by Toon Moene
...
Setting up libcaf-openmpi-3:amd64 (2.2.0-3) ...
Setting up libcoarrays-openmpi-dev:amd64 (2.2.0-3) ...
Processing triggers for libc-bin (2.27-6) ...
[ previously this led to apt errors, but not now. ]
and moved my own installation of the OpenCoarrays-2.2.0.tar.gz out of
drwxr-xr-x 6 toon toon 4096 Aug 10 16:01 OpenCoarrays-2.2.0.opzij
drwxr-xr-x 8 toon toon 4096 Sep 15 11:26 opencoarrays-build.opzij
drwxr-xr-x 6 toon toon 4096 Sep 15 11:26 opencoarrays.opzij
gfortran -g -fbacktrace -fcoarray=lib random-weather.f90
-L/usr/lib/x86_64-linux-gnu/open-coarrays/openmpi/lib -lcaf_mpi
[ Yes, the location of the libs is quite experimental, but OK for the
"Testing" variant of Debian ... ]
none -np 20 ./a.out
Decomposition information on image    7 is    4 *    5 slabs with   23
*   18 grid cells on this image.
Decomposition information on image    6 is    4 *    5 slabs with   23
*   18 grid cells on this image.
Decomposition information on image   11 is    4 *    5 slabs with   23
*   18 grid cells on this image.
Decomposition information on image   15 is    4 *    5 slabs with   23
*   18 grid cells on this image.
Decomposition information on image    1 is    4 *    5 slabs with   23
*   18 grid cells on this image.
Decomposition information on image   13 is    4 *    5 slabs with   23
*   18 grid cells on this image.
Decomposition information on image   12 is    4 *    5 slabs with   21
*   18 grid cells on this image.
Decomposition information on image   20 is    4 *    5 slabs with   21
*   18 grid cells on this image.
Decomposition information on image    9 is    4 *    5 slabs with   23
*   18 grid cells on this image.
Decomposition information on image   14 is    4 *    5 slabs with   23
*   18 grid cells on this image.
Decomposition information on image   16 is    4 *    5 slabs with   21
*   18 grid cells on this image.
Decomposition information on image   17 is    4 *    5 slabs with   23
*   18 grid cells on this image.
Decomposition information on image   18 is    4 *    5 slabs with   23
*   18 grid cells on this image.
Decomposition information on image    2 is    4 *    5 slabs with   23
*   18 grid cells on this image.
Decomposition information on image    4 is    4 *    5 slabs with   21
*   18 grid cells on this image.
Decomposition information on image    5 is    4 *    5 slabs with   23
*   18 grid cells on this image.
Decomposition information on image    3 is    4 *    5 slabs with   23
*   18 grid cells on this image.
Decomposition information on image    8 is    4 *    5 slabs with   21
*   18 grid cells on this image.
Decomposition information on image   10 is    4 *    5 slabs with   23
*   18 grid cells on this image.
Decomposition information on image   19 is    4 *    5 slabs with   23
*   18 grid cells on this image.
... etc. (see http://moene.org/~toon/random-weather.f90).
I presume other Linux distributors will follow shortly (this *is*
Debian Testing, which can be a bit testy at times - but I do trust my
main business at home on it for over 15 years now).
Kind regards,

Thanks, good to see it being tested (I'm the Debian/Ubuntu packager).

caf /cafrun has been dropped (for the moment ? ) in favour of mpirun,
but I've added pkg-config caf packages so that becomes an option.

$ pkg-config caf-mpich --libs

-L/usr/lib/x86_64-linux-gnu/open-coarrays/mpich/lib -lcaf_mpich -Wl,-z,relro -lmpich -lm -lbacktrace -lpthread -lrt

(My thinking is that for libraries in particular, the user need not know
whether CAF is being used, and if lib foobar uses CAF, then adding a:

    Requires: caf

into the pkg-config file gives you the correct linking transparently.

The "strange" paths are due to Debians multiarch : it is possible to
include libraries for multiple architectures simultaneously. This works
ok with pkg-config and cmake , etc (which allow you to set
PKG_CONFIG_PATH and have multiple pkgconfig files for different libs
simultaneously) , but currently break wrappers such as caf / cafrun.

I can add a new package for caf / cafrun but would rather not. (W e
currently don't do non-MPI CAF builds).

There is currently pkg-config files 'caf-mpich' and 'caf-openmpi' for
testing, and I'm adding a default alias caf -> caf-$(default-MPI)

regards

Alastair

[Richard Biener]

On Mon, Sep 24, 2018 at 12:58 PM Alastair McKinstry

Post by Alastair McKinstry

Post by Toon Moene
...
Setting up libcaf-openmpi-3:amd64 (2.2.0-3) ...
Setting up libcoarrays-openmpi-dev:amd64 (2.2.0-3) ...
Processing triggers for libc-bin (2.27-6) ...
[ previously this led to apt errors, but not now. ]
and moved my own installation of the OpenCoarrays-2.2.0.tar.gz out of
drwxr-xr-x 6 toon toon 4096 Aug 10 16:01 OpenCoarrays-2.2.0.opzij
drwxr-xr-x 8 toon toon 4096 Sep 15 11:26 opencoarrays-build.opzij
drwxr-xr-x 6 toon toon 4096 Sep 15 11:26 opencoarrays.opzij
gfortran -g -fbacktrace -fcoarray=lib random-weather.f90
-L/usr/lib/x86_64-linux-gnu/open-coarrays/openmpi/lib -lcaf_mpi
[ Yes, the location of the libs is quite experimental, but OK for the
"Testing" variant of Debian ... ]
none -np 20 ./a.out
Decomposition information on image 7 is 4 * 5 slabs with 23
* 18 grid cells on this image.
Decomposition information on image 6 is 4 * 5 slabs with 23
* 18 grid cells on this image.
Decomposition information on image 11 is 4 * 5 slabs with 23
* 18 grid cells on this image.
Decomposition information on image 15 is 4 * 5 slabs with 23
* 18 grid cells on this image.
Decomposition information on image 1 is 4 * 5 slabs with 23
* 18 grid cells on this image.
Decomposition information on image 13 is 4 * 5 slabs with 23
* 18 grid cells on this image.
Decomposition information on image 12 is 4 * 5 slabs with 21
* 18 grid cells on this image.
Decomposition information on image 20 is 4 * 5 slabs with 21
* 18 grid cells on this image.
Decomposition information on image 9 is 4 * 5 slabs with 23
* 18 grid cells on this image.
Decomposition information on image 14 is 4 * 5 slabs with 23
* 18 grid cells on this image.
Decomposition information on image 16 is 4 * 5 slabs with 21
* 18 grid cells on this image.
Decomposition information on image 17 is 4 * 5 slabs with 23
* 18 grid cells on this image.
Decomposition information on image 18 is 4 * 5 slabs with 23
* 18 grid cells on this image.
Decomposition information on image 2 is 4 * 5 slabs with 23
* 18 grid cells on this image.
Decomposition information on image 4 is 4 * 5 slabs with 21
* 18 grid cells on this image.
Decomposition information on image 5 is 4 * 5 slabs with 23
* 18 grid cells on this image.
Decomposition information on image 3 is 4 * 5 slabs with 23
* 18 grid cells on this image.
Decomposition information on image 8 is 4 * 5 slabs with 21
* 18 grid cells on this image.
Decomposition information on image 10 is 4 * 5 slabs with 23
* 18 grid cells on this image.
Decomposition information on image 19 is 4 * 5 slabs with 23
* 18 grid cells on this image.
... etc. (see http://moene.org/~toon/random-weather.f90).
I presume other Linux distributors will follow shortly (this *is*
Debian Testing, which can be a bit testy at times - but I do trust my
main business at home on it for over 15 years now).
Kind regards,

Thanks, good to see it being tested (I'm the Debian/Ubuntu packager).
caf /cafrun has been dropped (for the moment ? ) in favour of mpirun,
but I've added pkg-config caf packages so that becomes an option.
$ pkg-config caf-mpich --libs
-L/usr/lib/x86_64-linux-gnu/open-coarrays/mpich/lib -lcaf_mpich -Wl,-z,relro -lmpich -lm -lbacktrace -lpthread -lrt
(My thinking is that for libraries in particular, the user need not know
Requires: caf
into the pkg-config file gives you the correct linking transparently.
The "strange" paths are due to Debians multiarch : it is possible to
include libraries for multiple architectures simultaneously. This works
ok with pkg-config and cmake , etc (which allow you to set
PKG_CONFIG_PATH and have multiple pkgconfig files for different libs
simultaneously) , but currently break wrappers such as caf / cafrun.
I can add a new package for caf / cafrun but would rather not. (W e
currently don't do non-MPI CAF builds).
There is currently pkg-config files 'caf-mpich' and 'caf-openmpi' for
testing, and I'm adding a default alias caf -> caf-$(default-MPI)

So I've tried packaging of OpenCoarrays for SUSE and noticed a few things:

- caf by default links libcaf_mpi static (why?)
- the build system makes the libcaf_mpi SONAME dependent on the compiler
version(?), I once got libcaf_mpi2 and once libcaf_mpi3 (gcc7 vs. gcc8)

different SONAMEs definitely makes packaging difficult. Of course since
there's the first point I may very well elide the shared library
alltogether....?

Other than that it seems to "work" (OBS home:rguenther/OpenCoarrays).

Richard.

Post by Alastair McKinstry
regards
Alastair
--
Misentropy: doubting that the Universe is becoming more disordered.

[Jorge D'Elia]

----- Mensaje original -----

Enviados: Jueves, 27 de Septiembre 2018 9:28:47
Asunto: Re: OpenCoarrays integration with gfortran
On Mon, Sep 24, 2018 at 12:58 PM Alastair McKinstry

Post by Alastair McKinstry

Post by Toon Moene
...
Setting up libcaf-openmpi-3:amd64 (2.2.0-3) ...
Setting up libcoarrays-openmpi-dev:amd64 (2.2.0-3) ...
Processing triggers for libc-bin (2.27-6) ...
[ previously this led to apt errors, but not now. ]
and moved my own installation of the OpenCoarrays-2.2.0.tar.gz out of
drwxr-xr-x 6 toon toon 4096 Aug 10 16:01 OpenCoarrays-2.2.0.opzij
drwxr-xr-x 8 toon toon 4096 Sep 15 11:26 opencoarrays-build.opzij
drwxr-xr-x 6 toon toon 4096 Sep 15 11:26 opencoarrays.opzij
gfortran -g -fbacktrace -fcoarray=lib random-weather.f90
-L/usr/lib/x86_64-linux-gnu/open-coarrays/openmpi/lib -lcaf_mpi
[ Yes, the location of the libs is quite experimental, but OK for the
"Testing" variant of Debian ... ]
none -np 20 ./a.out
Decomposition information on image 7 is 4 * 5 slabs with 23
* 18 grid cells on this image.
Decomposition information on image 6 is 4 * 5 slabs with 23
* 18 grid cells on this image.
Decomposition information on image 11 is 4 * 5 slabs with 23
* 18 grid cells on this image.
Decomposition information on image 15 is 4 * 5 slabs with 23
* 18 grid cells on this image.
Decomposition information on image 1 is 4 * 5 slabs with 23
* 18 grid cells on this image.
Decomposition information on image 13 is 4 * 5 slabs with 23
* 18 grid cells on this image.
Decomposition information on image 12 is 4 * 5 slabs with 21
* 18 grid cells on this image.
Decomposition information on image 20 is 4 * 5 slabs with 21
* 18 grid cells on this image.
Decomposition information on image 9 is 4 * 5 slabs with 23
* 18 grid cells on this image.
Decomposition information on image 14 is 4 * 5 slabs with 23
* 18 grid cells on this image.
Decomposition information on image 16 is 4 * 5 slabs with 21
* 18 grid cells on this image.
Decomposition information on image 17 is 4 * 5 slabs with 23
* 18 grid cells on this image.
Decomposition information on image 18 is 4 * 5 slabs with 23
* 18 grid cells on this image.
Decomposition information on image 2 is 4 * 5 slabs with 23
* 18 grid cells on this image.
Decomposition information on image 4 is 4 * 5 slabs with 21
* 18 grid cells on this image.
Decomposition information on image 5 is 4 * 5 slabs with 23
* 18 grid cells on this image.
Decomposition information on image 3 is 4 * 5 slabs with 23
* 18 grid cells on this image.
Decomposition information on image 8 is 4 * 5 slabs with 21
* 18 grid cells on this image.
Decomposition information on image 10 is 4 * 5 slabs with 23
* 18 grid cells on this image.
Decomposition information on image 19 is 4 * 5 slabs with 23
* 18 grid cells on this image.
... etc. (see http://moene.org/~toon/random-weather.f90).
I presume other Linux distributors will follow shortly (this *is*
Debian Testing, which can be a bit testy at times - but I do trust my
main business at home on it for over 15 years now).
Kind regards,

Thanks, good to see it being tested (I'm the Debian/Ubuntu packager).
caf /cafrun has been dropped (for the moment ? ) in favour of mpirun,
but I've added pkg-config caf packages so that becomes an option.
$ pkg-config caf-mpich --libs
-L/usr/lib/x86_64-linux-gnu/open-coarrays/mpich/lib -lcaf_mpich -Wl,-z,relro
-lmpich -lm -lbacktrace -lpthread -lrt
(My thinking is that for libraries in particular, the user need not know
Requires: caf
into the pkg-config file gives you the correct linking transparently.
The "strange" paths are due to Debians multiarch : it is possible to
include libraries for multiple architectures simultaneously. This works
ok with pkg-config and cmake , etc (which allow you to set
PKG_CONFIG_PATH and have multiple pkgconfig files for different libs
simultaneously) , but currently break wrappers such as caf / cafrun.
I can add a new package for caf / cafrun but would rather not. (W e
currently don't do non-MPI CAF builds).
There is currently pkg-config files 'caf-mpich' and 'caf-openmpi' for
testing, and I'm adding a default alias caf -> caf-$(default-MPI)

- caf by default links libcaf_mpi static (why?)
- the build system makes the libcaf_mpi SONAME dependent on the compiler
version(?), I once got libcaf_mpi2 and once libcaf_mpi3 (gcc7 vs. gcc8)
different SONAMEs definitely makes packaging difficult. Of course since
there's the first point I may very well elide the shared library
alltogether....?
Other than that it seems to "work" (OBS home:rguenther/OpenCoarrays).
Richard.

The issue of the static libcaf_mpi reminded me of an old email
(from Lisandro Dalcin and me) about the coarray and Gfortran
integration strategies. I just hit one part below (possibly
something outdated). Maybe I can contribute something.

Regards,
Jorge.

#begin
About an MPI based library for the coarray model:

Nowadays, the coarray model is included in the lastest Fortran
2008 standard. However, since several Fortran compilers are built
on the basis of the C/C++ ones, there is not a native way to add
the coarray model.

One option is to build an MPI based library for supporting
the coarray model. In this case, it should be ensured that
the final library is as neutral as possible on regards to
a final user.

However, due to the fact that the MPI binaries are not yet
compatible among the available MPI implementations, this
issue should be overcomed somehow.

Among other possibilities, the following alternatives can be
considered, from the simplest to the more elaborated ones:

(1) A static library libcaf_mpi.a which uses a specific MPI
implementation when the library is built (e.g. OpenMPI or MPICH).
However, it only works with the distribution available when the
library is built.

(2) A dynamic library libcaf_mpi.so that uses a specific MPI
implementation when the library is built. However, again, it
only works with the MPI implementation available when the library
is built. From a practical point of view, this option is not very
different than the previous one.

(3) A symbolic link libcaf_mpi.so that points to dynamic libraries
libcaf_mpich.so or libcaf_openmpi.so. The sysadmin can manage the
link using system tools as alternatives. The GNU/Linux distributions
usually manage this infrastructure by themselves without additional
work required from the compiler side. However, regular (non-root)
users cannot switch the backend MPI. This is only available on POSIX
systems.

(4) Different dynamic libraries named libcaf_mpi.so built for each
MPI implementation, each of them installed in different directories, e.g.
/mpich/libcaf_mpi.so, or /openmpi/libcaf_mpi.so.
By using the modules tool, users can select the preferred MPI
implementation, e.g. module load mpich2-x86_64 in Fedora.
This works by adding entries in the LD_LIBRARY_PATH environment variable.
The GNU/Linux distributions usually manage this infrastructure by
themselves and do not require additional work from the Fortran compiler
side. Regular users are able to choose the preferred MPI implementation,
and the dynamic linker loads the appropriate libcaf_mpi.so.

(5) A dynamic library libcaf_mpi.so built by the dlopen() tool. This
option could be practical if the number of MPI functions were not too
large, and it can be built on both GNU/Linux and Windows OS.

(6) A dynamic library libcaf_mpi.so that is not linked with the MPI
library, but uses the dlopen() and dlsym() to load and access the
contents of a specific "MPI-linked" dynamic library, e.g.
libcaf_mpi_mpich2-openmpi-other.so. In this way, libcaf_mpi.so does
not depend on any MPI implementation, but acts as a thin wrapper to
the specific CAF+MPI library. By using environment variables or rc
configuration files, the user can choose the preferred library to
open at runtime with dlopen() in POSIX systems, or similar mechanisms
on Windows OS.
#end

[Janne Blomqvist]

Post by Toon Moene

Post by Janne Blomqvist
The argument must be of type size_type_node, not sizetype. Please instead
use
size = build_zero_cst (size_type_node);

Post by Toon Moene
* trans-intrinsic.c (conv_intrinsic_event_query): Convert

computed

Post by Janne Blomqvist

Post by Toon Moene
index to a size_t type.

Using integer_type_node is wrong, but the correct type for calculating
array indices (lbound, ubound, etc.) is not size_type_node but rather
gfc_array_index_type (which in practice maps to ptrdiff_t). So please use
that, and then fold_convert index to size_type_node just before

generating

Post by Janne Blomqvist
the call to event_query.

Post by Toon Moene
* trans-stmt.c (gfc_trans_event_post_wait): Likewise.

Same here as above.

How is the attached? I retested and found no regressions.
Andrew

Ok, looks good.

There are some other remaining incorrect uses of integer_type_node (at
least one visible in the diff), but that can be done as a separate patch
(not saying you must do it as a precondition for anything, though it would
of course be nice if you would. :) )

[Andrew Stubbs]

Post by Janne Blomqvist
Ok, looks good.

Thanks.

Post by Janne Blomqvist
There are some other remaining incorrect uses of integer_type_node (at
least one visible in the diff), but that can be done as a separate patch
(not saying you must do it as a precondition for anything, though it
would of course be nice if you would. :) )

I'm not confident I can tell what should be integer_type_node, and what
should not?

Once it gets to build_call_expr_loc it's clear that the types should
match the function signature, but the intermediate values' types are not
obvious to me.

Andrew