Quantcast

The freeplane-osgi-gradle crisis

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

The freeplane-osgi-gradle crisis

Felix Natter
Administrator
This post was updated on .
Dear devs,

I'd like to discuss some things with you.
Please don't "tl;dr" this ;-)

=== Current freeplane-gradle-osgi structure ===

Freeplane's (OSGi-)structure is simple:

- the 'freeplane' (core) project (OSGi plugin) contains core code as
  well as dependencies like commons-lang, commons-io, flamingo etc.

- the plugin projects ('freeplane_plugin_latex', 'freeplane_plugin_script',
  ...) make use of some core classes *and* third-party dependencies
  like commons-lang, commons-io, flamingo, ...

In order not to duplicate third-party-deps in all plugins [1], the old ant
code used to Require-Bundle: the core project in all plugin projects.

Since Require-Bundle: is not recommended practice any more (and
thus gradle-osgi doesn't support this), we decided to Export-Package: the
third-party-deps in the core project and Import-Package: them in the
plugin projects:

// core project MANIFEST.MF
Export-Package: [...]
 org.pushingpixels.flamingo.api.common,
 org.pushingpixels.flamingo.api.common.icon,
 org.pushingpixels.flamingo.api.common.model,
 org.pushingpixels.flamingo.api.common.popup,
 org.pushingpixels.flamingo.api.ribbon

// plugin project MANIFEST.MF (i.e. freeplane_plugin_script)
Import-Package: [...]
 org.pushingpixels.flamingo.api.common,
 org.pushingpixels.flamingo.api.common.icon,
 org.pushingpixels.flamingo.api.common.model,
 org.pushingpixels.flamingo.api.common.popup,
 org.pushingpixels.flamingo.api.ribbon

The current deployed structure of freeplane looks like this
(third-party-deps are only present where they are needed):

./core/org.freeplane.core
./core/org.freeplane.core/lib
./core/org.freeplane.core/lib/substance-7.2.1.jar
./core/org.freeplane.core/lib/flamingo-7.2.1.jar
./core/org.freeplane.core/lib/substance-swingx-7.2.1.jar
./core/org.freeplane.core/lib/swingx-action-1.6.3.jar
./core/org.freeplane.core/lib/commons-io-2.4.jar
[...]
./core/org.freeplane.core/META-INF
./core/org.freeplane.core/META-INF/MANIFEST.MF
[...]
./plugins
./plugins/org.freeplane.plugin.latex
./plugins/org.freeplane.plugin.latex/lib
./plugins/org.freeplane.plugin.latex/lib/plugin-1.4.1.jar
./plugins/org.freeplane.plugin.latex/lib/jlatexmath-1.0.2.jar
./plugins/org.freeplane.plugin.latex/META-INF
./plugins/org.freeplane.plugin.latex/META-INF/MANIFEST.MF
[...]
./plugins/org.freeplane.plugin.openmaps
./plugins/org.freeplane.plugin.openmaps/lib
./plugins/org.freeplane.plugin.openmaps/lib/plugin-1.4.1.jar
./plugins/org.freeplane.plugin.openmaps/lib/JMapViewer.jar
./plugins/org.freeplane.plugin.openmaps/lib/JMapViewer_src.jar
./plugins/org.freeplane.plugin.openmaps/META-INF
./plugins/org.freeplane.plugin.openmaps/META-INF/MANIFEST.MF
[...]

=== The problem ===

When we build Freeplane using gradle outside of eclipse, this works when
bypassing 'bnd' (a tool that helps generating OSGi MANIFESTs), details
are in [2], BUT there are problems when using the (modified) MANIFESTs
for starting from eclipse (eclipse's OSGi tool, PDE, won't let you
Export i.e. "org.pushingpixels.flamingo.api.common" and thus the
corresponding Import-Package in freeplane_plugin_script cannot be
resolved...).

=== Possible solutions ===

(a) on osgi-dev I was recommended to use bnd-gradle (an alternative
  to the builtin gradle-osgi plugin) and bndtools (an eclipse-OSGi
  plugin which replaces eclipe's PDE)
  --> problems: no obvious solution for IDEA (bndtools doesn't exist for
      IDEA) and needs rewrite of larger parts of freeplane-gradle
(https://mail.osgi.org/pipermail/osgi-dev/2015-January/004715.html
+ a private mail)

(b) get things to work by filtering eclipse MANIFEST.MFs and/or
configuring eclipse-osgi
(we already have a filter that replaces lib/foo.jar with
build/libs/foo.jar for eclipse manifests)
--> can anybody please look into this (instructions are below!)?
EDIT: one solution could be to filter the MANIFEST.MFs for eclipse
such that Require-Bundle: is used for eclipse (only).

(c) Tell gradle-osgi to put dependencies into MANIFEST.MFs and lib/
_transitively_:
./core/org.freeplane.core
./core/org.freeplane.core/lib
./core/org.freeplane.core/lib/substance-7.2.1.jar
./core/org.freeplane.core/lib/flamingo-7.2.1.jar
./core/org.freeplane.core/lib/substance-swingx-7.2.1.jar
./core/org.freeplane.core/lib/swingx-action-1.6.3.jar
./core/org.freeplane.core/lib/commons-io-2.4.jar
[...]
./core/org.freeplane.core/META-INF
./core/org.freeplane.core/META-INF/MANIFEST.MF
[...]
./plugins
./plugins/org.freeplane.plugin.latex
./plugins/org.freeplane.plugin.latex/lib
./plugins/org.freeplane.plugin.latex/lib/plugin-1.4.1.jar
./plugins/org.freeplane.plugin.latex/lib/jlatexmath-1.0.2.jar
./plugins/org.freeplane.plugin.latex/lib/substance-7.2.1.jar
./plugins/org.freeplane.plugin.latex/lib/flamingo-7.2.1.jar
./plugins/org.freeplane.plugin.latex/lib/substance-swingx-7.2.1.jar
./plugins/org.freeplane.plugin.latex/lib/swingx-action-1.6.3.jar
./plugins/org.freeplane.plugin.latex/lib/commons-io-2.4.jar
[...]
./plugins/org.freeplane.plugin.latex/META-INF
./plugins/org.freeplane.plugin.latex/META-INF/MANIFEST.MF
[...]
(and so on for the other plugins)

--> we would need to filter the duplicate jars when creating
distributions. In gradle this is simple:
  configurations.runtime.transitive = true
but it's not very elegant!

(d) Edit MANIFEST.MF files manually, commit them to git and use
Require-Bundle: instead of Import-Package:/Export-Package:.
--> should work because it works currently
--> totally against OSGi best practice (Require-Bundle:) and
    gradle-osgi best practice (hardcoded MANIFEST.MFs)...

There are probably more solutions...

=== Instructions for playing around ===

If you have time to play around with this,
here are simple instructions to make it run:
- mkdir workspaces/freeplane_gradle
- cd workspaces/freeplane_gradle
- git clone https://github.com/freeplane/freeplane.git
- cd freeplane
- git checkout Gradle_builds
- gradle clean build cleanEclipse eclipse
- start eclipse (I use luna) on workspaces/freeplane_gradle
  - File->Import->Existing projects into workspace
    (root = .../workspaces/freeplane_gradle/freeplane)
  - select all, hit finish
  - select all OSGI projects (freeplane, freeplane_plugin_*),
    right-click->configure->convert to plugin projects
  => 5 errors, freeplane/META-INF/MANIFEST.MF for some reason does not
     export org.pushingpixels, and freeplane_plugin_script wants
     to import this...
--> Maybe you find solution that is better than filtering the
MANIFEST.MFs for eclipse (or IDEA) further.

--> Please tell me if you have any problems with this!

[1] http://forums.gradle.org/gradle/topics/gradle-osgi-question-export-local-third-party-dependencies
[2] http://article.gmane.org/gmane.comp.ide.eclipse.pde.devel/1830
(unfortunately no answer yet...)

Thanks, Cheers and Best Regards,
--
Felix Natter
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: The freeplane-osgi-gradle crisis

Volker Börchers
Administrator
Hi Felix,

it's great news that you and Blair have managed to get Freeplane built
with Gradle - a big Thank you for that! Unfortunately I don't have time
to help but would like to share some thoughts about the questions you
have raised.

- I wouldn't choose any solution that only works for Eclipse since the
support for Idea was one of the unique selling points for the project.
(Not that there is something to be sold but I find Idea support very
attractive...)

- If duplicating the dependencies would be the simplest solution why
don't you choose it? What's the actual price? 1 MB, 2 MB, ...?

- If I understand correctly we currently Require-Bundle and it works for
us and it will almost certainly continue to work for us for all times.
The use of a deprecated feature is definitely not nice and the missing
support by Gradle is nasty but I wouldn't care if you get it to work
with any of the solutions you proposed.

- If it helps to manually write the MANIFEST files - do that! We have
them in Git already - who cares?? (Do I miss something here?)

- If you choose a workaround then I would rather choose one that might
potentially be fixed by improvements in Gradle. (Isn't extensibility one
of Gradle's biggest advantages?)

Best regards,
Volker

Am 18.01.2015 um 19:00 schrieb Felix Natter [via Freeplane Developer]:

> Dear devs,
>
> I'd like to discuss some things with you.
> Please don't "tl;dr" this ;-)
>
> === Current freeplane-gradle-osgi structure ===
>
> Freeplane's (OSGi-)structure is simple:
>
> - the 'freeplane' (core) project (OSGi plugin) contains core code as
>    well as dependencies like commons-lang, commons-io, flamingo etc.
>
> - the plugin projects ('freeplane_plugin_latex', 'freeplane_plugin_script',
>    ...) make use of some core classes *and* third-party dependencies
>    like commons-lang, commons-io, flamingo, ...
>
> In order not to duplicate third-party-deps in all plugins [1], the old ant
> code used to Require-Bundle: the core project in all plugin projects.
>
> Since Require-Bundle: is not recommended practice any more (and
> thus gradle-osgi doesn't support this), we decided to Export-Package: the
> third-party-deps in the core project and Import-Package: them in the
> plugin projects:
>
> // core project MANIFEST.MF
> Export-Package: [...]
>   org.pushingpixels.flamingo.api.common,
>   org.pushingpixels.flamingo.api.common.icon,
>   org.pushingpixels.flamingo.api.common.model,
>   org.pushingpixels.flamingo.api.common.popup,
>   org.pushingpixels.flamingo.api.ribbon
>
> // plugin project MANIFEST.MF (i.e. freeplane_plugin_script)
> Import-Package: [...]
>   org.pushingpixels.flamingo.api.common,
>   org.pushingpixels.flamingo.api.common.icon,
>   org.pushingpixels.flamingo.api.common.model,
>   org.pushingpixels.flamingo.api.common.popup,
>   org.pushingpixels.flamingo.api.ribbon
>
> The current deployed structure of freeplane looks like this
> (third-party-deps are only present where they are needed):
>
> ./core/org.freeplane.core
> ./core/org.freeplane.core/lib
> ./core/org.freeplane.core/lib/substance-7.2.1.jar
> ./core/org.freeplane.core/lib/flamingo-7.2.1.jar
> ./core/org.freeplane.core/lib/substance-swingx-7.2.1.jar
> ./core/org.freeplane.core/lib/swingx-action-1.6.3.jar
> ./core/org.freeplane.core/lib/commons-io-2.4.jar
> [...]
> ./core/org.freeplane.core/META-INF
> ./core/org.freeplane.core/META-INF/MANIFEST.MF
> [...]
> ./plugins
> ./plugins/org.freeplane.plugin.latex
> ./plugins/org.freeplane.plugin.latex/lib
> ./plugins/org.freeplane.plugin.latex/lib/plugin-1.4.1.jar
> ./plugins/org.freeplane.plugin.latex/lib/jlatexmath-1.0.2.jar
> ./plugins/org.freeplane.plugin.latex/META-INF
> ./plugins/org.freeplane.plugin.latex/META-INF/MANIFEST.MF
> [...]
> ./plugins/org.freeplane.plugin.openmaps
> ./plugins/org.freeplane.plugin.openmaps/lib
> ./plugins/org.freeplane.plugin.openmaps/lib/plugin-1.4.1.jar
> ./plugins/org.freeplane.plugin.openmaps/lib/JMapViewer.jar
> ./plugins/org.freeplane.plugin.openmaps/lib/JMapViewer_src.jar
> ./plugins/org.freeplane.plugin.openmaps/META-INF
> ./plugins/org.freeplane.plugin.openmaps/META-INF/MANIFEST.MF
> [...]
>
> === The problem ===
>
> When we build Freeplane using gradle outside of eclipse, this works when
> bypassing 'bnd' (a tool that helps generating OSGi MANIFESTs), details
> are in [2], BUT there are problems when using the (modified) MANIFESTs
> for starting from eclipse (eclipse's OSGi tool, PDE, won't let you
> Export i.e. "org.pushingpixels.flamingo.api.common" and thus the
> corresponding Import-Package in freeplane_plugin_script cannot be
> resolved...).
>
> === Possible solutions ===
>
> (a) on osgi-dev I was recommended to use bnd-gradle (an alternative
>    to the builtin gradle-osgi plugin) and bndtools (an eclipse-OSGi
>    plugin which replaces eclipe's PDE)
>    --> problems: no obvious solution for IDEA (bndtools doesn't exist for
>        IDEA) and needs rewrite of larger parts of freeplane-gradle
> (https://mail.osgi.org/pipermail/osgi-dev/2015-January/004715.html
> + a private mail)
>
> (b) get things to work by filtering eclipse MANIFEST.MFs and/or
> configuring eclipse-osgi
> (we already have a filter that replaces lib/foo.jar with
> build/libs/foo.jar for eclipse manifests)
> --> can anybody please look into this (instructions are below!)?
>
> (c) Tell gradle-osgi to put dependencies into MANIFEST.MFs and lib/
> _transitively_:
> ./core/org.freeplane.core
> ./core/org.freeplane.core/lib
> ./core/org.freeplane.core/lib/substance-7.2.1.jar
> ./core/org.freeplane.core/lib/flamingo-7.2.1.jar
> ./core/org.freeplane.core/lib/substance-swingx-7.2.1.jar
> ./core/org.freeplane.core/lib/swingx-action-1.6.3.jar
> ./core/org.freeplane.core/lib/commons-io-2.4.jar
> [...]
> ./core/org.freeplane.core/META-INF
> ./core/org.freeplane.core/META-INF/MANIFEST.MF
> [...]
> ./plugins
> ./plugins/org.freeplane.plugin.latex
> ./plugins/org.freeplane.plugin.latex/lib
> ./plugins/org.freeplane.plugin.latex/lib/plugin-1.4.1.jar
> ./plugins/org.freeplane.plugin.latex/lib/jlatexmath-1.0.2.jar
> ./plugins/org.freeplane.plugin.latex/lib/substance-7.2.1.jar
> ./plugins/org.freeplane.plugin.latex/lib/flamingo-7.2.1.jar
> ./plugins/org.freeplane.plugin.latex/lib/substance-swingx-7.2.1.jar
> ./plugins/org.freeplane.plugin.latex/lib/swingx-action-1.6.3.jar
> ./plugins/org.freeplane.plugin.latex/lib/commons-io-2.4.jar
> [...]
> ./plugins/org.freeplane.plugin.latex/META-INF
> ./plugins/org.freeplane.plugin.latex/META-INF/MANIFEST.MF
> [...]
> (and so on for the other plugins)
>
> --> we would need to filter the duplicate jars when creating
> distributions. In gradle this is simple:
>    configurations.runtime.transitive = true
> but it's not very elegant!
>
> (d) Edit MANIFEST.MF files manually, commit them to git and use
> Require-Bundle: instead of Import-Package:/Export-Package:.
> --> should work because it works currently
> --> totally against OSGi best practice (Require-Bundle:) and
>      gradle-osgi best practice (hardcoded MANIFEST.MFs)...
>
> There are probably more solutions...
>
> === Instructions for playing around ===
>
> If you have time to play around with this,
> here are simple instructions to make it run:
> - mkdir workspaces/freeplane_gradle
> - cd workspaces/freeplane_gradle
> - git clone https://github.com/freeplane/freeplane.git
> - cd freeplane
> - git checkout Gradle_builds
> - gradle clean build cleanEclipse eclipse
> - start eclipse (I use luna) on workspaces/freeplane_gradle
>    - File->Import->Existing projects into workspace
>      (root = .../workspaces/freeplane_gradle/freeplane)
>    - select all, hit finish
>    - select all OSGI projects (freeplane, freeplane_plugin_*),
>      right-click->configure->convert to plugin projects
>    => 5 errors, freeplane/META-INF/MANIFEST.MF for some reason does not
>       export org.pushingpixels, and freeplane_plugin_script wants
>       to import this...
> --> Maybe you find solution that is better than filtering the
> MANIFEST.MFs for eclipse (or IDEA) further.
>
> --> Please tell me if you have any problems with this!
>
> [1]
> http://forums.gradle.org/gradle/topics/gradle-osgi-question-export-local-third-party-dependencies
> [2] http://article.gmane.org/gmane.comp.ide.eclipse.pde.devel/1830
> (unfortunately no answer yet...)
>
> Thanks, Cheers and Best Regards,
> --
> Felix Natter
>
>
> ------------------------------------------------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://freeplane-developer.996965.n3.nabble.com/The-freeplane-osgi-gradle-crisis-tp596.html
>
> To start a new topic under Freeplane Developer, email
> [hidden email]
> To unsubscribe from Freeplane Developer, click here
> <
> NAML
> <
http://freeplane-developer.996965.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: The freeplane-osgi-gradle crisis

Felix Natter
Administrator
"Volker Börchers [via Freeplane Developer]"
<[hidden email]> writes:

> Hi Felix,

hello all,

Dimitry found out the reason why :freeplane refused to Export-Package:
third-party libraries:

        <classpathentry exported="true" kind="lib" path="/home/felix/.gradle/caches/artifacts-23/filestore/com.github.insubstantial/flamingo/7.2.1/jar/d1336d6861952bf85190c61d376fbe23f0cb243f/flamingo-7.2.1.jar" sourcepath="/home/felix/.gradle/caches/artifacts-23/filestore/com.github.insubstantial/flamingo/7.2.1/source/332fc797bbf79bffe28586c5f02870389db8e9eb/flamingo-7.2.1-sources.jar"/>

=> does not work because it's outside freeplane/ and it seems eclipse
considers it to be an _external_ dependency.

<classpathentry exported="true" kind="lib" path="build/libs/flamingo-7.2.1.jar"/>

=> works OK.

Many thanks at this point, yay!

It works when applied manually, but a lot of things still have to be
fixed:

- modify the gradle configurations.runtime.files for the eclipse plugin
- still pull the sources from ~/.gradle/caches/...
- jsyntaxpane does not work
- ...

> it's great news that you and Blair have managed to get Freeplane built
> with Gradle - a big Thank you for that! Unfortunately I don't have time
> to help but would like to share some thoughts about the questions you
> have raised.
>
> - I wouldn't choose any solution that only works for Eclipse since the
> support for Idea was one of the unique selling points for the project.
> (Not that there is something to be sold but I find Idea support very
> attractive...)

I agree it's better to support IDEA as well - in our current solution it
should work in theory (but there will be problems in real life like
we now get for eclipse...)

> - If duplicating the dependencies would be the simplest solution why
> don't you choose it? What's the actual price? 1 MB, 2 MB, ...?

org.freeplane.core contains 15M of libraries ;-)
(these can be filtered before creating a distribution, but that is ugly)

Anyway, Dimitry commented that there can be problems when classes are
loaded more than once for core/plugins, so it's not an option anyway.

> - If I understand correctly we currently Require-Bundle and it works for
> us and it will almost certainly continue to work for us for all times.
> The use of a deprecated feature is definitely not nice and the missing
> support by Gradle is nasty but I wouldn't care if you get it to work
> with any of the solutions you proposed.

We found out that Require-Bundle: also does not work in case of
"external" jars.

> - If it helps to manually write the MANIFEST files - do that! We have
> them in Git already - who cares?? (Do I miss something here?)

Fortunately we don't have to go that route with Dimitry's fix.

> - If you choose a workaround then I would rather choose one that might
> potentially be fixed by improvements in Gradle. (Isn't extensibility one
> of Gradle's biggest advantages?)

True, but when facing the problems with gradle-osgi and eclipse (also
see [1]), I sometimes doubt that.

[1] http://forums.gradle.org/gradle/topics/the_osgi_plugin_has_several_flaws

Best Regards,
--
Felix Natter
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: The freeplane-osgi-gradle crisis

Dimitry Polivaev
Administrator
In reply to this post by Volker Börchers
Hi,

even with relatively well known Eclipse we have got some troubles trying to create and configure our
OSGi projects. As Felix already mentioned we figured out that we need to copy the library
dependencies directly into the project in order so that they can be found from the manifest.

I have also found that alternatively we could automatically wrap dependency libraries as OSGi
plug-ins from gradle using e.g. https://github.com/stempler/bnd-platform or
https://github.com/akhikhl/wuff . Such OSGi plug-ins could be imported into eclipse workspace using
File->Import...->Plug-in Development->Plug-ins and Fragments wizard. So Felix and Blair can check if
they like this way which allows to separate all dependencies more than copying the libraries
directly into the project. Personally I think that this approach is better because it allows to
separate Freeplane from the other code in the most clear way.

In any case I think that if we want to support IntelliJ we need a volunteer who should try to create
IntelliJ projects without using Gradle just to see how all this would fit there. Without such check
we can not know for sure which of the above solutions could work with IntelliJ and which probably not.

Regards,
Dimitry

> - I wouldn't choose any solution that only works for Eclipse since the
> support for Idea was one of the unique selling points for the project.
> (Not that there is something to be sold but I find Idea support very
> attractive...)

> - If duplicating the dependencies would be the simplest solution why
> don't you choose it? What's the actual price? 1 MB, 2 MB, ...?
>
> - If I understand correctly we currently Require-Bundle and it works for
> us and it will almost certainly continue to work for us for all times.
> The use of a deprecated feature is definitely not nice and the missing
> support by Gradle is nasty but I wouldn't care if you get it to work
> with any of the solutions you proposed.
>
> - If it helps to manually write the MANIFEST files - do that! We have
> them in Git already - who cares?? (Do I miss something here?)
>
> - If you choose a workaround then I would rather choose one that might
> potentially be fixed by improvements in Gradle. (Isn't extensibility one
> of Gradle's biggest advantages?)
>
> Best regards,
> Volker

Loading...