Discussion:
[Freecol-developers] FeatureContainer
Enrico Weigelt, metux IT consult
2017-01-24 03:03:04 UTC
Permalink
Hi folks,


could any please give me some insight into the purpose of
FeatureContainer vs. similar functions (eg. hasAbility()) and
getAbilities()) in other objects (eg. Colony) ?

Should we perhaps consolidate all the ability handling inside
FeatureContainer and let FreeColObject just call it ?


--mtx
Michael T. Pope
2017-01-24 04:16:40 UTC
Permalink
On Tue, 24 Jan 2017 04:03:04 +0100
Post by Enrico Weigelt, metux IT consult
could any please give me some insight into the purpose of
FeatureContainer vs. similar functions (eg. hasAbility()) and
getAbilities()) in other objects (eg. Colony) ?
FeatureContainer is what the name implies, a container for Features (i.e.
Ability and Modifier). Many classes do not need one, so we do not add it
by default to FreeColObject. The classes that can use a simple container
implement getFeatureContainer (e.g. Player). However some classes (e.g.
Unit) are too complex for a simple container and override
get{Abilities,Modifiers} which the other feature-related routines
depend on.
Post by Enrico Weigelt, metux IT consult
Should we perhaps consolidate all the ability handling inside
FeatureContainer and let FreeColObject just call it ?
I am not sure what you intend here that is different from the current
implementation. Beware though that this is an area where it is easy to
break things.

Also, I have reviewed the first 50 or so of your patches. I do not
understand your objection to Streams. Nearly all the patches removing
Streams use bloat the code. Please do not do that *unless* you can
demonstrate a real benefit, and I think you will have to look hard if you
think that performance will significantly improved. I checked my
test results, and over the period from mid-2015 to early 2016 when I
added most of the Streams code, the average turn speed of my overnight
tests (all-AI simulations with no waiting for interactive users) decreased
by about 2%, which is within the bounds of error. This is obviously not
definitive as a lot of other code went in at the same time, but certainly
suggestive that Streams have negligible impact, and even if Streams are
fully to blame for the change, I am *not* worried that it now takes 7 AI
players 5.2s to move rather than 5.1s.

Cheers,
Mike Pope
Enrico Weigelt, metux IT consult
2017-01-24 10:25:28 UTC
Permalink
On 24.01.2017 05:16, Michael T. Pope wrote:

Hi,
Post by Michael T. Pope
Post by Enrico Weigelt, metux IT consult
Should we perhaps consolidate all the ability handling inside
FeatureContainer and let FreeColObject just call it ?
I am not sure what you intend here that is different from the current
implementation.
Better performance. I've seen that abilities are checked a lot, and
the current implementation isn't particularily efficient. In
FeatureContainer I've already optimized hasAbility() to do direct
lookup in the set(s) and bail out as fast as possible (w/o going
through getAbilities()). The downside is that I've currently have
a bit code duplication between the single-set and all-sets case
(id==null vs id!=null).

Looking at Colony, we're essentially combining two sets here (the colony
itself and the owner) - in both an ability can be absent or disabled.
To optimize it in a similar way as I've done in FeatureContainer, as
well as reducing the code in FeatureContainer, I'm thinkng of moving
out the single-set test with a 3-state result (yes,no,missing), so
the individual checks can be easily chained together.
Post by Michael T. Pope
Also, I have reviewed the first 50 or so of your patches. I do not
understand your objection to Streams.
It is sloooow and memory intensive. Just look at the jdk source, what
happens behind the scenes. Their streams api is bloaty and slow by
design, they

And it requires jdk8, which I have on none of my production machines,
running stable/lts distros (and properly packaging that monster is a
magnitudes larger job).

In the long run, I'd also like to get it running w/ AOT (to get the
real performance boost), as well as Dalvik.
Post by Michael T. Pope
Nearly all the patches removing Streams use bloat the code.
Note quite, a few of them do (yes, eg. finding list elements by min or
max on some attribute could be written more elegant.

OTOH, eg. replacing

| if (any(buildableType.getRequiredAbilities().entrySet(),
| e -> e.getValue() != hasAbility(e.getKey()))) {
| return NoBuildReason.MISSING_ABILITY;
| }

by

| for (Entry<String,Boolean> e :
buildableType.getRequiredAbilities().entrySet())
| if (e.getValue() != hasAbility(e.getKey()))
| return NoBuildReason.MISSING_ABILITY;

Doesn't really bloat it up (except a few more keystrokes for writing
down the actual types - but then the human reader doesn't have to
guess anymore), but it skips the function object instance and virtual
invocations per element.

Or even this:

| forEachMapEntry(buildingType.getRequiredAbilities(),
| e -> appendRequiredAbility(doc, e.getKey(), e.getValue()));

vs this:

| for (Map.Entry<String,Boolean> e :
buildingType.getRequiredAbilities().entrySet())
| appendRequiredAbility(doc, e.getKey(), e.getValue());

The old version is just a few chars longer (and more expressive, as it
actually shows the types involved), but way more expensive.
Post by Michael T. Pope
Please do not do that *unless* you can
demonstrate a real benefit, and I think you will have to look hard if you
think that performance will significantly improved. I checked my
test results, and over the period from mid-2015 to early 2016 when I
added most of the Streams code, the average turn speed of my overnight
tests (all-AI simulations with no waiting for interactive users) decreased
by about 2%, which is within the bounds of error.
Maybe you've just replaced even more inefficient code. On my site, my
branch ontop of mainline is notably faster than the version coming on
precise -- *without* any streams, on jre7.
Post by Michael T. Pope
This is obviously not
definitive as a lot of other code went in at the same time, but certainly
suggestive that Streams have negligible impact, and even if Streams are
fully to blame for the change, I am *not* worried that it now takes 7 AI
players 5.2s to move rather than 5.1s.
On my site it's magnitudes smaller. Haven't looked at the whole AI area
yet, my next 2do is the startup, which still is uncomfortably slow ...


--mtx
Michael T. Pope
2017-01-24 11:12:34 UTC
Permalink
On Tue, 24 Jan 2017 11:25:28 +0100
Post by Enrico Weigelt, metux IT consult
Post by Michael T. Pope
I am not sure what you intend here that is different from the current
implementation.
Better performance.
Always the performance... for the nth time, show me that it is a problem!
Post by Enrico Weigelt, metux IT consult
I've seen that abilities are checked a lot, and
the current implementation isn't particularily efficient. In
FeatureContainer I've already optimized hasAbility() to do direct
lookup in the set(s) and bail out as fast as possible (w/o going
through getAbilities()).
Which will break any class (e.g. Unit) that needs special handling and
relies on hasAbility calling the overridden getAbilities.
Post by Enrico Weigelt, metux IT consult
Post by Michael T. Pope
Also, I have reviewed the first 50 or so of your patches. I do not
understand your objection to Streams.
It is sloooow and memory intensive.
Your evidence remains absent.
Post by Enrico Weigelt, metux IT consult
Just look at the jdk source, what
happens behind the scenes. Their streams api is bloaty and slow by
design, they
I do not doubt that the implementation is krufty. Its new code after all.
But, once again, and I am getting sick of repeating this, show me where
FreeCol is being hurt by this. I have some rough numbers. Your turn.
Post by Enrico Weigelt, metux IT consult
And it requires jdk8, which I have on none of my production machines,
running stable/lts distros (and properly packaging that monster is a
magnitudes larger job).
Sorry. Java8 is not negotiable. Java7 is no longer getting security
updates.
Post by Enrico Weigelt, metux IT consult
In the long run, I'd also like to get it running w/ AOT (to get the
real performance boost), as well as Dalvik.
Again the performance... However I admit porting is interesting.
Post by Enrico Weigelt, metux IT consult
Post by Michael T. Pope
Nearly all the patches removing Streams use bloat the code.
Note quite, a few of them do (yes, eg. finding list elements by min or
max on some attribute could be written more elegant.
I think you will find the majority of them do, although you are correct
that many are fairly minor. Nevertheless, there are also many that are
significantly bloating. If you want to claim your code is "more
expressive" I would expect it to be shorter. It is not. It *is* more
"explicit" in that you are correct that the types are more obvious, but I
do not consider that an advantage over being succinct.
Post by Enrico Weigelt, metux IT consult
Post by Michael T. Pope
[Replacing]
Maybe you've just replaced even more inefficient code.
Not by much I am sure. Mostly your patches just put things back to
something very similar to what was there. Check the git record if you
like.
Post by Enrico Weigelt, metux IT consult
Post by Michael T. Pope
This is obviously not
definitive as a lot of other code went in at the same time, but certainly
suggestive that Streams have negligible impact, and even if Streams are
fully to blame for the change, I am *not* worried that it now takes 7 AI
players 5.2s to move rather than 5.1s.
On my site it's magnitudes smaller. Haven't looked at the whole AI area
yet, my next 2do is the startup, which still is uncomfortably slow ...
So *what* exactly is "magnitudes smaller"? I keep asking you to
substantiate your claims that performance both better and important.
Please explain precisely what you are measuring and why it matters.

Good luck with the startup. However I expect you will find that most of
the time is I/O, sending the XML for the Game to the client.

Cheers,
Mike Pope
Enrico Weigelt, metux IT consult
2017-01-24 17:10:19 UTC
Permalink
Post by Michael T. Pope
Always the performance... for the nth time, show me that it is a problem!
Well, eg. up to several seconds delay on map move, etc actually is a
problem - at least for me ...
Post by Michael T. Pope
Post by Enrico Weigelt, metux IT consult
I've seen that abilities are checked a lot, and
the current implementation isn't particularily efficient. In
FeatureContainer I've already optimized hasAbility() to do direct
lookup in the set(s) and bail out as fast as possible (w/o going
through getAbilities()).
Which will break any class (e.g. Unit) that needs special handling and
relies on hasAbility calling the overridden getAbilities.
Clarification: I was just talking about FeatureContainer - haven't seen
a derived class of that yet. fco+childs are a different issue (I'm
not there yet).
Post by Michael T. Pope
But, once again, and I am getting sick of repeating this, show me where
FreeCol is being hurt by this. I have some rough numbers. Your turn.
Haven't done any *real* benchmarking yet, just seeing a slow UE. And
when doing some individual traces here and there, they're looking
horribly complex.

I'll yet have to check how it performs on ARM. Assuming, jdk8 meanwhile
doesn't crash so quickly anymore.
Post by Michael T. Pope
Post by Enrico Weigelt, metux IT consult
And it requires jdk8, which I have on none of my production machines,
running stable/lts distros (and properly packaging that monster is a
magnitudes larger job).
Sorry. Java8 is not negotiable.
Does that mean, using j8 stuff whereever possible, just for its own
sake ?
Post by Michael T. Pope
Java7 is no longer getting security updates.
LTS distros still do it. (OTOH, for plain desktop applications, the
pressure isn't that high anyways.)
Post by Michael T. Pope
If you want to claim your code is "more expressive" I would expect it to be shorter.
Not if it hides important from the human reader. And in such procedural
languages, types are indeed important. (in functional languages, that
might be different, but Java is nowhere near to functional).
Post by Michael T. Pope
It *is* more "explicit" in that you are correct that the types are more obvious,
but I do not consider that an advantage over being succinct.
For me it *is* important. Everytime I see such an expression, I'll have
to look up which the actual types, to understand what's really going on
here, just because some people wanna save a few keystrokes ...

And I dont wanna waste so much memory for all the temporary objects,
(my notebook only has 8GB RAM, that has to be shared w/ lots of other
applications), and also I don't wanna burn too many cpu cycles - having
the fan on max all the time isn't actually comfortable.
Post by Michael T. Pope
Post by Enrico Weigelt, metux IT consult
Post by Michael T. Pope
This is obviously not
definitive as a lot of other code went in at the same time, but certainly
suggestive that Streams have negligible impact, and even if Streams are
fully to blame for the change, I am *not* worried that it now takes 7 AI
players 5.2s to move rather than 5.1s.
On my site it's magnitudes smaller. Haven't looked at the whole AI area
yet, my next 2do is the startup, which still is uncomfortably slow ...
So *what* exactly is "magnitudes smaller"?
Sorry, a bit fallen over --- AI is magnitudes slower here. In larger
games (when the AIs have lots of colonies and units), up to a minute
per turn. That's the point, where it really doesn't make fun anymore.
Post by Michael T. Pope
However I expect you will find that most of
the time is I/O, sending the XML for the Game to the client.
Yeah, already suspected that. Maybe I'll just replace the XML bloat
by something thinner and faster ... (or maybe just a faster xml
parser)


--mtx
Michael T. Pope
2017-01-25 05:00:51 UTC
Permalink
On Tue, 24 Jan 2017 18:10:19 +0100
Post by Enrico Weigelt, metux IT consult
Post by Michael T. Pope
Always the performance... for the nth time, show me that it is a problem!
Well, eg. up to several seconds delay on map move, etc actually is a
problem - at least for me ...
Apparently. Now do you recall what wintertime and I said a while back
about optimization? The GUI, specifically MapViewer is known to be slow.
Apparently *very* slow in your case, I suspect there may be something
particular to your system going on there. May I again suggest that you
look into why MapViewer is slow? I hope you can understand some of my
exasperation in that you have submitted 180 odd optimization patches none
of which touch the one area that has been reproducably shown to be bad.
Post by Enrico Weigelt, metux IT consult
Post by Michael T. Pope
Java8 is not negotiable.
Does that mean, using j8 stuff whereever possible, just for its own
sake ?
No. Why do you say that?
Post by Enrico Weigelt, metux IT consult
Post by Michael T. Pope
Java7 is no longer getting security updates.
LTS distros still do it. (OTOH, for plain desktop applications, the
pressure isn't that high anyways.)
Ahem. I am quite able to be a hard core Linux purist too, but our users
are not. Or were you volunteering to maintain Java7 for Windows, Macs and
non-LTE linux distros?

That said, you should also know that I do security professionally and have
spent far too much time cleaning after people who insist on hanging on to
their known vulnerable software. The FreeCol project generally was
burned a while back due to the website being exploited, which wasted
quite enough of my and other volunteer's time. I have no wish to waste
more.
Post by Enrico Weigelt, metux IT consult
Post by Michael T. Pope
It *is* more "explicit" in that you are correct that the types are more obvious,
but I do not consider that an advantage over being succinct.
For me it *is* important. Everytime I see such an expression, I'll have
to look up which the actual types, to understand what's really going on
here, just because some people wanna save a few keystrokes ...
Keystrokes are a simple proxy for expressiveness, not the aim in
themselves. I want to use high level constructs. Java may indeed not be
particularly functional, but that is no reason not to use some
functional-like constructs.

As to your preference for seeing all the types, I doubt we will agree on
this.
Post by Enrico Weigelt, metux IT consult
And I dont wanna waste so much memory for all the temporary objects,
(my notebook only has 8GB RAM,
Likewise. My main development machine is almost 11 years old. If you
think memory is an issue, enable the garbage collection logging
(IIRC its something like -verbose:gc) and see if it is being triggered a
lot. Note we tend to run FreeCol with -Xmx512M.
Post by Enrico Weigelt, metux IT consult
Post by Michael T. Pope
So *what* exactly is "magnitudes smaller"?
Sorry, a bit fallen over --- AI is magnitudes slower here. In larger
games (when the AIs have lots of colonies and units), up to a minute
per turn. That's the point, where it really doesn't make fun anymore.
Well I have to admit I do not see many games where AIs have lots of
colonies. Usually some human wipes them out. So what do you mean by
"lots"? Do you have a savegame that can be tested? If so please attach.

However, the simplest explanation is, again, the GUI. My tests are
headless.

That said, the next most likely explanation for your patches improving
matters is the Map handling (because that *does* impact MapViewer). What
happens if you apply just that patch?
Post by Enrico Weigelt, metux IT consult
Post by Michael T. Pope
However I expect you will find that most of
the time is I/O, sending the XML for the Game to the client.
Yeah, already suspected that. Maybe I'll just replace the XML bloat
by something thinner and faster ... (or maybe just a faster xml
parser)
That is underway, but going slowly. org.w3c.dom.* is very common,
and proving painful to remove. We try to work incrementally because the
skilled users that report bugs against trunk are of immense value.

Cheers,
Mike Pope
Caleb Williams
2017-01-25 06:34:42 UTC
Permalink
All:

Apparently. Now do you recall what wintertime and I said a while back
about optimization? The GUI, specifically MapViewer is known to be slow.
Apparently *very* slow in your case, I suspect there may be something
particular to your system going on there. May I again suggest that you
look into why MapViewer is slow? I hope you can understand some of my
exasperation in that you have submitted 180 odd optimization patches none
of which touch the one area that has been reproducably shown to be bad.


My take in this is that while optimization is always beneficial (to
someone), stopping to optimize everything can be detrimental to fixing bugs
and implementing features.

Just like any open source project all are free to work on whatever they
want, I jusy don't want the end user to be lost in all this philosophical
discussion.

A couple months back there was an open bug review on this list and many
those remain unfixed.
Post by Enrico Weigelt, metux IT consult
Post by Michael T. Pope
Java8 is not negotiable.
Does that mean, using j8 stuff whereever possible, just for its own
sake ?
No. Why do you say that?

...

That said, you should also know that I do security professionally and have
spent far too much time cleaning after people who insist on hanging on to
their known vulnerable software. The FreeCol project generally was
burned a while back due to the website being exploited...


Agreed. Using version 8 only makes sense and it seems to be useful for Mike
to use those new features when coding FreeCol (Streams API, Lambdas, etc.).
Post by Enrico Weigelt, metux IT consult
Post by Michael T. Pope
So *what* exactly is "magnitudes smaller"?
Sorry, a bit fallen over --- AI is magnitudes slower here. In larger
games (when the AIs have lots of colonies and units)...
I would like to raise this issue as a good one to tackle nearer to the v1
release (from a dev side), but before then I think some improvement is
always beneficial (from the user side).
Post by Enrico Weigelt, metux IT consult
From the UX side, the intial opening delay (both of the program and of a
game) can be a "long" time with no visual representation of actions
happening. Same with the AI turns delay.

Would progress bars help improve that experience?

Best,

Caleb

PS: Sorry for any typos, this was written on my phone.

Loading...