I suggested that applying the system no1marauder was told to use to gather evidence against suspecte3d cheats to a correspondence chess master who could not have used an engine might provide evidence for or against such players playing like engines. Someone suggested I should do it and post the results so here they are. I have not put this in the original thread to avoid it getting lost or possibly even deleted.
First, the how. I suspect most of you know how no1marauder has been gathering evidence by now but i shall describe the method here just to make sure you all know exactly what I did. I played each game through from the start, allowing an engine (HIARCS 12.1 in this case) to analyse each move for 30 seconds. I set it up to display the top 12 engine choices (no1 uses less but I wanted to see more of what the engine was doing) and then recorded all moves that matched the 1st, 2nd and 3rd choices. Finally, I calculated an engine match up percentage.
Second, the who. I decided to look at some of the games of Hans Berliner for the following reasons:
1. Berliner was CC World Champion in 1965 long before any chess engine would have been any use so he should come up clean as a whistle.
2. Berliner later went on to program chess engines (notably Hitech) and therefore should know something about chess engines and how they calculate.
3. Berliner wrote a book entitled "The System" that outlines his method for finding a move. He describes his system as a position evaluation function that ideally will give a numerical value for any position. He also comments that he has tried to program his Hitech system to use his evaluation function. On the basis of this I reckon that Berliner is the best candidate for possible engine-like play.
4. Berliner returned to play a 50 year Jubilee Waorld Champions Tournament in 2003 when engine use was already regarded as an issue.
5th CC World Championship Final
I had access to 15 games played by Berliner. He drew 4 and won 11.
311 moves total.
1st choice matches: 150
2nd choice matches: 55
3rd choice matches: 49
Total matches 254
Match up percentage = 82%
82% is fairly high but is just a number it does not really reveal much. I found that watching the games unfold and how the engine assessed the positions was very instructive.
Berliner always played 1. d4 as white and remarks on this in his book. As black he played Nf6 or g6 and aimed for King's Indian Defence type positions, not for the kingside attack possibilities but for solidity and safeness of position. Once out of the opening the engine generally assesses the position as about equal, giving a value within 0.3 of 0. At this point there will be many candidate moves that have approximately the same value to choose from. Berliner seems to choose from this 5, 6 or possibly more moves one that is as safe as possible. The chosen moves sometimes match, sometimes do not. In fact, in this stage of the game the match up percentage is about 50%. It seems that Berliner was in the habit of waiting for his opponent to make an error and then capitalising on it. This point is often easily seen in the engine analysis - the score suddenly changes from ~0 to ~1 (or ~-1 if playing black). At this point Berliner's play changes, his match up percentage increases dramatically and he often chooses the engine 1st choice. Berliner never blundered in any of these games although he did make mistakes, but these were not game losing mistakes unlike those of his opponents. Engine like? I don't think so but i have no doubt others will disagree.
Onwards then, to the age of engines.
50 Year Jubilee World Champions' Tournament.
This was a tournament played between past CC World Champions. It is important to remember that Berliner was 74 at the time of this tournament.
8 games played by Berliner. Won 1, drew 5, lost 2.
288 moves total.
1st choice matches: 220
2nd choice matches: 18
3rd choice matches: 6
Total matches: 244
Match up percentage: 85%
Again, this is quite high and is higher than in 1965, an again I found watching the games and the analysis instructive.
The first thing I noticed is that Berliner's opponents do not make mistakes for him to capitalise on anymore. The games tend to stay in that state of almost equality for many moves without really varying. I can think of two reasons for the lack of mistakes. First, this may be a consequence of having world champoions play each other. Maybe what separates a CC World Champion from the also-rans is simply the lack of mistakes. Second, maybe we are seeing evidence of engine use.
The second thing I noticed was that Berliner match up percentage in the sort of quiet position that dominates these games has shot up dramatically from 1965. He is now matching HIARCS' choices far more often whereas in 1965 in these quiet, equal positions he was only matching about 50% of the time. It is also notable that his highest match ups (both 92% ) happened in the two games he lost. I doubt this indicates total reliance on an engine since he is still making minor errors, and some that are major enough to lose. However, I strongly suspect he was making use of his own Hitech system.
So, make of that what you will.
Originally posted by KeplerI may comment more later, but regardless of any opinions, thanks for taking the time/effort to do this. 🙂
Are we saying...
- Berliner without engines = 82%
- Berliner with engines = 85%
? And hence only 3% of a difference?
Of course, other factors have changed too, e.g. more opening theory, etc.
EDIT: what evaluation threshold did you use for matching the moves?
Originally posted by VarenkaAll I am saying is that in 1965 Berliner matched 82% and in 2003 he matched 85%. I don't know why the increase or even if it is significant.
I may comment more later, but regardless of any opinions, thanks for taking the time/effort to do this. 🙂
Are we saying...
- Berliner without engines = 82%
- Berliner with engines = 85%
? And hence only 3% of a difference?
Of course, other factors have changed too, e.g. more opening theory, etc.
Opening theory will not have been a factor - I stripped all the database moves out and Berliner has little time for databases and opening books since he regards his system as THE way to play the opening. All moves flow from the system.
I didn't use an evaluation threshold, I just set the engine on infinite analysis and let it run. I literally did the whole thing manually in order to be able to see how the games unfolded and watch the engine in action.
Originally posted by KeplerAbout all I can say of the old mod team, those numbers are not alarming.
All I am saying is that in 1965 Berliner matched 82% and in 2003 he matched 85%. I don't know why the increase or even if it is significant.
Opening theory will not have been a factor - I stripped all the database moves out and Berliner has little time for databases and opening books since he regards his system as THE way to play the opening. All moves flow from the system.
P-
Originally posted by Kepleri believe your methodology perhaps might be severely flawed. Some moves are forced, some moves are common variations of well known openings. If you would post 1 game and give us the move by move breakdown, i think that would be helpful.
Opening theory will not have been a factor - I stripped all the database moves out and Berliner has little time for databases and opening books since he regards his system as THE way to play the opening. All moves flow from the system.
I didn't use an evaluation threshold, I just set the engine on infinite analysis and let it run. I literally did the whole ...[text shortened]... hing manually in order to be able to see how the games unfolded and watch the engine in action.
- Joined
- 26 May 02
- Moves
- 72546
Originally posted by KeplerIf we compare Berliner's first choice matches, the difference between the two sets of figures are astonishing:
I suggested that applying the system no1marauder was told to use to gather evidence against suspecte3d cheats to a correspondence chess master who could not have used an engine might provide evidence for or against such players playing like engines. Someone suggested I should do it and post the results so here they are. I have not put this in the original thr ...[text shortened]... y suspect he was making use of his own Hitech system.
So, make of that what you will.
5th CC World Championship Final 1965 (15 games)
1st choice matches: 150 / 311 = 48.2 %
50 Year Jubilee World Champions' Tournament 2003 (8 games)
1st choice matches: 220 / 288 = 76.4%
The first choice matches are very significant. To go from less than 50% to over 75% is an enormous improvement. It's very likely therefore that he did use an engine in the later tournament.
One thing which puzzles me though is that the average length of game in the first tournament is extremely low - about 20 non-book moves per game. I wonder whether he agreed a few quick draws. Very short games are usually useless for engine analysis.
Originally posted by eldragonflyI know the methodology is flawed. I have said so in the past, particularly with regard to forced or obviously best moves but no1marauder insists this is the way to do it because he was told so by a game mod. I just applied his methodology straight rather than modify it.
i believe your methodology perhaps might be severely flawed. Some moves are forced, some moves are common variations of well known openings. If you you post 1 game and give us the move by move breakdown, i think that would be helpful.
The opening moves were not analysed so are not a factor.
Sometime I'll likely redo the whole thing with better methodolgy but whether I post it here or not is another matter. Pearls before swine and all that.
Originally posted by eldragonflyIt would be very tiresome to evaluate all moves of a game to see what is forced. The percentage is a percentage based on the average number of forced moves in any given game.
i believe your methodology perhaps might be severely flawed. Some moves are forced, some moves are common variations of well known openings. If you you post 1 game and give us the move by move breakdown, i think that would be helpful.
If a game came out to say 92% (as a random example) you might want to evaluate the forced moves in that particular game.
P-
Originally posted by David TebbHe had two games in 1965 that contained 13 non-book moves, whatever that means in the context of 1965 and Berliner. Both had 100% match ups. There was also another 100% game, this time with 33 non-book moves. All of these games were wins for black. To be fair, one of those 13 non-book move games had 15 book moves in it.
If we compare Berliner's first choice matches, the difference between the two sets of figures are astonishing:
5th CC World Championship Final 1965 (15 games)
[b]1st choice matches: 150 / 311 = 48.2 %
50 Year Jubilee World Champions' Tournament 2003 (8 games)
1st choice matches: 220 / 288 = 76.4%
The first choice matches are ve ...[text shortened]... hether he agreed a few quick draws. Very short games are usually useless for engine analysis.[/b]
I know the limitations of the method and that short games are near useless but I am also aware that if I had changed anything or left anything out the arguments over that would have obscured the results.
- Joined
- 26 May 02
- Moves
- 72546
Originally posted by KeplerI think it's been quite an interesting exercise. I don't believe that too much can be read into the results, because the sample size from the second tournament (8 games) is a bit low.
He had two games in 1965 that contained 13 non-book moves, whatever that means in the context of 1965 and Berliner. Both had 100% match ups. There was also another 100% game, this time with 33 non-book moves. All of these games were wins for black. To be fair, one of those 13 non-book move games had 15 book moves in it.
I know the limitations of the method ...[text shortened]... d changed anything or left anything out the arguments over that would have obscured the results.
The main problem is that we don't know for certain whether he did use an engine in the later event, or how much he relied upon it and how much of the moves were his own. Even if he did use an engine, it would have been much weaker than the engines that are available today, especially when running on the latest hardware.
Originally posted by KeplerReally nice post and effort. got my rec.
I suggested that applying the system no1marauder was told to use to gather evidence against suspecte3d cheats to a correspondence chess master who could not have used an engine might provide evidence for or against such players playing like engines. Someone suggested I should do it and post the results so here they are. I have not put this in the original thr ...[text shortened]... y suspect he was making use of his own Hitech system.
So, make of that what you will.
But one thing I'd like to say. I was used to think CC more strategical/positional and classic timed OTB chess as more tactical. Just like Berliner says in his book: On CC tactical erros does not abound and then is all about milking the opponent till he's smothered. One other thing is that I think it is pretty safe to say that Berliner was a great calculator. On his game against Yakov demonstrates this quite beatifully. On The System he shows everything he calculated and everything is right. And the positions were so wild that even some chess engines calculating all by themselves make mistakes (at least that's what I've read from the guys that understand about chess and chess engines). He even anticipated what some people thought that was the bust to his line and busted the bust.
http://www.chessgames.com/perl/chessgame?gid=1224863&kpage=2 is the game Berliner won and http://www.chessgames.com/perl/chessgame?gid=1224989 has 14. Qb3! as a better move but still Berliner knew what would have to be his moves either to go for equality or to gain an advantage. Anyway if one checks the kibitzing on the first game a bust was actually found by an amateur player and Rybka. But here come what I think is the main point. No engine or player alone found the bust. It had to be the fact of a strong human player (albeit an amateur) could read the position and go for a new move and the engine calculating abilities combined that made it possible. What I'm trying to say here is that even though sometimes Berliner doesn't play what Hiarcs suggests he may in fact be playing something superior. I don't know much about this but I know that engines (so far at least) aren't too good understanding long ranged plans and sometimes they can't even correctly assess a position. O The System Berliner's gives a position that is a dead draw, even though one side is two pawns up, and at least the engines I have give a winning advantage to the up in material side.
And on the question of the possibility of him using his engine on the second time around I don't think that's likely. I remember reading somewhere him saying that he tried to program HITECH to play chess system wise and couldn't do it.
And one more thing to finish: if you go check the Estrin vs Berliner game just check out all the kibtzing.
Originally posted by KeplerYou shouldn't be evaluating ANY book/data positions. Each game should start evaluation from the first move out of data.... even checking a bit forward to be sure it didn't transpose back into known territory.
He had two games in 1965 that contained 13 non-book moves, whatever that means in the context of 1965 and Berliner. Both had 100% match ups. There was also another 100% game, this time with 33 non-book moves. All of these games were wins for black. To be fair, one of those 13 non-book move games had 15 book moves in it.
I know the limitations of the method ...[text shortened]... d changed anything or left anything out the arguments over that would have obscured the results.
P-