How to detect engine cheats - a guide

HI Varenka. Thanks.

I was aware of the 'Warriors' and found it wanting in a lot of departments.
It is also dated.

The other article I never knew about and will read it with interest.

I met Ivan Bratko a few times in the 1980's and was a guinea pig for the
Kopec-Bratko test mention on that article. What a small world.

Still think with the tremendous advance of computers over the past year
and using Squelch's top 3 method (which anybody with a reasonable computer can do).

I gave the article a quick going over and cannot see a top 3 match up score.
They have appear to have taken the blunder check method.

Cheers.

The post that was quoted here has been removed

You accused someone of being a cheater indirectly and you are whining about the mods?

Originally posted by greenpawn34
using Squelch's top 3 method

This method is too crude and is only suitable for blatant cheaters. Take a look at the work of people like Kenneth W. Regan and you'll start to get an idea of the real complexity of the problem.

http://www.cse.buffalo.edu/~regan/chess/fidelity/

Perhaps we could meet up for a pint or a coffee and play in person.😀🙂

Originally posted by greenpawn34
It's sometimes good to 'bump' a post if it has, as is the case here, relevant
and excellent information in it.

I'm positive there is a good chess book waiting to be written using this data
crunching method.

A scan could be run over every World Champion using the one match that won
them the World Title. This would give an indication who played t ...[text shortened]... he
greatest ever pawn pusher this book is simply wating to be written.

A good bump.

a few years ago there was a big engine shootout at most of the big boys, and if I remember right capablanca did indeed come out on top. fischer has also done well in these as I remember. -one interesting point that I remember about it, was that a lot of the old guys were barely mediocre compared to current players, punching away as low as the 2000 level. the really big boys did all okay, but their opponents... the genius of morphy wasn't the only reason why he was crushing the competition.

Originally posted by National Master Dale
Perhaps we could meet up for a pint or a coffee and play in person.😀🙂

Yeah sure 😛

Without posting pages of analysis (again!) here are the results of fairly recent benchmark testing using Fritz XI top 3 matchup with the above criteria for each:

Capablanca-Alekhine 1927

Alekhine
Top 1 Match: 467/852 (54,8% )
Top 2 Match: 622/852 (73,0% )
Top 3 Match: 685/852 (80,4% )

Capablanca
Top 1 Match: 470/853 (55,1% )
Top 2 Match: 632/853 (74,1% )
Top 3 Match: 703/853 (82,4% )

Fischer-Spassky 1972

Fischer
Top 1 Match: 385/658 (58,5% )
Top 2 Match: 509/658 (77,4% )
Top 3 Match: 563/658 (85,6% )

Spassky
Top 1 Match: 368/657 (56,0% )
Top 2 Match: 461/657 (70,2% )
Top 3 Match: 525/657 (79,9% )

7th CC World Championship 1972-1975
Top 3 finishers

1st Place Result:
Estrin (9 games)
Top 1 Match: 153/256 (59,8% )
Top 2 Match: 191/256 (74,6% )
Top 3 Match: 209/256 (81,6% )

2nd Place Result:
Boey (13 games)
Top 1 Match: 268/449 (59,7% )
Top 2 Match: 342/449 (76,2% )
Top 3 Match: 376/449 (83,7% )

3rd Place Result:
Zagorovsky V (10 games)
Top 1 Match: 153/252 (60,7% )
Top 2 Match: 190/252 (75,4% )
Top 3 Match: 208/252 (82,5% )

10th CC World Championship 1978-1981
Top 3 finishers

1st Place Result:
Palciauskas, V (11 games)
Top 1 Match: 209/355 (58,9% )
Top 2 Match: 268/355 (75,5% )
Top 3 Match: 291/355 (82,0% )

2nd Place Result:
Morgado, J (9 games)
Top 1 Match: 131/247 (53,0% )
Top 2 Match: 189/247 (76,5% )
Top 3 Match: 208/247 (84,2% )

3rd Place Result:
Richardson, K (10 games)
Top 1 Match: 204/361 (56,5% )
Top 2 Match: 258/361 (71,5% )
Top 3 Match: 284/361 (78,7% )

Kramnik-Topalov 2006

Kramnik
Top 1 Match: 306/552 (55,4% )
Top 2 Match: 417/552 (75,5% )
Top 3 Match: 461/552 (83,5% )

Topalov
Top 1 Match: 309/555 (55,7% )
Top 2 Match: 417/555 (75,1% )
Top 3 Match: 447/555 (80,5% )

Originally posted by Varenka
This method is too crude and is only suitable for blatant cheaters. Take a look at the work of people like Kenneth W. Regan and you'll start to get an idea of the real complexity of the problem.

http://www.cse.buffalo.edu/~regan/chess/fidelity/

Top 3 matchup will of course only catch blatant engine users, but as I said, these are the most annoying.
The already strong player who only uses Fritz to check the odd line or validity of a few moves in a few games won't really be caught by any detection method.
Top 3 matchup is just fine for 99% of the obvious cheats.

For instance, erik on www.chess.com has kicked their former #1 player a few weeks after I sent in analysis for him.
The results were

ouachita (20 games)
Top 1 Match: 495/719 (68,8% )
Top 2 Match: 603/719 (83,9% )
Top 3 Match: 650/719 (90,4% )

This player was well-established on that site & had the backing of many people in the forums who thought him legit.
It takes a lot of guts to kick a blatant engine user who tops the player rating lists.
Good on erik & chess.com!

Originally posted by Fat Lady
Despite having a degree in maths, I was never very interested in statistics.

However, if you consider that a very strong player has, say, a 0.65 chance of choosing a move which is Fritz' first choice in a particular position, then the probability of this player choosing Fritz' #1 move for the last 21 moves in a particular game would be 0.00012. I.e. you might expect to see this happen 1 game in every 8000.

You have to be careful because of the birthday paradox. The problem is that with millions of games played on the site the chances that someone has innocently replicated 20 engine moves in a row (I mean by chance rather than cheating) starts to get large. Since you'd expect a persistent engine user to be in the top couple of hundred players on the site and further checking is done before they ban someone it's not so much of a problem, but statistical proofs are notoriously tricky.

Originally posted by National Master Dale
Perhaps we could meet up for a pint or a coffee and play in person.😀🙂

Why not? We could play some rapid or blitz chess.

Originally posted by DeepThought
You have to be careful because of the birthday paradox. The problem is that with millions of games played on the site the chances that [b]someone has innocently replicated 20 engine moves in a row (I mean by chance rather than cheating) starts to get large. Since you'd expect a persistent engine user to be in the top couple of hundred players on th ...[text shortened]... they ban someone it's not so much of a problem, but statistical proofs are notoriously tricky.[/b]

hence the need for a large sample of games (i recall squelch recommended 20+) - but i expect Russ would want more for banning purposes.

as others have pointed out, it's fairly easy to be very accurate in non-complex positions.

btw - the "birthday paradox"?!

Originally posted by rja211077
btw - the "birthday paradox"?!

http://lmgtfy.com/?q=birthday+paradox

very interesting thread, however I think for the sake of really getting a good anti cheat system the following should be considered;

1) it seems that using capablanca, alekhine or any of the pre computer masters would give you lower than normal average % of matching moves since classical principles and lack of appreciation of dynamic qualities of games (something programmed into evals of modern engines) were not part of that era.

2) how do you seperate computer moves from home analysis? A great example of this, in a blitz game 6 months ago I had a very unusual move not covered in published theory played against me. I spend a few days analyzing variations until I am satisfied my mainline black defense isn't busted. 3 months later having started Correspondence chess, I get that exact line on the board. Now I can prove that I played a game in that line with an icc date etc... but to your system I will probably flag as a computer since I checked (and in some places improved) my analysis due to computer engine use? This seems to me to be a gray area in the TOS of online correspondence sites. I am sure the almost all serious players will due post mortem analysis with a computer to check for blunders and such.

3) All you examples are relatively well matched players against each other, in games where the playing strength is less balanced I am not sure if the match rate wouldn't be higher?

4) by nature OTB should have more "inaccuracy" built in- simply not being able to move the pieces should create a much higher level of error. Compare amber blindfold tournament match rates vs being able to see the pieces and you can extrapolate some degree of difference perhaps?

High pressure championship games are probably not the best choice of comparison. Even so what was Kramnik's match rate in the Braingames championship? He for certain would have flagged as cheating.

5) CC match rates would be the best indicator, and particularly ones in the computer era (read 1995 to date or so).

In general though, I think this is a very interesting idea and worth using further since anything to put in check computer engine use would be welcome.