Evidence of cheating

Originally posted by bbarr
I also have been playing and chatting with David Tebb for over three years now, and I trust his judgment. I think this provides a good, prima facie reason to believe that Exy cheated. Of course I don't know this with certainty, but certainty isn't the proper epistemic standard here. This is epistemology that deals with rational belief formation based on testimony.

If David Tebb said that you were a cheater, would you believe him? Probably not, because you have access to evidence to the contrary.

But you also have access to a plethora of evidence to the contrary of his finding about Exy, as well as contrary expert opinions such as no1marauder's. If you believe that Exy cheated given all of this, mustn't you also believe that you cheated if David Tebb were to claim that you did?

Originally posted by no1marauder
Are you saying merely because you believe the specific individuals involved are generally trustworthy that their findings are prima facie entitled to belief even if the available public evidence supports a contrary conclusion? If Dave Tebb told you he had been abducted by aliens would you feel his word entitled to belief until it was conclusively proven ...[text shortened]... never say when and where you robbed someone and that "evidence" be sufficient for punishment?

No, if the available public evidence suggests otherwise than that offered in the testimony of a generally reliable source, then it is proper to withold belief and gather further evidence. If the available public evidence is overwhelmingly contrary to that offered in such testimony, then it is proper to believe in accord with the balance of the evidence. The question here involves the extent to which the available public evidence strongly impugns the game mods' testimony.

I understand that in Exy's case, the games folk have independently checked thus far do not reveal evidence of engine use. Whether this is because these independent checks have not been run on the correct games, or not with the correct engines, I cannot say. It has not even been established that Exy was banned for using an engine, as opposed to other bannable offenses like consulting with a third party. I do not know if Exy used an engine in my two games with him. I know I did not suspect it at the time, and I see no reason now to suspect it, but these are only two games out of the many that he played. The game mods do not choose users to investigate, they investigate based on accusations submitted to them, so it is hard to say which games (if any) Exy was accused of using an engine in.

Suppose the defendant in your analogy was accused of robbing a particular store that is commonly robbed. Suppose the evidence collected by the trusted police officer was collected by use of a hidden video camera, and that by making public the video, other potential robbers would know how to evade video detection. In this case, would it be permissible not to release the video, or even the fact that there was a video camera? What if the police were elected by the general population, and were elected specifically to combat rampant theft, and that the populace agreed that some apparent violations of principles of due process were justifiable in light of the necessity of combatting rampant theft. Questions of justice need to be balanced by other competing values, like security.

Look, I agree that the lack of transparency here is lamentable. I'm having trouble, however, seeing any alternatives that don't undermine the effectiveness of the game mods. I'm willing to listen to suggestions for improving the process, if you or Herr Dr. have any...

Originally posted by DoctorScribbles
If David Tebb said that you were a cheater, would you believe him? Probably not, because you have access to evidence to the contrary.

But you also have access to a plethora of evidence to the contrary of his finding about Exy, as well as contrary expert opinions such as no1marauder's. If you believe that Exy cheated given all of this, mustn't you also believe that you cheated if David Tebb were to claim that you did?

The difference, of course, is that in my case I would have access to conclusive evidence of my innocence, based on introspection and memory. I do take No.1's concerns seriously, and I consider him a generally reliable source on many issues. I also know that he doesn't have access to a wide array of engines, and that he doesn't know which games are the ones at issue, and that the very supposition that it was engine use that was the bannable offense may be erroneous, and that there may be tools used by the game mods of which we are completely unaware and/or incapable of independently duplicating.

Originally posted by bbarr
No, if the available public evidence suggests otherwise than that offered in the testimony of a generally reliable source, then it is proper to withold belief and gather further evidence. If the available public evidence is overwhelmingly contrary to that offered in such testimony, then it is proper to believe in accord with the balance of the evidence. The ...[text shortened]... I'm willing to listen to suggestions for improving the process, if you or Herr Dr. have any...

Many months ago at the FW Round Table, I described a statistically sound and objective algorithm, based on Naive Bayesian Classification (http://en.wikipedia.org/wiki/Naive_Bayesian_classification), for classifying players into cheaters and non-cheaters. This algorithm could be tuned to achieve whatever false positive/false negative balance was desired, and could incorporate essentially any relevant evidence regarding players' moves and engines' moves, once appropriately modeled to plug in to the algorithm as a classification "feature."

The Game Mods would simply apply the algorithm I had in mind, and act upon the result. It is objective. They could tell the community, "The Standard Game Mod Cheating Classifier, whose description is given here ..., was run on this set of input games... and classified player ImABigCheater as a cheater and he has been banned accordingly."

Unfortunately, nobody took the time to understand the mathematics of the algorithm I described and thus the consensus about it was, "I doubt that will work."

Originally posted by bbarr
No, if the available public evidence suggests otherwise than that offered in the testimony of a generally reliable source, then it is proper to withold belief and gather further evidence. If the available public evidence is overwhelmingly contrary to that offered in such testimony, then it is proper to believe in accord with the balance of the evidence. The ...[text shortened]... I'm willing to listen to suggestions for improving the process, if you or Herr Dr. have any...

Since the Game Mods refuse to say where, when and how Exy cheated you are correct that I may be looking at the wrong games or for the wrong things, but that is purely the result of the extreme secrecy that has been adopted. Your analogy fails because we know when and where store X was robbed and a claim that Exy robbed it on that day could be checked against the known facts and evidence. Here, however, we are merely given a broad allegation ("Exy cheated"😉 and a complete refusal to state how and where he did so. We do not accept as a standard for justice an allegation that someone is guilty of robbery without being given salient facts in support of such allegations.
No level of personal trustworthiness is considered sufficient to convict without presenting facts in support of the charges.

I cannot conceive how a statement like "We believe Exy used engine Y in games 1,2, 3 .......... X" would prejudice the collection of evidence of cheating. There is a broad claim that presenting ANY evidence would necessarily lead to a total dissolution of the effectiveness of the Game Mods ability to find cheating. Surely that is not true and such a claim is Chicken Littleish and absurd. Surely the present system that relies on complete trust in a few individuals who's work product is completely unverifiable does not comport with any recognized system of justice nor any concept of fairness. The site is free to adopt whatever system they choose or none at all to stop cheating, but please let's not pretend that the Kafkaesque aspects of the present system are necessary or logical.

Originally posted by DoctorScribbles
Many months ago at the FW Round Table, I described a statistically sound and objective algorithm, based on Naive Bayesian Classification (http://en.wikipedia.org/wiki/Naive_Bayesian_classification), for classifying players into cheaters and non-cheaters. This algorithm could be tuned to achieve whatever false positive/false negative balance was des ...[text shortened]... cs of the algorithm I described and thus the consensus about it was, "I doubt that will work."

Do you still have that proposal available? If so, could you post it here? Would the algorithm need to be applied more than once to each game, given that different engines, different ply-depths, different search times, and different search parameters may be used by different engine users? Should this be the only tool game mods use, on your view, or do you think that if the game mods had other tools at their disposal it would be permissible to use those tools and keep them secret from the community?

The bottom line of the whole business is that when players get banned, possibly whole clans are leaving due to a perceived injustice, whether they are right or wrong.

Other players are feeling insecure, due to the believe that there could be a lot of false positive identifications.

These types of incidents are bad for business (loss of previously satisfied customers is ALWAYS bad for business)

I don't believe that the game mods should give away any information that might help cheats, but I do believe that a statement needs to be released. Despite the fact this is above and beyond the terms of the service, I believe it could only be a positive thing.

D

Originally posted by bbarr
Do you still have that proposal available? If so, could you post it here? Would the algorithm need to be applied more than once to each game, given that different engines, different ply-depths, different search times, and different search parameters may be used by different engine users? Should this be the only tool game mods use, on your view, or do you th ...[text shortened]... eir disposal it would be permissible to use those tools and keep them secret from the community?

FW has been wiped from the Internets and I no longer have access to the original. I suppose I could attempt to formulate a reconstruction of it in the immediate future if I could receive some indication from the powers that be that it would be given due consideration. But Gatecrasher indicates that the Game Mods are not even open to considering a public evidentiary standard; he said he would quit if they decided to reveal their evidence.

The algorithm could be as simple or as intensive as the Game Mods desire. If they currently always examine a variety of engines with various configurations, then the classification model could account for each of those. If they currently only use Fritz with a 10 minute analysis per move looking for a match in the top two candidates, the model could use that evidence only. It would be up to the Game Mods.

Their expert knowledge would essentially be modeled as a set of "features", as described in the citation. For each way in which they currently use their expertise to distinguish cheating from non-cheating, the automated decision making model would have a corresponding feature.

I am yet to be convinced of the need for any amount of secrecy. I don't think that my suggestion need be the only tool, but I do think that the Game Mods would not require secret complementary tools.

Originally posted by no1marauder
Since the Game Mods refuse to say where, when and how Exy cheated you are correct that I may be looking at the wrong games or for the wrong things, but that is purely the result of the extreme secrecy that has been adopted. Your analogy fails because we know when and where store X was robbed and a claim that Exy robbed it on that day could be checked aga ...[text shortened]... e let's not pretend that the Kafkaesque aspects of the present system are necessary or logical.

Look, you know from our personal communications that I agree with you in spirit. I am also troubled by the Kafkaesque properties of the current game moderation system.

The analogy doesn't fail. In the analogy, we (the public) do not know where and when the store was robbed. The authorities know where and when the store was robbed by virtue of having caught the theft on video. The claim is that if the public were aware of where and when the store was robbed, and of the manner in which the determination of guilt was made, then this information could be put to use by others who wish to rob the store.

Of course you are right that it is not necessarily the case that making any piece of evidence public will undermine the effectiveness of the game mods. After all, if everybody just decided to never ever use engines, then making public all the evidence that supported past bannings would be perfectly benign. But I never claimed otherwise, and I don't remember the game mods claiming otherwise. Their claim, which I find plausible, is that making any evidence public would, as a matter of empirical fact (not necessity) make cheating easier. I think it is right to question this claim, and try to determine whether there are evidence types that would not undermine game mod effectiveness if their instances were made public. Perhaps, as you mention, making public the specific games at issue would be benign. Then again, if the evidence to be found in such games was collected using tools to which we don't have access, then the community will be continuously outraged that their own independent engine checks of these games fail to reveal evidence of cheating.

In the end, as I've said to you before, the current system is neither perfectly just nor perfectly fair. There are elements of the current system that comport with justice and fairness (e.g., the election of the game mods, their inability to initiate charges on their own, the refund of subscription fees to banned users, etc.). But, as I've also said before, justice and fairness are not the only values at issue in this debate. There is also the value of being secure from those that use engines, and the community seems to think this is a value worth taking seriously. If, of course, the community decides that the cure here is worse than the disease, then that would be a good reason to rethink either the way in which the game mods function or, more radically, whether there ought to be game mods at all.

Originally posted by bbarr
Then again, if the evidence to be found in such games was collected using tools to which we don't have access, then the community will be continuously outraged that their own independent engine checks of these games fail to reveal evidence of cheating.

But if the cheater can pass the Turing Test, such that his opponents can't distinguish him from a non-cheater, why should his opponents want him banned anyway? The resulting banning would be a result of mere obstinance. It's putting the cart before the horse.

Suppose a cheater were to alternate engines, such that both he was de facto unbeatable even against the very best opponents, even when moving very rapidly, and that independent checks of his moves against any particular engine would not reveal a match over some specified threshold. You may have good reason to think somebody is cheating even though independent checks in line with your proposed algorithm would fail to supply evidence of cheating.

Originally posted by no1marauder
I cannot conceive how a statement like "We believe Exy used engine Y in games 1,2, 3 .......... X" would prejudice the collection of evidence of cheating.

Unfortunately, a statement like "We believe Exy used engine Y in games 1,2, 3 .......... X" may represent only partial evidence.

It is certainly true that several tools have been developed for game moderation at RHP. But I cannot see how any of the evidence gathered using these tools could be publicized without jeopardising their effectiveness.

Having said that, a good case could be made for giving a more detailed reason for a banning, other than 3(a), 3(b) or 3(c).

I know that if I were on the other side of the divide I would be hopping up and down with a mixture of curiosity and indignation. I'm sure any suggestion that might lead to a workable compromise would be welcome.

Originally posted by bbarr
Suppose a cheater were to alternate engines, such that both he was de facto unbeatable even against the very best opponents, even when moving very rapidly, and that independent checks of his moves against any particular engine would not reveal a match over some specified threshold. You may have good reason to think somebody is cheating even though independent checks in line with your proposed algorithm would fail to supply evidence of cheating.

My proposed algorithm would in fact be very likely to classify such player as a cheater! In particular, when examining the games of the user in question, it would observe several features:
"Matches engine X on every other move."
"Matches engine Y on every other move."
"Matches engine X with a rate of 50%."
"Matches engine Y with a rate of 50%."

That's the beauty of Naive Bayes. Any one of these is probably not damning. Maybe even the conjunction of them isn't damning. But obseving them all and possibly others, and also observing that known non-cheaters don't exhibit these features, can lead to a confident and precise determination of just how likely it is that such features would be observed in the game of a cheater or non-cheater, and the player can be classified accordingly.

In fact, I'd be willing to speculate that the cheater you describe would be more likely to be classified as a cheater by my algorithm than a player who uses only one engine with a 60% match.

Originally posted by Gatecrasher
Unfortunately, a statement like "We believe Exy used engine Y in games 1,2, 3 .......... X" may represent only partial evidence.

That's fine. That at least constitutes a specific public charge, which would be a step in the right direction.

Originally posted by bbarr
Suppose a cheater were to alternate engines, such that both he was de facto unbeatable even against the very best opponents, even when moving very rapidly, and that independent checks of his moves against any particular engine would not reveal a match over some specified threshold. You may have good reason to think somebody is cheating even though independent checks in line with your proposed algorithm would fail to supply evidence of cheating.

Aren't you revealing something which might make cheating easier?😠

Seriously, I refuse to believe that cases of blatant cheating are soooooooo difficult to find evidence of that even the specific conclusions of an investigation must be kept totally secret for fear that some possible technique could be developed by potential cheaters to hamper their detection. I find such an assertion highly unlikely. I think that the present situation, where the community is asked simply to trust group A (the Game Mods) rather than Group B (the accused) but are given no reason to do so is simply begging for greater controversy. That is precisely what has happened. If the Game Mods are going to be an effective deterrent than the banning of individuals must be shown to be more than the seemingly whimisical process it appears to be when someone like Exy is banned but other usernames who match up far higher with engines and rarely, if ever, lose, are allowed to rocket to the top of the ratings charts.