February 2014

Mike Cane wrote a blog post in response to Chuck Wendig’s and mine, saying that he thinks the self-publishing shit volcano will come to an end, because Amazon will end it. I left a comment on Cane’s blog, but it was starting to get longer than his initial blog post and I had more to say, so I’m expanding upon it here.

Cane’s thesis is that Amazon will act to remove bad ebooks that don’t sell because all that crap clogs up their site and is bad for business. He thinks that there will come a time where Amazon feels the pain so removes poor quality books and ban further submissions from terrible authors.

I wish he were right, but I don’t think Amazon will do anything within the foreseeable future. There is one circumstance which might fix this whole problem.

So, first, why won’t Amazon act?

Amazon is not a rational actor

At least, not in any way that you or I might consider rational. It’s pretty much the only company I can think of that can consistently not make a profit and not be punished by Wall Street. In the past, we’ve seen that it only takes action when it is cornered, and then it takes the smallest action it can get away with.

Take the bestiality/rape/incest/pseudoincest furore of last year. Amazon only acted when it felt cornered, and even then it did as little as it could get away with. There’s still plenty of dodgy porn on Amazon and will continue to be, because Amazon has no interest in really properly clearing it up.

Same with the sockpuppet review affair. And when Amazon did take action, it was to put in place stupid and ill-considered rules about whether Kindle authors could review or not. It has done nothing substantial about improving the quality of reviews, even though that would be something that you’d think would affect their bottom line quite significantly. After all, if you can’t trust the reviews on Amazon, how do you know whether to buy or not?

So at the moment, there is no force pushing Amazon to act, nothing making it whip out the banhammer. Yes, the shite clogs up Amazon’s arteries, but they have shown no interest in dealing with shite in other areas of their business, because clearly having heart disease isn’t producing any painful symptoms for them. Yet.

Amazon does make money out of bad books 

50 Shades of Grey. Not a masterpiece of literature, but it tapped into a market desperate for soft porn, did well, then broke out of that niche to became a cultural touchstone, bought not because it is good but because everyone wanted to know what the fuss was all about. Other areas of shitty writing, niche erotica in particular, do well again because people want stuff that the traditional publishers won’t touch with a bargepole.

So there is no 1:1 correlation between shitty self-published books and sales. The idea that self-publishing is a meritocracy where the good writing naturally floats to the top is at best a happy fairytale and at worst a delusion. If Amazon can make money out of monster porn without getting slapped about by the law, it will.

Storage is cheap and getting cheaper

Amazon has turned cloud storage into a business, and book files are small, so there’s no real reason for them to worry about how much space the long tale of self-published dross is taking up.

If your average ebook file takes up 500kb, then you can fit 2,147,483 in a single terabyte. Amazon charges $0.010 per gb per month for its “Glacier” storage. So if you’re hiring Amazon’s cloud directly, you can store 2097 averagely-sized files for a month for a cent. You could store 5 million books for just $2384 per month, which is certainly more than it actually costs Amazon, because they obviously mark up their commercial cloud storage offerings.

It is undoubtedly cheaper for Amazon to just store all ebooks uploaded than it is for them to pay someone to figure out how best to get rid of the ones that don’t sell AND are badly written, and then deal with the resultant backlash from offended authors.

That offended backlash

If there’s one thing Amazon isn’t interested in, it’s alienating hundreds of thousands of self-published authors. A few hundred noisy gasbags it can, and does, ignore. (Including the ones in the press.) But if you consider that most books don’t sell, and there is probably more than half a million self-published ebooks getting uploaded each year and growing, that’s a lot of shit and a lot of angry authors they’d have to deal with.

Whether there would be enough angry authors to hurt Amazon’s overall sales in any meaningful manner is something I couldn’t say. But it’s certainly enough to hurt Amazon’s brand (even more than they do themselves – they don’t seem to give a crap about brand), and hurt ebook and possibly paper book sales. Not to mention the deluge of angry email that would cripple their customer support department.

So whilst I would love Amazon to take a long, hard look at their self-publishing platform, I have absolutely no confidence that they will, because I cannot see any motivator big enough to push them to action.

What might change the calculation?

There is one thing that might change all this, and when it comes online it will revolutionise the book industry in ways we cannot even imagine.

Artificial Intelligence.

When we have meaningful AI, not necessarily all the way to full consciousness, but computers sophisticated enough to be able to learn to read and be programmed to develop a reliable taste, then the whole game changes. Everything. Amazon’s pathetic recommendation engine, which is the most overrated algorithm on the planet, will become utterly irrelevant. So will reader reviews. Because when we have a computer capable of reading a book and accurately scoring it for grammar, punctuation, plot, character development, style and genre, then we have a chance to be able to sift out the good from the bad.

Of course, then the question becomes, what do we mean by ‘accurately’? Or ‘good’? Whose standards will be used to draw the lines?

If past experience with technology is anything to go by, as soon as we have AI capable of doing this, we’ll have multiple interpretations of what ‘good’ is, and suddenly all books will become discoverable. Love monster porn? But really, really love velociraptor porn? AI will be able to scan the whole corpus and give you the very best in small dinosaur erotica. Want to read books that are just like Agatha Christie’s? Easy. Want to set your standards to embrace only the most obscure literary fiction? Piffle. Here I am, brain the size of a planet and you ask me to find you some literary fiction.

When we have AI, Amazon stops being the canonical catalogue of all books on the planet. Reader reviews become irrelevant. Sockpuppetry becomes impossible. Only quality – defined however the reader wants – matters.

Is this what Google is attempting with its mass book digitisation program? In 2005, Google played down that exact rumour. Last month, nearly ten years later, Google acquired Mind Deep, an artificial intelligence company based in London. I think we can all draw our own conclusions from that.

{ Comments on this entry are closed }

Author Chuck Wendig has written a long post about how self-publishing is turning into a shit volcano. Vast quantities of terribly written rubbish is being published, and this is damaging to everyone in self-publishing. He says (emphasis as original):

[…] one of the features of self-publishing is that the door is open to anyone. Everyone. Always. No bouncers at this nightclub door, which is fine, but that also means you get folks with no shirt and no shoes. You’ll get folks dressed to the nines in sharkskin suits and you’ll also get wild-eyed dudes who are eating goulash out of rubber boots and who are quietly masturbating in the corner. You let anybody swim in the pool and, well, anybody can swim in the pool.

He goes on to make a number of arguments as to why this is a bad thing, and asks what we can do about it. If you haven’t read it yet, do so, because Wendig makes some very good points.

Right.

I’m afraid I have some bad news for Wendig, and for everyone else in the industry, self-published or otherwise. The shit volcano is not going to stop erupting, and there’s nothing we can do about it. There are a number of reasons for my pessimism, but the main one is this:

“Ignorance more frequently begets confidence than does knowledge.”

Charles Darwin was dead on the money when he said that, and it’s now known as the Dunning-Kruger Effect. Wikipedia says:

[…] unskilled individuals suffer from illusory superiority, mistakenly rating their ability much higher than is accurate. This bias is attributed to a metacognitive inability of the unskilled to recognize their ineptitude. Actual competence may weaken self-confidence, as competent individuals may falsely assume that others have an equivalent understanding.

David Dunning and Justin Kruger of Cornell University conclude, “the miscalibration of the incompetent stems from an error about the self, whereas the miscalibration of the highly competent stems from an error about others”.

I’ve seen it over and over again in social media. People believe that because using Twitter and Facebook is easy, that producing a meaningful long-term strategy for a multinational company is therefore easy, and that their intern can do it. After a decade as a social technologist, I can tell you from experience that it’s really not easy, and no, your intern cannot do it.

You see it in web design, which is what I did before I moved into social media. Because anyone can learn to throw a bit of HTML together, they think that it’s easy to design a website. Again, from experience, I can promise you it isn’t.

The problem is that people are generally very bad at accurately assessing their level of skill in any given area, especially an area in which they are inexperienced. That’s bad enough in a field where there’s an objective measure of capability. You may think you’re the bees knees at tennis, but if you keep losing every game you play, that’s a fairly clear indicator that you’re crap. And it’s not just an indicator to you, it makes it obvious to everyone that you’re crap, so it becomes hard, though not impossible, to maintain the delusion that you’re good.

With writing, however, there is no such clarity. The factors influencing the quality of a book can be broken down into three categories:

1. Objective factors: Spelling, grammatical and punctuation errors. Formatting errors. Inconsistencies. Issues that, no matter the reader, are obvious and should have been avoided. Much of this stuff could be picked up by a well-written algorithm.
2. Subjective factors: Poorly drawn characters, unconvincing plots, poor dialogue, cliche-ridden prose. Problems that many people will find problematic, but that some people will be able to successfully gloss over when reading. More experienced and professional readers/writers will notice these more than those who are less experienced. A computer couldn’t spot these problems, but us humans can, although the extent to which we are bothered by them varies.
3. Matters of taste: Tone, genre, aspects of plot or character. Other issues that really can’t be said to be good or bad, but which either fit your taste or don’t. Computers have no sense of taste.

The problem is that if you’re unskilled, it can be hard enough to spot the objective errors, but the subjective problems are well beyond your ken. Yet what often happens is that the unskilled are so overconfident that they try to classify subjective (and even objective) errors as a matter of taste, and thus something that they don’t need to address because hey, not everyone likes everything.

The Dunning-Kruger Effect is intractable, because it requires the unskilled to develop a high level of self-awareness to counteract their tendency towards overconfidence, and self-awareness doesn’t come easily. Again, from Wikipedia:

Dunning and Kruger proposed that, for a given skill, incompetent people will:

1. tend to overestimate their own level of skill;
2. fail to recognize genuine skill in others;
3. fail to recognize the extremity of their inadequacy;
4. recognize and acknowledge their own previous lack of skill, if they are exposed to training for that skill.

So, according to Dunning and Kruger, in order to combat the massive shit volcano, we would need to train every self-publisher who produces shit, and hope that they realise that they aren’t as good as they think they are and need to try a bit harder. Well, good luck with that one.

Now, it’s true that not every self-published author is on the wrong side of Dunning-Kruger. Some are on the only slightly less wrong side: Good writers whose confidence is shot because they understand that they could be better, and are over-sensitive to the gap between the quality of the work they do produce and the quality they want to achieve. Those people are better than they think they are and will publish less than they should.

Of course, there are self-published authors who have an accurate view of their own competence, and others who are moving up the competence ladder and developing a better appreciation for their own skills and what more they need to learn. Here, it’s useful to think about the Four Stages of Competence (again, from Wikipedia):

1. Unconscious incompetence
The individual does not understand or know how to do something and does not necessarily recognize the deficit. They may deny the usefulness of the skill. The individual must recognise their own incompetence, and the value of the new skill, before moving on to the next stage. The length of time an individual spends in this stage depends on the strength of the stimulus to learn.

2. Conscious incompetence
Though the individual does not understand or know how to do something, he or she does recognize the deficit, as well as the value of a new skill in addressing the deficit. The making of mistakes can be integral to the learning process at this stage.

3. Conscious competence
The individual understands or knows how to do something. However, demonstrating the skill or knowledge requires concentration. It may be broken down into steps, and there is heavy conscious involvement in executing the new skill.

4. Unconscious competence
The individual has had so much practice with a skill that it has become “second nature” and can be performed easily. As a result, the skill can be performed while executing another task. The individual may be able to teach it to others, depending upon how and when it was learned.

But if the quality of self-published books is anything to go by, most self-publishers are at stage 1. Very few have made it through to stage 4., though I think that’s true of all authors, even the traditionally published ones. It’s a very high bar after all. What we really need is more people getting as far as stage 3. Conscious competence is a perfectly fine place to be, but it is hard to get to with Dunning-Kruger in the way.

There is no intervention that I can think of that will help people, en masse, transcend the Dunning-Kruger effect and elevate themselves to a state of conscious competence as writers. Thus, we can expect the shit volcano to keep on spewing for the foreseeable future, and this without even beginning to think about the cultural reasons why there might be many people who are so eager to be authors.

Notes for commenters: I’m not talking here about people who just write for fun and give their work away on sites like Wattpad or in fanfic communities. I’m talking about people who are selling their books and, through asking for money for their work, presenting themselves as professional writers.

{ Comments on this entry are closed }