Letter #13 the Twitter kerfuffle part 2 -and a note on why mDAU is not an odd metric

Whistleblower disclosure and why mDAU is not such a bad metric

(image created using Dall E)

A lot has happened on Twitter vs Musk saga since my last letter

I had written about how a lot of people were misreading twitter’s 5% claim (Potentially deliberately) , and how twitter claimed 5% of it’s mDAU users were spam and NOT all users. A good chart to summarise it is below.

Very recently, Twitter’s Ex Head of Security became a whistleblower(Source) and revealed a lot of details about its security practices and also Spam accounting.

Keeping the security bits asides, it seems that even the whistleblower, who typically would be very antagonist to twitter, more or less confirmed that what twitter was reporting all along was correct.

SPAM in mDAUs are ONLY the users who slip through their existing spam filters

Twitter, Zatko’s disclosure claims, actually considers bots to be a part of a category of millions of “non-monetizable” users that it does not report. The 5% bots figure that Twitter shares publicly is essentially an estimate, based on human review, of the number of bots that slip through into the company’s automated count of monetizable daily active users, the disclosure states. So while Twitter’s 5% of mDAU bots figure may be useful in indicating to advertisers the number of fake accounts that might see but be unable to interact with their ads, the disclosure alleges that it does not reflect the full scope of fake and spam accounts on the platform.

Executives are incentivized to avoid counting spam bots as mDAU, because mDAU is reported to advertisers, and advertisers use it to calculate the effectiveness of ads. If mDAU includes spam bots that do not click through ads to buy products, then advertisers conclude the ads are less effective, and might shift their ad spending away from Twitter to other platforms with higher perceived effectiveness.

However there are many millions of active accounts that are not considered “mDAU,” either because they are spam bots, or because Twitter does not believe it can monetize them. These millions of non-mDAU accounts are part of the median user’s experience on the platform. And for this vast set of non-mDAU active accounts, Musk is correct: Twitter executives have little or no personal incentive to accurately “detect” or measure the prevalence of spam bots.

Twitter announced a new, proprietary, opaque metric they called “mDAU” or
“Monetizable Daily Active Users,” defined as valid user accounts that might click through ads and actually buy a product. 19 From Twitter’s perspective, “mDAU” was an improvement because it could internally define the mDAU formula, and thereby report numbers that would reassure shareholders and advertisers. Executives’ bonuses (which can exceed $10 million) are tied to growing mDAU.

Unless you’re a Twitter engineer responsible for calculating mDAU, you probably wouldn’t know what Agrawal is talking about. He is not saying that fewer than 5% of all accounts on the platform are spam. He’s saying, more or less, that Twitter starts with all the accounts on the platform, tries to automatically put all the human accounts that could be convinced by advertisers to buy products (but no spam accounts) into mDAU, and then uses humans to estimate the error rate of spam accounts that nevertheless slip through into mDAU. And naturally, Twitter “can’t share” its special sauce for determining mDAU.

Even though it’s written in a very antagonist fashion, what Zatko is saying should be music to Advertisers and twitter BD teams.

It says that Twitter took great care in making sure ads were not shown to suspected fake users and voluntarily removed them from its monetizable pool. It further claims that Exec comp was tied to increasing this specific metric rather than the “Vanity metric” DAU.

This is a GOOD thing. Anyone who works in Ad tech or marketing would tell you that.

Sure twitter can do more to fight spam, sure spam makes user experience worse, but there is currently no evidence that twitter lied in it’s SEC filing..

He does reveal a lot of things that can potentially give Musk an escape hatch from the deal , or musk would need to prove that there were issues with tagging spam in mDAU

---

Is mDAU really that bad?

There is another aspect that I now see some people raising: That mDAU is a “made up” metric and has no validity. While you can have your opinions on this, I disagree on this assertion and this is what I really want to talk about in this pos

There are three premises to this argument that I hope to address in this article

Premise 1: Twitter is using a non- standard metric

Premise 2: It is a bad metric

Premise 3: Twitter is not measuring it right

Premise 1: Twitter is using a non- standard metric

Let’s be clear, there is NO standard usage metric that you are supposed to report . The government mandates public companies disclose financial data in certain format, but not how a company measures usage.
Every company decides what are the most important measures for it and reports them. Even the basic metric like what is a defined as “Active” can vary from company to company based on it’s footprints

Eg: Snapchat only looks at people who opened their app(Annual report)

We define a DAU as a registered Snapchat user who opens the Snapchat application at least once during a defined 24-
hour period.”

From snapchat annual report

Whereas Pinterest(Annual report) accounts for all kinds of actions including web visits

We define a monthly active user as an authenticated Pinterest user who visits our website,
opens our mobile application or interacts with Pinterest through one of our browser or site extensions, such as the
Save button, at least once during the 30-day period ending on the date of measurement

From Pinterests annual report

Companies, especially large ones, routinely create their own combination metrics that make most sense to them.

Eg: You won’t care much about how many times Uber app was opened, you care about how many users actually took a trip.

Uber has it’s own metric called Monthly Active Platform consumer

Monthly Active Platform Consumers. MAPCs is the number of unique consumers who completed a Mobility or New Mobility ride or received a Delivery order on our platform at least once in a given month, averaged over each month in the quarter. While a unique consumer can use multiple product offerings on our platform in a given month, that unique consumer is counted as only one MAPC. We use MAPCs to assess the adoption of our platform and frequency of transactions, which are key factors in our penetration of the countries in which we operate.

From Uber’s annual report

Similarly Facebook, which is a direct competitor of twitter, is also introducing a new metric called Daily Active People (Annual report)

Family metrics represent our estimates of the number of unique people using at least one of Facebook,Instagram, Messenger, and WhatsApp

From Facebook’s annual report

Premise 2: It is a BAD metric

In short: Twitter’s mDAU metric is the number of people who it can show ad to . Twitter removes potentially suspected bots, spam accounts, and also accounts only posting via APIs etc from mDAU count(See more details)

This is an ABSOLUTELY GOLD metric. If you are in the business of selling ads, making sure you only show ads to real humans is an extremely important measure.

Every company would have some measure of this. Twitter just chose to disclose this and make this their key metric. It is a strong signal that they are in the business of selling Ads.

Twitter can absolutely disclose and measure overall spam accounts, but that does not take away the validity of the mDAU metric.

I also keep hearing that twitter removes Bots and SPAM from it’s calculations of mDAU, do you know who else does that ? Pinterest . Here is a direct quote from their annual report

We regularly deactivate false, spam and malicious automation accounts that violate our terms of service, and
exclude these users from the calculation of our MAU metrics;

From Pinterest’s annual report

Pinterests Daily Active user seems functionally equivalent to twitters monetisable Daily active users. I see nothing inherently bad in this metric.

Another point to note: Facebook and Snapchat seem to not take out spam accounts from their Daily active user count. Facebook does mention how many active users it suspects to be spam, but I did not find anything related to that in Snapchat’s filings.

There is no consistency or rule on what is a”Daily active user”

Premise 3: Twitter is not measuring it right

Here is the process twitter follows, it has a bunch of AI / ML algos that automatically remove as many accounts as possible that. it suspects are spam . Twitter than samples 100 accounts per day(9000/quarter) from the rest of the accounts and have manual reviewers rate if these accounts were SPAM or not(Triple checked I read somewhere).

They do this everyday to get a trend and find that approx number of spam users that pass through their filters is <5%.

There are three objections I hear in this regard

Objection 1: The Sample size is too small

Statistical significance is not related too much to sample size, but rather to sample selection. Typically a sample size of 100 can give great representative results for a large population, twitter is doing 9000(over a quarter).

Eg: CNN did 2020 presidential election exit poll with a sample size of just 15,590

Objection 2: They do manual review

Of course they do. They already used their AI/ ML algos to filter out all spam they could and now manual is the last step. Infact, even facebook uses manual reviewers to tag spam

Facebook defines them as “Violating accounts” (Another non standardised name)

We define “violating” accounts as accounts which we believe are intended to be used for purposes that violate our terms of service, including bots and spam.

From meta’s annual report

It goes on to explain how they determine if an account is violating

Such estimation is based on an internal review of a limited sample of accounts, and we apply significant judgment in making this determination. For example, we look for account information and behaviors associated with Facebook and Instagram accounts that appear to be inauthentic to the reviewers

From meta’s annual report

Objection 3: it can’t be 5%

Do you know how much TOTAL spam facebook claims it has? 3%. NO Kidding

we estimated that approximately 3% of our worldwide MAP consisted solely of violating accounts”

From meta’s annual report

Assuming there is no lying, if facebook can have only 3% of all its users as SPAM, twitter’s 5% SPAM after SPAM filters does not seem off.

I do suspect though that I may have missed something and facebook also removes suspected SPAM accounts before calculating “Violating accounts”, which makes it potentially functionally equivalent to mDAU of twitter

Also remember, mDAU is NOT a user facing metric.Your own experience is immaterial. Twitter can have 20% SPAM and still have only 5% SPAM in mDAU.

So NO mDAU is not really that non standard, nor is it specifically bad , nor does it seem twitter’s revealed methodology is anything shady.

There can obviously be something deeply wrong with twitters count if they are hiding something, but I am unable to see any info about that yet.

Want to know about a real BAD metric that’s super popular and is also stated in annual reports? Read why NPS scores are useless