The Centrelink Robo-Debt Error Rate Is Higher Than 20%

When is an error not an error? According to the Department of Human Services, it’s when

Initial notices request information to explain differences in earned income between the Australian Taxation Office and Centrelink records.

This number has been bandied about a lot in the last few weeks. And the Department has been sticking to its line that

The remaining 20 per cent are instances where people have explained the difference and don’t owe any money following assessment of this updated information.

Centrelink is taking income information from the ATO and matching it with its own records. It sends a letter when it finds the figures don’t line up. Exactly how it does this is something we’re all trying to find out. We know that—in at least some cases—the income information doesn’t line up because the matching process was simplistic and ham-fisted. Income has been counted twice due to the same business having multiple names, or income earned in only part of the year has been averaged over the entire year.

The 20 percent figure is only those who correct Centrelink’s incorrect information and end up with zero debt owing. We have no figures on how many in the remaining 80 percent have some debt, but are wrong in the amount. I’ve asked the Department of Human resources this question on several occasions now and have yet to receive an answer.

This means we don’t know, and I suspect DHS doesn’t know, some really important information:

  • Of all these letters, how many were sent because DHS didn’t catch obvious mistakes like “averaged income over the year for someone working casually/part-time” or “income has been double counted because the business names don’t match but they’re actually the same company”? These are letters that should never have been sent, and so are very clearly errors created by Centrelink.
  • Of the 80% with “some debt”, how many of them, and how much of the ‘debt’ amounts, were wrong because Centrelink made a mistake as compared to customers incorrectly reporting their income (mostly by accident, some on purpose)?

All The Data Is Suspect

We don’t seem to know how many of these “income discrepancies” are there because of the flaws in the the data matching process itself.

Some of the information Centrelink has is likely to be incorrect. When you have humans entering data, this is bound to occur, and trying to fix up those errors is a worthy goal. Indeed, it’s required by Australian Privacy Principle number 10 as part of the Privacy Act 1988. Most of these errors are going to be innocent misunderstandings, forgetfulness, or just typos. Some smaller amount will be deliberate fraud.

But what we have here is the data matching process creating brand new errors! It’s actively making the data Centrelink has worse not better.

And then, to add insult to injury, Centrelink is forcing its customers—citizens, tax payers, because we’re matching data with the ATO let’s not forget—to spend their time and energy dealing with Centrelink to correct data errors that were, in many cases, created by Centrelink while trying to fix data errors.

Without some idea of how often the data matching process is going wrong, we can’t have any confidence that any of the letters are right.

I just don’t see how you can justify continuing to throw money away ($1 a stamp now!) harassing your customers when you know that a lot (20%? 30%? Who knows!?) of your harassment is caused by your own cockup!

The Robo-Debt That Doesn’t Exist

After looking at where the headline $4 billion figure came from previously, we now turn our attention to another big number that’s getting bandied about the place: $1.7 billion in over-payments that the Government wants to claw back.

This number comes from the 2015-16 Federal Budget. The Hockey/Cormann budget. The “lifters and leaners”, let’s go smoke cigars within view of the cameras budget. You know the one.

Anyhoo, on page 116 of Budget Paper No. 2 [PDF] we find this statement:

The Government will achieve savings of $1.7 billion over five years by enhancing the
Department of Human Services (DHS) fraud prevention and debt recovery capability,
and improving assessment processes.
From 1 July 2015 DHS will implement an integrated package of compliance and
process improvement initiatives including improved automation and targeted
strategies for fraud prevention in areas of high risk.
The savings from this measure will be redirected by the Government to repair the
Budget and fund policy priorities.

Just under this table:

Strengthening the Integrity of Welfare Payments

The sum of that Total — Expense line over five years is $1,687 million, which rounds up to $1.7 billion. The actual amount the Government reckons will get clawed back is in that Department of Social Services line, a total of $1,926 million over five years.

Okay, how was that worked out? It’s not spelled out in the Budget, which is where Senate Estimates comes in, and questions asked of the Government about what this all means.

Improving Data Matching

The Budget was handed down on 12 May 2015, and a couple of weeks later, the Senate Standing Committee on Community Affairs asked some questions about it.

DHS has performed data matching of income data from the ATO for some time as part of its audit and risk management process, which is fair enough. It’s supposed to find fraud, as well as simple mistakes, so that DHS isn’t giving out taxpayer money to the wrong people.

The process had traditionally been pretty labour intensive, and only done a couple of times a year, because it took too long and cost too much to do it more often. Ms Kathryn Campbell, Department Secretary for DHS, told the committee “Using the newly developed streamlined intervention approach, the savings now far outweigh the cost of undertaking the activity and far outweigh the cost of the overall measure.” This new approach is the Welfare Payments Infrastructure Transformation (WPIT) Programme.

Cue Devo Karaoke.

It’s worth reading Hansard for 3 June 2015 [PDF] to get a good idea of what the programme is really all about, starting on page 67. It’s about finding mismatches between what people have declared their income was to DHS, and what they declared to the ATO.

Senator CAMERON: This is just targeting welfare recipients basically?
Mr Withnell: Yes. It is a target on those welfare recipients who have underdeclared their
income.

Page 77 also provides some insight into the programme.

Mr Withnell: Probably the easiest way to describe it is that most of the strengthening
integrity in the welfare system measure deals with historic overpayments but also provides
some money for development into the future, which will connect into the welfare payment
infrastructure.
Senator SIEWERT: I am having trouble hearing you.
Mr Withnell: With the budget measure most of the savings are historic overpayments, so
they are overpayments that have already occurred. Those savings, in a sense, are retrospective
because they have already occurred. In terms of benefits going forward there is a small
amount of funding within the compliance measure which allows us to take advantage into the
future of the developments in the welfare payment infrastructure transformation program.

So it’s all about getting people to pay back money that DHS paid them already, but shouldn’t have because the information in the DHS systems wasn’t accurate, for one reason or another. At least, that’s the theory.

But here’s where things start to go wrong.

Numbers That Don’t Add Up

On page 78 of Hansard, we find that the Budget figures above are probably wrong.

Senator SIEWERT: And they all add up to $1.7 billion?
Ms Campbell: I have not got that.
Ms Golightly: No, the $1.7 is purely to do with the business integrity measure.
Senator SIEWERT: Which is the fraud?
Mr Withnell: It is both.
Ms Campbell: I think the net number is $1.5 billion, because it is $1.7 billion in savings
and $200 million in expenses, which is a net $1.5 billion.

In the Budget, we saw $1.9 billion become $1.7 billion after the $200 million in expenses to set the system up were taken out of the debt recovered. But here we discover that the revenue is actually $1.7 billion, so we end up with $1.5 billion after taking out the costs of settings things up.

Senator Cameron asked about the business case underlying this program. It was taken on notice (Question 15) and an answer was supplied on 24 July 2015 [PDF]. The answer show us how this $1.7 billion was worked out. I’ve summarised it in the following table.

  DHS analysis Headline Version Error
Discrepancies with Debt (%) 85.00% 100.00% 15.00%
Number of potential debts 918,000 1,080,000 162,000
Likely average debt value $1,440 $1,440  
Predicted debt value ($’000s) 1,321,920 1,555,200 233,280
Additional recovery fee (10%) 10.00% 10.00%  
% of debt subject to recovery fee 100.00% 100.00%  
Recovery fee value ($’000s) 132,192 155,520 23,328
Total value estimated ($’000s) 1,454,112 1,710,720 256,608

To explain, in the answer to Question 15, DHS explained that, based on their analysis, there were about 1,080,000 instances of a mismatch between what people had declared to Centrelink they’d earned, and what the ATO says they earned.

Of those, about 85% would result in a debt, with an average value of $1,440. If you use the “mismatches” instead of the reduced 85% figure, you get $1.55 billion, but the real amount should be $1.32 billion (more on this in a moment. Stay tuned, it’s a doozy.)

Recent legislation also adds a 10% debt recovery fee to this money, which bumps our wrong total to $1.7 billion. If we add it to our 85% based figure, we end up with $1.45 billion.

Take away the $200 million the project will allegedly cost, and we end up at $1.2 billion. Half a billion dollars gone because of some dodgy maths.

These amount will also need to be further discounted based on how recoverable they are. We’ve already seen a big write-down in the receivables on DSS’ balance sheet for these alleged debts, so the real number is even lower than this. The discounts to the real number certainly do add up, don’t they?

And this is the “business case” justifying the programme.

Underpayments

Note how there’s nothing in the answer to Question 15 about the other 15% of the mismatches? If they’re a mismatch with the ATO, that means the amounts are not the same, so the debt can’t be zero. Since 85% are overpayments, that means this 15% must be underpayments. But what’s their average value? That’s money DHS should be paying people, so the value of the debt should be discounted by this amount.

At least, that’s what DHS should be doing, if Ministers Tudge and Porter, and the DHS actually meant the “no less” part of the line they’ve been spouting for weeks.

Let’s pretend DHS were planning to pay people who got underpaid. How big a problem is it, really? Well, 15% of the discrepancies is 162,000. The discrepancy to customer ratio is about 1:0.8, so if we assume the 15% has roughly the same mapping of discrepancy:customer that’s 129,900 people the Government owes money.

Now, the average underpayment could be very different from the average overpayment, so we’re getting even further out into inaccurate figures now, but let’s assume the average per debt is the same: $1,400. That would mean DHS owes people over $233 million. If DHS actually paid people the money it owes them, the savings from this robo-debt fiasco would be reduced down to under $1 billion. A far cry from the $1.7 billion in the headlines, eh?

Of course, it isn’t being included, because DHS has no intention of paying anyone that money.

So much for the integrity of the system.

Robo-Debt Targets Youth Allowance and Newstart

Another article in the ongoing #notmydebt saga today, this one increasing the level of fail.

Today we learn, via The Age, that the data matching process is targeting Newstart and Youth Allowance, and specifically not Family Tax Benefit recipients. Department of Human Service spokesperson Hank Jongen:

“This has primarily involved Newstart and Youth Allowance.

“This compliance activity does not apply to non-income support payments like FTB.

This is all about getting back money that shouldn’t have been paid out, right? It’s about balancing the budget, being sensible with money and all that, yeah?

Really?

Not The Bulk Of The Money

Let’s take a look at where all the overpayments are, since we just recently dived into the Department of Social Services’ financial reports to find where this magic $4 billion number came from.

Here’s the value of the receivables for “Recovery of personal benefit payments”.

Personal Benefits Receivables (Source: Department of Social Services’ Annual Report for FY 2015-16).

There’s more detail on what this $4 billion figure is made up of in a footnote.

Components of Personal Benefits Receivables (Source: Department of Social Services’ Annual Report for FY 2015-16).

Here it is in a table.

Payment Type 2016
($millions)
% total 2016 2015
($millions)
% total 2015
Family Tax Benefit 1,476 36.61% 1,419 35.20%
Parenting Payments 783 19.42% 698 17.31%
Newstart 647 16.05% 485 12.04%
Disability Support Pension 289 7.16% 254 6.31%
Total 3,195 79.24% 2,857 70.86%

And here it is as a chart.

Human Test Subjects

Why target Newstart recipients when Family Tax Benefit and Parenting Payments make up 56% of the total amount owing? Why go after the part that’s only 16%?

This might imply that the purpose of data matching isn’t really about getting money back, but about punishing specific groups of people. I think it’s something far more insidious: This is the test group for the first run of data matching. They’re the human test subjects for this experiment.

Either way, it shows that Centrelink is deliberately targeting a subset of all those who might have been overpaid.

Why this specific group was targeted is a line of questioning others might like to explore.

Centrelink And The Mystery Of The $4 Billion

The Australian Derpartment Department of Human Services continues to cover itself in glory* this week hundreds of people go public with their stories of incorrect debts being raised against them and the Kafka-esque bureaucracy that awaits anyone caught in its dragnet.

I’ve been digging into the financial justifications for the automation effort, and it’s a convoluted beast. The numbers being bandied about sound good in a headline, but figuring out the real numbers has been surprisingly difficult. Little wonder that the stories containing any of the figures, particularly the ones used by Ministers Alan Tudge and Christian Porter, are confusing at best.

Here’s my attempt to unravel what’s really going on here. I’ll provide links to primary sources, rather than media reports, where I can.

The Headline Number: $4 billion.

As best I can work out, this number comes from the balance sheet value of the amount of money paid to people that shouldn’t have been, for whatever reason.

The Department of Human Services operates the social welfare payment system (aka Centrelink) on behalf of the Department of Social Services (DSS). Both of them therefore report on the finances of social welfare payments, but they do it in slightly different ways. DSS has more of a governance and oversight role, while Centrelink is more operational.

We can therefore find financial information about Centrelink in DSS’ Annual Reports. It’s here that we can see, in the financial statements, a bunch of information about the money administered by DSS on behalf of the Government. This is different from the financials of DSS itself, which is more concerned with how DSS runs itself than how it looks after the Government’s money (even though it’s all ultimately Government money, i.e. our money).

For 2015-16, the important figure is in Part 4 (Financial Statements), specifically note 4.1A: Receivables.

Note 4.1A from the Department of Social Services’ Financial Statements from the Annual Report for FY 2015-16.

Here we can see that DSS expects that there is about $4bn of money paid as personal benefits payments that it should be able to recover, i.e. overpayments.

Importantly, there’s also an impairment allowance:

Receivables impairment allowances (Source: Department of Social Services’ Annual Report for FY 2015-16).

An impairment allowance how much money you don’t think you’ll be able to actually get back, for whatever reason. It’s writing off a bad debt, essentially.

Note the substantial change from FY 2014-15 (at the right) to FY 2015-16: $765 million dollar, or about 25% of the total balance. That means that DSS has re-evaluated how much of the debts (receivables) it thought it would be able to get back, and decided that it wouldn’t be able to recover quite a lot of it. 25% is a big change.

Why did this happen? Good question, and I haven’t figured that out yet.

There’s also this detail on Aged Receivables, which shows how overdue the debts are. As we can see, most of them, again around $4bn, are not at all overdue.

Aged Receivables (Source: Department of Social Services’ Annual Report for FY 2015-16).

Finally we get to a consolidation of all this stuff to see how much money DSS reckons it will be able to actually get paid in the coming year, and how much will take longer than that:

Net Receivables (Source: Department of Social Services’ Annual Report for FY 2015-16).

Just under $1.2 billion this year, and another $2.64 billion ‘eventually’.

An important thing to note here: with accruals accounting (which is what is being used here, like most organisations) you never get your receivables to zero. Receivables are used to track the difference in timing between when a debt is incurred (you owe the money) and when you actually get the cash. Whenever anyone buys something on credit, this happens. The bigger the organisation, the larger this number will be, because the total volume of payments is larger.

What we’d need to look at (and I haven’t yet) is how large this running balance of receivables should be. Is it too high? Maybe. But by how much?

The bottom line here is that it will not get to zero unless DSS/Centrelink can manage to never overpay anyone for any reason, ever. That’s unrealistic, because it would require Centrelink’s systems to be perfect, and no system is perfect.

And that’s okay! As long as the running balance isn’t too large (and we need to know what that amount is and why) and stays relatively constant over time, then the money is flowing in and out of the system nice and smoothly. There will always be errors and mistakes, because we’re dealing with humans and human made systems, which always have flaws. It’s how you manage those flaws that matters.

Double-entry book-keeping and accruals accounting have been developed for just this reason. Provided you do it properly, and that’s where we should be focusing our attention. What does a properly run system look like, and how different is that from what we currently have.

Unless you can clearly articulate that, then you cannot fix the system. And this is the big, big problem with the Government’s current approach.

The Other Number: $1.7 Billion

Or is it $1.5 billion? That’s the other number being bandied about, and I’ll address that in another post because this one is already long enough.

Questions?

Centrelink Response to Questions on #notmydebt

I emailed the Department of Human Services about the Centrelink #notmydebt fiasco that’s been in the news of late. The content of my emailed questions, and the DHS response are reproduced below. I’ve trimmed personal data and some header/footer guff, and played with some of the styling to make it more readable in this blog, but the content itself is verbatim.

My Initial Email

My first email of questions, sent at 10:39pm Thursday 5 Jan 2017:

Hi folks!

Busy time, I’m sure.

I’m a contributor at Forbes.com (http://www.forbes.com/sites/justinwarren) and I’m working on a story about Centrelink/ATO data matching you’d be well aware of at the moment.

Some questions:

  • Is the data matching system new, or an existing system that has been made ‘fully automatic’? If the latter, how much changed to make it fully automatic?
  • Who developed the automated data matching software and process? Was it done in-house, by external suppliers, or some mix of the two? Which suppliers?
  • What technology stack is used to underpin the data matching process? What is the main software system used, e.g. SAP?
  • Is the system maintained in-house by IT within DHS/Centrelink, or are external IT suppliers used? I am thinking of the example of IBM as used by the ABS for the Australian Census.
  • The false-positive rate (the rate at which debt notices are sent, yet the debts are found to be incorrect or non-existent) has been reported as at least 20%. Was this error rate known prior to going live with the more automated approach? Or did the error rate increase more recently?
  • Is there an SLA or similar for the system that defines the acceptable false-positive rate? If so, what is this rate?
  • The Minister is on record as saying “We’re determined to ensure welfare recipients get no more, and no less than they’re entitled to.” [emphasis mine]. Does the data matching process detect under-payment, and are people then contacted to ensure they receive any missing payments they are entitled to? Please describe the process by which this happens.
  • The rate at which letters have been sent out appears to have increased over time. A total of ~169,000 have been sent, and lately at a rate of 20,000 per week. How was the sending rate increase planned: linear growth from an initial number, or a non-linear growth pattern (5% of max, 10%, then 25%, 50%, 100% kind of pattern)? I’m trying to get a sense of how much volume the system was sending each week/month from July to December.
  • Does the system plan to sustain a sending rate of 20,000 letters per week indefinitely?

My deadline is 5pm Friday 6 Jan 2016, Australia/Melbourne time.

My direct number is [redacted].

Staffer Phonecall

I got a phone call from a DHS media staffer at 1:30pm on Friday 6 Jan 2017 explaining they were busy and I probably wouldn’t get an official response before deadline; I’d probably get an email over the weekend. I don’t do scoops, and I’m looking at this more from a “how to manage enterprise technology projects” angle, so I was okay waiting a bit for clear information.

The staffer did say a bunch of interesting things, and then, after talking to me for just on eight minutes and having said some very interesting things, he then said “That stuff was just on background. Don’t quote me.” I was very clear that I was press, what with emailing the media contact address planning to write something in Forbes.com and all.

Followup Email

I then sent a followup email:

Thanks for the call earlier. I’d like to confirm my understanding/clarity about the 80% real, 20% “needs more info” part of the data matching:

  • Please confirm that the 20% figure refers to letters that are sent that do not result in any debt owing after people provide additional information (mostly online).
  • With the 80% figure: is this letters that are completely correct and require no adjustment, or does it include a subset that do result in a debt, but not the amount matched originally? So, the letter might say “it looks like you owe us $2,500” but after they supply more info the actual debt amount is $1,000.
  • What proportion of letters sent end up as “debt exists and is totally correct” versus “a debt exists, but amount is adjusted after more information supplied”?
  • Of amounts adjusted, what’s the distribution of adjustment amounts (mean adjustment of $X with a variance/standard deviation of $n)?

DHS Response

I just got a response, at 4:20pm Monday 9 Jan, Australia/Melbourne time. Here it is:

Hi Justin,

Apologies for the delay. You may attribute the following response to Department of Human Services General Manager Hank Jongen.

Regards,

Kevin Room

Media Section

Response:

We are confident in the online compliance system.

The data matching capability has been developed and is maintained internally.

The data matching software has been used by the department to successfully carry out data matching operations for a number of years.

The business rules used to identify non-compliance are well established and have not been altered in the new system.

If there is a difference in the income recorded by the ATO compared to the income reported to Centrelink for a particular time period, the department is obliged to seek clarification of the difference in the data.

This is part of ensuring the integrity of the welfare system. If someone receives benefits to which they are not entitled, the department is required by law to recover the money.

The online compliance intervention system doesn’t automate debt recovery – it is a system which automates part of the standard compliance review process. The department always provides assistance where people request it and this has not changed.

While the new online compliance system automates part of this process, it also does not change how income is assessed or how debts are calculated. The system is an easy way for people to confirm details and resolve any outstanding matters.

The first letter generated by the system is a letter advising people that there is a difference between the information reported to Centrelink and the information reported to the ATO. This is not a debt letter and simply asks them to either confirm or update the information.

At this stage of the process no assumptions about debt or no debt has been raised. No decision to raise a debt or otherwise is undertaken until the person has been provided an opportunity to confirm or update the information.

The easiest way for a person to confirm or update their information is online, through their myGov account.

The letter they receive explains how they can go online. The letter also has a number they can call if they would rather speak with someone. At any time, people can seek assistance from the department.

By going online, people can view their information including checking the dates over which their income was earned, and if necessary update any information.

People have 21 days from the date of the initial letter to go online and confirm or update their information. People who do not respond at the 14 day mark will receive a reminder.

It’s possible to go online and ask for an extension to the 21 day timeframe – this is automatically granted on the first two occasions, before a third request for extension triggers a phone call from our staff to provide assistance.

There is also the option through the system to request a phone call at any time to discuss any issues the person may have. This is a call back option and the call back will occur within 7 days.

If the employment income was earned before they began receiving or after they stopped receiving income support payments, then they will not incur a debt.

When a person completes an online intervention, the system will advise the person of the outcome. If the event that money is owing, the outcome will be explained to the person on how the debt was calculated.

If a person fails to respond to the initial request and the reminder to confirm or update the information, a debt will be raised using data from the ATO as provided to them by the employer. This is not new and is consistent with procedures which have been in place for many years.

Even after a debt has been raised the person can change their details or provide more information to have the debt reassessed or they can request to have a formal review of their case, if required.

People have further appeal rights should they disagree with the outcome of the formal review.

There is no time limit to request a review of the decision with regards to debts raised by the department.

When a debt is raised a person will receive an account payable letter advising of the debt, and provided 28 days to pay the debt in full or contact the department to discuss payment options.

Current recipients receiving a payment who have incurred a debt that has not been paid in full by the 28th day are subject to automatic withholding arrangements, which can be negotiated depending on their circumstances.

If they are not receiving a payment that is they are not a current customer and contact has not made been made after 42 days (that is 28 days plus an additional 14 days), further recovery action may apply such as referral to an external collection agent.

Since July, 169 000 reviews have been completed. In approximately 80 per cent of cases people have a debt owing to the Commonwealth. The remaining 20 per cent are instances where the customers have resolved the matter by providing updated information to clarify the difference in reported income.

Only 2.2 per cent of customers were requested to supply supporting documentation, which means 97.8 per cent did not need to supply supporting documentation.

The department is committed to ensure that people get what they are entitled to, nothing more, nothing less.

My Followup Email

I sent a followup email straight away (4.44pm on 9 Jan 2016), because DHS haven’t answered some important questions from my original email, and my post-phonecall email.

Thank you for your response.

Please clarify the following:

  • When you refer to “Only 2.2 per cent of customers”, what do you mean by ‘customers’? Is this all customers of Centrelink, or something else? How many customers constitute 2.2 per cent?

I would still like an answer to the following questions:

  • Does the data matching process detect under-payment, and are people then contacted to ensure they receive any missing payments they are entitled to? Please describe the process by which this happens.
  • Is the plan to have the system sustain a sending rate of 20,000 letters per week indefinitely?
  • For the 80% of the 169,000 reviews conducted where a debt was owed, how many cases were entirely correct, i.e. “a debt exists and no adjustment was needed to the amount”, and how many required an adjustment to the amount “a debt exists, but amount is adjusted after more information is supplied”?
  • For debts that are adjusted, what is the distribution of the adjustment amount (mean adjustment of $X with a variance/standard deviation of $n)? Is the distribution statistically Normal, or skewed?