‘Technician error’ led to 3 percent loss of data storage this week
By JERRY NOWICKI
Capitol News Illinois
SPRINGFIELD — In its first two years of existence, the state’s lead technology agency was not equipped to handle technology disasters, maintained servers and computers with inadequate or nonexistent anti-virus protection, failed to implement cybersecurity controls, and did not properly document purchases or property inventory, according to a report from the Illinois Auditor General’s office.
The audit of the Illinois Department of Innovation and Technology — a state agency created in January 2016 through an executive order signed by former Republican Gov. Bruce Rauner — also found that an effort to consolidate financial, human capital and procurement functions for all state agencies will cost $150 million more than initially estimated over a six-year implementation period.
The Enterprise Resource Planning System, launched during former Democratic Gov. Pat Quinn’s administration and overseen by the Illinois Department of Central Management Services before being taken over by DOIT, will cost just under $400 million by 2021, up from an initial estimate of $250 million.
These findings were among 30 listed in Auditor General Frank Mautino’s report for fiscal years 2017 and 2018, the first two years of operation for the department created to “deliver best-in-class innovation and technology to client agencies.”
Jennifer Schultz, a spokeswoman for DOIT, said failure to execute the requirements of the executive order was due to a number of factors, including state government dysfunction.
“The executive order establishing DOIT called for an immediate unification, which did not fully account for the transition period needed to bring the IT offices of 38 different agencies into one department,” she said in an emailed statement. “These challenges were exacerbated by the budget crisis under the previous administration. Since that time, the department has made significant progress and we are confident the new secretary (Ron Guerrier) and a fully funded state government will allow us to operate at full capacity.”
Guerrier, a former chief information officer for Toyota, was nominated by Pritzker in March to replace Jack King, who resigned as acting director in February. He is serving in an acting capacity as he awaits confirmation by the Senate.
DOIT’s mission is to “transform information technology systems,” “protect state data and systems from cyberattacks and breaches,” enable agencies to “provide better services” and to “reduce costs and avoid inefficiencies through innovation,” according to the auditor’s summary of DOIT’s annual report to the General Assembly.
But auditors detailed several findings that showed the department directly failed at executing some of its most basic core functions, while also failing in several other accounting and managerial functions.
The audit, released Tuesday, July 9, said DOIT “had not developed detailed disaster recovery plans for the mainframe and midrange environments.”
“Failure to have tested comprehensive plans could result in agencies not being able to process critical transactions for an extended period of time in the event of disaster,” the report said.
On the same day the report was released, the Department of Employment Security was in the middle of a more than two-day outage of the system it relies on to distribute unemployment checks. The system was back online Thursday, with checks expected to be released Friday after a two-day delay.
Schultz said attempts to identify the cause of the service disruption led to a 3 percent loss of data storage on the state’s mainframe system due to “technician error.”
“The mainframe environment houses a variety of applications utilized by many state agencies, but does not include every state system,” she said in a follow-up email Friday. “While determining the cause of the service disruption, mainframe operations were ceased. A technician error resulted in loss of a portion of data in storage.”
She added: “(T)he vast majority of impacted systems are successfully operating. A small number of applications are currently in the final stages of being restored to operation.”
Auditors also detailed servers running with unsupported operating systems and without anti-virus protection. At the time of the report, 551 computers maintained by the department did not have up-to-date anti-virus software, and 3,692 computers did not have the latest anti-virus definitions.
According to the audit, DOIT could not produce documentation of weekly reviews of security records, annual reviews of which users had powerful security privileges or assessments of newly discovered vulnerabilities.
The report also found initial intergovernmental agreements between DOIT and other state agencies did not address “the security, processing integrity, availability and confidentiality of the user agencies systems and data.”
Another finding details the department’s lack of internal controls over third-party service providers.
According to the report, DOIT agreed with all 30 findings and attributed them to causes such as “oversight,” “lack of resources,” “poor communication” and “cash flow difficulties.” The audit also noted procedures had been, or would be, put in place to address several of the material weaknesses and deficiencies in the report, but it is unclear which of the findings have been rectified.
The department’s tracking of property was particularly lax, according to the report, which made note of 17 “missing computers” totaling $57,424 worth of property. Fourteen of those computers may have contained confidential information, as DOIT was unable to provide documentation as to what was contained on them.
DOIT was also unable to provide auditors with the purchase dates or prices for 2,305 items in its inventory, while another 6,928 items worth $5.7 million did not have purchase prices or dates initially logged.
Another 4,965 inventory items totaling $19 million were transferred to the department from other agencies but not recorded in DOIT’s records.
The department also failed to maintain records of more than $1 million of property that was deleted from its inventory, while hundreds of thousands of dollars in other property transfers and deletions were conducted without proper oversight.
Schultz said new systems are in place for tracking property.
“When DOIT was established, the IT offices of 38 different agencies were brought under the same roof. Many of these offices operated under different property control policies or lacked policies entirely. DOIT now operates under uniform established procedures and utilizes a computer application for intake and reporting. Inaccurate data has been and continues to be corrected,” she said in an email.
The report also faulted the agency for not cooperating with Democratic Comptroller Susana Mendoza, who questioned DOIT expenditures after defeating Rauner-ally Leslie Munger in a 2016 election.
The report said Mendoza was holding $124.9 million in vouchers submitted to her office without proper documentation from DOIT, accruing $20 million in interest. A spokesman for Mendoza said the comptroller’s office was working with DOIT to determine which of those vouchers are valid.
“More than two years ago, we flagged some of the problems cited in this audit,” Mendoza said in a statement. “We noted the ‘ERP’ computerizations was a program in crisis. This independent investigation validated our actions, which included asking tough questions, bringing it to the attention of the public and stopping payment to vendors. While the audit finding is embarrassing for the prior administration, what's worse is that under Governor Bruce Rauner's leadership, his agency cost taxpayers hundreds of millions of dollars with no substantial results to show for it.”
The audit also detailed $4.1 million in overtime paid to staff without proper approval, poor documentation of emergency purchases, inadequate controls for state vehicles and employee cellphone contracts which were not canceled until nearly a year after employees left the department.