Some of the coolest things I did in my Information Technology career occurred during my time as a student and employee of Wright State University, and those I wrote about back in 2018 can be found at https://lifeexperiences.paulishing.com/2018/08/wright-state-days_10.html. In this blog, I’ll cover the ten coolest things I did during my employment at a string of paper companies, including Mead, MeadWestvaco, NewPage, and Catalyst Paper, selected from a list of thirty that I fondly remember.
1. Zellerbach Network Design
What made this cool is that I beat AT&T at their game.
The Mead Merchants division had just acquired the Zellerbach distribution business, and as the Manager of Mead’s Network Services group, it was my responsibility to get their thirty locations added to our Wide Area Network (WAN). We needed a design that would (a) only have four locations on a single line, (b) each line would run at no more than fifty percent utilization, (c) their major hub locations had to have two separate lines for redundancy, and (d) be at the lowest monthly cost. We based each location’s utilization on comparable sales data from Mead Merchants locations and gave the above requirements to AT&T to design the best solution.
At the same time, I was taking an MBA-level course in Operations Research, where we were learning how to use Integer Programming (IP) and Linear Programming (LP) to solve business problems. Our final assignment was to find a problem to solve and use one of those two techniques to solve it. I thought that maybe the Zellerbach network design would be a candidate, and spent a week thinking it through. I decided to make that my project even though I knew I would need to solve a very, very large IP. I decided to use Mead’s mainframe-based software as I would need a lot more computer resources than I could get via the university. I called up the company’s expert in that software, and he gladly showed me the format of the input card deck to describe the matrix and set the proper solution constraints. To generate that matrix, I wrote two PL/1 (Programming Language/1) programs that produced over 20,000 columns of potential city combinations. Each row represented a city, with two rows used for the major hubs. The intersection of each row/column was the monthly cost of using that line if that one was selected. Then the software was told that each location had to be selected only once and solve the matrix for the lowest cost. I submitted my program to run at the mainframe’s lowest priority to avoid causing interference with our production systems and left for the day.
When I came in the next morning, I was surprised to see my program was still running. I cancelled the program and went about trying to find out what was going on. I finally figured out that it was spending all of its time reading and writing to its temporary file because, as I previously stated, it was a very, very large IP. But since I was a technical mainframe guy, I knew the solution; I just put that temporary file in memory. I submitted the job again that evening, and it successfully ran to completion. There were a few dozen more iterations as I tweaked the matrix, some that ran successfully, and others did not. I was pushing the mainframe and the software to its absolute limit, but eventually found the optimal solution to the problem. We asked AT&T to price the solution I had created, and it was $100 less per month than the best solution their world-class INOS tool had found. The two lines to each major hub location tripped up INOS.
I gathered all the documentation for my final report, about an inch thick, and gave it to the professor as the class began the final exam. I saw the professor looking at the various submissions. He would look at one for a minute or less and flip to the next one and do the same. Then he came to mine. He read for about five minutes, then stood up, walked around the room, still reading, and walked out the door, returning several minutes later. I think the scope of the problem I tackled and all the various skills I had to use just blew his mind. And yeah, I got an A+.
2. The Business Meeting
What made this cool was that I figured out how two people could both be right.
Of all the cool things I’ve done, this one took, by far, the least amount of time to accomplish, about forty minutes. I was invited to a meeting with a bunch of business folks, and I had no idea why I was invited. As the meeting began, I had no clue what they were talking about; I had not heard the business terms they were throwing around before. After a few minutes, the only clear thing was that two guys had competing proposals for some kind of business change. At that point, I faced two choices. I could just sit there and listen politely, or I could try to figure out what they were talking about. I chose the latter and further decided I would try a novel tactic: I would try to figure out how both guys could be right. What small detail could there be that one saw working one way and the other differently?
Over the course of the next half-hour, I slowly started to understand their business terminology. Fortunately, the conversation eventually turned to something I was familiar with: how several business systems were integrated. I slowly built a mental model of how each guy saw it working, and after thirty minutes, I thought I understood that one small detail they saw differently. I continued to listen for another ten minutes to validate my hypothesis as the two guys continued to advocate for their solution. At the forty-minute point, I said I might have something and asked if I could draw on the whiteboard. Permission granted, I drew one integration diagram on the left side and a slightly different one on the right. I then asked one guy if the left side accurately represented his view, and he said yes. I did the same with the other guy and the right side. He said yes also. Then I pointed to the one version that I knew was the way it worked. The two guys stood up, agreed on the solution based on their new, shared understanding, and the meeting was over.
3. NewPage’s IT Objectives
What made this cool was how simply organizing objectives led to so much more.
I joined the NewPage Corporation in early 2008, and one of the first meetings I attended was the CIO covering the IT department’s objectives for the year. It was a great list and was going to be a challenging year, but I felt it was trying to tell me something, and I just couldn’t put my finger on it. After the meeting, I asked the CIO if he would send me the slide deck, as I wanted to try to figure out what was bugging me. He sent me the PowerPoint deck, and I spent a few hours over the next few days staring at it. Slowly, things began to coalesce and make more sense. I started moving the objectives around into groups, and then it hit me. All the objectives sorted themselves into three buckets: Changing the Business, Changing the Business of IT, and Delivering IT Services. Under these categories, there were several common themes, which I listed. Under those, I placed all the objectives the CIO had listed. Now it all made sense and was far easier to understand. I sent the updated slide deck back to the CIO, and he appreciated the extra clarity. But that was just the beginning.
In the new format, Changing the Business was listed first and for good reason. That was the primary reason the IT department existed and the reason we had a CIO in the first place. Now, that message was delivered with renewed clarity. The first thing the executives saw, the first thing the business units saw, and the first thing all the IT employees saw was Changing the Business. That was a very powerful message. It also served as a reminder that resources from the business units would be needed to make those changes a reality.
Changing The Business of IT told everyone that we were spending time, money, and energy making ourselves better. Delivering IT Services is about the importance of the day-to-day operation that every business unit counts on. Everything we did could be listed under one of these three high-level objectives.
The NewPage Human Resources (HR) department ran the system where all objectives and their progress were tracked. They wanted all groups to link every objective for every employee to an objective for the group’s leader, but they never gave us any guidance on how to actually accomplish that. Now we had a way. We loaded all the CIO’s objectives using the new format, and each employee’s objectives would link to a lower-level objective, but if one didn’t exist, it could be linked to one of the three highest-level objectives: Changing the Business, Changing the Business of IT, or Delivering IT Services.
We used the new format for other purposes, for example, scorecards on project progress. What started as an idea that a list of great ideas could be better organized led to a profound shift in how we viewed ourselves and how others viewed us.
4. Four-Hour Monthly Outage
What made this cool was how it turned into a sense of pride.
Until Mead implemented the SAP ERP (Enterprise Resource Planning) system in 2000, major maintenance of the computer systems occurred between midnight and 8:00 am on Sunday mornings when the business did not need access. The paper mills did not need the centralized applications to make or ship paper, so they were good. That all changed with SAP because the design decision was made to ship paper out of SAP. Any SAP outage would impact their ability to keep the flow of paper going, and, worst case, the paper machine might have to be shut down, an event that would cost Mead a lot of money. I had to find a way to minimize the outages, knowing they could not be eliminated.
After some thought, I decided that we could have a single monthly outage lasting four hours. We would have to work together to fit all the needed changes in that window and perform as many changes as possible concurrently. I presented that idea to the technical people, figuring they would really like the idea of only having to come in at midnight once a month. Their first reaction was that there was no way we could fit all the changes in a shorter, single monthly outage. They really bristled at not having the systems to themselves. They would gladly continue interrupting their every weekend to be left alone. But I insisted we had to try.
The process we used to plan the outage started by having all change requests submitted a couple of weeks prior, and an initial planning meeting from 3 pm to 5 pm ten days before the outage date. We discussed the changes, and the group would design an initial plan. The Outage Coordinator would take all the input and create a Microsoft Project project plan, which was reviewed by the group the following Thursday, where any small corrections would be made. It wasn’t a bad process, and at best it worked okay, but I knew we needed something better.
At the next month’s planning meeting, I told the group we were going to try a different method. For the first hour, we would discuss all the changes, their importance, and their operational requirements (e.g., certain systems up or down). For the second hour, the Outage Coordinator, two volunteers from the meeting's participants, and I would work on a more detailed plan. After that first hour, I asked the group (about a dozen people) for two volunteers to stay for the second hour and help design the plan. Nobody offered, so I had to pick the two based on the changes being made. Everyone else hustled to the exit. The four of us designed a much better plan, and we had our best outage so far.
We followed the new process at the next meeting, and when I asked for two volunteers, I got more than that. Apparently, word was getting around that the second hour was fun. At the next month’s meeting, everyone raised their hand. After another few meetings, I told the Outage Coordinator that I was bowing out. I really wasn’t adding any value at that point. A year later, I decided to check up on things and visited at the end of the second hour. I found the manager of the mainframe group, dry-erase marker in hand, staring at the whiteboard with all the changes written on sticky notes, saying out loud, “I just know we can fit all this in.” All was good.
There are hundreds of steps during an outage, for example, locking out users or bringing down a server, and I noticed that several of those steps would not go smoothly. With only four hours from start to finish, these operational missteps were costing us ten or more minutes each outage and were often the same issue over and over. I suggested to the Outage Coordinator that we write down everything that went wrong, no matter how small, and he agreed. He brought in a large paper flipchart and placed it in the middle of the room. We would listen to everything going on, and when something didn’t go right, we wrote it on the flipchart. On Monday, he would take each item and turn it into a problem ticket and assign it to the appropriate group to resolve. After a few months, the outages ran smoothly, and the wasted time was eliminated.
A couple of years later, MeadWestvaco would outsource all computer operations to Xerox Business Systems (XBS), including the people involved in the above process. I was told later on that they had talked to other XBS support teams that also supported SAP outages, and they wanted to know how we managed a once-a-month, four-hour outage. They proudly explained all the steps and how everyone worked together to make it happen. Other teams just shrugged and said they could never do that.
5. Data Center Consolidation
What made this cool was how I came to lead the project.
I joined Mead’s Technical Services group in 1981 as a systems programmer after having similar positions at Wright State University and Hobart Corporation. I didn’t have primary responsibility for any of the major system components, but served as a backup or an extra resource as needed. Mead had several data centers, and in 1982, it was decided to fold the Cincinnati data center into the Dayton data center, where I worked. That was announced to us at a meeting one Friday in Dayton. After the meeting, I mentioned to someone that it would be a cool project to lead; although we had several people with far more experience than I, I hoped to be involved in some way.
A follow-up meeting was held the following Friday in Cincinnati. I parked my car, and as I was walking to the venue, somebody came up to me and said they had heard I was running the project. That was news to me, and I said I hadn’t heard that, but I would be excited if that was true. Surely someone would have told me, or maybe asked me, before that. It turned out to be true. I don’t know if they chose me because someone had heard me express interest the week before or simply because I was the easiest person to make available.
It took about three months to plan, test, and execute the move of the Cincinnati data center. I worked seventy hours per week over that stretch, collaborating with various teams to identify the differences between the two data center setups and stage the necessary changes. Instead of one big move, we decided to move one division per weekend, starting with the division that would have the smallest business impact in case things went awry. The first move went well, and we had some lessons learned that we could incorporate into the second move. That move also went well, and then took the next weekend off for Thanksgiving. The last move was the largest, most complicated, and amazingly, it also went with no major issues. Time to go back to my day job.
A few weeks later, my boss stopped by my office and said the CIO wanted to see me and Jerry, my right-hand guy during the project, in his office. We were so nervous about being called to the big boss's office, and we were sweating bullets, wondering what we had done wrong to be called on the carpet. The CIO just wanted to thank us for our efforts and presented us with spot bonus checks. On the way back down the elevator, we couldn’t decide if we were happier about not being in trouble or that we had some extra spending money.
6. Network Silo Design
What made this cool was how an unexpected outage turned into a better network.
I have this saying that I call “looking for the root, root cause.” When problems happen in the Information Technology world, most people will find the “root cause”, which they define as the things that broke. So it was one day back in the early 2000s when we experienced a network outage in Mead’s World Headquarters building, impacting a number of critical business applications. The problem was quickly determined to be a network switch that had gone haywire, and a simple reset of the device cleared up the issue. To the technical folks, they had found the “root cause” and fixed it.
But I dug deeper and found that the network switch was part of a Local Area Network (LAN) that served PCs in employee offices. How could that affect business applications that run in the computer room? I needed to find the “root, root cause” of the problem’s impact. After digging deep, the best metaphor I could come up with was that our network had grown larger over the years, and it resembled a balloon that had grown bigger and bigger, with all the network equipment in that one balloon. I had no clue how to fix that, so the network team scheduled a session with the IBM network experts in Raleigh, NC. We had used them several times in the past, and they proved themselves as excellent facilitators and network experts. In the first meeting, I described our network and how the network switch caused the large outage.
Robert Brinkman, IBM’s lead network expert, said that other larger customers had moved to a “five-level network”. After he explained that a few times, I finally understood the concept. The network is divided into “silos” based on their purpose, for example, a silo for production systems, a silo for test systems, a silo for the Wide Area Network (WAN), etc. The devices in the silo are attached to high-speed switches, and they can directly communicate with each other. But to communicate with devices in another silo, they had to exit the silo via a router. The router not only moved the data, but more importantly, protected the silo from the type of issue we had encountered. Armed with this new network design, we headed home ready to implement it.
The first task was to document all of the existing network devices and how they interconnected. I created a one-page diagram with boxes (i.e., routers, switches, etc.) and wires (e.g., cables) and moved them around until I minimized the number of wires that crossed each other. I find crossing wires visually confuses people, and I think I only had two or three that crossed. As soon as I validated the accuracy of the diagram, I built a second diagram that had the silos we had chosen, about nine of them, with vertical lines separating the silos. Then I moved all the boxes to the silo they belonged in. Every place a wire crossed the vertical silo lines was a device we had to move to the proper silo. This created a visual roadmap of the changes we needed to make, and there were quite a few of them. I had it printed out on a plotter, creating a huge 4’x6’ printout that I hung on my office wall. The team worked diligently over several months to make those changes, and I would reflect that on the diagram until at last everything was in its proper place.
As we made those changes, the number of network outages decreased almost magically, and their impact was localized to their silo. We had always performed most network changes during the Saturday night outage window, and we were now allowed to make changes inside the silos that supported non-production systems during normal business hours, which the team really appreciated. I used the final diagram to educate our internal audit staff, taking our big, complex network and reducing it to ingestible pieces. The project was a lot of work for my team, but the results were totally worth it.
7. Excel Performance Problem
What made this cool was how I solved it while kneeling in church.
One of the consultants the NewPage Corporation contracted with gave me a call one Friday afternoon, asking me to stop by and take a look at a performance problem with Microsoft Excel. The user said it took 2-3 hours to recalculate the spreadsheet, and they were requesting a faster PC to reduce that wait time. I brought up the Windows performance monitor and had them kick off the recalculation. Instantly, all four cores on the CPU went to 100% utilization, and no disk input or output was seen. A lot of people think PCs are just slow, but when it comes to their CPU, that’s rarely the bottleneck that causes this type of problem. I had them send me the spreadsheet so I could take a deeper dive over the weekend.
I briefly looked at the spreadsheet, which was very large with lots of data in it, on Saturday, but nothing immediately jumped out at me. During church on Sunday morning, my mind drifted back to the problem, and I started thinking of what could cause this type of multi-hour CPU burn. This is very much not my favorite way of solving problems. I don't like the “let’s make a guess and see if that’s it” approach. I prefer gathering data and looking at the details, but since I was kneeling in a church, that wasn’t going to happen, so I thought of what could cause Excel to loop over and over again. I knew from experience that large table lookups can take some time, particularly if a linear search is used. A linear search is where you start at the top of the table and go row-by-row looking for a match. But even if you had a million rows of data, that wouldn’t take all that long. But if you search those millions of rows over and over and over, it will eventually add up. In Excel, those table lookups use a function called VLOOKUP. So I needed to look at the spreadsheet and find all the VLOOKUPs in search of the cause. I thanked God for the inspiration and returned to paying attention to the mass.
When I got home, I searched through the spreadsheet and found the VLOOKUPs. I found the VLOOKUPs; some were contained in loops that called other VLOOKUPs, and those contained more loops with even more VLOOKUPs. I had found the source of the problem, but how to fix it? I knew I had to replace the linear search form of the VLOOKUP with its binary search form. In a binary search, the data must be sorted from lowest to highest values, which allows the search to cut the table in half, looking to see if the desired match is in the top half of the table or the bottom half. The table gets cut in half over and over until the value is found. A linear search of one million rows takes, on average, five hundred thousand compares. A binary search of the same million rows takes no more than twenty compares to find the value, a 99.996% reduction in the number of compares. Now the plot thickens. The binary search form of VLOOKUP returns the closest value, whereas the linear search form will return “not found”. I didn’t know how to resolve that, so I did a few Internet searches and found someone with a solution. Each linear VLOOKUP would need to be replaced with two binary VLOOKUPs. While that would double the number of total VLOOKUPs performed, the binary form is so much faster that it probably wouldn’t matter much.
Getting into the office on Monday morning, I sent an email to the spreadsheet owner explaining what I had found and that he needed to sort the data and change the VLOOKUPs to the new format. Early that afternoon, he sent a message back that the spreadsheet was now recalculating in 2-3 seconds and that he was so shocked he had to double-check the answer to make sure things were still working properly. And he no longer wanted a new PC.
8. The Oracle Contract
What made this cool was how we saved big money with a timely question.
One of the many cost-saving opportunities I took on during my career was to review our Oracle contracts to identify potential savings. I was surprised to find we spent about $250,000 a year on Oracle’s database software, since we primarily used two other database systems, Microsoft’s SQL Server and IBM’s DB2. I pulled all the contract information, and I was surprised we were paying for maintenance on several Oracle licences we had not used in quite a while. I thought my job was going to be easy, just cancel those licenses and save a bunch of money. But Oracle made that much harder due to the structure of their contracts.
What I came to learn is that we had three Oracle contracts, and each of those contained multiple software licenses. We could not simply cancel a license; we had to cancel an entire contract, so even if a contract had one active license, you paid maintenance on all the licenses, active or not. Now it looked like it was going to be impossible to save any money. But I wasn’t done trying.
I analyzed the contracts more deeply, looking at the applications using each license, then deciding which might be the easiest to eliminate and which the most difficult. The easiest contract had only one active license, used by one of the applications that connected to our ERP system and supported by the same on-site team. I walked over to that team’s lead, Mark, and explained we could save $80,000 by canceling an Oracle contract if it was possible to convert that application to SQL Server. Mark was ecstatic! His team didn’t like Oracle very much, preferring SQL Server. Mark then said that his team was at the beginning of a project to upgrade that application and that the vendor provided automated scripts to convert from Oracle to SQL Server. He said he would add the conversion to the plan and that he would let me know when they turned off that Oracle database. It took a couple of months for the upgrade to be completed, and I let the contracts manager know we could cancel the first of the three contracts, saving $80,000 per year. I think this personifies the saying, “The harder you work, the luckier you are.”
9. Replacing The Mead Building Coax
What made this cool was not getting fired.
In the 1980s, all Mead offices had a phone and an IBM mainframe terminal. The phone was connected to the PBX (Private Branch eXchange) system using unshielded twisted pair wiring, the kind used in your average house. The terminal was connected to the computer room using a coax cable that ran all the way from that office to the computer room on the 16th floor. That setup was still good when Personal Computers (PCs) were added; we just put a coax board in the PC. But that setup was a challenge when Local Area Networks were introduced, which required a Token-Ring or Ethernet connection. We began running a third wire to these offices, one that used shielded twisted pair wires. In fact, we didn’t need the coax or the unshielded twisted pair wiring anymore; we could use the shielded twisted pair for all three needs.
We drafted a Capital Project Request (CPR) to remove the old wiring and replace it with shielded twisted pair. We presented to the I.T. Director (my boss's boss), and he unexpectedly turned it down. My team came up with another way. We knew that major office remodeling happens every 3-5 years on all the Mead floors, so we got together with the facilities people. They agreed to pull the old wiring and put in the new wiring whenever a major remodel occurred. It cost less money to put in the new wiring, so they were happy to save money. We were happy that the capital costs incurred were charged to the project and did not use I.T.’s capital budget. Over the next several years, remodeling projects slowly removed all the old wiring. A real win-win.
One day, the I.T. Director came to my office on a mission. He was tasked to get rid of coax cables because if they burn, they can release toxic polyvinyl chloride (PVC) gas. He asked me how much coax we had in the building. Half in a state of shock, I told him the truth. There were four coax cables in the building, and they served cable TVs on the executive floors. I explained how we worked with the facilities people over the past years, that we saved Mead money, and didn’t use I.T.’s capital dollars. He just sat there, looking at me. For a long, uncomfortable time. Maybe he would fire me on the spot for doing a project that he had rejected. Then he just got up and walked out of my office. I never knew what he was thinking, but I think I might have missed a bullet on that one.
10. Writing Digital Filing Apps
What made this cool was that its greatest value was never imagined.
When Canadian-based Catalyst Paper bought the Rumford, Maine, and Biron, Wisconsin paper mills from Verso, the people who went to Catalyst Paper, including me, moved into a single-floor office, where everybody got to know each other. It was the first time that I mingled every day with the business folks instead of mostly other I.T. people. One of the things I noticed was that their business processes involved a lot of paper. Now, being a paper company, that might seem okay, but I knew digitizing some of these processes would be valuable. There wasn’t a lot of interest until the two-person Damage Control group started running out of filing cabinet space and asked me to take a look to see if they could go digital. As we discussed the need, I was happily surprised to find that almost all of the papers they filed away had come to them digitally. They would get customer forms and pictures of damage attached to emails. They would print out orders and other documents from SAP. They would download other documents from websites. We embarked on a proof-of-concept project to see if a digital filing system could meet their requirements.
I wrote the first iteration of the application using Google App Script and used a Google Drive for data storage. I built a small PC application to create a digital “stamp” that could be added to documents to make sure all the needed control information was attached. I wrote a couple of scripts that would automatically download files from a couple of large customers’ websites. Over the course of a few months, we continued to refine the digital filing system to meet their needs. Now that the concept had been proven, I looked for another application platform, knowing that Google App Script wasn’t mainstream enough, and selected Microsoft PowerShell. It was better than Google App Script in some ways, and worse in others, but it was also free. I rewrote the application in PowerShell and supported the application when the very occasional problem cropped up.
I retired a year or so later after China-based ND Paper acquired the business, knowing it was the right time to end my career. But it wasn’t quite over yet.
Two other business groups, Customer Service and Accounts Payable, contacted me the following year and asked if I would be interested in writing digital filing applications for them. They had the same problem; they were running out of filing cabinet space. Having the time on my hands, I agreed and spent much of 2019 tailoring a pair of applications for them. Another pair of happy campers. One of my former co-workers told me they showed management how much paper they saved by creating a folded-up, eighteen-foot banner that they spread apart at a meeting. I thought that was such a creative idea and really made the savings visible.
The three digital filing systems all met their objectives, and no more office space or filing cabinets were purchased. But we all knew there was going to be a lot more value than that. Avoiding printing all the paper, at ten cents a page, was a nice saving. Being able to cover for a co-worker out sick for a day, on vacation, or on maternity leave was easier since everyone could access the files. An unexpected benefit was that fulfilling audit requests for data went from hours to minutes. But the greatest value occurred when the COVID lockdowns occurred, and everyone had to work at home. They worked from home, using a VPN connection, and all three digital filing applications worked just as well as in the office. I don’t know what the business would have done if these major business processes were still paper-based. It might not have shut the business down, but it would have certainly slowed it down quite a bit.
I began my career at Wright State University, writing their first online Admissions system while still a student. I ended my career writing three digital filing applications while my actual job was an I.T. Strategist. But as you can see from this collection of cool stuff, everything in between was a wild ride.
Here are a couple of bonus cool stories that I wrote years ago on my TechnologyViewPoint blog. They both belong in the above list, but I didn’t want to repeat myself. Along with the six Wright State stories at the link at the beginning of this blog, that makes 18 cool memories. But really, there were dozens more. Maybe someday I’ll get to them.
https://technologyviewpoint.paulishing.com/2016/05/the-pocket-directory-story.html
https://technologyviewpoint.paulishing.com/2012/07/5-50-500-5000.html
