How to Drastically Improve IT Department Efficiency

Before discussing the best approach, it is worth noting that the CIO must be willing to dedicate a lot of time and resources. Existing development and maintenance doesn’t have to stop, but it may slow down depending on available staff. It is a very large endeavor, and requires big planning, patience and persistence to see things through. It is critical that the CIO completely understands this and is onboard. If not, the process will for sure fail.

1) Hire (or identify) a qualified Enterprise Architect who is capable of doing enterprise domain analysis. Depending on the size of the organization, this process may take anywhere from a month to over a year. This is the most important step toward improving things and cannot be sacrificed.

2) Once domain analysis is completed, begin translating the domains to real world entities. For example, if someone calls a person a “client” in one department, and a “customer” in another, investigate to see what the differences are and see what is in common. You may discover they are exactly the same thing, or you may discover a “person” domain entity would have almost all the required information and a “client” inherits from that and has an additional attribute called “client relationship” or something of that nature. Organizations often come up with their own terminology too. Don’t make your real world domains out of those. Instead, translate the organization’s terms to real world (based on a definition from a dictionary). The reason this is important is because a real world entity represents reality. If you begin creating entities based on an organization, they may be flawed because of lack of understanding by an end user, or simply because someone new gets hired that doesn’t want it to be that anymore. If you create real world domain entities that you translate organization entities to, your domains will be safe. Example: Someone calls something a “schedule”. It isn’t a schedule by the definition in a dictionary. It’s a client calendar day tied to a specific process in their eyes. You can always make a user interface say “schedule” but don’t fall victim of the domain entities actually being the flawed representation. Call it “personnel client schedule” or something like that instead, and again, map it to real world (may be a combination of 5 or more real world entities at that point).

3) After step 2, you may identify some flawed processes in the organization. If possible (and within reason), change the flawed processes. No good reason to develop around bad processes. Sometimes this isn’t possible due to authorization, politics, etc.

4) Begin to design databases with the domains separated for each. For example, most businesses deal with people, so have a “person” database that has common information about people: first name, last name, home address, home phone, etc. You may discover that some of the things you thought were simple domains, like “home address” may end up needing to be in its own “address” database. For instance, if you deal with land and property, you may need addresses that don’t belong to people (owned by a bank for example). This would be a reason to move addresses to its own database. As you can see, domain analysis is extremely complex and requires a good deal of time. But once things are separated out, you will see all the duplicate data you’ve been dealing with for so long.

5) Come up with a plan to begin translating existing systems to use the new databases. This process is highly dependent on the staff, system capabilities, architectures, etc. Web services and Service Oriented Architecture can greatly aid and simplify this process because of the loose coupling concepts. You may needs to have multiple systems up at the same time during transition. For example, you may have an old Mainframe application for dealing with employee information. You may have to create a new Java or .NET system that pulls this information in a batch process to a more available SQL Server or Oracle database (technologies are just listed as examples). You may also discover some of the data from the existing systems will no longer be needed once the new ones are in place. It could be there were duplicate feeds for a given process (or separate processes). All of these factors will help determine the timeline for shutting off existing/older systems.

6) Now that you’ve figured out what new systems you need and are very familiar with the domain entities, take a look at developing a solid framework. A framework should consist of not only persistence layers, but also user interface layers, logic layers, etc. You can use other 3^rd party frameworks as part of yours, but make sure you do have your own as well. It might end up just being wrappers around the others, but it will ensure you can solidify and protect your new systems since it will be under your control. Also, ensure the enterprise architect is familiar with various technologies and understands the lifecycle and risk associated. Stick with technologies that are sound and have a long future ahead to minimize re-writing in the future. Don’t be “bleeding edge” and incorporate something just because it’s cool or a new buzzword. Figure out first if it will be around for the long haul and determine its popularity. Don’t invest in technologies that your staff can’t support/understand or ones that will be impossible to find other hires to manage.

7) Begin development of the new framework.

8) Simultaneously with the previous step, begin architecting the new systems (back end and front end) based on the framework.

9) After the framework is solid and tested, begin implementation. The planning and architecture of the framework is actually the most difficult part, if done correctly. The implementation of systems may take some time, but it should all go smoothly if the plan and framework were constructed well. It can’t all be done at once, so implementation will have many phases. Start with something small first to familiarize yourself with the transition process. Once you have a couple successful transitions in place, tackle the larger domains/systems. New development of web services can begin to supply this information to new systems in development. As the Mainframe system is phased out, all other systems have started relying on the new web service (which gets its data from the original source, the Mainframe). Once a new system is in place to input data into the SQL Server/Oracle database, and staff trained on a new system, the Mainframe system can be shut off. It’s also important to mention that there are 2 pieces to this puzzle. Firstly, the data. Secondly, the user interface/interactions with the data. If going the web services route, you can actually have all the data handled before evening beginning the design of a user interface.

Now, during this whole process, it is very important to be mentoring and training the staff. Make them feel involved and get them excited about the changes coming up. Many people don’t like change, mainly because they are scared of it. Help them to feel important and needed in the process and most will gladly give it their all. It’s also important to help ones not feel insecure about the systems “they” maintain. Make sure they understand the goal isn’t to eliminate their job, but to help them get away from maintenance and get excited about doing new and better things.

Now that the framework has been developed, and all existing systems phased out or transitioned to the new framework, you have an extremely robust, reliable, consistent, and maintainable set of systems (if architected well of course). Congratulations, the efficiency of the IT department will be through the roof and continue to improve day by day.

The Technology Fever

I'm constantly surprised by the number of people obsessed or fascinated with new technology. Specifically, with new software "technology". I used quotes because most of what people think is new is actually the same old same old.

What makes things even more strange is how people flock to a specific company's technologies, whether it be Microsoft, Oracle, IBM, etc. I've worked with many people who have had this "issue" and it rules their career. I'm not sure if it's fear of not knowing much about another set of technologies from a different company that prevents them from venturing out or what... It also could be that they have just been a sucker for a marketing campaign that pulled them in.

When you've done software development for over 20 years you identify trends that others fall victim to. Some companies, such as Microsoft, are experts at manipulating their "followers". I thought I'd post some of the "trends" I've seen with specific companies.

Let's start with Microsoft:

1) Likes to take a standard and change it a tiny bit and act like they came up with something brand new.

2) Buys out start-up companies with new technologies and put the Microsoft brand on it making it appear they did it.

3) Take a product they have that works just fine, and make a few "beneficial tweaks" that work totally different so developers have to change their code for compatibility. I put beneficial tweaks in quotes because usually it's worse than the previous behavior. It also forces new training for employees to get used to the new features. Example: Office 2007 and the ribbon.

4) Comes up with "new" products that are the biggest and best thing since sliced bread. The clients want to be on the bleeding edge so they switch over to using the new products (and they develop for it). The product has a short lifespan before Microsoft comes out with a "better" product and thus the client/customer is left with having to constantly redevelop the same systems/products over and over.

Apple:

1) Crank out tiny improvements to their hardware (iPhone, iMac, etc) very frequently to constantly sucker people into buying the latest. Their main philosophy works off the herding concept.

2) Act like they are innovative with ideas, but in reality they are just very small improvements. It's more about convincing people the ideas are revolutionary than them actually being revolutionary.

3) Shun out competition by not allowing it to run on their hardware/operating system. Take Flash for example... Apple excluded from the iPad even though it could run perfectly fine. Apple's excuse? It crashes. I know many internet browsers that have built in protection from crashing plug-ins, so nice try Apple. This was really about trying to minimize Adobe's influence in the Apple world.

IBM and Oracle:

1) Extremely painful upgrade paths that require constant training of developers to even be qualified to make such upgrades successfully. Nice way to keep people locked in to product lines.

2) Over-complicating their products and development tools to make it appear "better" than the competition. Also requires "certifications" and training to keep up to date in the industry.

The Industry:

1) Flip-flop between server-centric processing vs distributed processing. It started with the mainframe, then flipped to pc's, then flipped back to the Internet web servers, then flipped back to peer-to-peer sharing, then flipped back to "clouds", etc.

I chose to pick on the major players in the software development world. There are many more. I hope that this blog brings attention to some facts about how it isn't a good thing to be a follower of a specific company and their set of technologies. Some may say that you have to stick with one company because otherwise there is too much to learn. I say that is bogus. Spend more time in the evenings when you're not working learning some new things. Focus on learning concepts as they are much more important and universal. Don't become a victim to the technology hopping race that the majority of the industry participates in.

Searching for Freelance Developers

I belong to one of the most popular freelance developer sites, guru.com. Guru.com is a great resource for attempting to find work if you're a freelancer. The site will notify you when there are new opportunities for you to bid on via email. There are typically several emails sent per day. Most of the jobs are either web site creations (often the requesters want you to clone or mimic another site), small database changes, or off the wall ideas.

As a freelance developer in the United States, I've come to realize that people are mainly driven by cost, not quality. The requesters can specify their budget, or leave it confidential or unknown. I often have to shake my head at how ridiculous the budgets are for most of the projects. Sometimes the requesters will be asking for something like complex artificial intelligence with neural networks and specify a budget of $250. Seriously? 250 bucks??? Who in their right mind would attempt such a project for a ridiculously low budget? To make matters worse, the requester usually has an unreasonable timeline of a week or two. Come on people!

Well, it just so happens there are thousands and thousands of vendors perfectly happy with such a budget. They are almost all entirely over seas! They live in areas of the world where economies are so pitiful that $250 may be a large sum of money. Unfortunately the money is being dumped into another country's economy (not the US) when the "cheapskate" requesters accept the bids.

What many of the "noobish" requesters will discover though is you often get what you pay for (notice the word often). I'm not claiming that only people in the United States are qualified to do freelance development work. That would be as ridiculous as a $250 budget for a software development project. What I am saying though is the requester may get someone who can barely communicate with English or has very little real world experience with successful projects.

Because of the foolish move to put cost over quality, the requester's project will most likely fail, have serious issues, or have to be completely redone. In the end, it causes resentment toward sites like guru.com, freelance developers (thinking they are all "rouges"), etc. What this means for freelance developers like myself is that we have to wade through garbage project requests and waste a lot of time (it is very rare that any project is worth bidding on even though I'm qualified to do almost all of them that come my way).

There are some companies/individuals who have experienced what I've described above, and come to the conclusion that they shouldn't specify the budget when they have no idea how difficult or time consuming a task is (i.e. create a space ship that operates off PHP and Javascript). Some of them try to control the situation by putting tremendous constraints on the project to the point where it's impossible to accomplish. The wise ones, however, realize that you should look at a company's reputation and cost. This again is bad news for new freelance developers out there trying to establish a reputation. It goes back to the days where a youngster just gets out of college and can't find a job because they all require 3+ years experience!

In conclusion, I wrote this article to vent some of the frustrations freelance developers experience (especially ones that offer quality solutions and want to make a living in the United States) in hopes that some requesters will read this. There are frustrations on both sides. Why don't some of you do us a favor and fix both our frustrations by actively looking for freelance developers that offer quality solutions at prices that aren't ridiculously low. You get what you pay for people.

The Open Source Software Society

There are two main types of software. Closed source (proprietary) and open source. With closed source, a person (or business) keeps the source code as the holy grail and never allows anyone else to obtain it. Open source, is of course the opposite. The source is released for anyone who desires to have it.

Proponents of open source say that it only improves the software having it open. Multiple people can contribute and there are many testers. Ok fine, but why can't a large corporation actually pay people to do this for a living? It baffles my mind that open source proponents (I'm mainly speaking of people with mentality like Richard Stallman - founder of GNU GPL) can't understand that people need to make money to live.

Creating software that can be sold and distributed to millions of people (key being the money made on the software license itself) is far more appealing to me than having to deal with a lot of people that I know nothing about. Most coders that I've run across aren't that great. Why do I want them contributing to the software?

What am I getting at? I see two ways to make money with software:

1) I only develop software and I sell it to people.

2) I have to spend a lot of time providing service to people to make my money.

I don't know about you, but I'd rather develop software than having to deal with tech support. I don't mean tech support isn't of utmost importance, it is (funny enough, that's what you DON'T get with open source software unless you pay for it). I'd rather have others that specialize and enjoy doing tech support do it. I enjoy doing architecture and design. If I come up with a radical way to do something, I don't want every one else to know how to do it. Why? Because they will turn around and do it and essentially take my money and steal my idea. I worked hard to come up with the idea, so why should someone else be able to take all the credit without me getting what I feel is fair?

I don't want it to sound like I'm completely against open source software. That is not the case. Sun Microsystem's (now Oracle) Java is one of the best open source products out there. BUT, guess what, it wasn't open source when it was being created and all the core libraries built. It didn't have rogue programmers in there messing around and creating their own versions.

That leads me to another topic that bugs me. Do you know why Linux hasn't even come close to overtaking Microsoft or Apple in regards to operating systems? Since it is open source, there are multitudes of "builds" and versions. Well guess what. That makes it almost impossible to organize and support all the different versions. Something like this is typical:

• Person trying to just use software: "How come my image printer driver silently fails on Mandrake but not Red Hat?"
• Linux fanatic: "Because they added an extra security feature in version xxx of Mandrake."
• Person trying to just use software:"Ok... So I've spent 2 hours trying to get it to work and I haven't been successful."
• Linux fanatic: "All you have to do is ..."

Fill in the "..." with 7 complex steps. Guess what. Most people just want something that works. They will pay extra for it. They don't want to mess around with something for 2 hours, let alone 2 hours and it still not work. They don't want to be forced to learn everything about Linux. They want to spend their time with their kids, wife, friends, whatever. Shoot, they may want to spend their time coding so that they can make extra money to go on a vacation to Hawaii!

I hate calling my Internet provider when they screw up my bill or something of that nature. I spend a lot of time listening to an annoying voice saying "Please hold... Your call is important to us." I often think to myself "why do I have to waste all this time for THEIR mistake?" I should be able to easily send off a note in 1 minute describing the problem, and they should look into it on THEIR time. I'm paying them for service! They don't pay me for my wasted time.

What does all that have to do with open source software? The majority of the open source software that isn't backed by a financial entity/business is absolute muck to wade through. The documentation is usually so detailed that you fall asleep trying to even find the section you're looking for. Then you have to read the terrible humor that some open source nerds try to inject into the documentation. You can picture the nerd sitting in front of their green screen with pizza crust laying around and a coffee stain on the printed copy of their manual. Time is of utmost importance to me. If I spend more time trying to get some piece of junk open source product to work than just purchasing something that is fully supported and well coded/designed I kick myself. It's just not worth it in most cases.

The key to making software successful is knowing when to use open source and when not to. Some products (such as Java) are a fantastic resource to utilize to build your products (that you can sell). There should be a balance. Unfortunately, a lot of open source projects is released under the infamous GNU GPL which is like a parasite. If you're not familiar with it, look up Richard Stallman and maybe do an Internet search for "why gpl sucks". You may have even found this blog because of that search!

Also, one more thing to mention in order to be successful with a hybrid of closed and open source for your products. Only use open source software that has been around for a while and is still strongly supported and has been controlled well. I see so much software rewritten over and over because of being "hooked" into some technology that either dies off or transitions. And of course, avoid GPL at all costs because your software will no longer be closed. Don't listen to Richard Stallman's nonsense about how closed source software is ruining society and how it "broke the system". I'd rather be able to afford to go to Hawaii for vacation than be on a starvation list.

My advice to any reader of this blog: Don't fall for the open source "community/society" mentality. There needs to be a balance of closed and open source. In fact, I'd rather have all closed source than all open source so that I can go on that trip...

What Exactly Are We Learning?

I am always surprised by the number of software developers and programmers out there that are at a stand-still with their progression and skills. I don't know how many times I've helped others that ask almost the same questions over and over. Often times the questions are dealing with some language syntax or library in the latest and hottest development environment/language.

I've learned over the years that most people stall because of a lack of understanding about fundamentals and concepts. I always say concepts are universal, but technologies are specific. What I mean by this is once you learn a concept, it will always be applicable and helpful. The more concepts one understands, the easier it is for one to relate situations, issues, and other concepts to each other. It makes it much easier to solve problems and learn new "tools".

I am the type of person that never memorizes all the details about a program, language, etc. I memorize the basics and most useful aspects, but then learn how to quickly find what I'm looking for (Google is everyone's friend). I've noticed that those having issues with progression often times try to memorize useless things that could be left in a dictionary for lookup purposes. I finally reached the conclusion that they try to memorize so many things because they don't know the difference between what is important to know and what isn't. Concepts are important. Knowing specific methods, libraries, etc. is not important. If one is efficient with searching, and knows what to look for, they will quickly find it.

I believe that we are teaching things incorrectly for the most part. We are trying to teach people specific programming languages to get them going as soon as possible instead of helping them learn universal concepts. I'm not just talking about the college level, but high school and even grade school. Once the fundamentals are understood, the details can be filled in much easier. One issue is a majority of people are impatient and want to produce something right away. This can be addressed by having demonstrations of what could be created based on concept instead of technology.

I'll admit, this is by far no easy task. But most things worthwhile aren't!

So what are some concepts that I think are paramount to continuous progression? Here are just a few:

• Hardware (CPU vs Hard Disk vs Hard Drive vs Memory vs Tower Case)
• Software Development vs Programming vs Architecture
• Binary vs Base 10 vs Octal vs Hexidecimal
• Machine Language Instructions
• Low Level Language (Assembly)
• High Level Language (C++)
• 3rd, 4th, and 5th Generation Languages
• Computer's Can't Think For Themselves
• Operating System
• Applications vs Services vs Server vs Utility vs Library
• Networking
• Protocols (HTTP, FTP, etc)
• Formats (XML, Text, Binary, etc)
• Character Codes (ASCII, Unicode, etc)
• Process (Heavy Weight)
• Thread (Light Weight)
• Multi-Threading
• Parallel Processing
• Heap vs Stack
• Memory Allocation and Deallocation
• Basic programming concepts (Conditions, Jumps, Variables, Loops, Reads vs Writes, etc)
• Crash vs Lockup vs Infinite Loop
• Peripherals and Interfaces (Mice, Graphics Card, USB Port, etc)

This is a very small list, but you get the idea. Most of the concepts don't have anything to do with programming directly. They are all things that should be known and understood however for continual progression. Once many concepts are understood, you'll see that when Microsoft comes out with a "new" technology or fad, it has already been done and probably done 20 to 30 years ago.

History Repeats Itself

Anyone who's been in this industry for a long enough period of time will notice the trends that happen over and over.

In the days of the mainframe, centralized processing ruled. Then, things turned more to distributed processing with the popularity of PC's in the 80's. The 90's come along, and with it, the Internet became popular. All of the sudden centralized processing was back in the picture. Now we are beginning to see rich Internet applications becoming popular once again. And so on...

When will people stop to think that there is no perfect solution? There is a time and place when server side processing is better. There is also a time and place when client side processing is better.

I, for one, am extremely tired of the dull and dumbed down web applications out there. I have developed my fair share. Don't get me wrong, they are useful. But most of the time it is no way to write software.

What happened to the good old days of Forte Tool development? Anyone who has experience with it will remember the fantastic design and architecture of it. Why did things go from that to dumbed down web applications with HTML postbacks and poor mechanisms?

I think the answer is because of a trend. Something "new" is developed, that is the best thing since sliced bread. In the 80's it was DOS and the PC's. Suddenly one could afford their own personal computer instead of needing a mainframe (anyone out there remember good old Tron?). Of course, the Internet became popular in the 90's. This suddenly required a major change in the way we think. Unfortunately HTML is what became the standard. Next it was Javascipt that tried to "remove" the dumbness of the web browsers. AJAX came along, which is a client on crutches (i.e. Javascript). The whole HTML process needs to be slowly phased out and replaced with something better. We need rich and powerful client applications again.

So far, I'm very impressed with Adobe's Flex for creating Flash applications. The Flash player really has its act together. It is lightweight, appears to be very robust, and very high performance. On top of that, it is crossplatform and can even run outside a web browser (AIR). Microsoft is trying to sneak into the market (what else is new) with their Silverlight. I'm not sure how much share they're grab, but I'm sure it will be like anything else they do (lock in some clients by selling how great it is, only to abandon it later and make the clients shift to the next greatest thing).

What needs to happen is for people to begin focusing on architecture more than development. By layering a system or application, one is "safer" when investing in a new technology because they don't have all their eggs in one basket. I used to be paranoid about having a product written in more than one language. I have recently realized this isn't as bad as it seems. If the system/product is architected well, various layers can be phased out and replaced fairly easily.

Let's develop systems that give the user what they want (i.e. not a dumb user interface that is HTML in a browser). A pretty and high performance user interface, backed by a robust network and server. I personally am leaning to Flex (Flash) on the front end, with Web Services as the network/receiving layer along with Java on the backend for persistence and processing.

Flash is very lightweight, so it downloads very quickly for machines that don't have it. It utilizes the CPU very efficiently. It also happens to have an excellent XML parser, which lends itself to utilizing Web Services on the back end. The Web Services are of course crossplatform wizards, so this layer is safe in case you need to change the front or back end. By being Java on the back end, one has the ability to run on pretty much any popular platform (plus take advantage of many open source or third party products because of the maturity). Plus you don't have to pay Microsoft anymore for development tools or operating systems. Don't get me wrong, I'm by no means a Microsoft basher. I develop in ASP.NET, C#, and VB.NET almost every day. I think C# has some advantages over Java. .NET makes it extremely easy to create Web Services and to consume them (Java, listen up - AXIS is the next best thing but it isn't as good). But the crossplatform capabilities of Java just makes it worth it.

Anyway, I think we're going to continue to see a shift toward rich Internet applications. Possibly seeing desktop icons eventually that takes one straight to an Internet application (doesn't run in the browser but utilizes the Internet never-the-less). But how long will it last? Probably just long enough for the next big fad to come along and start shifting it toward being "dumb" once again (wait, has Microsoft already been doing this with SharePoint?)...

Artificial Intelligence (Or Ignorance)

I have a deep passion for artificial intelligence theories as a software developer and architect. This passion existed as a child, and was one of the emotions that drove me to becoming what I am today. Hollywood and science fiction movies/novels helped this motivation. Now, after many years of being in the industry, I have a deep understanding of our technical capabilities as a human race of today's times.

I want to share some of my thoughts about this topic to anyone who may be interested.
As a child, I remember watching Arthur Clarke's 2001 (a Space Odyssey) in absolute amazement. The special effects blew me away. But being sort of a strange child, I was more fascinated with the way the computer, HAL 9000, could carry on a conversation with a person. I wondered if a computer could actually do this, and if it could, how fun it would be. I had similar feelings of awe from 2010, The Explorers, Tron, etc.

I was so enthralled that my dad purchased a desktop computer for me, a Leading Edge (very expensive at the time), and hired a personal computer teacher to teach me how to program in GW-Basic. I was about 11 years old at the time. I struggled with trying to understand the most basic programming concepts. However, just like any concept that we learn, I had moments of clarity where the light bulb would go off. Even so, I desperately tried to communicate with alien lifeforms by typing in silly commands and hoping the combinations would somehow magically do something.

Unfortunately, no response should have been my sign that there is no such thing as magic. Today, I am all to well aware of this fact. Sometimes I wish I could believe in magic again because of the fun and deep imaginary thoughts that would occur. Logic now rules my thoughts. Don't get me wrong, I love and treasure logic. It is one of the few things in life that can always be trusted and consistent. I use my imagination now in other ways, such as trying to invent, come up with new processes and ways of doing things, etc.

I said all of that to lead up to the fact that artificially intelligence is one subject that somehow squeezes in between logic and imagination for me. I know what is possible today, with our current knowledge and understanding. However, I also know what may be possible in the future.

As of right now, don't expect to be carrying on a free form conversation about various topics with a computer any time soon. We are so far away from having that type of intelligence in software. Not only has no one been able to create a program that even comes close to mimicking HAL in 2001, but our current hardware limitations would make the computer too slow to even be useful.

So how can we get to the point where we have a computer behave like HAL? Inventions. We need lots of inventions of technologies and theories that have yet to be discovered. Here are a few ingredients that will be needed (keep in mind I'm talking about HAL quality here):

1. A program that has a huge dictionary of "words" and definitions.
2. A program that can understand concepts from these definitions (this is a huge challenge to over come). This statement is so profound that many won't understand why this is so important.
3. A program that can begin to relate the concepts to each other and form a greater understanding. Again, extremely difficult task.
4. The ability to determine good information from bad information based on source and reliability. Another very difficult task (are you starting to see a pattern?).
5. The ability to apply "fuzzy logic" and rankings to possible answers instead of having a "definite" answer like 99% of programs do today. This has been studied to death so it probably has the most progress.
6. Extremely stable and reliable base libraries for the core functions of the program. These libraries must have gone through years and years of rigorous testing from any possible angle with zero defects (yeah right). Believe it or not, this will be one of the easier tasks.
7. The concept of sub-conscious vs. conscious would have to be implemented in the program. This concept is easy to understand. It is similar to an Operating System and a User Application. But the concept must be implemented with a huge amount of safety mechanisms (crashing and infinite loops are not allowed), priorities (sub-conscious always overrides conscious), multi-tasking, event notification (from sub-conscious to conscious), etc.
8. Atomic computing (or similar technology) to parallel process a huge amount of instructions at once. This probably needs the most inventions.
   

An so on... This is just a very small list of over all requirements. Each of these would have many details.

I wish I could be the bearer of good news. Alas, I must face reality and say that we are perhaps 100+ years away from having a program behave like HAL. I do want to say it is possible, just don't get your hopes up for seeing this in your lifetime...