I always find it interesting how market demand for a specific job in a particular time influences the perception of how good a job really is. In the late in the 80's, before the World realised how twisted and fragile the financial market was, a job in Finance had a very different social connotation than it does today.
As things evolved and people became more aware of how toxic that industry was and perception changed. Suddenly working as a broker in an investment firm was no longer viewed as something that was positive to society, over time this has not changed much and it is still common to hear that It takes a "especial kind of person" to be in Finance.
Whether this is true or not is besides the point, what is interesting to me is how external circumstantial factors have the potential to impact and influence the perception of how valuable, respectful, or even worthy a profession is. We see this effect wide spread in every industry today. The evolution of software engineering as a profession is no exception.
Currently, there is a high demand for software engineers, this is mostly driven by microeconomics; supply and demand, there is scarcity of a skill-set that the market requires that raises the value of the job resulting in a shift in the social perception of that profession.
As with most things, this will change, and perhaps the moment will come when artificial intelligence is able to create mathematical paradigms without Human Intervention. If (when) that happens, the creative aspect of being an IT engineer will progressively subside and be replaced by a more "supportive" function, filling the gaps of the few things machines are not able to do independently.
Creativity meets repetition
While this future is not too far ahead, we are not quite there yet, but more and more we are living in a time where the symbioses between Technology and Humanity is making one or the other indistinguishable, the need for an underlying "invisible" technological layer that is able to support the next generation of Human Beings is creating a high demand on Human brain power, there has never been a better time to be an IT engineer.
It is not that the profession has dramatically changed, but external circumstances certainly made a dramatic effect on what being an Engineer is all about and how valuable the profession is to Society.
This is perhaps best illustrated by the perception of how much creative work is done by a software engineer, I will try and stay away from dwelling too much on the concept of creativity itself, as it is a highly subjective and debatable, so let's assume that in this context creative work is the process and the ability to creating things with little or no restraint and/or obstacles that are not directly related to the process of creation itself.
Breaking Down Creativity
There is an image imprinted in our minds of an engineer sipping coffee from a cup in a trendy "cafe" changing the faith of an entire industry from his laptop. While it is not completely infeasible that this could ever happen, the reality is that as with most professions there is a certain conventional labour that makes up for most of the work done by a software engineer.
It can start with the restrains imposed by a coding frame-work, to a set of conventional rules, tests and protocols that need to be followed in order to create something. More often than not, it is a tedious process that is more acquainted with the labour of a factory worker than the creative work of an artist.
My Grandfather was a writer and I can still recall my Mother mentioning how he dreaded the process of writing, it took an tremendous amount of discipline and will power to get through the creative process. It is not a spontaneous feverish process born our of passion, creative work is a result of careful planning backed-up by a very clear vision and the means to deliver it.
Creative thoughts lead to plans that may or not happen in the future, only actions are able to create things, getting the creative process from a thought to reality takes rigorous labour and structure, this is where great ideas come to die and where entrepreneurs tend to fail.
Software Engineers have the potential to create great things, but assuming that it happens overnight in an eureka moment is not realistic. I used to believe in sporadic moments of genius and brilliance until I came across Daniel Kahnemans Thinking, Fast and Slow publication. (also available in Audible)
And then there is the autonomy, the painful reality is that, the luxury of deciding on the end to end scope of a project is only possible when someone has no stakeholders to respond to, perhaps in the very early stages of a Start-up when there is no board or Angel Investors.
The false perception that a person can determine its own destiny working for someone else creates a big problem; i.e. a wave of talented self-driven engineers are put in root of collision with a corporate industry that is not willing to be dictated to, in the end, the result is usually frustrating mess where expectations fail to meet reality from both ends.
Now the problem is not the absence of autonomy, the main issue is around what autonomy really means in the real world. As with most areas in life, it usually means that unless funding is not a problem, the person that gets to sign the check is the person that makes the final call, this may seem as logical at first, but in the mind of a young passionate engineer or entrepreneur this usually comes as a shock.
Great ideas need a spark to start but they also need money to run, and this is where a conditioned autonomy takes over.
This of course has a much wider impact, as it will change the decisions students will make as they go through the education system creating a chain effect further down the line that leads to discrepancies between the talent available at any given time and what the market is demanding, purely because we are basing career decisions on perception and by present circumstances steered by a fragile and volatile market.
So the speed in which things change know-days demands for a better predictability model in place to help understand what are the skills that the market will need in the future.
A good solution would be to focus on the foundations of learning and accept right from the start the reality that the market is always going to change and that education should be about giving the structure and the tools to embrace change, not fight against it.
This does not mean that we should ignore specific talents and desires of a particular individual, but it does put in question a tendency for an overly specialisation at an early age. Assuming that a teenager has enough data and experience to decide the career before having ever had an exposure to it on the first place is not logical.
The bigger the impact of this investment is the less likely the student will be to change its major even if there is a realisation that it was a poor choice or that there may another opportunity that brings more advantages long term.
We need to refocus on the basic foundations that enable flexible growth. The ability of a person to change career at 40 years old needs to be something that not only is possible but supported academically, accepting and embracing change is a principle that needs to be set in students early on as opposed to committing to a single discipline for 4-5 years and finding out that in the Word has changed.
Adaptability and the ability to learn and process information is the new discipline. I am not proposing we should turn everyone into the "jack of all traits", but specialization too early on without "learning how to learn" is not the right way to go.