Big data streams and the analytics systems that can make sense of them are becoming easier to deploy due to the advent of cloud services, better tools and more data scientists, but new challenges are emerging such as increasing data privacy regulation.

One constant with big data is that it just keeps getting bigger.

As PwC Australia Partner Rob Parker puts it, there’s a wave of data pounding on the doors of organisations these days and the surge is only going to get stronger in the future.

“We have got this data tsunami where everyone has realised that the amount of data we have generated in the last five years is just growing tremedously,” he said.

“We know we need to keep it, so the storage requirements are increasing exponentially. Trying to leverage all that data has been a challenge for organisations.

Mr Parker sees modern day cyber security systems as representative of how big data and analytics work together these days.

The massive streams of data hitting the outer ring of an organisation’s cyber security defence pose a big data analytics challenge.

“In terms of the security aspects, it actually becomes more and more challenging to improve the posture of your security because you have to sift through all that information.

“Arguably it was easier ten years ago –when the data volumes were not so massive – to find the bad guys in the system. Now there is just so much data, it’s easier to miss them”

“That’s where the analytics comes into play – being able to sift through that data and apply some intelligence to it,” says Mr Parker.

“It’s like a balancing act – ingesting all this data and finding a way to make it useful.”

That balancing act is becoming somewhat simpler compared to the early days when organisations had to roll their own systems from scratch as well as track down teams with then rare skill sets such as data scientists.

These days the toolsets for big data and analytics are more mature and there are a lot more data scientists than five years ago, although the job category remains in high demand.

“Data scientist is still one of the highest demand jobs there is in ICT today,” says Mr Parker.

Ready-to-run cloud services are also helping organisations with their big data and analytics projects.

“Being able to leverage the cloud makes a difference. To harness big data and analytics you need a lot of computers and storage, and having cloud environments now has allowed organisations to do that, particularly in the ASX top 20 company list and large-scale government departments.”

At the forefront of consumer privacy concerns is the European Union’s General Data Protection Regulation (GDPR), which became enforceable in May this year and touches entities that process EU citizen data anywhere in the world.

Mr Parker does not see the GDPR regulations as an inhibitor of big data and analytics projects, but rather a force that should help the global business community avoid privacy disasters.

“I don’t see GDPR as an inhibitor, I see GDPR in the EU as a great example of trying to get ahead of the curve in terms of technological growth before it becomes too difficult,” he says.

David Moore, Senior Technology Partner at PwC Australia sees lots of market need from inbound and outbound call centre analytics.

On the inbound side, call success rates are around the 40 per cent to 60 per cent mark but outbound call success rates are still low.

“People want to know some pretty basic stuff,” from their investments in data and analytics systems says Mr Moore.

“Things like unanswered rates, ring off rates, time on call, call completion, call satisfaction and a whole lot of things that are direct inputs off the call itself,” says Mr Moore.

“My anecdotal understanding of outbound call centres is that they are becoming a very ineffective means of communicating with potential consumers and markets because everyone is bugged by overseas call centres, unwanted calls and phishing operations.”

Legitimate outbound call centre operations are giving way to online self-service portals.

The analytics engine sitting behind these portals are getting smarter every day, says Mr Moore.

“When I hear of the opportunities that enterprises are creating out of this, they are definitely getting smarter. These engines are sensitive to size, colour, style, fit, age and the potential for on-selling co-products.”

Automated sales techniques such as brewing up combination offers on the fly, warning of low stock levels or creating spot deals are becoming more and more compelling for consumers.

Another sales analytics trend centres around time of day, particularly with food, says Mr Moore.

“Businesses know that when they have an inbound enquiry at certain times of the day, they have a highly likely success rate on food orders and co-dependant food orders – like pizza going with cola or ice creams.

“Analytics engines are becoming very attuned to these types of combinations, and in more complex relationships than that of pizza is to ice-cream or cola.”

Syndication and co-sharing customer data is another data analytics trend and customers signing up for login rights to an online sales site often agree, sometimes unknowingly, to agree to co-sharing which is mentioned in the sign-on fine print.

“The value in syndication means that five participants spending a twentieth each are getting one hundred percent of value out of this.”

“Some of these back-end services are in clouds, they are pretty easy to stand up, they have a standard solution stack, they have a big data engine and an analytics engine that can be experimented with through their own algorithms.

“This is all becoming finely tuned,” says Mr Moore.

With the advent of analytics systems and services to better manage and understand big data streams, and recent changes to the regulatory landscape, it is clear that businesses must equip themselves with the tools and talent to manage cyber risks, gain data insights and create value for their customers.

PwC Australia has partnered with InnovationAus.com to present the Civic Nation 2018 on September 27 in Sydney and will host an invitation-only Data & Analytics Policy Roundtable on the same day.

Source : https://www.innovationaus.com/2018/08/Big-data-analytics-meets-big-business

Towards the end of 2017, we at Hortonworks had predicted that Industrial Internet of Things (IIoT) will impact the manufacturing sector further through Big Data technologies. Mid-way through the year, it is a good time to delve further into this prediction and talk about the role of Big Data in making manufacturing ‘smarter’. Through low-cost sensors and IoT devices, manufacturers are creating a connected shop floor. But what about the data that is gathered across the manufacturing supply chain? This data needs to be stored in an inexpensive and secure manner. And more importantly, how can this data create more efficiencies? After all, manufacturing is all about efficiency at scale.

By using a combination IoT and Big Data technologies, players in the manufacturing sector can create better efficiencies and better products while improving speed-to-market and customer satisfaction. And reduce their costs of course. So, how can IoT and Big Data make manufacturing smarter? Here are some use cases:

 Manufacturers always want to keep their inventories at minimum possible levels and prefer just-in-time delivery of raw materials. The danger with this model is that lack of raw material supplies can lead to production delays. By using sensors, RFID tags, supply chain data can be easily captured. Such historical data will lead to patterns which will enable manufacturers to follow a just-in-time model thereby reducing supply chain costs and improve their margins.

 Every manufacturer wants to maximize the first-time yield for a production run. First-time yield is basically a measure of the number of products that are made correctly the first time they come through the production line. However, the key challenge that manufacturers face today is the lack of visibility into operations. Sensors can provide raw data for improving that visibility, if the sensor data can be integrated with other existing data stores. Hortonworks has increased yields in drug manufacturing for Merck Research Laboratories through its Hortonworks Data Platform.

By Improving Process and Product

 Sometimes, manufacturers face situations of product malfunction after the product is sold. If manufacturers can deploy sensors to capture data at various stages in the manufacturing process, they can detect the issue which caused the malfunction. This, by combining the forensic data with the original sensor data from when the product was manufactured. Thus, a manufacturer can improve the process and product across the product range.

Another way, IoT and Big Data can improve efficiencies and reduce costs is by creating maintenance schedules based on real-time and historical data drawn from machinery and equipment in the factory. Machine learning algorithms can compare maintenance events and machine data for each piece of equipment to its history of malfunctions. Manufacturers will also soon look beyond the shop floor and leverage IoT and Big Data across the value chain. For instance, they can fit sensors on their trucks or on those of their vendors to collect data like location of the truck, weather, and even incidents like speeding, or lack of lane discipline. Such data can be streamed back to the manufacturers’ servers in real time. This way, they can make sure the goods in the truck are safe and no major traffic violations happen through analytics. Real-time sensor data-in-motion would also find use on the shop floor in connected factories soon.

Interestingly, few of these use cases have started being deployed in the manufacturing sector in a few countries and in India too, they will be utilized in connected factories soon. With the push on manufacturing through initiatives like Make in India, the need for improving efficiencies at scale will arise. For manufacturers, the key will be choosing the right solutions to build successful enterprise grade IoT platforms.

Another important factor, manufacturers need to consider is interoperability. Sadly, standards of connectivity between the various devices were not in place at the start of the IoT boom. However, open source software offers a cohesive mechanism for data exchange. With open source software, the developer community can work together to create solutions that will allow data to flow across IoT networks. In the next few years, the ecosystem will mature and we will see greater adoption of IoT and Big Data in the manufacturing sector. ‘Smart’ will be the way forward.

Source : https://www.dqindia.com/iot-coupled-big-data-can-make-manufacturing-smarter/

Decision-making remains one of the ultimate tests for leadership in new entrepreneurs. Even experienced leaders who have a track record of sound decision-making have, at some point, made a drastically poor decision that shook their reputation.

As the talk about AI promises a radical transformation of the organization, leaders are especially curious to know if it will make it easier for them. While a lot of them are excited, some of them don’t want decision-making made easier. Their ability to make sound decisions without complex technology is the very foundation of their reputation as good leaders.

Related: Why 'Cognitive Computing' Is the Next Big Thing for Business Growth

The good news is that AI is quite unlikely to make it easier for decision makers as they’ll be required to input judgment in the machine predictions. As the real impact remains to be seen, there are ways in which AI is set to inevitably affect business decision-making.

Through data mining, many businesses are already using predictive analytics to make better decisions. Predictive analytics allows businesses to anticipate events by looking at a data set and trying to guess accurately what will happen at a certain time in the future.

AI brings with it machine learning, another technique used in predictive analytics. The variation is that while data mining involves merely identifying patterns in large data sets, in machine learning, machines are not just designed to learn from the data, they are also built to react to it by themselves.

With the information provided, decisions can be made on such issues as:

Various psychological studies have shown that when we’re faced with many decisions to make within a short period of time, quality declines because we gradually deplete our mental energy.

A case application of this is when supermarkets place candy and snacks at the cash register. Marketers know you’ll be making decisions throughout your short shopping trip and will be less likely to resist the sugar rush by the time you’re done. But you know who can resist the sugar? A machine.

lgorithms, not prone to decision fatigue, can make an infinite number of decisions per day, each as accurate as possible. Executives who use AI will be at an advantage by using it to bypass human weakness.

Related: The Sooner You Get Your First AI Job, the Better for Your Career

When making complex decisions, executives typically need to look at a set of different factors. Where there’s too much data to be considered, the decision-maker may get overwhelmed, leading to disastrous decisions.

On the contrary, a machine can easily handle multiple inputs without exhaustion or confusion. All that’s needed is a set of instructions or programs that guide the machine to use probability and suggest or implement the most logical decision.

Recent advances in artificial intelligence (AI) and computer technology are causing us to think again about some really basic questions: what is a firm?  What can firms do better than markets?  And what are the distinctive qualities of firms in a world of smart contracts and AI?

While there has been a lot of discussion about “what’s left for humans?” as AI improves at exponential rates — the customary answer is that humans need to focus on the things they are uniquely good at, such as creativity, intuition, and personal empathy — I think we now have to ask, “what’s left for firms?”

In many ways this is an old question, because it takes us back to the arguments of Nobel Laureates Ronald Coase and Oliver Williamson that firms exist to coordinate complex forms of economic activity in an efficient way.  If computer technology has the capacity to simplify and streamline transaction costs, more and more work can be done through these smart-contract arrangements, making traditional human-managed firms obsolete.  For example, when you say to Alexa “order more dog food,” a chain of activities is initiated that leads to the delivery of a fresh supply of Kibble 24 hours later, with little or no human intervention. This work is coordinated by a single firm, Amazon, but it often involves third parties (makers of dog food, delivery companies) whose systems interact seamlessly with Amazon’s.

But is this coordination logic, this ability to internalize transactions to make them more efficient, really the raison d’etre of firms? I would argue that it is just one among many reasons that firms exist. And as computer technology simplifies and reduces transaction costs further, it is these other things that firms do uniquely well that will come more to the forefront. Here are four areas where firms excel.

1. Firms create value by managing tensions between competing priorities. 

In today’s parlance, firms have to exploit their established sources of advantage (to make profits today) while also exploring for new sources of advantage (to ensure their long-term viability).  However, getting the right balance between these two sets of activities is tricky because each one is to a large degree self-reinforcing. Hence the notion of organizational ambidexterity — the capacity to balance exploitation and exploration.

Artificial intelligence is evidently helping many firms to exploit their existing sources of advantage — whether through process automation, improved problem-solving or quality assurance.  Artificial intelligence can also be useful in exploring new sources of advantage: in the famous case of AlphaGo, the winning “strategy” was one that no human player had ever come up with; and computers are increasingly writing new musical scores and painting Picasso-like landscapes.

But AI is not helpful in managing the tension between these activities, i.e. knowing when to do more of one or the other.  Such choices require careful judgment — weighing up qualitative and quantitative factors, being sensitive to context, or bringing emotional or intuitive factors into play. These are the capabilities that lie at the heart of organizational ambidexterity and I don’t believe AI can help us with them at all right now. IBM’s recently-announced Project Debater is a case in point: it showed just how far AI has come in terms of constructing and articulating a point of view, but equally how much better humans are at balancing different points of view.

2. Firms create value by taking a long-term perspective.

As a variant of the first point, firms don’t just manage trade-offs between exploitation and exploration on a day to day basis, they also manage trade-offs over time. My former colleagues Sumantra Ghoshal and Peter Moran wrote a landmark paper arguing that, unlike markets, firms deliberately take resources away from their short-term best use, in order to give themselves the chance to create even more value over the long term.  This “one step back, two steps forward” logic manifests itself in many ways — risky R&D projects, pursuing sustainability goals, paying above-market wages to improve loyalty, and so on. We actually take it for granted that firms will do many of these things, but again they involve judgments that AI is ill equipped to help us with.  AI can devise seemingly-cunning strategies that look prescient (remember AlphaGo) but only when the rules of the game are pre-determined and stable.

An example:  the “Innovator’s Dilemma” is that by the time it’s clear an invasive technology is going to disrupt an incumbent firm’s business model, it’s too late to respond effectively. The incumbent therefore needs to invest in the invasive technology before it is definitively needed. Successful firms, in other words, need to be prepared to commit to new technologies in periods of ambiguity, and to have a “willingness to be misunderstood,” in Jeff Bezos’s terms.  This isn’t an easy concept for AI to get used to.

3. Firms create value through purpose — a moral or spiritual call to action.

There is a second dimension to long-term thinking, and that is its impact on individual and team motivation. We typically use the term purpose here, to describe what Ratan Tata calls a “moral or spiritual call to action” that leads people to put in discretionary effort — to work long hours, and to bring their passion and creativity to the workplace.

This notion that a firm has a social quality — a purpose or identity — that goes beyond its economic raison d’etre is well established in the literature, from March and Simon through to Kogut and Zander.  But it still arouses suspicion among those who think of the firm as a nexus of contracts, and who believe that people are motivated largely through extrinsic rewards.

My view is that you just need to look at charities, open source software movements, and many other not-for-profit organizations to realize that many people actually work harder when money is not involved. And it is the capacity of a leader to articulate a sense of purpose, in a way that creates emotional resonance with followers, that is uniquely human.

Successful firms, in other words, institutionalize a sense of identity and purpose that attracts employees and customers. Ironically, even though blockchain technology is — by definition — about building a system that cannot be hacked, or misused by a few opportunists, people still prefer to put their faith in other people.

4. Firms create value by nurturing “unreasonable” behavior.

There are many famous cases of mavericks who succeeded by challenging the rules, such as Steve Jobs, Elon Musk, and Richard Branson. With apologies to George Bernard Shaw, I think of these people as unreasonable — they seek to adapt the world to their view, rather than learn to fit in. And if we want to see progress, to move beyond what is already known and proven, we need more of these types of people in our firms.

Unreasonableness is antithetical to the world of AI. Computers work either through sophisticated algorithms or by inference from prior data, and in both cases the capacity to make an entirely out-of-the-box leap doesn’t exist. Consider the case of investment management, where robo advisors are not just making trades, they are also providing investment advice to investors, and at a fraction of the cost of human financial advisors. But as the Financial Times said last year, “when it comes to investing, human stupidity beats AI.”  In other words, if you want to beat the market, you need to be a contrarian — you need to make investments that go against the perceived wisdom at the time, and you need to accept the risk that your judgment or your timing might be wrong.  Both qualities that — at the moment — are distinctively human.

So one of the distinctive qualities of firms is that they nurture this type of unreasonable behavior.  Of course, many firms do their best to drive out variance, by using tight control systems and punishing failure. My argument is that as AI becomes more influential, though the automation of basic activities and simple contracts, it becomes even more important for firms to push in the other direction — to nurture unorthodox thinking, encourage experimentation, and tolerate failure.

In a recent Fast Company article, Vitalik Buterin described how all the elements of Uber’s ride-sharing service could be provided through Ethereum-based applications that worked seamlessly with one another:  “the whole process is basically as before, but without the middleman [Uber].”  This is may be true, but it doesn’t necessarily follow that a computer-mediated service is the better option.

For example, in 2016 a distributed autonomous organization (DAO) was launched on Ethereum. This idea was that it would run without human intervention, using pre-established rules and blockchain technology to operate seamlessly. But it had a small technical flaw, which allowed an un-named user to siphon of $55 million of the money raised in a matter of days.   Faced with the meltdown of their entire creation, the founding fathers of Ethereum intervened, creating a so-called hard fork in the blockchain that allowed investors to get their money back, and for the development of Ethereum applications to continue.

No matter how powerful the technology, sometimes a little human judgment is necessary to keep things moving in the right direction.

Source : https://hbr.org/2018/08/whats-the-purpose-of-companies-in-the-age-of-ai