Hadoop – the word – is everywhere, it’s ubiquitous.  But, knowing how to spell it, and knowing what it means for your company, are two very different things.  So, should supply chain professionals be concerned about Hadoop?  Yes, because although at the moment, Hadoop is probably being used most widely for marketing applications, the marketing and demand management functions are becoming more tightly integrated in companies striving to become demand driven.

Hadoop is used widely in marketing because it is best suited to unstructured and semi-structured data:  Images, social media data, call center transcripts, and clickstream data, for example.   So marketers use Hadoop to improve their understanding of customers and prospects, and their ability to sell them the right product, at the right time, using the right channel.  These additional insight can be a gold mine for companies striving to use social media and internet traffic to forecast new product introductions or omni-channel retailers seeking to understand a promotion’s potential lift by channel.

Unstructured data is potentially useful for emerging supply chain risk management applications, to drive a better understanding of the supply chain programs at leading competitors, and for recruiting and retaining supply chain talent.

Discussing Hadoop can be tricky because it’s a bit like the blind men and the elephant.  Hadoop is lots of things.  Depending on which way you want to look at it, Hadoop is:

  • A distributed data management platform – really a cut-down distributed operating system. It is designed to manage and work with immense volumes of data, and scale linearly from just a few to thousands of commodity computers.  In its earliest incarnation, it consisted of three parts, one for data management, one for programming, and one to make it all hang together.  The Hadoop Distributed File System (HDFS), Map/Reduce, and Hadoop Common respectively.
  • Open source. Hadoop originated at Yahoo in 2005 as the infrastructure to support a web search project.  Since then, Hadoop has migrated over to the Apache Software Foundation (“Apache”).  As such, it is available for anyone to download and use, free of charge.
  • An ecosystem. Like many open source projects, Hadoop has spawned a diverse and evolving ecosystem of enhancements, add-ons, and alternatives.  Just to name a few, these include Pig, Hive, YARN, ZooKeeper, and Avro.  The ecosystem also includes commercial vendors that provide value-added services based on Hadoop.
  • Hadoop is really a software project, not a software product.  As noted, you can download it free of charge.  But, unless you have fairly rare technical skills – or plenty of time on your hands – implementing, scaling and supporting that distribution can be a bit of a challenge.  Consequently, a number of companies now provide a more polished software distribution and supporting services.  Hadoop is available as a managed service too.

Putting those definitions and technobabble to one side, it’s always important in the technology game to follow the money:

  • Commercial Hadoop startups such as Cloudera, HortonWorks and MapR have recently scored massive venture capital investment. Cloudera closed a $900m round of funding in June.   Not to be outdone, Hortonworks announced a $100m funding round in March, with an additional $50m investment in June.  Likewise, MapR raised $110m in June, with Google Capital leading that round of investment.
  • Large mature enterprise IT vendors such as HP, Intel and IBM are backing Hadoop too. HP invested $50m in Hortonworks in June (see above) to drive closer integration between Hadoop and HP’s other big data technologies.  For its part, Intel was part of Cloudera’s recent $900m financing round, owns 18% of Cloudera, and has a seat on the board too.  IBM has its very own Hadoop distribution and also offers Hadoop in the cloud.

So your IT department really shouldn’t be on the fence about Hadoop because it’s a given.  It’s a done deal. It’s going to happen.  Hadoop has so much momentum at the moment it’s hard to see an alternate data management infrastructure emerging in the foreseeable future.  Almost anyone that wants to manage massive amounts of unstructured (or semi-structured) data will have Hadoop.  So, instead of wondering what Hadoop is and whether it’ll be part of your future, get ahead of the game and ponder these more important questions instead:

  1. What’s the best Hadoop approach for my company? There are three main approaches that each trade off different cost profiles and the technical skills required:  Downloading the free distribution from Apache requires intensive and ongoing technical skills; using a commercial distribution reduces the skills burden; pursuing the Hadoop-as-a-Service approach minimizes the technical skills needed.
  2. What analytic infrastructure are we going to use on top of Hadoop? Hadoop is just a data management platform, a cut-down operating system.  By itself, it adds little value to an enterprise.  In earlier IT generations, relational databases breathed life into the Unix operation system, and productivity applications made Microsoft Windows pre-eminent.  In the same vein, choosing the right analytic database and toolset for Hadoop is more important than Hadoop itself.

Planning further ahead, it behooves supply chain managers to start asking pointed questions of their favorite supply chain application vendors:  What hooks are they providing to integrate Hadoop databases, and what plans to they have to incorporate Hadoop as part of the supporting technology behind their own applications?

David White is ARC’s expert on Big Data, analytics, and Business Intelligence.

Be Sociable, Share!
Categories Business Intelligence
Comments (0)

The World Economic Forum (WEF) released its annual Global Competitiveness Report last week. The WEF bases its competitiveness analysis on its Global Competitiveness Index (GCI). The index considers numerous determinants of national economic competitiveness. It outlines 12 pillars of competitiveness, categorized into three stages of economic development where the given pillars are most important. While reviewing the article for specific mentions of logistics and supply chain, it occurred to me that the CGI framework is also an applicable structure for analyzing the sophistication of an organization’s logistics processes.

Source: World Economic Forum  The Global Competitiveness Index framework

Source: World Economic Forum
The Global Competitiveness Index framework

Basic, Factor-Driven
The CGI basic requirements subindex includes pillars that are essential for executing basic tasks that often rely on manual approaches. Manual approaches to basic tasks often provide limited productivity improvements, but can be established quickly with minimal planning or capital investment. From a human resource perspective, the needed skills may be widely available with little required training. From a financial planning perspective, the costs are often variable with respect to volume and directly related to operations.

A manual warehouse with limited technology investment is a logistics operation that would fall into the Basic, Factor-Driven category. The warehouse would execute basic tasks, without the use (or limited) of technology. It would be a low-risk investment, and would likely achieve limited productivity gains. To achieve additional productivity gains, this operation should look to the pillars in the Efficiency Enhancers subindex.

The CGI Efficiency Enhancers subindex includes pillars that are essential for developing more efficient processes and increasing product and process quality. Included in this subindex are supply chain relevant factors such as higher education, to develop workers that can perform more complex tasks and adapt quickly to the changing demands of the working environment; and the deployment and use of technology, to enhance productivity, efficiency, and adaptability. From a human resource perspective, the workers must be able to perform more advanced tasks and use technology effectively. From a financial planning perspective, there are upfront, fixed capital expenditures that will require extended time to achieve time-to-value. This increase time until payback increases the risk of the investment.

A warehouse that utilizes a real-time warehouse management system (WMS), enhanced with supporting technology such as speech recognition and optimization functions, would fall into the efficiency-driven category. ARC Research by my colleague Steve Banker indicates that basic real-time WMS solutions provide efficiencies in the form of inventory accuracy, while more advanced WMS solutions provide additional efficiencies in the form of labor productivity enhancements. This more advanced warehouse can achieve substantial efficiency, accuracy, and overall productivity gains. And the fixed expenses from deploying the WMS can provide scalable improvements after investment payback has been achieved. The use of other supply chain software such as supply chain planning and optimization would also fall into this category. The use of optimization technology often requires advanced skill and knowledge, but the improvements in operational efficiency can be substantial.

The CGI Innovation and Sophistication subindex includes two pillars that are essential to delivering new, differentiated, and superior products and services. Included in this subindex are innovation and sophisticated business processes that support the creation of unique and sophisticated products and methodologies.  From a human resource perspective, workers must be capable of improving on existing, advanced processes, and creating differentiation that will command a premium in the market. This differentiation will arise from developing superior or unique offerings that are not easily replicated by others. From a financial perspective, upfront investment in processes and products can be substantial and the potential returns from the investment are more abstract, less certain, and more risky than those from proven capital expenditures.

A company that develops a complex supply chain strategy to create a strategic advantage would fall into the innovation-driven category. As an example, leading supply chain organizations are currently developing omni-channel fulfillment strategies that will increase product availability, improve customer satisfaction, and ultimately increase market share. These operations serve more than the fulfillment function, they’re central to how these companies compete in today’s integrated commercial environment. Additionally, companies that work with their supply chain technology vendors to continually improve features and functionality in support of advanced processes are also examples of innovation-driven operations. Finally, the technology vendors themselves are innovators, as they must obtain a talented and knowledgeable workforce to produce products that structure and formalize knowledge into standardized processes for customers to deploy within their organizations.

Be Sociable, Share!
Categories Learning & Leadership, Warehousing
Comments (0)

fab_brasca_jda_2013_1_medNo, this is not yet another article about capturing network milestones or establishing better carrier connectivity, but rather the more strategic issue of how companies can raise the visibility of transportation within the domain of corporate initiatives.

I recently attended the American Shipper Executive Summit in New York City and early in the agenda, Eric Johnson, research director and IT editor for American Shipper, presented a view on the transportation information technology landscape.  One of the things that really struck a chord with me was that a key barrier to the adoption of new technology is a lack of corporate priority for transportation as a functional domain, and consequently, a failure for transportation projects to rise above the fray against competing projects.

As I hit my twenty-year milestone in the transportation and supply chain technology field, I think it is timely to share what I have seen as successful characteristics and approaches in raising the importance of transportation technology in the corporate landscape.

 Go Beyond the Easy

“Think little goals and expect little achievements. Think big goals and win big success.” David Joseph Schwartz, motivational writer

When I started in this space, transportation was generally a little-considered part of a company’s corporate strategy, if considered at all.  In many organizations today, however, it has earned a seat at the strategic table.  What I have observed in those organizations that have been successful in raising awareness of this domain is the will to go beyond solving the basic “bread and butter” challenges to also address the tougher problems.

For example, over the past several years I have witnessed several instances in which consumer goods companies have had their transportation initiatives mentioned in their annual reports.  In each case, the initiative in question went beyond the basics of carrier assignment or better carrier tracking and freight bill auditing.  Instead, they focused on advanced strategies for establishing a true shared service logistics organization spanning divisions and geographies, better utilization of large pools of assets holistically and concurrently with commercial freight, and in some cases, even creating a revenue stream from their excess capacity. These more advanced strategies garnered both management and stakeholder attention.

Roadmap versus Project

The counter argument to thinking big and tackling tougher problems is that doing so takes time and greater investment.  It is normal for those of us who are naturally risk averse to shy away from taking on such big projects.  This leads to a second concept in which I am a firm believer and consistently advocate: to think in terms of roadmap rather than project.

Successful companies in transportation, or in any business for that matter, do not think in terms of projects.  They have a long-term view of where they want to be and use that to create a roadmap for growth.  Taking this view provides two key benefits for raising the visibility of transportation in the management approval process.  The first is that having a long-term vision of how the project maps to the corporate growth strategy provides a better foundation for approval of individual project-level funding than a project presented in isolation.  The second is that a long-term vision provides the engine for continuous improvement, and hence, sustainable value that a short-term project focus lacks.

End the Isolation

Another emerging trend for which I am certainly a big proponent is the leveraging of transportation beyond its current role as an isolated, execution-level function.  Certainly, we have seen some convergence with the warehousing domain, with some companies adopting more comprehensive strategies and approaches than others.  But what about the world of upstream supply chain planning?

Transportation is a unique entity in the world of supply chain in that it truly straddles the worlds of planning and execution. Yet organizations rarely leverage the former beyond a near-term implementation.  Again, I have seen that those organizations that leverage their decision points across a broader time horizon (than just execution) have also been more successful raising the visibility of the transportation domain.  Considering capacity, modal decisions, and route alternatives across a diverse set of time-based and functional decision points creates a level of importance beyond simple execution.

Beyond Technology

The last thought I will leave you with, and certainly an interesting one coming from someone in the technology field, is to not consider technology as a solution on its own.  Technology is only one piece of the puzzle when it comes to improving performance. It must be complemented by process as well as people.  This may seem like a cliché, and yet time and again I encounter projects that focus on embedding new technology into an existing process without understanding whether the process itself is good or flawed.  The achievement of sustainable value, which creates corporate visibility, has to include being intellectually honest in evaluating current processes, particularly in view of the corporate long-term strategy.

By going beyond simple cost factors to take on the larger transportation challenges, mapping transportation projects to long-term corporate strategic growth plans, integrating planning and execution elements for greater synergies, and evaluating processes and human factors; transportation management professionals can raise the visibility of transportation to corporate executives and stakeholders and increase the likelihood of getting transportation initiatives approved and funded.


As Vice President of solution strategy at JDA Software, Fabrizio Brasca (@FabBrasca) is responsible for developing innovative strategies across all industry verticals, strengthening executive-level relationships with JDA’s key customers and prospects, and advising companies on best practices. He holds an Honors Bachelor of Mathematics co-op degree with a specialization in business and information systems from the University of Waterloo, Waterloo, Ontario. The author invites comments or questions from readers. He can be reached via email at fab.brasca@jda.com.

Be Sociable, Share!

I’ve been doing some research on the unconventional oil supply chain. Unconventional oil includes crude oil from shale and oil sands. I’ve talked to over 20 oil executives and experts about the inbound supply chain that supports oil well production. This is part of the supply chain that the oil industry refers to as “upstream.”

The key players involved in bringing in the sand, piping and casings, production chemicals, and other necessary raw materials are the oilfield services companies. A good argument can be made that the well services industry has the least mature supply chain capabilities of any industry in North America.

The industry does attain fairly high service levels. Not supplying needed materials to an oil field would cost the average owner operator (the oil company) forty thousand dollars per day in lost production per well. But an owner operator may be supplying dozens of wells from an oil field warehouse. If that warehouse runs out of a critical supply, it could cost the company over a million dollars per day. Because of this, a core part of many well service companies’ value proposition is that customers will always have products when they need them.

Well services firms are really very large, global distributors of industrial bulk supplies for oil fields. These companies purchase goods globally from hundreds of suppliers. The goods flow into warehouses where they are treated and their quality is certified, and then they flow out to oil fields all over North America. In many cases the oil fields are in very remote locations served by dirt roads that are not on standard maps. Because these are bulk products, the goods move by ocean and rail, with trucking necessary for last mile deliveries.

This is a supply chain with long lead times. Sand, for example, is used to keep a hydraulic fracture open, during or following a fracturing treatment. Sand of the right quality comes from Wisconsin, but because of demand and train capacity issues, it can actually be quicker to purchase a proppant – a man-made ceramic material – from China or Russia. Chinese proppants ship out of the port of Tianjin and into the port of Seattle to serve North Dakota’s Bakken oil fields; Russian proppants ship from St. Petersburgh to the port of Duluth. It can take 30 days for goods to arrive at port, clear customs, and ship across an ocean; another week to 10 days to secure rail capacity; and up to 10 days to certify the products at the central warehouse in North Dakota. And yet in many cases customers are promised 30-day deliveries.

But these service levels are achieved, in most cases, despite very poor forecast accuracy, and without the use of a Sales & Operations Planning (S&OP) process – which means that the supply chain team never knows when orders will be placed. Not surprisingly this is a supply chain with a lot of inventory and sales people making just-in-case bulk orders for more inventory than is needed. And some firms will tell you that to meet service levels in rush situations, they will use air. But, of course, air is a very expensive way to ship pallets that often weigh over 2,000 pounds.

Not all well services firms lack a S&OP process. I’ve been able to uncover one division of a very large well services firm that is using S&OP. But this firm’s process is at a very low maturity level.

Achieving high service through extremely high inventory levels, means that the inventory carrying costs get passed along to their clients.

oil field

And in fairness to the well services firms, the upstream supply chain capabilities of most of the independent producers they are dealing with are very poor. The owner operator often has outsourced different parts of the production process to different firms and lacks visibility to a good forecast of what they will be produced in the coming days and weeks. If they can’t provide a good forecast to the well service suppliers, it is not surprising that the oilfield suppliers can’t forecast accurately.

Nevertheless, I have never come across a North American industry with such a dysfunctional supply chain.

Be Sociable, Share!
Train on BNSF Rail in North Dakota (courtesy of Fox Business)

Train on BNSF Rail in North Dakota (courtesy of Fox Business)

The World Economic Forum (WEO) released its annual Global Competitiveness Report earlier this week. The 565 page PDF file is available for download here, if you dare to read it cover-to-cover. It evaluates the competitiveness of 144 economies across the globe. The index outlines 12 pillars of competitiveness, categorized into three stages of economic development. It came to my attention while viewing this framework, that it can be a useful tool for evaluating one’s own supply chain and logistics operations.  I will write a blog post in the near future discussing the framework the WEO used in its index and explain how I see its relevance and usefulness in evaluating logistics operations.

But for now, on to this week’s news:

An article on Fox Business discusses the rise of crude-by-rail volumes in the US, resulting from the increase in US production driven by the shale oil boom. The report states that rail transportation of petroleum for the first 7 months of 2014 is up 9 percent from last year. The report also states that US production hit 8.5 million barrels per day in June, the highest level since 1986. The article delves into aspects of the boom in North Dakota. Here is a quote from the article:

Drillers in North Dakota, which trails only Texas in oil production, rely heavily on railroads to transport oil. According to the North Dakota Pipeline Authority, between 60% and 70% of the more than one million barrels per day produced in the state was shipped to refineries by rail throughout the first half of 2014.

However, there have been negative side effects from the increased demand for rail transportation of crude. The New York Times published an article last week on the backlog of grain shipments in North Dakota resulting from the rail capacity shortage due to oil shipment demand squeezing out capacity for agriculture transports. The farmers stated that the backlog is only going to get worse as a record crop of wheat and soybeans is being harvested in the region.

The Atlantic published a detailed article on Google’s drone program (Yes Google, not Amazon). The article discusses the development, key players in the process, and the recent test flights in Australia. The program is known as Project Wing and has been ongoing at Google X for the last couple years. The drone’s design is known as a “tail sitter,” a hybrid plane and helicopter that takes off vertically then rotates horizontally for flight. Here is a link to a brief video of a recent test flight.

An equity analyst named  Ming-Chi Kuo who is following Apple is apparently predicting that Apple’s launch and distribution of its anticipated iwatch will likely be disrupted by a shortage of sapphire glass. Sapphire glass is substantially more scratch resistant than the current screen glass used in iPhones. However, sapphire glass is also reportedly planned for use in the iphone 6. GT Advanced Technologies is noted as Apple’s manufacturing partner for the special glass.

OmniTracs, acquired from Qualcomm at end of 2013 by private equity firm Vista Equity Partners, has announced its intent to purchase XRS Corporation. XRS Corporation, formerly known as Xata, is a fleet telematics provider. OmniTracs also recently acquired Roadnet Technologies, a provider of fleet management software. When Vista Equity acquired OmniTracs, I was curious about the company’s plans for the business unit, as its sales had been declining for a number of years. Given the recent acquisitions, it appears that the strategy for the new OmniTracs organization is one which delivers value from offering a broad footprint of solutions focused on the management of private fleets. It will be interesting to see what steps they take next in this journey.

The Journal of Commerce  discusses a pending decision by the EU 28 on potential new measures to take against Russia for its intervention in the Ukranian conflict. The article notes that the EU measures are unlikely to target industries or trade that will directly affect Baltic shipping. However, the Russian response is likely to trigger a reduction in trade flows. The report notes a number of trade routes with declining volumes, including those of DFDS between the Baltics and Russia, and container traffic between Hamburg and Russia.

And with that have a great weekend! This week’s video, The Rock Island Line by Johnny Cash.





Be Sociable, Share!
Categories This Week in News
Comments (0)