Customer Success Stories with the Hadoop Gateway

The following customers are customers that the Advanced Technology team visited to highlight the value of the Hadoop Gateway with Strategy 10.7. 
ONE OF THE LARGEST DIGITAL MEDIA COMPANIES IN THE WORLD
The first customer that I will highlight is one of the largest digital media companies in the world. This customer was looking to publish a rapidly growing set of viewership data into memory from all of their digital media platforms. It was taking hours to fully refresh this cube from HIVE via a thin ODBC pipe. Before our team came on-site, the BI team saw that 150M rows from this dataset was taking over 6 hours to publish in Hive and in about 30 minutes against Teradata. That being said, Teradata did require about 30 minutes additional time to “normalize” the tables to improve performance. 
So our requirement for success was to showcase the Hadoop Gateway’s ability to publish cubes directly against HDFS without having to create hive tables/normalize tables in Teradata AND in less time than Teradata. Strategy did just that. We published a 150M row cube in 24 minutes and by the end of a three day engagement with the customer, we were able to publish 500M rows of data in less than an hour with the Hadoop Gateway. This same request took over double that time with Teradata. 
Problem

Solution

A couple notes about this customer: 
This customer was using a dev cluster with Hortonworks 2.4.2 and only 2 data nodes that had 380 GB RAM each and 40 cores.  Another thing we observed while on-site was that parquet files were able to be processed in the cluster faster than orc files. This is something to note that not all data formats (even ones with the same data) will perform at the same speed. However, don't worry too much. Once the data is brought into Strategy it takes the same amount of time to load into an intelligent cube.  I just figured this would be strong insight as you investigate this new gateway. 
ONE OF THE LARGEST RETAILERS IN EUROPE
Another company I went to about a month ago was one of the largest retailers in Europe. Last year, executives at this company had begun heavily investing in big data initiatives that would allow this customer to more proactively respond to market trends and sales anomalies. They began loading daily transaction data (which amounted to about 12M+ rows per day) into parquet and avro files. However, they wouldn’t be able to report on this data until it was all loaded into Hive. They were excited to hear about the Hadoop Gateway because it would give their end users direct access into publishing data directly from files in HDFS. 
So our requirement for success with this customer was to prove that #1 the Hadoop Gateway would be able to publish cubes faster than their other ODBC drivers AND showcase the Hadoop Gateway's ability to publish these files in a consumable format.  On both fronts, the Hadoop Gateway succeeded. First, we took the raw daily transaction HDFS files and loaded this data into Hive. I then published cubes from Hadoop Gateway, Hive on Spark. Spark was being used as the executor engine for Hive.  We saw the Hadoop Gateway publish the cubes in the fastest amount of time. We then applied various Data Wrangling operations against the transaction cube and saw something interesting. If you looked at the time that it took for the transaction data to be “wrangled” and then published into a cube (the bottom left bar), it took less than half the time that the transaction cube that was published directly against Hive on Spark. (the top right)  Add data wrangling to Hive on Spark and the comparison is just not fair. The Hadoop Gateway blows it out of the water. This is because the Hadoop Gateway is able to directly push these data wrangling operations down to Spark and the ODBC connection to Hive on Spark does not. 
This customer was leveraging Cloudera 5.8.3. 
ONE OF THE LARGEST BANKS IN THE WORLD
The last customer I will talk about is one of the largest banks in the world that I went to. The team we worked most closely with was responsible for Strategy certification. Any new feature needs to go through this team before it can be disseminated to the application teams. The #1 concern that this bank had was if our new feature could handle the configuration of their hadoop cluster that had been organized to optimize their security and stability.  So our requirement for success was to simply to connect to their Hadoop cluster and publish a cube. 
They had enabled the following:

• MIT Kerberos had been enabled for data access and authentication. With MIT Kerberos, Hadoop Gateway can identify itself as a Kerberos principal and have access to the required services: HDFS, Spark Manager
• SSL/Secure Socket Layer had been enabled for data transfer:  For some more context - Hadoop clusters exchange internal data such as file system images and the user data through the HTTP protocol. Since HTTP transfers the data in clear-text, an attacker might be able to tap into the network and put your valuable data at risk. To protect their data, HTTPS had also been enabled. 
• High Availability had also been enabled: This can be set in HDFS and YARN cluster services. A server node can be set as Active and enabled while an additional can be set on Stand By to replace the Active at any time. 

Despite all of these security and stability measures that had been enabled on the cluster, the Hadoop Gateway was able to publish a cube and we were able to label this visit as a success as well.
This customer was leveraging Cloudera 5.9.1.