![]() ![]() “FAILED: Execution Error, return code 1 from .ql.exec.DDLTask. Once done, you can now start to create the table schema from the Hive script – “Create Twitter Schema.hql” (Note: This can be found in the Github repository)īefore you can run the Hive script to create the tables, you must ensure that the JSON serdes (Serializer / Deseralizer) library is available otherwise you will get the following error: Use the File Browser in Hue to view the folder you have just created. If you do an “ls” command in HDFS you should see the directory you have just created as shown below. See this site for more details on the HDFS architecture. Hence HDFS files are referred to by their fully qualified name which is a parameter of many of the elements of the interaction between the Client and the other elements of the HDFS architecture. However, one of the key differences is that there is no concept of a current directory within HDFS. Run the above command instructs HDFS to create a folder “twitteranalytics” in the top level of the HDFS directory. A standard directory structure is used in HDFS that is similar to a typical file system in Unix. First, create the following directory structure in HDFS. In this project, we will access the Twitter API to download the tweets and the downloaded files will be saved onto HDFS and access through Hive tables. Once the server is successfully started, login to Hue and click on Query Editors > Hive to view the Query Editor page. Run the following command to start Hive2 server. In the VM environment, ensure that Hive2 server is started. This tutorial assumes that you are familiar with hdfs commands. The version of HDFS used in this tutorial is Hadoop 2.6.0-cdh5.4.3, however the instructions and steps here should be application for any subsequent versions. However the issues encountered in this tutorial may differ for different distributions. If you have another distribution, you should still be able to run this how-to. We begin by first setting up and installing Cloudera Hadoop CDH5.4.3a. Ensure that you run any preconfigured scripts to ensure that Flume, Spark, Python, HDFS, Hive, Hue, Impala, Zookeeper, Kafka, Slor are setup and configured.įor this exercise, I am using a pre-configured Hadoop stack setup from Cloudera. Ready? Lets start! Step 1: Getting Cloudera Hadoop CDH5.4.3a ready ![]() Remember to save them in your local folders on the Cloudera VM. You can download the source files in this how-to for your easy reference here. While many other blogs do cover a great deal on how to do the above, I wanted to also share some of the errors I encountered and how to resolve them, hopefully saving you time from searching the web and trying all kinds of solutions. Recently I had the opportunity to do some simple Twitter sentiment analytics using a combination of HDFS, Hive, Flume and Spark and wanted to share how it was done. Its been some time since my last post but am excited to be sharing about my learnings and adventures with Big Data and Data Analytics. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |