How to create a DataFrame from a text file in Spark

asked10 years ago
last updated7 years ago
viewed175.1k times
Up Vote23Down Vote

I have a text file on HDFS and I want to convert it to a Data Frame in Spark.

I am using the Spark Context to load the file and then try to generate individual columns from that file.

val myFile = sc.textFile("file.txt")
val myFile1 = myFile.map(x=>x.split(";"))

After doing this, I am trying the following operation.

myFile1.toDF()

I am getting an issues since the elements in myFile1 RDD are now array type.

How can I solve this issue?