Associate-Developer-Apache-Spark-3.5 試験問題を無料オンラインアクセス

A data scientist at a financial services company is working with a Spark DataFrame containing transaction records. The DataFrame has millions of rows and includes columns for transaction_id, account_number, transaction_amount, and timestamp. Due to an issue with the source system, some transactions were accidentally recorded multiple times with identical information across all fields. The data scientist needs to remove rows with duplicates across all fields to ensure accurate financial reporting.
Which approach should the data scientist use to deduplicate the orders using PySpark?

• A.df = df.dropDuplicates()
• B.df = df.dropDuplicates(["transaction_amount"])
• C.df = df.groupBy("transaction_id").agg(F.first("account_number"), F.first("transaction_amount"), F.first("timestamp"))
• D.df = df.filter(F.col("transaction_id").isNotNull())

20 of 55.
What is the difference between df.cache() and df.persist() in Spark DataFrame?

• A.persist() - Persists the DataFrame with the default storage level (MEMORY_AND_DISK_DESER), and cache() - Can be used to set different storage levels.
• B.Both cache() and persist() can be used to set the default storage level (MEMORY_AND_DISK_DESER).
• C.Both functions perform the same operation. The persist() function provides improved performance as its default storage level is DISK_ONLY.
• D.cache() - Persists the DataFrame with the default storage level (MEMORY_AND_DISK_DESER), and persist() - Can be used to set different storage levels to persist the contents of the DataFrame.

31 of 55.
Given a DataFrame df that has 10 partitions, after running the code:
df.repartition(20)
How many partitions will the result DataFrame have?

• A.Same number as the cluster executors
• B.5
• C.10
• D.20

A data engineer writes the following code to join two DataFrames df1 and df2:
df1 = spark.read.csv("sales_data.csv") # ~10 GB
df2 = spark.read.csv("product_data.csv") # ~8 MB
result = df1.join(df2, df1.product_id == df2.product_id)

Which join strategy will Spark use?

• A.Shuffle join, because AQE is not enabled, and Spark uses a static query plan
• B.Shuffle join, as the size difference between df1 and df2 is too large for a broadcast join to work efficiently
• C.Broadcast join, as df2 is smaller than the default broadcast threshold
• D.Shuffle join because no broadcast hints were provided

1 of 55. A data scientist wants to ingest a directory full of plain text files so that each record in the output DataFrame contains the entire contents of a single file and the full path of the file the text was read from.
The first attempt does read the text files, but each record contains a single line. This code is shown below:
txt_path = "/datasets/raw_txt/*"
df = spark.read.text(txt_path) # one row per line by default
df = df.withColumn("file_path", input_file_name()) # add full path
Which code change can be implemented in a DataFrame that meets the data scientist's requirements?

• A.Add the option lineSep to the text() function.
• B.Add the option lineSep=", " to the text() function.
• C.Add the option wholetext to the text() function.
• D.Add the option wholetext=False to the text() function.