Databricks-Certified-Professional-Data-Engineer 試験問題を無料オンラインアクセス

An hourly batch job is configured to ingest data files from a cloud object storage container where each batch represent all records produced by the source system in a given hour. The batch job to process these records into the Lakehouse is sufficiently delayed to ensure no late-arriving data is missed. The user_id field represents a unique key for the data, which has the following schema:
user_id BIGINT, username STRING, user_utc STRING, user_region STRING, last_login BIGINT, auto_pay BOOLEAN, last_updated BIGINT New records are all ingested into a table named account_history which maintains a full record of all data in the same schema as the source. The next table in the system is named account_current and is implemented as a Type 1 table representing the most recent value for each unique user_id .
Assuming there are millions of user accounts and tens of thousands of records processed hourly, which implementation can be used to efficiently update the described account_current table as part of each hourly batch job?

• A.Overwrite the account current table with each batch using the results of a query against the account history table grouping by user id and filtering for the max value of last updated.
• B.Use Auto Loader to subscribe to new files in the account history directory; configure a Structured Streaminq trigger once job to batch update newly detected files into the account current table.
• C.Use Delta Lake version history to get the difference between the latest version of account history and one version prior, then write these records to account current.
• D.Filter records in account history using the last updated field and the most recent hour processed, as well as the max last iogin by user id write a merge statement to update or insert the most recent value for each user id.
• E.Filter records in account history using the last updated field and the most recent hour processed, making sure to deduplicate on username; write a merge statement to update or insert the most recent value for each username.

A junior data engineer has configured a workload that posts the following JSON to the Databricks REST API endpoint 2.0/jobs/create .

Assuming that all configurations and referenced resources are available, which statement describes the result of executing this workload three times?

• A.The logic defined in the referenced notebook will be executed three times on new clusters with the configurations of the provided cluster ID.
• B.Three new jobs named " Ingest new data " will be defined in the workspace, but no jobs will be executed.
• C.The logic defined in the referenced notebook will be executed three times on the referenced existing all purpose cluster.
• D.Three new jobs named " Ingest new data " will be defined in the workspace, and they will each run once daily.
• E.One new job named " Ingest new data " will be defined in the workspace, but it will not be executed.

A data engineer is optimizing a managed Delta table that suffers from data skew and frequently changing query filter columns . The engineer wants to avoid costly data rewrites when query patterns evolve. The table size is under 1 TB.
How should the data engineer meet this requirement?

• A.Apply Z-ordering , since it allows flexible reorganization of data layout without rewriting existing files and adapts easily to new filter columns.
• B.Combine partitioning and Z-ordering to maximize flexibility and minimize maintenance as query patterns change.
• C.Enable liquid clustering , as it efficiently handles data skew, allows clustering keys to be changed without rewriting existing data, and adapts to evolving query patterns.
• D.Use Hive-style partitioning , as it provides efficient data skipping and is easy to change partition columns at any time.

Which approach demonstrates a modular and testable way to use DataFrame.transform for ETL code in PySpark?

• A.def upper_value(df):
  return df.withColumn( " value_upper " , upper(col( " value " )))
  def filter_positive(df):
  return df.filter(df[ " id " ] > 0)
  pipeline_df = df.transform(upper_value).transform(filter_positive)
• B.class Pipeline:
  def transform(self, df):
  return df.withColumn( " value_upper " , upper(col( " value " )))
  pipeline = Pipeline()
  assertDataFrameEqual(pipeline.transform(test_input), expected)
• C.def transform_data(input_df):
  # transformation logic here
  return output_df
  test_input = spark.createDataFrame([(1, " a " )], [ " id " , " value " ]) assertDataFrameEqual(transform_data(test_input), expected)
• D.def upper_transform(df):
  return df.withColumn( " value_upper " , upper(col( " value " )))
  actual = test_input.transform(upper_transform)
  assertDataFrameEqual(actual, expected)

An external object storage container has been mounted to the location /mnt/finance_eda_bucket .
The following logic was executed to create a database for the finance team:

After the database was successfully created and permissions configured, a member of the finance team runs the following code:

If all users on the finance team are members of the finance group, which statement describes how the tx_sales table will be created?

• A.An managed table will be created in the storage container mounted to /mnt/finance eda bucket.
• B.A logical table will persist the query plan to the Hive Metastore in the Databricks control plane.
• C.An external table will be created in the storage container mounted to /mnt/finance eda bucket.
• D.A managed table will be created in the DBFS root storage container.
• E.A logical table will persist the physical plan to the Hive Metastore in the Databricks control plane.