Databricks-Certified-Professional-Data-Engineer 試験問題を無料オンラインアクセス

Which Python variable contains a list of directories to be searched when trying to locate required modules?

• A.pylib.source
• B.os-path
• C.pypi.path
• D.,sys.path
• E.importlib.resource path

A Delta table of weather records is partitioned by date and has the below schema:
date DATE, device_id INT, temp FLOAT, latitude FLOAT, longitude FLOAT
To find all the records from within the Arctic Circle, you execute a query with the below filter:
latitude > 66.3
Which statement describes how the Delta engine identifies which files to load?

• A.The Hive metastore is scanned for min and max statistics for the latitude column
• B.The Parquet file footers are scanned for min and max statistics for the latitude column
• C.The Delta log is scanned for min and max statistics for the latitude column
• D.All records are cached to an operational database and then the filter is applied
• E.All records are cached to attached storage and then the filter is applied

A data engineer is configuring a Databricks Asset Bundle to deploy a job with granular permissions. The requirements are:
* Grant the data-engineers group CAN_MANAGE access to the job.
* Ensure the auditors group can view the job but not modify or run it.
* Avoid granting unintended permissions to other users or groups.
How should the data engineer deploy the job while meeting the requirements?

• A.permissions:
  - group_name: data-engineers
  level: CAN_MANAGE
  - group_name: auditors
  level: CAN_VIEW
  resources:
  jobs:
  my-job:
  name: data-pipeline
  tasks: [...]
  job_clusters: [...]
• B.resources:
  jobs:
  my-job:
  name: data-pipeline
  tasks: [...]
  job_clusters: [...]
  permissions:
  - group_name: data-engineers
  level: CAN_MANAGE
  permissions:
  - group_name: auditors
  level: CAN_VIEW
• C.resources:
  jobs:
  my-job:
  name: data-pipeline
  tasks: (...)
  job_clusters: (...)
  permissions:
  - group_name: data-engineers
  level: CAN_MANAGE
  - group_name: auditors
  level: CAN_VIEW
• D.resources:
  jobs:
  my-job:
  name: data-pipeline
  tasks: (...)
  job_clusters: (...)
  permissions:
  - group_name: data-engineers
  level: CAN_MANAGE
  - group_name: auditors
  level: CAN_VIEW

Which approach demonstrates a modular and testable way to use DataFrame transform for ETL code in PySpark?

• A.def upper_value(df):
  return df.withColumn( " value_upper " , upper(col( " value " )))
  def filter_positive(df):
  return df.filter(df[ " id " ] > 0)
  pipeline_df = df.transform(upper_value).transform(filter_positive)
• B.class Pipeline:
  def transform(self, df):
  return df.withColumn( " value_upper " , upper(col( " value " )))
  pipeline = Pipeline()
  assertDataFrameEqual(pipeline.transform(test_input), expected)
• C.def upper_transform(df):
  return df.withColumn( " value_upper " , upper(col( " value " )))
  actual = test_input.transform(upper_transform)
  assertDataFrameEqual(actual, expected)
• D.def transform_data(input_df):
  # transformation logic here
  return output_df
  test_input = spark.createDataFrame([(1, " a " )], [ " id " , " value " ]) assertDataFrameEqual(transform_data(test_input), expected)

The following code has been migrated to a Databricks notebook from a legacy workload:

The code executes successfully and provides the logically correct results, however, it takes over 20 minutes to extract and load around 1 GB of data.
Which statement is a possible explanation for this behavior?

• A.Python will always execute slower than Scala on Databricks. The run.py script should be refactored to Scala.
• B.Instead of cloning, the code should use %sh pip install so that the Python code can get executed in parallel across all nodes in a cluster.
• C.%sh does not distribute file moving operations; the final line of code should be updated to use %fs instead.
• D.%sh triggers a cluster restart to collect and install Git. Most of the latency is related to cluster startup time.
• E.%sh executes shell code on the driver node. The code does not take advantage of the worker nodes or Databricks optimized Spark.