Databricks-Certified-Data-Engineer-Professional 試験問題を無料オンラインアクセス

An upstream source writes Parquet data as hourly batches to directories named with the current date. A nightly batch job runs the following code to ingest all data from the previous day as indicated by the date variable:

Assume that the fields customer_id and order_id serve as a composite key to uniquely identify each order.
If the upstream system is known to occasionally produce duplicate entries for a single order hours apart, which statement is correct?

• A.Each write to the orders table will run deduplication over the union of new and existing records, ensuring no duplicate records are present.
• B.Each write to the orders table will only contain unique records, but newly written records may have duplicates already present in the target table.
• C.Each write to the orders table will only contain unique records; if existing records with the same key are present in the target table, these records will be overwritten.
• D.Each write to the orders table will only contain unique records, and only those records without duplicates in the target table will be written.
• E.Each write to the orders table will only contain unique records; if existing records with the same key are present in the target table, the operation will tail.

In a Databricks Asset Bundle project, in the file resources/app.yml, the data engineer would like to deploy a Databricks Apps databricks_app_deployed and Volume volume_deployed and grant the Service Principal behind Databricks Apps permissions to READ and WRITE to the Volume.
How should the data engineer achieve the deployment?

• A.
• B.
• C.
• D.

A user wants to use DLT expectations to validate that a derived table report contains all records from the source, included in the table validation_copy.
The user attempts and fails to accomplish this by adding an expectation to the report table definition.

Which approach would allow using DLT expectations to validate all expected records are present in this table?

• A.Define a view that performs a left outer join on validation_copy and report, and reference this view in DLT expectations for the report table
• B.Define a function that performs a left outer join on validation_copy and report and report, and check against the result in a DLT expectation for the report table
• C.Define a temporary table that perform a left outer join on validation_copy and report, and define an expectation that no report key values are null
• D.Define a SQL UDF that performs a left outer join on two tables, and check if this returns null values for report key values in a DLT expectation for the report table.

The business reporting tem requires that data for their dashboards be updated every hour. The total processing time for the pipeline that extracts transforms and load the data for their pipeline runs in 10 minutes.
Assuming normal operating conditions, which configuration will meet their service-level agreement requirements with the lowest cost?

• A.Schedule a Structured Streaming job with a trigger interval of 60 minutes.
• B.Schedule a job to execute the pipeline once an hour on a dedicated interactive cluster.
• C.Schedule a job to execute the pipeline once an hour on a new job cluster.
• D.Configure a job that executes every time new data lands in a given directory.

A user new to Databricks is trying to troubleshoot long execution times for some pipeline logic they are working on. Presently, the user is executing code cell-by-cell, using display() calls to confirm code is producing the logically correct results as new transformations are added to an operation. To get a measure of average time to execute, the user is running each cell multiple times interactively.
Which of the following adjustments will get a more accurate measure of how code is likely to perform in production?

• A.Scala is the only language that can be accurately tested using interactive notebooks; because the best performance is achieved by using Scala code compiled to JARs. all PySpark and Spark SQL logic should be refactored.
• B.Calling display () forces a job to trigger, while many transformations will only add to the logical query plan; because of caching, repeated execution of the same logic does not provide meaningful results.
• C.Production code development should only be done using an IDE; executing code against a local build of open source Spark and Delta Lake will provide the most accurate benchmarks for how code will perform in production.
• D.The Jobs Ul should be leveraged to occasionally run the notebook as a job and track execution time during incremental code development because Photon can only be enabled on clusters launched for scheduled jobs.
• E.The only way to meaningfully troubleshoot code execution times in development notebooks Is to use production-sized data and production-sized clusters with Run All execution.